A question on elementary number theory

Just came across the following question:

Let $S=\{2,5,13\}$. Notice that $S$ satisfies the following property: for any $a,b \in S$ and $a \neq b$, $ab-1$ is a perfect square. Show that for any positive integer $d \not\in S$, $S \cup \{d\}$ does not satify the above property.

This question can be done by considering modulo 4.

Here comes my question:

What is the greatest value of $|A|$ if all elements in $A$ are different and for any $a,b \in A$ and $a \neq b$, $ab-1$ is a perfect square?

Remark: $A$ may not contain any of $\{2,5,13\}$, example: $\{17, 26, 85\}$

Edit: From the link here, there are infinite many 3-element sets statisfy the poperty. These sets are of the form $\{a, b, a+b+2r\}$ where $r^2 = ab-1$. Are we able to find a 4-element set that statisfies the property?

Solutions Collecting From Web of "A question on elementary number theory"

By the paper A. Dujella and C. Fuchs, Complete solution of a problem of Diophantus and Euler, J. London Math. Soc. 71 (2005), 33-52. (see Theorem 1b),
there does not exist 4-element set the considered property and with all elements greater than 1.

For results on sets contaning 1, see e.g. N. C. Bonciocat, M. Cipu, M. Mignotte, On D(-1)-quadruples, Publ. Mat. 56 (2012), 279-304. In particular, if {1,b,c,d} has considered property, then b>10^{13}.

I have been trying to find a 4-element set that satisfies the property. I ran a computer program to check all 4-element sets of integers <= 20,000 and found nothing, so I am guessing that there are no such sets, although of course I have nothing conclusive.

The program also listed all of the 3-element sets that it found, and I’ve noticed that each set, taken modulo 4, is either {1, 1, 1} or {1, 1, 2}. I can now prove this:

It is fairly easy to see that, if a*b-1 is square, then {a, b} modulo 4 is one of {1, 1}, {3, 3}, {1, 2}, and {2, 3}.

Next, looking through some notes on number theory that I printed from MIT OpenCourseware, there is a lemma which states that, because a*b is the sum of two squares (n^2 + 1^2), any prime factors of a*b which are congruent to 3 mod 4 must divide both squares, i.e., n and 1. Since no primes divide 1, this implies that a*b has no prime factors congruent to 3 mod 4, from which it follows that a and b are both not congruent to 3 mod 4.

Thus, {a, b} modulo 4 is one of {1, 1}, {1, 2}. From this, it is easy to see that any 3-element set satisfying the property is of one of the forms {1, 1, 1}, {1, 1, 2}, as desired.

@pipi: how did you prove the original problem “modulo 4”? Does your result imply that any set of the form {1, 1, 2} modulo 4 cannot be extended to a 4-element set? If so, by using your argument and then modifying it to work on {1, 1, 1}, we may be able to prove that no 4-element sets satisfying the property exist.