An annoying Pell-like equation related to a binary quadratic form problem

Let $A,B,C,D$ be integers such that $AD-BC= 1 $ and $ A+D = -1 $.

Show by elementary means that the Diophantine equation
$$\bigl[2Bx + (D-A) y\bigr] ^ 2 + 3y^2 = 4|B|$$

has an integer solution (that is, a solution $(x,y)\in\mathbb Z^2$). If possible, find an explicit solution (involving $A,B,C,D$, of course).

Motivation: I arrived at this equation after trying to find explicitly the matrix $g$ suggested by Will Jagy on his answer to this question of mine. Concretely, if $\gamma=\binom{A\ \ B}{C\ \ D}$, then $\gamma$ has order $3$ in $\operatorname{SL_2}(\mathbb Z)$. By indirect methods it can be shown that $\gamma$ is conjugated in $\operatorname{SL_2}(\mathbb Z)$ to one of the matrices $P$ or $P^{-1}$, being $P=\binom{\ \ \,0\quad1}{-1\ \ -1}$ (see studiosus’ answer to the same question.). Unfortunately this argument is rather sophisticated to my knowledge, and besides I think that a direct argument is possible.

Because of this I tried to find a explicit matrix $g=\binom{x\ \ y}{z\ \ w}\in\operatorname{SL_2}(\mathbb Z)$ such that $gP=\gamma g$ or $gP^{-1}=\gamma g$. The matricial equalities lead to a system of $4$ linear equations in the unknowns $x,y,z,w$ , which can be easily solved. Plugging these solutions $(x,y,z,w)$ (recall that we are considering the two possibilities of conjugation, to $P$ or $P^{-1}$) into the equation $xy-zw=1$ yields $Bx^2+(D-A)xy+(-C)y^2=\pm1$. Completing the square and using the equalities $AD-BC=1$ and $A+D=-1$ we obtain the required equation. I tried to solve it explicitly, with no success.

Solutions Collecting From Web of "An annoying Pell-like equation related to a binary quadratic form problem"

Reduction can also be accomplished in a conjugacy class in $SL_2 \mathbb Z,$ in a manner very similar to that for quadratic forms. As matrices, there is a little difference: for quadratic forms, we start with symmetric $G$ and work with $P^T G P, $ for conjugacy classes we work with $P^{-1} \gamma P.$

Lemma: given real numbers with $|V| \geq |W| > 0,$ then either $|V+W| < |V|$ or $|V-W|< |V|.$

Proof: $$ |V+W| |V-W| = |V^2 – W^2| = |V|^2 – |W|^2 < |V|^2. $$

Now, we have your matrix $\gamma,$ with $(A-D)^2 -4 |BC| = -3,$ or
$$ 4|BC| = (A-D)^2 + 3. $$ Since $A+D = -1,$ we know that $A-D$ is also odd. If both $2|B|, 2|C| \geq |A-D|,$ then actually $2|B|, 2|C| \geq 1 + |A-D|.$ If so,
$4|BC| \geq (A-D)^2 + 2 |A-D| + 1, $ whence
$$ (A-D)^2 + 3 = 4 |BC| \geq (A-D)^2 + 2 |A-D| + 1, $$ or
$$ 3 \geq 2 |A-D| + 1 $$ and
$$ |A-D| = 1. $$ So, the assumption that both $2|B|, 2|C| \geq |A-D|,$ then $|A-D| = 1.$

The other case is either $2|B| < |A-D|$ or $2|C| < |A-D|.$ In this case, we use either of
$$
\left(
\begin{array}{cc}
1 & -n \\
0 & 1
\end{array}
\right)
\left(
\begin{array}{cc}
A & B \\
C & D
\end{array}
\right)
\left(
\begin{array}{cc}
1 & n \\
0 & 1
\end{array}
\right) =
\left(
\begin{array}{cc}
A-Cn & B+(A-D)n – C n^2 \\
C & D+Cn
\end{array}
\right)
$$
or
$$
\left(
\begin{array}{cc}
1 & 0 \\
-n & 1
\end{array}
\right)
\left(
\begin{array}{cc}
A & B \\
C & D
\end{array}
\right)
\left(
\begin{array}{cc}
1 & 0 \\
n & 1
\end{array}
\right) =
\left(
\begin{array}{cc}
A+Bn & B \\
-(Bn^2 + (A-D)n-C) & D-Bn
\end{array}
\right)
$$
with $n=\pm 1$
to strictly reduce $|A-D|,$ as we can have the new difference of diagonal terms either $|A-D-2Cn|$ or $|A-D+2Bn|.$ Keep doing these steps, eventually the difference of the diagonal terms is exactly $1$ in absolute value; so one of them is $0$ and the other is $-1,$ and the off-diagonal terms are $1,-1.$

So far, each possible integer matrix with determinant $1$ and trace $-1$ is conjugate to at least one of these four:
$$
\left(
\begin{array}{cc}
0 & 1 \\
-1 & -1
\end{array}
\right), \;
\left(
\begin{array}{cc}
-1 & 1 \\
-1 & 0
\end{array}
\right), \;
\left(
\begin{array}{cc}
0 & -1 \\
1 & -1
\end{array}
\right) , \;
\left(
\begin{array}{cc}
-1 & -1 \\
1 & 0
\end{array}
\right)
$$
Now, the first and second are conjugate to each other, and the third and fourth are conjugate to each other, because
$$
\left(
\begin{array}{cc}
0 & 1 \\
-1 & 0
\end{array}
\right)
\left(
\begin{array}{cc}
0 & -1 \\
1 & 0
\end{array}
\right) =
\left(
\begin{array}{cc}
1 & 0 \\
0 & 1
\end{array}
\right)
$$
and
$$
\left(
\begin{array}{cc}
0 & 1 \\
-1 & 0
\end{array}
\right)
\left(
\begin{array}{cc}
A & B \\
C & D
\end{array}
\right)
\left(
\begin{array}{cc}
0 & -1 \\
1 & 0
\end{array}
\right) =
\left(
\begin{array}{cc}
D & -C \\
-B & A
\end{array}
\right)
$$
As it happens, the first and the third, which are transposes, are not $SL_2 \mathbb Z$ conjugate, so we get two pairs, one pair just the transposes of the other pair.
Why not. IF
$$
\left(
\begin{array}{cc}
s & -q \\
-r & p
\end{array}
\right)
\left(
\begin{array}{cc}
0 & 1 \\
-1 & -1
\end{array}
\right)
\left(
\begin{array}{cc}
p & q \\
r & s
\end{array}
\right) =
\left(
\begin{array}{cc}
0 & -1 \\
1 & -1
\end{array}
\right)
$$
THEN
$$ s^2 + s q + q^2 = -1, $$ which is impossible in real numbers, since $ s^2 + s q + q^2$ is a positive definite quadratic form.

You have the binary quadratic form
$$ \color{red}{ f(x,y) = B x^2 + (D-A)xy – C y^2} $$ in your last paragraph. The discriminant is $$ \Delta = (D-A)^2 + 4 B C. $$
You also have $AD-BC = 1$ and $A+D = -1.$ So, $BC – AD = -1$ and
$$ A^2 + 2 AD + D^2 = 1, $$
$$ 4BC – 4AD = -4, $$
$$ A^2 – 2 AD + D^2 +4BC = -3, $$
$$ \Delta = (A – D)^2 +4BC = -3. $$
If we had $\gcd(A-D,B,C) > 1$ we would have a square factor of $\Delta,$ so that is out, the coefficients of $f(x,y)$ are relatively prime (as a triple, not necessarily in pairs). Next, $-3$ is not a square, so we cannot have $B=0$ or $C=0.$

Finally $f$ is definite. If, say, $B > 0,$ it is positive definite. With discriminant $-3,$ it is then equivalent in $SL_2 \mathbb Z$ to
$$g(u,v) = u^2 + u v + v^2$$ and both $g$ and $f$ integrally represent $1.$

If $B < 0,$ then $f$ is negative definite. With discriminant $-3,$ it is then equivalent in $SL_2 \mathbb Z$ to
$$h(u,v) = -u^2 – u v – v^2$$ and both $h$ and $f$ integrally represent $-1.$

That is enough for what you asked. I should point out that the reduction for positive binary forms is very similar to the Euclidean algorithm for finding GCD is very similar to the algorithm for finding the matrix $W$ in $SL_2 \mathbb Z$ that is conjugate to your original and has, for example, the smallest value of
$(w_{22} – w_{11})^2.$ I recommend Buell, Binary Quadratic Forms if I have not already. I was going to answer with a reduction that minimized $(A-D)^2,$ then looked again at your question and realized there was a shortcut. Note that many books do reduction for positive definite binary forms, and give short lists for discriminants of small absolute value.