any solution

If $\alpha>1$ then set $x_1=x_2=\alpha-1 \to x_3=\alpha-1 \to x_n=\alpha-1$ - infinite sequence. So, let $ \alpha \leq 1$ $\alpha x_{n+1}> x_n$ and $x_{n+1}<\sqrt{\alpha x_n} \to x_n < a^3 \leq 1$ $x_{n+1}-x_n\geq \alpha x_{n+1}-x_n>0$ $\alpha x_{n+2}-x_{n+1}=x_{n+3}^2>x_{n+2}^2=\alpha x_{n+1}-x_n$ $\alpha (x_{n+2}-x_{n+1})>x_{n+1}-x_n$ $x_{n+2}>x_{n+1}+ \frac{x_2-x_1}{\alpha^n} \geq x_{n+1}+(x_2-x_1) >x_n+2(x_2-x_1)>...>x_2+n(x_2-x_1)>(n+1)(x_2-x_1)$ Let $m> \frac{1}{x_2-x_1}-1 \to x_{m+2}>(m+1)(x_2-x_1)>1$ -contradiction, with $x_n <1$ Answer : $\alpha>1$

When $\alpha >1$, then the numbers $x_n = \alpha -1$ satisfies the recurrence; $\alpha-1 = \sqrt{\alpha(\alpha-1)-(\alpha-1)}$. Since the argument of the square root should be positive, we have $\alpha x_{n+1} >x_n$ for every positive $n$, and $x_{n+1} < \sqrt{\alpha x_n}$ for every integer $n \ge 2$. Therefore, $x_n < \alpha^3$ for every $n \ge 2$. When $\alpha<1$, however, $x_{n+1} > \frac{1}{\alpha} x_n$ for every $n$ implies $x_{n} > \frac{1}{\alpha^{n-1}} x_1$ which is an absurdity. Finally, when $\alpha =1$, we have $x_{n+2} = \sqrt{x_{n+1}-x_n}$. Hence $x_{n+1}>x_n$ for each $n \ge 1$. So $\sqrt{x_{n+1}-x_n} = x_{n+2}>x_{n+1}$. It follows that $x_{n+1} > x_n + {x_{n+1}}^2 > x_n + {x_n}^2 = x_n (1+x_n) \ge x_n (1+x_1)$. Therefore $x_n > x_1 ( 1+x_1)^{n-1}$ which is an absurdity.

Note that if we are allowed to use limit argument, the solution may be more natural. When $\alpha \le 1$, we have boundedness of $\left\{ a_n \right \}$ and monotonicity of $x_n$. Hence the sequence should converges which is impossible when $\alpha <1$. For the case $\alpha =1$, $x_n$ is motonic and hence it converges to zero which is strictly less than $x_1$ which is impossible too.

Replace $\alpha$ with $a$ due to laziness. The answer is $a>1$. For $a>1$ then set the entire sequence to $a-1$, which works. If $a<1$, then since the terms in the square root are positive we should have $ax_{n+1}>x_n \implies x_{n+1}>a^{-1}x_n$, hence $(x_n)$ is unbounded and increasing. On the other hand we should have $x_{n+2}^2 \leq ax_{n+1} \leq ax_{n+2}$, which is a contradiction if $x_{n+2}$ is large. If $a=1$, then $(x_n)$ is still increasing. If it's bounded then it has some limit $L$, but then we should have $L=\sqrt{L-L}$ which is false. Otherwise, it's unbounded, and the same proof as above applies.

If $\alpha > 1$, set the entire sequence to $\alpha - 1$. If $\alpha \leq 1$, then the sequence $\{x_n\}$ must be strictly increasing. But if some $x_{n+1} > 1$, then $x_{n+2} < \sqrt{x_{n+1}} < x_{n+1}$, contradiction. So $\{x_n\}$ is bounded above and thus has a limit $L$. Then, for $\alpha < 1$, choose an $\varepsilon < \frac{1+\alpha}{1-\alpha}$ such that for sufficiently large $n$, $x_n \in (L-\varepsilon, L + \varepsilon)$, hence \[\alpha x_{n+1} - x_n < \alpha(L+\varepsilon) - (L - \varepsilon) < 0.\] If $\alpha = 1$, the same argument yields $x_{n+1} - x_n < (\alpha + 1) \varepsilon$, so taking $\varepsilon < \frac {L^2}{10^{100}(\alpha + 1)}$, it follows suit that $x_{n+2} < L - \varepsilon < x_{n+1}$, contradiction again.