I like this problem. Rearrange a bit and we get that $$y - x^2 -xy + \sqrt{y - x^2 -xy} \le -(x-y)^2\le 0$$Obviously, $y - x^2 -xy \geq 0$, otherwise square root would be complex. Also, by the definition, $\sqrt{y - x^2 -xy}\geq 0$, so $$0\leq y - x^2 -xy + \sqrt{y - x^2 -xy}\le 0\implies y - x^2 -xy + \sqrt{y - x^2 -xy}=0$$If $y - x^2 -xy>0$, then $\sqrt{y - x^2 -xy}>0$, which is contradiction, so $y - x^2 -xy=0$, which means that $0\le -(x-y)^2\le 0\implies x=y$. Now solving $x-2x^2=0$ gives us that $x=y=0$ or $x=y=\frac{1}{2}$. Both solutions work with the given inequality. Answer. $\boxed{x=y=0 \text{ and }x=y=\frac{1}{2}}$.