My idea is from here and here. There is a faster approach that from Dirichlet, $2xf(1)+1$ is prime for infinitely $x.$ I just dont know if it can be used in the contest

Let me add the alternative approach laikhanhhoang_3011 suggests. Let $P(x,y)$ be the assertion that the given expression is a perfect square. $P(x,1)$ gives that $f(x)^2+2xf(1)+1$ is a perfect square. By Dirichlet's theorem, the set $A= \{x: 2f(1)x+1 \, \rm{is \, prime} \}$ is infinite. For each $x \in A$, we have that $2f(1)x+1=p$ with $p$ prime, and so $f(x)^2+p$ is a perfect square $k^2$, implying that $p=(k-f(x))(k+f(x))$, and so $k-f(x)=1$ and $k+f(x)=p$. Thus, $f(x)=\dfrac{p-1}{2}=f(1)x$. Hence, $f(x)=f(1)x$, for all $x \in A$. Now, take $y \in A$ in the given assertion to obtain that $f^2(x)+2f(1)xy+y^2$ is a perfect square, or equivalently that $(f^2(x)-f(1)^2x^2)+(y-f(1)x)^2$ is a perfect square. Now, fixing $x$ and letting $y \in A$ vary, we obtain that $f^2(x)-f(1)^2x^2$ can be written as a difference of squares in infinitely many ways (since set $A$ is infinite), which is impossible unless $f^2(x)-f(1)^2x^2=0$. Thus, to all integers $x$ arbitrarily assign one of the values $f(1)x$ and $-f(1)x$ to $f(x)$, and this function evidently satisfies the original assertion. Edit: corrected the solution set.

We claim that the answer is any function $f$ satisfying $|f(x)|=|cx|$ for an integer $c$. In other words, for each integer $x$, we have $f(x)=cx$ or $f(x)=-cx$. Putting $x=p$, where $p$ is an odd prime number, in the original equation tells us that $f(p)^2+y^2$ is a quadratic residue in $\pmod{p}$ for any integer $y$. So all the numbers $$f(p)^2+0^2, f(p)^2+1^2, f(p)^2+2^2, \cdots, f(p)^2+\left(\dfrac{p-1}2\right)^2$$are quadratic residues. All these numbers are pairwisely noncongruent in $\pmod{p}$ so they cover all quadratic residues and hence $$\left\{f(p)^2+0^2, f(p)^2+1^2, \cdots, f(p)^2+\left(\dfrac{p-1}2\right)^2\right\}= \left\{0^2, 1^2, \cdots, \left(\dfrac{p-1}2\right)^2\right\}$$in $\pmod{p}$. In particular, the sum of elements of these two sets must be congruent so $$\left(\dfrac{p+1}2\right)f(p)^2\equiv 0\Rightarrow f(p)\equiv 0\pmod{p}$$ Now, putting $y=1$ in the original equation tells us that $$f(x)^2+2xf(1)+1$$is a perfect square. If $|f(x)|>|f(1)x|$, then $$(|f(x)|-1)^2<f(x)^2+2xf(1)+1<(|f(x)|+1)^2$$Hence, we need to have $$f(x)^2+2xf(1)+1=(|f(x)|)^2\Rightarrow 2xf(1)=-1, \text{ contradiction}$$So, $|f(x)|\leq |f(1)x|$. In particular, $|f(p)|\leq |f(1)p|$ for all odd primes $p$. Also, we proved that $p|f(p)$. Then, $0\leq \left|\dfrac{f(p)}p\right|\leq |f(1)|$. Thus, there exist a positive integer $c$ such that $|f(p)|=cp$ for infinitely many odd primes $p$. Let $a$ be an arbitrary integer. Choose a sufficiently large odd prime $q$ such that $|f(q)|=cq$. Put $x=a, y=q$ in the original equation. $f(a)^2\pm 2acq+q^2$ is a perfect square. But, $$(q\pm ac-1)^2<f(a)^2\pm 2acq+q^2<(q\pm ac+1)^2$$since $q$ is large enough. Then, we must have $$f(a)^2\pm 2acq+q^2=(q\pm ac)^2\Rightarrow f(a)^2=(ac)^2\Rightarrow |f(a)|=|ac|$$Thus, we get $f(x)=cx$ or $f(x)=-cx$ for all integers $x$.

For some integer $c$, the function $f(x)=\pm cx$ satisfies the condition of the problem. (Note that for any integer $c$ we have $2^{\mathbb{Z}}$ many functions i.e. we can choose $f(x)=cx$ or $f(x)=-cx$ for each integer $x$, independently to any other value of function.) Now let us show that if $f(1)=c$ we have $|f(x)|=cx$ for all integers $x$. Firstly let $c=0$. Then put $y=1$ in the equation the we have $$f(x)^2+2xf(y)+y^2=f(x)^2+1$$is a perfect square which is only possible when $f(x)=0$ since only consecutive perfect squares are 0 and 1. Then let $c\neq 0$. From Dirichlet's Theorem we can find infinitely many primes $p$ with $2c|p-1$. Then choose $x=\frac{p-1}{2c}$ and put $y=1$ then $$f(x)^2+2xf(y)+y^2=f(x)^2+2\frac{p-1}{2c}+1=f(x)^2+p$$which is a perfect square. Since $p$ can be factorized in one way (4 ways if you count with negatives and order) but from factorization we can see that $f(x)=\pm\frac{p-1}{2}$ which means that $|f(x)|=|cx|$. So we have shown that we can find integers $x$ with arbitrarily large absolute value and satisfies $|f(x)|=|cx|$. Now let $x$ be an arbitrary integer, $y$ be an integer with $|f(y)|=|cy|$ and $|y|$ is very large compared to both $|x|$ and $|f(x)|$. We have $$f(x)^2+2xf(y)+y^2=f(x)^2+2cxy+y^2$$is a perfect square. Then there is an integer $a$ such that $$f(x)^2+2cxy+y^2=(y+cx+a)^2 = y^2+2cxy+c^2x^2+2ay+2acx+a^2$$then we have $$f(x)^2-c^2x^2=a(2y+2cx+a)$$Thus if $f(x)^2-c^2x^2\neq 0$ we have $|a|\leq |f(x)^2-c^2x^2|$ so $|a|$ is bounded in terms of $x,f(x),c$. Then we also have $$|f(x)^2-c^2x^2| \geq |2y+2cx+a|\geq 2|y|-|2cx+a|$$so we found that $|y|$ is bounded in terms of $a,x,f(x),c$ and since $a$ is bounded in terms of $x,f(x),c$ we get $|y|$ is bounded in terms of $x,f(x),c$, more rigorously we have $$2|y|\leq |f(x)^2-c^2x^2|+|2cx+a| \leq |f(x)^2-c^2x^2|+|2cx|+|a|\leq 2|f(x)^2-c^2x^2|+2|cx| $$So we ultimately get $$|y| \leq |f(x)^2-c^2x^2|+|cx| $$but we have shown that we can choose $|y|$ arbitrarily large so we get the desired contradiction thus we must have $f(x)^2=c^2x^2$ so we have $|f(x)|=|cx|$ and we are done.