The functional equation $ f\left(f(x^2)+y+f(y)\right)=x^2+2f(y) $ is equivalent to $ f\left(f(x)+y+f(y)\right)=x+2f(y) $, if $x\geq 0$. Let $y=x \geq 0$. We have $f\left(x+2f(x)\right) = x + 2f(x)$ and $f(0)=a \Rightarrow f\left(0+2f(0)\right)=f(2a)=2a$. Since $f$ is non-decreasing, $x\geq 0 \Rightarrow f(x) \geq f(0)=a \Rightarrow x+2f(x) \geq 2a$. So $f(x)=x$, if $x\geq M = \max(0, 2a) $. Let $M(y) \geq \max\left (M,M-y-f(y) \right)$ be a sufficiently large number. So $M(y) \geq M$ and $M(y) + y + f(y) \geq M$. We have $ f\left(f(x)+y+f(y)\right)=x+2f(y) $, if $x\geq 0$. For a real number $y$, take $x=M(y)$. We will have \[ \begin{array}{rcl} f\left(f\left(M(y)\right)+y+f(y)\right) &=& M(y)+2f(y)\\ f\left( M(y) + y + f(y)\right) &=& M(y)+y+f(y). \end{array} \] Thus, $M(y)+2f(y)=M(y)+y+f(y) \Rightarrow f(y)=y$. Since there is no constraint on $y$, The only candidate for $f$ is $f(x)=x$. \[ \begin{array}{rcl} f\left(f\left(x\right)+y+f(y)\right) &=& x+2f(y)\\ f(x+y+y)&=& x+ 2y \end{array} \] So, the only solution is $f(x)=x$. Edit: I'm not sure for the red part.

Easy to see that: if $x,z\geq 0, f(x)=f(z)$ then $x=z$(*) Let $f(0)=a$. If $a<0$ with x^2=-2a then exist a real b and f(b)=0 And f non-des then $b \leq 0$ With $\sqrt{x}=y=b$ then b=0. Contradition. If $a\geq 0$ : $F(2a)=2a,f(3a)=4a,f(5a)=4a$ then a=0 Hence $f(f(x))=x$ all nonnegative x. Then f is surjective in nonnegative.(**) (*)(**) and f non-decresing then f continuous in real nonnegative.(***) And $f(2f(y))=y+f(y)$ then y+f(y) is surjective. Hence $f(f(x^{2})+y+f(y))=f(f(x^{2})+f(f(y)+y)$ all x,y Then $f(u+v)=f(u)+f(v)$ all u,v nonnegative with (***) then $f(u)=cu$ u in real nonnegative. Easy to see c=1. Let $f(y)=-\frac{x^{2} }{2}$ then $f(x^{2})+y+f(y)=0$ Then $y=-\frac{x^{2} }{2}$ And of all : we have only $f(x)=x$

Easy to see that: if $x,z\geq 0, f(x)=f(z)$ then $x=z$(*) Let $f(0)=a$. If $a<0$ with x^2=-2a then exist a real b and f(b)=0 And f non-des then $b \leq 0$ With $\sqrt{x}=y=b$ then b=0. Contradition. If $a\geq 0$ : $F(2a)=2a,f(3a)=4a,f(5a)=4a$ then a=0 Hence $f(f(x))=x$ all nonnegative x. Then f is surjective in nonnegative.(**) (*)(**) and f non-decresing then f continuous in real nonnegative.(***) And $f(2f(y))=y+f(y)$ then y+f(y) is surjective. Hence $f(f(x^{2})+y+f(y))=f(f(x^{2})+f(f(y)+y)$ all x,y Then $f(u+v)=f(u)+f(v)$ all u,v nonnegative with (***) then $f(u)=cu$ u in real nonnegative. Easy to see c=1. Let $f(y)=-\frac{x^{2} }{2}$ then $f(x^{2})+y+f(y)=0$ Then $y=-\frac{x^{2} }{2}$ And of all : we have only $f(x)=x$

Could you please fix typos. It makes it really hard to understand.

vntbqpqh234 wrote: If $a\geq 0$ : $F(2a)=2a,f(3a)=4a,f(5a)=4a$ then $ a=0$ Can you explain the lines?

First you insert $x=0$ $y=0$ then $x^2=2a$ $y=0$ and then $x=0$ $y=2a$.

Claim 1: $f$ is one-to-one. To see why this holds, first, let $a=x^2$. Fixing a $y$, we get that, $f(f(a)+y+f(y))=a^2+2f(y)$, hence, if $f(a)=f(a')$ for $a,a'\geq 0$, we must necessarily have $a=a'$. Hence, $f(\cdot)$ is injective on $[0,\infty)$. For negative numbers, first, note that $f(x)$ can be made arbitrarily large (simply observe that, keeping $y$ at a fixed level and letting $x\to\infty$ yields this). Now, for any given $y,y'$, suppose $f(y)=f(y')$. Taking $x$ large enough that, both, $$ f(x^2)+y+f(y)>0 \quad \text{and} \quad f(x^2)+y'+f(y')>0, $$we obtain that $y=y'$ (since, these inputs are positive, and giving the same value, using the fact that $f$ is an injection, when restricted on non-negative reals, we conclude). Claim 2: $f(0)=0$. For this part, we need to carry out a few case analysis. First, taking $x=y=0$, we obtain $f(2f(0))=2f(0)$. $\bullet$ $f(0)\geq 0$. In this case, taking $x=\sqrt{2f(0)}$, we get, $$ f(2f(0)+y+f(y))=2f(0)+2f(y)\underbrace{\implies}_{y=0}f(3f(0))=4f(0). $$Now, taking $x=0$, and $y=2f(0)$, we get, $$ f(f(0)+2f(0)+f(2f(0)))=2f(2f(0))\underbrace{\implies}_{f(2f(0))=2f(0)} f(5f(0))=4f(0), $$hence, $3f(0)=5f(0)$ (due to injectivity), thus $f(0)=0$. $\bullet$ $f(0)<0$. Taking $x^2 = -2f(0)$, and $y=2f(0)$, we get, $$ f(f(-2f(0))+2f(0)+f(2f(0)))=-2f(0)+2f(2f(0))\implies f(f(-2f(0))+4f(0))=2f(0), $$and since $f(2f(0))=2f(0)$, and $f$ is one-to-one, we have,$\boxed{f(-2f(0))=-2f(0)}$. Now, taking again $x^2 = -2f(0)$, and $y=0$, we get, $$ f(f(-2f(0))+0+f(0))=0\implies \boxed{f(-f(0))=0}. $$Finally, letting $y=-f(0)$, and $x=0$, we have, $$ f(f(0)-f(0)+f(-f(0)))=0+2f(-f(0))\implies f(0)=0. $$Thus, $f(0)$ is necessarily $0$. Claim 3: $f(x)=x$. For this, first take $y=0$, we get, $f(f(a))=a$, for any $a\geq 0$. Also, note that if $a\geq 0$, we must also have $f(a)\geq f(0)=0$. Now, suppose, there exists an $b>0$ such that $f(b)<b$. Taking $y=f(b)$, and $x<b$, we have, $y\leq x\implies b=f(f(b))=f(y)<f(x)=f(b)$, contradicting with $f(b)<b$. Similarly, if there exists a $c>0$ such that $f(c)>c$, again taking $y=c$ and $x=f(c)$, since $x>y$, we have $c=f(f(c))=f(x)>f(y)=f(c)$, giving a contradiction. Hence, for every non-negative real-number $x$, $f(x)=x$. Finally, we need to study the case of negative numbers. Take a $y$ such that $y<0$. For a fixed, positive, $x_0$, $$ f(f(x_0^2)+y+f(y))=2f(y)+x_0^2\implies f(x_0^2+y+f(y))=2f(y)+x_0^2. $$Now, for any given $y$, taking $x_0$ sufficiently large, we can make the input $x_0^2 + y+f(y)$ eventually positive. Since we have already proven that $f(a)=a$, and $f(\cdot)$ is injective, this would yield, $$ x_0^2 + y+f(y)=2f(y)+x_0^2 \implies y=f(y), $$with the aid of $x_0$. Hence, the only such function is $f(x)=x$.

Let $P(x,y)$ be the assertion. $P(0,0)$ gives us $f(2f(0))=2f(0)$. Take $y=0$ and vary $x$. See that all sufficiently large real numbers are in the range of $f$. Claim 1: $f$ is injective Proof: Let $f(a)=f(b)$ where $a\neq b$. By $P(x,a)$ and $P(x,b)$, one can find that $f(f(x^2)+a+f(a))=x^2+2f(a)=x^2+2f(b)=f(f(x^2)+b+f(b))=f(f(x^2)+b+f(a))$. Take $x$ large enough such that $f(x^2)+a+f(a), f(x^2)+b+f(a)>0$. Then, there exist distinct positive real numbers $u$ and $v$ such that $f(u)=f(v)$. Now, by $P(\sqrt{u},y)$ and $P(\sqrt{v},y)$, one can find that $u^2=v^2$. But, $u\neq v$ and $u, v>0$. Contradiction, done. This shows us that $f$ is increasing. Claim 2: There exist a real number $r$ such that $f(r)=0$. Proof: If there exist a real number $t$ such that $f(t)\leq 0$, then $P\left(\sqrt{-2f(t)},t\right)$ gives us the desired result. Suppose that $f(x)>0$ for all $x\in\mathbb{R}$. By $P\left(\sqrt{2f(0)}, y\right)$, one can find that $f(2f(0)+y+f(y))=2f(0)+2f(y)>f(2f(0))$. Since $f$ is increasing, we find that $y+f(y)>0$ for all $y\in\mathbb{R}$. Now $P(\sqrt{2f(0)-2f(y)}, y)$ where $y<0$ gives us $f(f(2f(0)-2f(y))+y+f(y))=2f(0)=f(2f(0))\Rightarrow f(2f(0)-2f(y))+y+f(y)=2f(0)$. Then, $2f(0)>f(2f(0)-2f(y))$ for all $y<0$. Hence, $2f(0)+2f(0)-2f(y)>f(2f(0)-2f(y))+2f(0)-2f(y)>0\Rightarrow 2f(0)>f(y)\Rightarrow 2f(0)+y>f(y)+y>0\Rightarrow y>-2f(0)$. But, $y$ can be any negative real number, contradiction. Claim 3: $f(0)=0$. Proof: By $P(0, r)$, one can find that $f(f(0)+r)=0=f(r)\Rightarrow f(0)+r=r\Rightarrow f(0)=0$, as desired. Now, $P(x, 0)$ gives us $f(f(x^2))=x^2$. If $f(x^2)>x^2$ for a real number $x$, then $f(f(x^2))>f(x^2)>x^2$, contradiction. If $f(x^2)<x^2$ for a real number $x$, then $f(f(x^2))<f(x^2)<x^2$, contradiction. Hence, $f(x^2)=x^2$ for all real numbers $x$. Thus, $f(x)=x$ for all $x>0$. Finally, fix $y$ and take $x$ large enough such that $f(x^2)+y+f(y), x^2+2f(y)>0$. We have $f(f(x^2)+y+f(y))=x^2+2f(y)=f(x^2+2f(y))\Rightarrow f(x^2)+y+f(y)=x^2+2f(y)\Rightarrow y=f(y)$. Hence, the only solution is $\boxed{f(x)=x}.$

similar problem BMO 2021 p2.according to this problem, $f (x) = x$ at $ x> 0 $. For $y$ we choose a sufficiently large $x$ where $f(x^2) + y+f(y)>0.$ $\implies f(y) =y$ but non-decreasing was not required.

https://artofproblemsolving.com/community/c6h2667350p23111675

\[f(f(x^2)+y+f(y))=x^2+2f(y)\]Only such function is $f(x)=x$ which clearly works. Let $S$ be the image set of $f$. Plugging $x=0$ gives $f(y+f(y)+f(0))=2f(y)$ hence $\frac{f(x)}{2}\in S$. Since $f$ is surjective for $\geq 2f(0)$, $f$ is surjective over positive reals since $2^k\in S\rightarrow 2^{k-1}\rightarrow S\rightarrow \dots \rightarrow \frac{1}{2^m}\in S$ and if $f(x)\in S$, then $y\geq f(x)$ is an element of $S$. Case $I$: There exists $f(a)=0$. Then, subsitute $y=a$ to verify $f(f(x^2)+a)=x^2$. Pick $x=0$,we have $f(a+f(0))=0$ so if $f(a)=0$, then $f(a+nf(0))=0$ for integers $n$. If $f(0)$ is positive, since $f$ is non-decreasing we see that $f(x)=0$ for $x\geq C$ however this implies $f\leq 0$ which contradicts by $0\geq (f(f(x^2)+y+f(y))=x^2+2f(y)$ since we can choose $x$ sufficiently large. If $f(0)$ is negative, then $f(x)=0$ for $f\leq C$ and this implies $f\geq 0$. Pick $y<C$ in order to get $x^2=x^2+2f(y)=f(f(x^2)+y)$ but decreasing $y$ causes right hand side to be $0$ which is a contradiction. Thus, $f(0)=0$. \[f(f(x^2)+0+f(0))=x^2+2f(0)\iff f(f(x^2))=x^2\]Hence $f$ is involution over positive reals. Comparing $x,f(y)$ with $x,y$ yields \[x^2+2f(y)=f(f(x^2)+y+f(y))=f(f(x^2)+f(y)+y)=x^2+2y, \ \ y>0\]So $f(x^2)=x^2$. We have $f(x^2+y+f(y))=x^2+2f(y)$ and pick $x$ sufficiently large to get $x^2+y+f(y)>0$. This implies $x^2+y+f(y)=x^2+2f(y)$ or $f(y)=y$ for all reals which is a solution.$\square$ Case $II$: There doesn't exist $f(a)=0$. If $f(t)<0$, then $2f(t)+x^2\in S$ and this contradicts with $0\not \in S$. Thus, $f>0$. Note that $f$ is surjective over positive reals. Let $\epsilon>f(k)$ and choose $y<<k$. \[x^2<x^2+2f(y)=f(f(x^2)+y+f(y))\leq f(f(x^2)+y+\epsilon)\]If we let $y<k-f(x^2)-\epsilon$ then $x^2<f(f(x^2)+y+\epsilon)<f(k)<\epsilon$ however this is impossible for large $x$ as desired.$\blacksquare$