jasperE3 wrote: Let $P(x,y)$ be the assertion. $P(0,x)\Rightarrow f(f(x))=x$ $P(x,f(y)-x)\Rightarrow Q(x,y):f(x^2+y)=f(y)+x(f(x+1)-1)$ $Q(x,0)\Rightarrow f(x^2)=f(0)+x(f(x+1)-1)$ $Q(-x,-x^2)\Rightarrow f(-x^2)=f(0)+x(f(-x+1)-1)$ So $f(x+1)=f(-x+1)$ for $x\ne0$. This gives a contradiction with injectivity from $f(f(x))=x$. Doesn't $f(x)=x$ work?

Using your progress @jasperE3, after $Q(x,0)$ you get $f(x^2+y)+f(0)=f(x^2)+f(y)$, which can easily become $f(x)+f(y)=f(x+y)+f(0)$. Going back into the original eqn, From $f(x^2)=f(0)+x(f(x+1)-1)$ and putting $x\rightarrow x+1$, we get $f(x^2)+2f(x)+f(1)-3f(0)=f(0)+(x+1)(f(x+1)+f(1)-f(0)-1)$. Hence, $2f(x)+f(1)-3f(0)=f(0)+f(x+1)+(x+1)(f(1)-f(0))-1$. Note $f(x+1)=f(x)+f(1)-f(0)$, so $f(x)=ax+b$ for some constants. Hence $f$ is linear. I may have messed up some constants, but this should work.

My solution in the camp: Answer: $f(x)=x \ \forall x\in \mathbb{R}$, $f(x)=2-x \ \forall x\in \mathbb{R}.$ We can easily verify that these two functions are indeed solutions. Let $P(x,y)$ denote the given FE, we have \[P(0,x)\implies f(f(x))=x.\]This gives us $f$ is bijective as it is an involution. In particular, $f(f(0))=0.$ Also, \[P(-1,1)\implies f(1+f(0))=1-f(0)\]\[P(1,f(0)-1) \implies f(1)=f(0)-1+f(2)\]\[P(1,-1)\implies f(1+f(0))=-1+f(2),\]so $f(1)=f(0)-1+f(2)=f(0)+f(1+f(0))=f(0)+1-f(0)=1$. Now, \[P(f(0)-1,1) \implies f((f(0)-1)^2)=1\]since $f(1)=1, f(0)=0 \ \text{or} \ 2.$ Case 1: $f(0)=0.$ We have \[P(-1,x+1) \implies f(1+f(x))=x+1\]as $f(f(x+1))=x+1, f(x+1)=f(x)+1 \ \forall x\in \mathbb{R}.$ Then, \[P(x,-x)\implies f(x^2)=x+xf(x+1)=xf(x).\]The original FE can be rewritten as \[f(x^2+f(x+y))=f(x^2)+f(f(x+y))\]\[P(x,f(y)-x)\implies f(x^2+y)=f(x^2)+f(y) \ \ \ \forall x,y\in \mathbb{R},\]this shows that $f(u+v)=f(u)+f(v) \ \forall u\in \mathbb{R}_{\ge 0}, v\in \mathbb{R}$. We now show that $f$ is an odd function, \[P(-x,x) \implies f(x^2)=-xf(-x)=xf(x) \implies f(x)=-f(-x)\]combining with $f(0)=0,$ we indeed have $f$ being odd. Hence, $\forall u\in \mathbb{R}_{\ge 0}, v\in \mathbb{R},$ \[f(-u+v)=-f(u-v)=-f(u)-f(-v)=f(-u)+f(v),\]this proves the additivity of $f$. Finally, \[xf(x)+2f(x)+1=f(x^2)+2f(x)+1=f((x+1)^2)=(x+1)f(x+1)=xf(x)+x+f(x)+1\]yields $f(x)=x\ \ \forall x \in \mathbb{R}$ in this case. Case 2: $f(0)=2.$ We let $g(x)=2-f(x),$ then let $Q(x,y)$ denote the new FE by expressing $f(x)$ in terms of $g(x),$ \[Q(x,y): 2-g(x^2+2-g(x+y))=y+2x-xg(x+1),\]and $g(0)=0, g(1)=1.$ We have \[Q(0,0) \implies g(2)=2\]\[Q(0,x) \implies g(2-g(x))=2-x\]\[Q(x,2-x) \implies g(x^2)=xg(x+1)-x\]\[Q(1,x) \implies g(3-g(x+1))=2-x=g(2-g(x)) \implies g(x+1) = g(x)+1\]so $g(x^2)=x(g(x)+1)-x=xg(x)$ and $2-x=g(2-g(x))=2-g(g(x)) \implies g(g(x))=x$. Similarly, we can show that $g$ is odd, \[xg(x)=g(x^2)=-xg(-x) \implies g(x)=-g(-x), x\ne 0\]and since $g(0)=0,$ $g$ is indeed odd. We can write $Q(x,y)$ as \[g(x^2-g(x+y))=-(2-g(x^2+2-g(x+y)))=xg(x+1)-y-2x=xg(x)-x-y=g(x^2)-g(g(x+y))\]\[Q(x,-g(y)-x) \implies g(x^2+y)=g(x^2)+g(y).\]This is the same FE that we have dealt in Case 1, $g(x)=x \implies f(x)=2-x \ \forall x \in \mathbb{R}. \blacksquare$

Let $P(x,y)$ denote the given assertion. $P(0,x): f(f(x))=x$. Thus, $f$ is bijective. Let $k$ satisfy $f(k+1)=0$. $P(k,1): f(k^2)=1$. So $k^2=f(1)$. $P(1,k^2-1): f(2)=k^2-1+f(2)\implies k^2=1$. Case 1: $k=-1$. Then $f(0)=0$. $P(-1,x): f(f(x-1)+1)=x\implies f(x-1)+1=f(x)$. FE can be rearranged to $f(x^2+f(x+y))=xf(x)+x+y$. $P(x,-x): f(x^2)=xf(x)$. $P(-x,x): f(x^2)=-xf(-x)$. So $xf(x)=-xf(-x)$, so $f$ is odd. Now $f(x^2+f(x+y))=f(x^2)+f(f(x+y))$. Claim: $f(a+b)=f(a)+f(b)$ for all reals $a$ and $b$. Proof: Casework on parity of $a$ and $b$. Case 1: $a,b\ge 0$. Then set $x$ so that $x^2=a$ and $y$ so that $f(x+y)=b$. Case 2: $a\ge 0, b<0$. Do the same process as the above case. Case 3: $a<0, b\ge 0$. Set $x$ so that $x^2=b$ and $y$ so that $f(x+y)=a$. Case 4: $a,b<0$. Then \[f((-a)+(-b))=f(-a)+f(-b)\implies f(-(a+b))=f(-a)+f(-b)\implies -f(a+b)=-(f(a)+f(b))\implies f(a+b)=f(a)+f(b).\]$\blacksquare$ Now we have \[f((x+1)^2)=f(x^2+2x+1)=f(x^2)+2f(x)+1=(x+1)f(x+1)=(x+1)(f(x)+1)=xf(x)+f(x)+x+1=f(x^2)+f(x)+x+1\] So $2f(x)+1=f(x)+x+1\implies \boxed{f(x)=x}$. Case 2: $k=1$. Then $f(2)=0, f(0)=2, f(1)=1$. Let $g(x)=2-f(x)$ with $g(0)=0, g(1)=1, g(2)=2$. $g$ is also bijective. Then let the new assertion $Q(x,y)$ be \[2-g(x^2+2-g(x+y))=y+2x-xg(x+1).\] $Q(0,x): 2-g(2-g(x))=x$. $Q(x,2-x): 2-g(x^2)=2-x+2x-xg(x+1)\implies g(x^2)=xg(x+1)-x$. $Q(1,x): 2-g(3-g(x+1))=x\implies g(3-g(x+1))=g(2-g(x))\implies 3-g(x+1)=2-g(x)\implies g(x)+1=g(x+1)$. So $g(x^2)=xg(x)$. $Q(-x,x): 2-g(x^2+2)=-x+xg(-x+1)\implies 2-(g(x^2)+2)=-x+xg(-x)+x\implies -g(x^2)=xg(-x)\implies xg(x)=-xg(-x)$, so $g$ is odd. Thus, $2-x=g(2-g(x))=g(-g(x))+2\implies g(-g(x))=-x\implies g(g(x))=x$. The FE rearranges to $2-g(x^2-g(x+y))-2=2x+y-xg(x)-x\implies -g(x^2-g(x+y))=x+y-xg(x)\implies g(x^2-g(x+y))=g(x^2)-(x+y)$. Now $g(x^2-g((-x)+(-y)))=g(x^2)-((-x)+(-y))\implies g(x^2-g(-x-y))=g(x^2)+x+y\implies g(x^2+g(x+y))=g(x^2)+x+y$. This is the same as in Case 1, so $g$ is additive and linear, which implies $g(x)=x$. So $\boxed{f(x)=2-x}$.

Solution without additivity. Solved with Mirhabib. First of all, $P(0,x)$ and $P(x,0)$ gives us that $f(f(x))=x$(and so f is bijective) and $f(x^2+f(x))=xf(x+1) (*)$. So $f(f(0))=0$. Let $f(0)$ be $a$ ,then $f(a)=0$. Then $P(a-1,1)$ gives us that $f((a-1)^2)=f(f(1))$. Note that $x=1$ in $(*)$ gives us that $f(1+f(1))=f(2)$ and so $f(1)=1$.Therefore $(a-1)^2=f(1)=1$ since f is injective. Therefore $a=0$ or $2$. Case 1:$f(0)=0$: $P(1,-1)$ yields that $f(2)=2$. Use this in $P(1,x-1)$ to get $f(1+f(x))=1+x$. So, $f(x+1)=f(f(1+f(x))=1+f(x)(**)$. Let $P(x,-x)$: $f(x^2)=xf(x+1)-x=xf(x)$ by $(**)$. Let $x=f(x)$ in the last equality: $f(f(x)^2)=xf(x)=f(x^2)$ , so $f(x)^2=x^2$. Let $f(a)=a, f(b)=-b$ for some $a,b$ not equal to $0$. Then $P(a,b-a)$ in the original equation gives us that $f(a^2-b)=a^2+b$ ( since $f(a+1)=f(a)+1=a+1$). İdentifying 2 cases, we get contradiction for both. So,by verifying, we get $f(x)=x$ for all $x$. Case 2:$f(0)=2$: First of all, note that $P(1,f(0)-1)$ gives us that $f(2)=2$. Then define the function $g(x)=2-f(x)$. So $g(1)=1,g(2)=2,g(0)=0$. The original equation becomes $xg(x+1)+2=y+2x+g(x^2+2-g(x+y))$.Denote this as $Q(x,y)$. Compare $Q(0,x)$ and $Q(1,x)$ to get $g(2-g(x))=g(3-g(x+1))$. Since $g$ is injective as $f$ was injective, we get that $g(x)+1=g(x+1)$ for all $x$. Now let $Q(x,-x)$ to obtain $-g(x^2)=2-g(x^2+2)=x-g(x+1)x=-g(x)=x \implies g(x^2)=xg(x)$. So $g$ is odd. $Q(-1,x)$ gives us that $4-x=g(3-g(x-1))=g(4-g(x))=g(-g(x))+4$. Since $g$ is odd, we get $g(g(x))=x$. Again, similar to the other case, note that $g(g(x)^2)=g(x)g(g(x))=xg(x)=g(x^2) \implies g(x)^2=x^2$. We ended up into a pointwise trap similarly. Let $g(a)=a, g(b)=-b$($a,b$ are different from 0) and look at $Q(a,b-a)$. We get $a^2+2-b=g(a^2+2+b)$. Looking at both cases we derive into a contradiction. Therefore, verifying in $Q(x,y)$, we get $g(x)=x$ for all $x \implies f(x)=2-x$ for all $x$. Done.