Given $x=y=0 \Rightarrow (f(0))^2=1$ $*)f(0)=1$ Given$y=0\Rightarrow f(x)\equiv 1 \forall x$ $)f(0)=-1$ Given $x=-y$we have: $f(x)f(-x)-1=f(-x^2)-2x^2+1$(*) Given $x=1$we have $f(-1)(1-f(1))=0\Rightarrow f(-1)=0$or$f(1)=1$ $+)f(1)=1$ Given y=1,we have: $f(x+1)=2x+1$ $\Rightarrow f(x)=2x-1 \forall x$ $+)f(-1)=0$ Given $y=-1$,we have: $f(-x)=f(x-1)+2x-1$ Given$y=-1-x$we have: $f(x)(f(x)+2x+1)=f(x^2+x-1)$ Given$x=1$we have: $f(1)=f(1)(f(1)+3)$ $\Rightarrow f(1)=0$or$f(1)=-2$ .)$f(1)=0$ Given$y=1$we have $f(x+1)=f(x)+2x+1$ $\Rightarrow f(2)=3$ Given$y=2$we have: $f(x+2)+3f(x)=f(2x)+4x+1$ $f(x+2)=f(x)+4x+1 \Rightarrow f(2x)=4f(x)+3$ Given $x=x+1,y=x-1$we have: $f(2x)+f(x-1)f(x+1)=f(x^2-1)+2x^2-1$ But $f(x^2-1)=f(x^2)\2x^2+1)$ $\Rightarrow 4f(x)+3+(f(x)-2x+1)(f(x)+2x+1)=f(x^2)$ $\Rightarrow (f(x))^2+6f(x)+4-4x^2=f(x^2)$(1) Given $x=y$we have: $f(2x)+(f(x))^2=f(x^2)+2x^2+1$ $\Rightarrow f(x^2)=(f(x))^2+4f(x)+2-2x^2$(2) (1)(2)$\Rightarrow 2f(x)+2-2x^2=0$ $\Rightarrow f(x)=x^2-1\forall x$ .)Similar with $f(1)=-2$ we have:$f(x)=-x-1$ The function satisfly: $f(x)=2x-1\forall x$ $f(x)=x^2-1\forall x$ $f(x)=-x-1\forall x$

vietnamesegauss89 wrote: $\Rightarrow 4f(x)+3+(f(x)-2x+1)(f(x)+2x+1)=f(x^2)$ How you get that ? I get $4f(x)+3+(f(x)-2x+1)(f(x)+2x+1)=f(-x^2)$ Anyone help me please!

Is there a shorter solution?

can anyone explain how he could get : Given $y=-1-x$ we have , $f(x)(f(x)+2x+1)=f(x^2+x-1)$ thanks

Hi Here you have found as follows vietnamesegauss89 wrote: Given$y=-1-x$we have: $f(x)(f(x)+2x+1)=f(x^2+x-1)$ Given$x=1$we have: $f(1)=f(1)(f(1)+3)$ $\Rightarrow f(1)=0$or$f(1)=-2$ $\Rightarrow f(1)=0$or$f(1)=-2$. you used "or" but it could be the same or I am wrong. explain it how to discover that they are different Abdurashid

They are allowed to be the same; the or is not exclusive in that scenario. Here's my solution: Let $ P(x,y)$ be the assertion that $ f(x + y) + f(x)f(y) = f(xy) + 2xy + 1$. $ P(0,0)$ yields $ f(0) = \pm 1$. If $ f(0) = 1$, then $ P(x,0)$ yields $ f(x) = 1$ for all $ x$, which $ P(3,3)$ shows is impossible, so $ f(0) = - 1$. $ P(x, - x)$ now gives $ f(x)f( - x) = f( - x^2) - 2x^2 + 2$. Setting $ x = 1$ yields $ f(1)f( - 1) = f( - 1)$, so $ f(1) = 1$ or $ f( - 1) = 0$. If $ f(1) = 1$, then $ P(x,1)$ yields $ f(x + 1) + f(x) = f(x) + 2x + 1$, so $ f(x + 1) = 2x + 1$, so $ f(x) = 2x - 1$ for all real $ x$. It can easily be seen that $ f(x) = 2x - 1$ indeed satisfies this functional equation. Suppose now that $ f(1) \neq 1$; then $ f( - 1) = 0$. $ P(1, - 2)$ yields $ f( - 1) + f(1)f( - 2) = f( - 2) - 3$, that is, $ f(1)f( - 2) = f( - 2) - 3$. $ P( - 1,2)$ gives $ f(1) + f( - 1)f(2) = f( - 2) - 3$, that is, $ f(1) = f( - 2) - 3$. Therefore, $ f(1) = f(1)f( - 2)$, so $ f(1) = 0$ or $ f( - 2) = 1$. $ f(1) = 0$ will give $ f(x) = x^2 - 1$. $ P(x, - 1)$ and $ P( - x,1)$ yield $ f(x - 1) = f(1 - x) = f( - x) - 2x + 1$, so $ f$ is even. $ P(x, - x)$, combined with the fact that $ f$ is even, yields $ f(x)^2 = f(x^2) - 2x^2 + 2$, while $ P(x,x)$ gives $ f(2x) + f(x)^2 = f(x^2) + 2x^2 + 1$. Since $ f(x)^2 = f(x^2) - 2x^2 + 2$, we see that $ f(2x) = 4x^2 - 1$, so $ f(x) = x^2 - 1$ for all $ x$; it can easily be verified that $ f(x) = x^2 - 1$ satisfies the functional equation. If $ f( - 2) = 1$, then $ f(1) = f( - 2) - 3 = - 2$. $ P(1, - x)$ yields $ f(1 - x) = 3f( - x) - 2x + 1$, while $ P( - 1,x)$ yields $ f(x - 1) = f( - x) - 2x + 1$. Subtracting these two equations gives $ 2f( - x) = f(1 - x) - f(x - 1)$. Substituting $ x = 1 - y$ yields $ 2f( - 1 + y) = f(y) - f( - y)$. Replacing $ y$ with $ - y$ yields $ 2f( - 1 - y) = f( - y) - f(y) = - 2f( - 1 + y)$, so $ f( - 1 + y) = - f( - 1 - y)$ for all real $ y$. $ P(x, - 1)$ yields $ f(x - 1) = f( - x) - 2x + 1$, and $ P( - x, - 1)$ yields $ f( - x - 1) = f(x) + 2x + 1$. Since $ f( - x - 1) = - f(x - 1)$, we have that $ f(x - 1) = - f(x) - 2x - 1 = f( - x) - 2x + 1$, so $ f(x) + f( - x) = - 2$ for all real $ x$. $ P(0,0)$ yields $ f(x)f( - x) = f( - x^2) - 2x^2 + 2$, so $ f(x)( - 2 - f(x)) = - 2 - f(x^2) - 2x^2 + 2$. Thus, $ 2f(x) + f(x)^2 = f(x^2) + 2x^2$. On the other hand, $ P(x,x)$ yields $ f(2x) + f(x)^2 = f(x^2) + 2x^2 + 1$. Subtracting $ 2f(x) + f(x)^2 = f(x)^2 + 2x^2$ gives $ f(2x) - 2f(x) = 1$. $ f(2) = - 2 - f( - 2) = - 3$, so $ P(x,2)$ yields $ f(x + 2) - 3f(x) = f(2x) + 4x + 1$. But $ f(2x) = 2f(x) + 1$, so $ f(x + 2) = 5f(x) + 4x + 2$. On the other hand, $ P(x,1)$ gives $ f(x + 1) = 3f(x) + 2x + 1$; substituting $ x = y + 1$ here yields $ f(y + 2) = 3f(y + 1) + 2y + 3$. But $ f(y + 1) = 3f(y) + 2y + 1$, so $ f(y + 2) = 9f(y) + 8y + 6$. However, $ f(y + 2) = 5f(y) + 4y + 2$ as well, so $ 4f(y) = - 4y - 4$, yielding $ f(y) = - y - 1$ for all real $ y$. It can easily be verified that $ f(x) = - x - 1$ satisfies the functional equation. Hence, our three solutions to this functional equation are $ \boxed{f(x) = 2x - 1, x^2 - 1, - x - 1}$.

This is my solution from WOOT some time ago... Let $P(x,y)\implies f(x+y)+f(x)f(y)=f(xy)+2xy+1$. First, note that $P(1,-1)\implies f(-1)[f(1)-1]=0$. Case 1: $f(1)=1$. Then \[P(x,1)\implies f(x+1)=2x+1\implies f(x)=2x-1\]for all $x$. Case 2: $f(1)\ne1\implies f(-1)=0$. Note that $P(x,0)\implies f(0)=-1$ (otherwise, $f(x)$ is constant, which is clearly impossible since $2xy$ is surjective). Now let $c=1-f(1)$ (by assumption, $c\ne0$). Then \[P(x,1)\implies f(x+1)=cf(x)+2x+1.\]The second order difference gives us \[f(x+3)-(c+2)f(x+2)+(2c+1)f(x+1)-cf(x)=0\]for all $x$. Subcase 2.1: If $c=1$ (i.e. $f(1)=0$), so $f(x+1)=f(x)+2x+1$ for all $x$ and we find by simple induction that $f(n)=n^2-1$ and $f(x+n)=f(x)+(x+n)^2-x^2$ for all integers $n$, whence \[P(x,\pm n)\implies f(-xn)=f(xn)+1=n^2[f(x)+1].\]Thus $f$ is even, and \[P(x,x)\implies f(x^2)=[f(x)]^2+4f(x)-2x^2+2\]while \[P(x,-x)\implies f(x^2)=f(-x^2)=[f(x)]^2+2x^2-2.\]Equating, we arrive at \[f(x)=x^2-1,\]which indeed satisfies the original equation. Subcase 2.2: If $c\ne1$ (i.e. $f(1)\ne0$), then the characteristic polynomial of the sequence $f(n)$ has roots $1,1,c$ and \[f(n)=\alpha+\beta n+\gamma c^n\]for all integers $n$ (for some real constants $\alpha,\beta,\gamma$). Considering $f(-1),f(0),f(1)$, we find that \[0=\alpha-\beta+\frac{\gamma}{c},\quad -1=\alpha+\gamma,\quad 1-c=\alpha+\beta+\gamma c.\]Solving this system, \[f(n)=-\frac{c+1}{(c-1)^2}+\frac{2n}{1-c}-\frac{c(c-3)}{(c-1)^2}c^n.\]Now, \[P(2,2)\implies f(2)=\pm3.\]If $f(2)=3$, then \[3=f(2)=cf(1)+2(1)+1=cf(1)+3\implies cf(1)=0.\]But we have $c\ne0$ and $f(1)\ne0$. So $f(2)=-3$, and \[-3=f(2)=cf(1)+2(1)+1\implies [f(1)-3][f(1)+2]=0.\] Subcase 2.2.1: $f(1)=3\implies c=-2$. Then \[f(n)=\frac{1}{9}+\frac{2n}{3}-\frac{10}{9}(-2)^n\]for all integers $n$, so $f(2)=-3$, $f(-2)=-3/2$, and $f(-4)=-21/8$. This contradicts $P(2,-2)\implies f(2)f(-2)=f(-2^2)$. Subcase 2.2.2: $f(1)=-2\implies c=3$. Then \[f(n)=-1-n\]for all integers $n$. By induction, we find $f(x+n)=3^nf(x)+(3^n-1)x+(3^n-n-1)$. Thus \[P(x,n)\implies f(xn)+xn+1=(3^n-n-1)[f(x)+x+1].\]So \begin{align*} (3^n-n-1)^2[f(x)+x+1]&=(3^n-n-1)[f(xn)+xn+1]\\ &=f(xn^2)+xn^2+1=(3^{n^2}-n^2-1)[f(x)+x+1] \end{align*}for all real $x$ and integers $n$. Take $n=-1$. We find \[f(x)=-x-1.\] Finally, our solutions are $f(x)=2x-1\forall x$, $f(x)=x^2-1\forall x$, and $f(x)=-x-1\forall x$ (all three work).

andre.l wrote: Find all functions $ f: \mathbb{R}\to\mathbb{R}$ such that $ f\left(x+y\right)+f\left(x\right)f\left(y\right)=f\left(xy\right)+2xy+1$ for all real numbers $ x$ and $ y$. Let $P(x,y)$ be the assertion $f(x+y)+f(x)f(y)=f(xy)+2xy+1$ Let $f(0)=a$ Let $f(1)=b$ Let $f(-1)=c$ (e1) : $P(x,-1)$ $\implies$ $f(x-1)+cf(x)=f(-x)-2x+1$ (e2) : $P(x-1,1)$ $\implies$ $f(x)+(b-1)f(x-1)=2x-1$ (1-b)e1 + e2 $\implies$ $(1+c-bc)f(x)+(b-1)f(-x)=-2bx-b$ changing $x\to -x$, we get $(1+c-bc)f(-x)+(b-1)f(x)=2bx-b$ And so the system : $(1+c-bc)f(x)+(b-1)f(-x)=-2bx-b$ $(b-1)f(x)+(1+c-bc)f(-x)=2bx-b$ If the determinant of the system is non zero, this gives $f(x)=ux+v$ for some real $u,v$ and plugging this in original equation, we get two solutions $\boxed{f(x)=-x-1}$ and $\boxed{f(x)=2x-1}$ If the determinant of the system is zero, this means : either $1+c-bc=b-1$ and so $-2bx-b=2bx-b$ and so $b=0$ either $1+c-bc=1-b$ and so $-2bx-b=-(2bx-b)$ and so $b=0$ So $b=0$ and : $P(0,1)$ $\implies$ $b+ab=a+1$ and so $a=-1$ $P(-1,1)$ $\implies$ $a+bc=c-1$ and so $c=0$ The system above becomes then $f(x)=f(-x)$ and the function is even. $P(\frac x2,\frac x2)$ $\implies$ $f(x)+f(\frac x2)^2=f(\frac {x^2}4)+\frac{x^2}2+1$ $P(\frac x2,-\frac x2)$ $\implies$ $-1+f(\frac x2)^2=f(\frac {x^2}4)-\frac{x^2}2+1$ Subtracting these two lines implies $\boxed{f(x)=x^2-1}$ which indeed is a solution

Let $P(x,y)$ the assertion : $f(x+y)+f(x)f(y)=f(xy)+2xy+1$ $P(x,0) : (f(0)+1)(f(x)-1)=0$, we can see easily that the constant function $f \equiv 1$ is not a solution then $f(0)=-1$ $P(1,-1): f(-1)(f(1)-1)=0$ - If $f(1)=1$ then : $P(x,1): f(x+1)=2x+1$ which give us $\forall x \in \mathbb{R} : f(x)=2x-1$ which is indeed a solution. - If $f(-1)=0$ : $P(2,-1) : f(-2)=f(1)+3$ $P(-2,1): f(-2)(1-f(1))=3$ hence either $f(1)=0$ either $f(1)=2$ -- If $f(1)=0$ $P(x,1): f(x+1)=f(x)+2x+1$ $P(x,-1):f(x-1)=f(-x)-2x+1$ $P(-x,1):f(-x+1)=f(-x)-2x+1$ hence we get $f(-x+1)=f(x-1)$ which mean that f is even. $P(x,-x): (f(x))^{2}=f(x^{2})+2-2x^{2}$ $P(x,x): f(2x)+(f(x))^{2}=f(x^{2})+2x^{2}+1$ hence $ \forall x \in \mathbb{R} : f(2x)=4x^{2}-1=(2x)^{2}-1$ hence we get $\forall x \in \mathbb{R}: f(x)=x^{2}-1$ -- If $f(1)=-2$: $P(x,1):f(x+1)=3f(x)+2x+1$ Hence :$f(x+2)=9f(x)+8x+6 \forall x$ (*) Otherwise : $P(x,-1): f(x-1)=f(-x)-2x+1$ $P(-x,1): f(-x+1)-2f(-x)=f(-x)-2x+1 \Rightarrow f(-x)=f(x-1)+2x-1=\frac{f(-x+1)+2x-1}{3}$ Hence : $f(-x+1)=3f(x-1)+4(x-1)+2$ which mean $\forall x \in \mathbb{R} f(-x)=3f(x)+4x+2$ $P(x,-x): -1+f(x)(3f(x)+4x+2)=3f(x^{2})+4x^{2}+2$ which is equivalent to : $f(x^{2}) = \frac{-3-4x^{2}+f(x)(3f(x)+4x+2)}{3}$ (1) From (*) we get $f(2)=-9+6=-3$ $P(x,2) f(x+2)-3f(x)=f(2x)+4x+1 \Rightarrow f(2x)=6f(x)+4x+5$ (2) $P(x,x) : f(2x)+(f(x))^{2}=f(x^{2})+4x^{2}+1$ (3) Using (1) +(2) +(3) we get : $ \forall x \in \mathbb {R} : f(x)=-x-1$ Finally the F.E have 3 solutions which are : $\forall x \in \mathbb{R} : f(x)=-x-1 , f(x)=x^{2}-1, f(x)=2x-1$

Let $P(x,y)$ be the assertion. First see that no constant functions exist. $P(x,0)\implies f(x)+f(x)f(0)=f(0)+1\implies (f(0)+1)(f(x)-1)=0$ As $f$ is non constant $\exists x$ such that $f(x)\neq 1\implies f(0)=-1$ $P(1,-1)\implies f(0)+f(1)f(-1)=f(-1)-1\implies f(-1)(f(1)-1)=0$ If $f(1)=1$ $P(x,1)\implies f(x+1)=2x+1\implies \boxed{f(x)=2x-1\text{ for all } x\in\mathbb{R}}$ If $f(-1)=0$. $P(-1,-1)\implies f(-2)+f(-1)^2=f(1)+3\implies f(-2)=f(1)+3$. $P(1,-2)\implies f(-1)+f(1)f(-2)=f(-2)-3\implies f(-2)(f(1)-1)=-3$ $\implies (f(1)+3)(f(1)-1)=-3$ $\implies f(1)^2+2f(1)=0\implies f(1)=0\text{ or } f(1)=-2$ $\text{ If }f(1)=0$ $P(x,1)\implies f(x+1)=f(x)+2x+1\implies f(x-1+1)=f(x-1)+2x-2+1$ $\implies f(x)=f(x-1)+2x-1$ $P(x,1)\implies f(x-1)=f(-x)-2x+1$ but we have just proved $f(x)=f(x-1)+2x-1$. Comparing these two equations, we get $f(x)=f(-x)\text{ for all } x\in\mathbb{R}$ $P(x,-y)\implies f(x-y)+f(x)f(-y)=f(-x)-2xy+1\text{ but as }f(x)=f(-x)$ $f(x-y)=f(xy)-f(x)f(y)-2xy+1$ $P(x,y)\implies f(x+y)+f(x)f(y)=f(xy)+2xy+1$ Comparing these two equations, we get $f(x+y)-f(x-y)=4xy$ Take $x=y$ to get $f(2x)=4x^2-1$, so, $x\rightarrow \frac{x}{2}\implies \boxed{f(x)=x^2-1\text{ for all } x\in\mathbb{R}}\text{ is another solution}$ If $f(1)=-2$ $P(x,1)\implies f(x+1)=3f(x)+2x+1\implies f((x-1)+1)=3f(x-1)+2x-1\implies f(x)=3f(x-1)+2x-1$ $P(x,-1)\implies f(x-1)=f(-x)-2x+1$ $\implies 3(f(x-1))=3f(-x)-6x+3=f(x)-2x+1$ $\implies f(x)+4x=3f(-x)+2$ Now $x\rightarrow -x\implies f(-x)-4x=3f(x)+2$ Adding these two, we get, $f(x)+f(-x)=-2$ $P(x,-y)\implies f(x-y)+f(x)f(-y)=f(-xy)-2xy+1$ $\implies f(x-y)+f(x)\left[-2-f(y)\right]=-2-f(xy)-2xy+1$ $\implies f(x-y)-2f(x)+2xy+1=f(x)f(y)-f(xy)$ But $P(x,y)\implies f(x)f(y)-f(xy)=2xy+1-f(x+y)$ Therefore, $2xy+1-f(x+y)=f(x-y)-2f(x)+2xy+1$ $\implies f(x-y)+f(x+y)=2f(x)$ $x=y\implies -1+f(2x)=2f(x)\implies f(2x)-2f(x)=1$ We have already proved $f(x)=3f(x-1)+2x-1\stackrel{x=-1}{\implies} f(-2)=1$ As $f(x)+f(-x)=-2\implies f(-2)=3$ $P(x,2)\implies f(x+2)+f(x).f(2)=f(2x)+4x+1\implies f(2x)=f(x+2)-3f(x)-4x-1$ We have already proved $f(x+1)=3f(x)+2x+1$ $\implies f((x+1)+1)=3f(x+1)+2x+3=9f(x)+6x+3+2x+3$ $\implies f(x+2)=9f(x)+8x+6$ Therefore $9f(x)+8x+6-3f(x)-4x-1=f(2x)$ But $f(2x)=1+2f(x)\implies 6f(x)+4x+5-2f(x)=1\implies 4f(x)=-4x-4$ $\implies \boxed{f(x)=-x-1\text{ for all }x\in\mathbb{R}}$

Why $f(x)=1$ for all $x$ don't satisfy in this problem?

Goblik wrote: Why $f(x)=1$ for all $x$ don't satisfy in this problem? Because with $f(x)=1$ $\forall x$ : $LHS$ of functional equation is $2$ $RHS$ of functional equation is $2+2xy$ And obviously we dont have $LHS=RHS$ $\forall x,y$

I hate casework.

very beatiful function.

This problem took a really long while for me... Well, here's my solution. Feedback would be greatly appreciated - it is loooong (4 o's to indicate it's pretty long, but not insane).

IMO ShortList 2005 A4 wrote: Find all functions $ f: \mathbb{R}\to\mathbb{R}$ such that $ f(x+y)+f(x)f(y)=f(xy)+2xy+1$ for all real numbers $ x$ and $ y$. Proposed by B.J. Venkatachala, India

Let $P(x,y)$ denote the assertion that $ f(x+y)+f(x)f(y)=f(xy)+2xy+1$. $$P(x,0) \implies (1+f(0))(f(x))=1+f(0) \implies f(0)=-1$$If $f(1)=1$, then $P(x-1,1) \implies f(x)=2x-1$. Now suppose that $f(1) \neq 1$ and let $c=f(1)$. $$P(1,-1) \implies f(1)f(-1)=f(-1) \implies f(-1)=0$$$$P(x,-1) \implies f(x-1)=f(-x)-2x+1$$$$P(x-1,1) \implies f(x)+cf(x-1)=f(x-1)+2x-1 \implies f(x)+c(f(-x)-2x+1)=f(-x) \implies (c-1)f(-x)=2cx-c-f(x) \ \ (Q(x))$$$$Q(-1) \implies c(c-1)=-3c \implies c=0 \ \text{or} \ c=-2$$Let's check the case $c=-2$. $$(c-1)Q(-x)-Q(x) \implies (c-1)f(-x)=-(c-1)(2cx+c+(c-1)f(x))=2cx-c-f(x) \implies 3(-4x-2-3f(x))=-4x+2-f(x) \implies f(x)=-x-1$$Now let's check the case when $c=0$. Note that $Q(x) \implies f(x)=f(-x)$ $$P(x,x)-P(x,-x) \implies f(2x)+1=4x^2 \implies f(x)=x^2-1$$To conclude, the only solutions are $f(x)=x^2-1$ , $f(x)=-x-1$ and $f(x)=2x-1$.

The solutions are $f\equiv 2x-1$, $f\equiv x^2-1$, and $f\equiv -x-1$. It is easy to check that each works. Set $x=y=0$ to yield $f(0)^2=1$. Set $y=0$ to yield $f(x)(f(0)+1)=f(x)+f(x)f(0)=f(0)+1$. Thus either $f(0)=-1$ or $f\equiv1$, which is clearly absurd. Then $f(0)=-1$. Let $y=-x$ to yield \[-1+f(x)f(-x)=f(-x^2)-2x^2+1.\]In particular, for $x=1$ this yields $f(1)f(-1)=f(-1)$, so either $f(1)=1$ or $f(-1)=0$. In the case of $f(1)=1$, we have $f(x+1)=f(x)=f(x)+2x+1$ so $f(x+1)=2[x+1]-1$, yielding the solution $f\equiv 2x-1$. Now suppose $f(-1)=0$. Then we write the equation $f(-x-1)=f(x)+2x+1$. Take $y=-x-1$ so \[f(-x^2-x)-2x^2-2x+1=f(x)f(-1-x)=f(x)f(x)+2xf(x)+f(x)\qquad (\diamondsuit).\]So for $x=1$, this reads $f(-2)+1=f(1)^2+3f(1)$. But we also have $f(-2)=f(1)+3$, so \[f(1)=f(1)^2+3f(1)\iff f(1)\in \{0,-2\}.\]The first case yields the equation $f(x+1)=f(x)+2x+1$ so $f$ is even, so then $f(2x)+f(x)f(-x)=f(-x^2)+2x^2+1$, which implies $f(2x)=4x^2-1$ and $f\equiv x^2-1$. Now we have $f(1)=-2$, $f(-1)=0$, and $f(0)=-1$. Take $x=1$ to yield $f(1+y)=3f(y)+2y+1$. Now, write \[f(y)=3f(y-1)+2y-1=3[f(-y)-2y+1]+2y-1 = 3f(-y)-4y+2 = 3[3f(y)+4y+2]-4y+2=9f(y)+8y+8\implies f\equiv -y-1.\]

Let $P(x,y)$ be the assertion $f(x+y)+f(x)f(y)=f(xy)+2xy+1$. $$P(x,0)\implies\left(1+f(0)\right)\left(f(x)-1\right)=0$$If $f(0)\ne-1$ then $f(x)=1$, which doesn't work. Therefore, we must have $f(0)=-1$. Also note that: $$P(1,-1)\implies f(1)f(-1)=f(-1)$$So either $f(1)=1$ or $f(-1)=0$. Case 1: $f(1)=1$ We have that: $P(x-1,1)\implies\boxed{f(x)=2x-1}$, which works. Case 2: $f(-1)=0$ \begin{align*} P(-x,-1)&\implies f(-x-1)=f(x)-2x+1\qquad(1) \\ P(x,1)&\implies f(x+1)+f(x)f(1)=f(x)+2x+1\qquad(2) \\ &\implies f(-x-1)+4x=f(x+1)+f(x)f(1)\qquad(3) \end{align*} Taking $x\mapsto-x-1$ in $(3)$, we get: $f(x)=f(-x)+f(-x-1)f(1)-4x$ Substituting $(1)$, we can get an expression in terms of just $f(x)$, $f(-x)$, $x$, and constants. $f(x)=f(-x)+f(x)f(1)+(2f(1)-4)x+f(1)$ $\implies f(x)\left(1-f(1)\right)=f(-x)+(2f(1)-4)x+f(1)$ Now, taking $x\mapsto-x$ and adding will cancel out the $2xf(1)$ term: $f(-x)\left(1-f(1)\right)=f(x)+(4-2f(1))x+f(1)$ Adding both of these: $\left(f(x)+f(-x)\right)\left(1-f(1)\right)=\left(f(x)+f(-x)\right)+2f(1)$ $-f(1)\left(f(x)+f(-x)\right)=2f(1)$ This means that either $f(1)=0$ or $f(x)+f(-x)=-2$. Case 2.1: $f(x)+f(-x)=-2$ We get $f(-x)=-2-f(x)$. Comparing $P(x,y)$ with $P(-x,-y)$ proves: \begin{align*} &\phantom{\implies}f(-x-y)+f(-x)f(-y)=f(x+y)+f(x)f(y) \\ &\implies -2-f(x+y)+4+2f(x)+2f(y)+f(x)f(y)=f(x+y)+f(x)f(y) \\ &\implies f(x+y)=f(x)+f(y)+1\end{align*} Note that the substitution $g(x)=f(x)-1$ proves that $g$ is additive, but there's no way to prove that it's linear. Substituting this into $P(x,y)$, we get: $$f(x)+f(y)+1+f(x)f(y)=f(xy)+2xy+1$$ Now let $h(x)=f(x)+1$. We have: $$Q(x,y):h(x)h(y)=h(xy)+2xy$$Since $f(-1)=0$ and $f(x)+f(-x)=-2$, we know that $h(-1)=1$ and $h$ is odd. The latter means that $h(1)=-1$. Now, $Q(x,1)\implies h(x)=-x$ Therefore, $f(x)=h(x)-1=\boxed{-x-1}$, which indeed works. There are no other solutions in this case. Case 2.2: $f(1)=0$ Substituting this into $(3)$, we see that $f(-x-1)=f(x+1)\forall x$, and $x\mapsto x-1$ shows that $f$ is even. \begin{align*} P(x,-y)&\implies f(x-y)+f(x)f(-y)=f(-xy)-2xy+1 \\ &\implies f(x-y)+f(x)f(y)=f(xy)-2xy+1 \end{align*}We know that $f(x+y)+f(x)f(y)=f(xy)+2xy+1$, so we can say that: $$f(x-y)+2xy=f(x+y)-2xy$$$$f(x-y)+4xy=f(x+y)$$Here, taking $x=y$ gives: $$f(2x)=4x^2-1$$And doing the transformation $x\mapsto\frac x2$, we get: $$\boxed{f(x)=x^2-1}$$which also works. Thus, our solutions are $\boxed{f(x)=2x-1}$, $\boxed{f(x)=x^2-1}$ and $\boxed{f(x)=-x-1}$. $\square$

Let $P(x,y)$ denote the assertion $f(x+y)+f(x)f(y)=f(xy)+2xy+1$. First, $P(x,0)$ gives $(f(0)+1)(f(x)-1)=0$, so either $f\equiv 1$ or $f(0)=-1$. The former fails, so $f(0)=-1$. Next, $P(1,-1)$ gives $f(-1)f(1)=f(-1)$, so $f(-1)=0$ or $f(1)=1$. Case 1: $f(1)=1$. Then $P(x,1)$ gives $f(x+1)+f(x)=f(x)+2x+1$, so $f(x+1)=2x+1$, so $f(x)=2x-1$. This is a valid solution for all $x$. Case 2: $f(-1)=0$. Firstly, \[ P(x,-1): \quad f(x-1)=f(-x)-2x+1. \quad (\heartsuit)\]We can use the cancellation trick on $P(x,y)$, to cancel $f(x+y)$ and $f(xy)$ with the following. $P(x,\tfrac{x}{x-1})$ gives $f(x)f(\tfrac{x}{x-1})=\tfrac{(2x-1)(x+1)}{x-1}$. Now, $x=\tfrac{x}{x-1}$ when $x=2$, so in particular $P(2,2)$ gives $f(2)^2=9$. Hence $f(2)=\pm 3$. Case 2.1: $f(2)=3$. We can use this information to find $f(1)$ via $P(1,1)$: $f(1)^2=f(1)$, so $f(1) \in \{0,1\}$. Case 2.1.1: $f(1)=0$. Then $P(x-1,1)$ gives $f(x)=f(x-1)+2x-1$. Combined with $(\heartsuit)$, this implies $f$ is even. Then compare $P(x,y)$ and $P(x,-y)$: \begin{align*} P(x,y)&: \quad f(x+y)+f(x)f(y)=f(xy)+2xy+1 \\ P(x,-y)&: \quad f(x-y)+f(x)f(y)=f(xy)-2xy+1. \end{align*}Subtracting, $f(x+y)-f(x-y)=4xy$. Plug in $y=x$ into this: $f(2x)-f(0)=4x^2$, so $f(2x)=4x^2-1$, so $f(x)=x^2-1$. This is a valid solution for all $x$. Case 2.1.2: $f(1)=1$. Then $P(x-1,1)$ gives $f(x)=2x-1$ for all $x$, a valid solution. Case 2.2: $f(2)=-3$. Then $P(1,1)$ gives $f(1)^2=f(1)+6$, so $f(1)\in \{-2,3\}$. Case 2.2.1: $f(1)=-2$. Then $P(x-1,1)$ gives $f(x)=3f(x-1)+2x-1$. Combined with $(\heartsuit)$, this implies $f(x)=3f(-x)-4x+2$. These two both give us that $f(1-x)=3f(-x)+2(1-x)-1=f(x)+2x-1$. The RHS's of $P(x,1-x)$ and $P(-x,x-1)$ are the same since $x(1-x)=(-x)(x-1)$, so the LHS's are equal too. This gives a relation between all the f's we currently do not know how to relate. This gives \[ f(1)+f(x)f(1-x) = f(-1)+f(-x)f(x-1). \]We have \begin{align*} f(-x) &= \tfrac13(f(x)+4x-2) \\ f(x-1) &= \tfrac13(f(x)-2x+1) \\ f(1-x) &= f(x)+2x-1. \end{align*}Plugging these into the above equation: \begin{align*} &\qquad -2 + f(x)(f(x)+2x-1) = \tfrac19(f(x)+4x-2)(f(x)-2x+1) \\ &\implies -18+9f(x)^2 + 18xf(x)-9f(x)=f(x)^2+2xf(x)-f(x)-8x^2+8x-2 \\ &\implies 8f(x)^2+16xf(x)-8f(x)+8x^2-8x-16=0 \\ &\implies f(x)^2 + 2xf(x) - f(x)+x^2-x-2 = 0 \\ &\implies (f(x)+x)^2 - (f(x)+x) - 2 = 0 \\ &\implies f(x)+x \in \{2,-1\}. \end{align*}Therefore, for each $x$, either $f(x)=-x+2$ or $f(x)=-x-1$. Now we have to deal with the pointwise trap. Suppose $f(a)=-a+2$ for some $a$. Then $P(a,2)$ gives $f(a+2)+(-a+2)(-3)=f(2a)+4a+1$, so $f(a+2)=f(2a)+a+7$. But $f(a+2) \in \{-a,-a-3\}$ and $f(2a)\in \{-2a+2,-2a-1\}$, so we cannot have $f(a+2)-f(2a) =a+7$. Therefore, $f(a)\not = -a+2$ for any $a$, so $f(x)=-x-1$ for all $x$. This is a valid solution. Case 2.2.2: $f(1)=3$. We show this is impossible. We know $f(0)=-1$ and $f(-1)=0$ and $f(2)=-3$. $P(2,-1)$ gives $f(1)+f(2)f(-1)=f(-2)-3$ so $f(1)=f(-2)-3$, so $f(-2)=6$. $P(-2,1)$ gives $f(-1)+f(-2)f(1)=f(-2)-4+1$, so $f(-2)(f(1)-1)=-3$, so $f(-2)=-3/2$. This is a contradiction. Finally, the solutions we recovered are $f(x)=2x-1 \forall x$, and $f(x)=x^2-1\forall x$, and $f(x)-x-1\forall x$.

Another solution since I re-did this problem without realizing it. Let $P(x,y)$ be the assertion $f(x+y)+f(x)f(y)=f(xy)+2xy+1$. $P(x,0)\Rightarrow f(x)+f(x)f(0)=f(0)+1$ so $f(0)=-1$ (else $f(x)=1$, which doesn't work). $P(1,-1)\Rightarrow f(1)f(-1)=f(-1)\Rightarrow f(-1)=0\vee f(1)=1$ Case 1: $f(-1)=0$ $P(1,-2)\Rightarrow f(1)f(-2)+3=f(-2)$ $P(-1,-1)\Rightarrow f(-2)=f(1)+3$ So $f(1)^2+2f(1)=0$, thus either $f(1)\in\{0,-2\}$. Case 1.1: $f(1)=0$ Let $g(x)=f(x)-x^2+1$, so that $g$ has zeroes at all elements of the set $\{-1,0,1\}$. We get the assertion $$Q(x,y):g(x+y)+g(x)g(y)-g(x)-g(y)+x^2g(y)+y^2g(x)=g(xy)$$$Q(x,1)\Rightarrow g(x+1)=g(x)$ $Q(-x,-1)\Rightarrow g(-x-1)=g(x)$ thus $g$ is even, since $g(x+1)=g(-x-1)$. $Q(x,-y)\Rightarrow g(x-y)+g(x)g(y)-g(x)-g(y)+x^2g(y)+y^2g(x)=g(xy)$ (using evenness) So $g(x+y)=g(x-y)$, which we obtain when we compare the above with $Q(x,y)$. Setting $x=y$ yields $g(2x)=g(0)=0$, so $g(x)=0$, and then $\boxed{f(x)=x^2-1}$, and it's easy, but a bit annoying to verify that this works. Case 1.2: $f(1)=-2$ \begin{align*} P(x-1,1)&\Rightarrow f(x)-2x+1=3f(x-1)\\ P(-x+1,-1)&\Rightarrow f(-x)=f(x-1)+2x-1\\ &\Rightarrow f(-x)=\frac{f(x)+4x-2}3\\ &\Rightarrow3f(x)=f(-x)-4x-2=\frac{f(x)+4x-2}3-4x-2\\ &\Rightarrow8f(x)=-8x-8\\ &\Rightarrow\boxed{f(x)=-x-1},\\ \end{align*}which works. Case 2: $f(1)=1$ $P(x-1,1)\Rightarrow\boxed{f(x)=2x-1}$, which also works. Solutions, which all work: $\boxed{f(x)=x^2-1}$, $\boxed{f(x)=-x-1}$, $\boxed{f(x)=2x-1}$

I redid this problem yet again without realizing it. Let $P(x,y)$ denote this assertion. If $f(0)\ne-1$, then: $P(x,0)\Rightarrow f(x)=1$ which doesn't work. So $f(0)=-1$. $P(1,-1)\Rightarrow f(1)f(-1)=f(-1)$ If $f(1)=1$ we have: $P(x-1,1)\Rightarrow\boxed{f(x)=2x-1}$, which is a solution. Otherwise, assume $f(-1)=0$. $P(-2,1)\Rightarrow f(-2)f(1)=f(-2)-3$ $P(-1,-1)\Rightarrow f(-2)=f(1)+3$ Substituting $P(-1,-1)$ into $P(-2,1)$, we have $f(1)\in\{-2,0\}$. Case 1: $f(1)=0$ $P(x,1)\Rightarrow f(x+1)=f(x)+2x+1$ $P(x,y+1)-P(x,y)\Rightarrow 2x^2+4xy+2y^2+2yf(x)+f(x)=f(xy+x)-f(xy)+2x-1$ Setting $y=\frac1x$ and making $x\ne0$, we obtain $f(x)=x^2-1$ for $x\ne0$. Since $f(0)=-1$, it completes the solution $\boxed{f(x)=x^2-1}$ which fits. Case 2: $f(1)=-2$ $P(x,1)\Rightarrow f(x+1)=3f(x)+2x+1$ $P(x,y+1)-3P(x,y)\Rightarrow 2y+3+2yf(x)+f(x)=f(xy+x)-3f(xy)-4xy$ We repeat with $y=\frac1x,x\ne0$, we have $f(x)=-x-1$ for $x\ne0$. Since $f(0)=-1$ this is $\boxed{f(x)=-x-1}$, which is a solution.

I rather dislike casework to be honest. Let the assertion be $P(x,y).$ $P(0,0)$ gives $f(0)+f(0)^2=f(0)+1$ which implies $f(0)^2=1.$ If $f(0)=1$ then $P(x,0)$ implies $f(x)+f(x)=2$ so $f(x)=1$ for all $x.$ Note that this construction doesn't work when plugging into original. Now, we work under the assumption that $f(0)=-1.$ $P(-1,1)$ gives $-1+f(-1)f(1)=f(-1)-1,$ which implies either $f(-1)=0$ or $f(1)=1.$ If $f(1)=1$ then $P(x-1,1)$ gives $f(x)+f(x-1)=f(x-1)+2x-1$ so $f(x)=2x-1.$ Note that this satisfies the original condition. Now, we assume $f(-1)=0.$ $P(-1,-1)$ gives $f(-2)=f(1)+3.$ On the other hand, $P(-2,1)$ gives $f(-2)f(1)=f(-2)-3$ so $f(-2)f(1)=f(1).$ Thus, $f(1)=0$ or $f(-2)=1.$ If $f(1)=0$ then $P(x,1)$ implies $f(x+1)=f(x)+2x+1$ and so $f(2)=3.$ $P(x,2)$ implies $f(x+2)+3f(x)=f(2x)+4x+1.$ On the other hand, $P(x+1,1)$ implies $f(x+2)=f(x+1)+2(x+1)+1=f(x+1)+2x+3=f(x)+2x+1+2x+3=f(x)+4x+4.$ This implies $f(2x)+4x+1-3f(x)=f(x)+4x+4$ so $f(2x)=4f(x)+3.$ $P(x,x+2)$ gives $f(2x+2)+f(x)f(x+2)=f(x^2+2x)+2x(x+2)+1.$ This implies \[f(2x+2)+f(x)f(x+2)=f((x+1)^2)-2(x+1)^2+1+2x^2+4x+1.\]The RHS cancels out to $f((x+1)^2)$ so \[f((x+1)^2)=4f(x+1)+3+(f(x+1)-2(x+1)+1)(f(x+1)+2(x+1)+1)\]This simplifies down to $f(x^2+2x+1)=f(x+1)^2+6f(x+1)-4(x+1)^2+3$ so $f(x^2)=f(x)^2-4x^2+6f(x)+4.$ However, $P(x,x)$ gives $f(2x)+f(x)^2=f(x^2)+2x^2+1$ so $f(x^2)-f(x)^2=-4x^2+6f(x)+4$ while it also equals $4f(x)+2-2x^2.$ This all implies that $f(x)=x^2-1.$ Note that this is a valid solution. If $f(-2)=1$ then $f(1)=-2.$ Now $P(x-1,1)$ gives $f(x)=3f(x-1)+2x-1$ and $P(-x+1,-1)$ gives $f(-x)=f(x-1)+2x-1$ (remember that $f(-1)=0$). Note that equivalently $f(-x-1)=f(x)+2x+1$ Thus, $f(x)-f(-x)=2f(x-1).$ Also, $f(-x)-f(x)=-2f(x-1)$ and $2f(-x-1)$ at the same time. Thus $f(x-1)=-f(-x-1).$ We see that $f(x-1)=-f(x)-2x-1$ so $f(x)=3(-f(x)-2x-1)+2x-1=-3f(x)-4x-4$ so $f(x)=-x-1,$ and this also works so $f(x)=-x-1,x^2-1,2x-1$

Let $P(x,y)$ denote the assertion. $P(x,1)\implies f(x+1)=cf(x)+2x+1,~~~~~(**)$ where $c=1-f(1).$ $P(x,y+1)\implies c(f(x+y)+f(x)f(y))+(2y+1)(1+f(c))=f(x(y+1))+2xy+1.$ $P(2k,-1/2)\implies c(f(-k)-2k+1)=f(-k)+2k+1.~~~~~(*)$ Combining yields, $(-c^2)f(k)=2(1-c)^2k+c^2-1.$ Note that $c\neq -1.$ Case 1: $c\neq 1$ Then $1-c^2\neq 0 \implies f(k)=2(\frac{1-c}{1+c})k-1.$ $k\to 1\implies c=0$ or $c=3\implies \boxed{f(x)=2x-1}$ or $\boxed{f(x)=-1-x},$ both work. Case 2: $c=1.$ $(*)\implies f(k)=f(-k).$ $P(x,x)-P(x,-x)\implies f(2x)=4x^2+f(0).$ Setting $x=0$ in $(**)\implies f(0)=-1.$ It follows that $\boxed{f(x)=x^2-1},$ which also fits.

Really straightforward problem. Let $P(x;y)$ denotes the assertion of given functional equation. Note that $P(0;0)$ gives us: $$ f(0)+f(0)^2 = f(0)+1 \implies f(0)= 1 \quad \text{or} \quad f(0)=-1 $$Suppose that $f(0)=1$. Then from $P(x;0)$ we have: $$ f(x)+f(x) =1 +1 \implies f(x)=1 $$It is easy to see that this function does not work; therefore we can assume that $f(0)=-1$. Consider $P(1;-1)$: $$ f(0)+f(1)f(-1)=f(-1)-2+1 \implies f(1)f(-1)=f(-1) \implies f(-1)=0 \quad \text{or} \quad f(1)=1 $$Suppose that the latter is the case. Then from $P(x;1)$ we have: $$ f(x+1) +f(x)=f(x)+2x+1 \implies f(x+1)=2x+1 \implies f(x)=2x-1$$It is easy to see that this function works; thus we can assume that $f(-1)=0$. We would like to determine the value of $f(1)$. To do so we consider $P(2;-1)$, which gives us: $$ f(1)=f(-2)-3 $$On the other hand, $P(1;-2)$ reveals that: $$ f(1)f(-2)=f(-2)-3 \implies f(1)(f(1)+3)=f(1)+3-3 \implies f(1) =0 \quad \text{or} \quad f(1)=-2 $$Let's consider the case when $f(1)=0$. From $P(x;1)$ we have: $$ f(x+1)=f(x)+2x+1 \implies f(x)=f(x-1)+2x-1 \implies f(x)-2x+1=f(x-1)$$On the other hand, $P(x,-1)$ gives us: $$ f(x-1) = f(-x)-2x+1 = f(x)-2x+1 \implies f(x)=f(-x) $$Now it is quite natural to consider $P(x;-x)$: $$ -1 +f(x)f(-x)=f(-x^2)-2x^2+1 \implies f(x)^2=f(x^2)-2x^2 +2 \qquad (\star)$$Also note that $P(x;x)$ gives us: $$ f(2x) +f(x)^2 = f(x^2) +2x^2 +1 \qquad (\star \star) $$Subtracting $(\star)$ from $(\star \star )$: $$ f(2x) = 4x^2 -1 \implies f(x) = x^2 -1 $$It is easy to check that this function indeed works. Now comes the painful part, when $f(1)=-2$. Since $f(1)=f(-2)-3$, we must have $f(-2)=1$. Also note that from $P(1;1)$ we have $f(2)=-3$. First of all, observe that from $P(x;1)$ we have: $$ f(x+1) -2f(x) = f(x)+2x+1 \implies f(x+1) = 3f(x)+2x+1 $$Now $P(x;2)$ gives us: \begin{align*} f(x+2) -3f(x) =f(2x)+4x+1 \\ 3f(x+1)+2x+3 =3f(x)+f(2x)+4x+1 \\ 9f(x)+6x+3+2x+3 =3f(x)+f(2x)+4x+1 \\ 6f(x)+4x+5=f(2x) \implies 6f(-x)-4x+5 = f(-2x) \end{align*}It also makes sense now to consider $P(x;-2)$: \begin{align*} f(x-2) +f(x)=f(-2x) -4x+1 \\ 9f(x-2)+9f(x) =9f(-2x)-36x+9 \\ 3(f(x-1)-2(x-2)-1)+9f(x) = 9f(-2x)-36x+9 \\ 3f(x-1)-6x+9+9f(x) =9f(-2x)-36x+9 \\ 3f(x-1)+9f(x) =9f(-2x)-30x \\ f(x)-2(x-1)-1+9f(x)=9f(-2x)-30x \\ f(x)-2x+1+9f(x)=9f(-2x)-30x \\ 10f(x) +1 = 9(-2x)-28 x \\ 10f(x)+1 = 9(6f(-x)-4x+5) -28x \\ 10f(x) =54f(-x)-64x+44 \\ 5f(x) =27f(-x)-32x+22 \end{align*}Our pain ends here, since now we can relate $f(x)$ and $f(-x)$ using only $x$ and some constants. The final step is to take a look at $P(x;-1)$: \begin{align*} f(x-1) =f(-x)-2x+1 \\ 27f(x-1)=27f(-x)-54x+27 \\ 27f(x-1)=5f(x)+32x-22-54x+27 \\ 27f(x-1)=5f(x)-22x+5 \\ 9(3f(x-1)) =5f(x)-22x+5 \\ 9(f(x)-2(x-1)-1) = 5f(x)-22x+5 \\ 9f(x) -18x+9 = 5f(x)-22x+5 \\ 4f(x) =-4x-4 \\ f(x) = -x-1 \end{align*}It is easy to check that this function also satisfies given equations. With all cases being considered, we conclude that only possible functions are $f(x)=x^2-1$; $f(x)=2x-1$ and $f(x)=-x-1$.

Let $P(x,y)$ be the assertion.

Let $P(x,y)$ be the assertation that $f(x+y)+f(x)f(y)=f(xy)+2xy+1$ $P(0,0) \Rightarrow f(0)=\pm1$ If $f(0)=1$ then $P(x,0) \Rightarrow f(x)=1$ $P(x,y) \Rightarrow 0=2xy \Rightarrow$ contradiction $\Rightarrow f(0)=-1$ $P(-1,1) \Rightarrow f(0)+f(-1)f(1)=f(-1)-1 \Rightarrow f(-1)(f(1)-1)=0$ Case 1: f(1)=1 $\Rightarrow$ $ P(x,1) \Rightarrow f(x+1)+f(x)f(1)=f(x)+2x+1 \Rightarrow f(x+1)=2x+1 \Rightarrow \boxed{f(x)=2x-1}$ Case 2: f(-1)=0 $\Rightarrow P(-1,-1) \Rightarrow f(-2)=f(1)+3$ $P(-2,1) \Rightarrow f(-2)f(1)=f(-2)-3 \Rightarrow f(-2)=\frac{-3}{f(1)-1} \Rightarrow f(1)+3=\frac{-3}{f(1)-1} \Leftrightarrow $ $f(1)=0$ or $f(1)=-2$ Subcase 2.1: f(1)=0 $\Rightarrow P(x,1) \Rightarrow f(x+1)=f(x)+2x+1$ $P(x+1,-1) \Rightarrow f(x)=f(-x-1)-2x-1 \Rightarrow f(x+1)=f(-x-1) \Rightarrow f(x)=f(-x)$ for all $x \in \mathbb{R}$ $P(x,-x) \Rightarrow f(x)^2=f(x^2)-2x^2+2 \Rightarrow P(x,x) \Rightarrow f(2x)+f(x)^2=f(x)^2+2x^2-2+2x^2+1 \Rightarrow \boxed{f(x)=x^2-1}$ Subcase 2.2: f(1)=-2 $\Rightarrow P(1,1) \Rightarrow f(2)=-3$ $P(x,1) \Rightarrow$ $f(x+1)=3f(x)+2x+1$ (1) $\Rightarrow f(x+2)=9f(x)+8x+6$ (2) $P(x,-1) \Rightarrow$ $f(x)=f(-x-1)-2x-1 \Rightarrow f(x)=3f(-x)-4x+2$ (3) Using (1) and (3): $P(x,-x) \Rightarrow$ $f(x)f(-x)=f(-x^2)-2x^2+2 \Rightarrow f(x^2)=\frac{3f(x)^2+4xf(x)+2f(x)-2x^2-4}{3}$(4) Using (1),(3) and (4): $P(x,x) \Rightarrow f(2x)+f(x)^2=f(x^2)+2x^2+1 \Rightarrow f(2x)=\frac{4xf(x)+2f(x)+4x^2-1}{3}$(5) Using (2) and (5): $P(x,2) \Rightarrow f(x+2)-3f(x)=f(2x)+4x+1\Rightarrow$ after substituting $\boxed{f(x)=-x-1}$ After checking we see that all 3 solution work.

Standard FE Denote $P(x,y)$ by the assertion $f(x+y)+f(x)f(y)=f(xy)+2xy+1$ Consider $P(0,0) \implies f(0)=\pm 1$ If $f(0)=1$, we consider $P(x,0) \implies f(x)=1 \forall x \in \mathbb{R}$ which is clearly fails. So, $f(0)=-1$ Consider $P(1,-1) \implies f(-1)(1-f(1))=0$ We divide into 2 cases case 1) $f(1)=1$ Consider $P(x,1)$ yields $f(x+1)=2x+1 \forall \in \mathbb{R}$ Plug $x \mapsto x-1$, we conclude that $f(x)=2x-1 \forall x \in \mathbb{R}$ case 2) $f(-1)=0$ Consider $P(2,-1)$ yields $f(1)+f(2)f(-1)=f(-2)-3 \implies f(1)=f(-2)-3~~(\clubsuit)$ Consider $P(-2,1)$ yields $f(-1)+f(-2)f(1)=f(-2)-3 \implies f(-2)f(1)=f(-2)-3~~(\spadesuit)$ By$(\clubsuit)$ and $(\spadesuit)$, $f(1)(1-f(-2))=0$ We divide into 2 subcases case 2.1 $f(1)=0$ Consider $P(x,1)$ yields $f(x+1)=f(x)+2x+1 \forall x \in \mathbb{R}$ So, $P(x,y)$ is equivalent to $Q(x,y): f(x+y)+f(x)f(y)=f(xy+1) ~~\forall x,y \in \mathbb{R}$ Consider $Q(x+1,-1)$ yields $f(x)=f(-x) \forall x \in \mathbb{R}$ Then we compare $Q(x,y)$ and $Q(x,-y)$, $$f(x+y)-f(x-y)=f(xy+1)-f(-xy+1)=f(xy+1)-f(xy-1)=[f(xy+1)-f(xy)]+[f(xy)-f(xy-1)]=2xy+1+2(xy-1)+1=4xy ~~\forall x,y \in \mathbb{R}$$Plug $y=x$ in the latest equation yields $f(2x)+1=4x^2 \forall x \in \mathbb{R}$ So, $f(x)=x^2-1 \forall x \in \mathbb{R}$ case 2.2 $f(-2)=1$ By $(\clubsuit)$, $f(1)=f(-2)-3=-2$ Consider $P(x-1,1)$ yields $f(x)=3f(x-1)+2x-1$ Consider $P(x,-1)$ yields $f(x-1)=f(-x)-2x+1 \implies 3f(x-1)=3f(-x)-6x+3$ So, $f(x)+4x-2=3f(-x)$ Plug $x \mapsto -x$, $f(-x)-4x-2=3f(x)$ By solving these two equations, we conclude that $f(x)=-x-1 \forall x \in \mathbb{R}$ So, there are exactly three functions which satisfy the problem condition which are $\boxed{f(x)=2x-1 \forall x \in \mathbb{R}},~~\boxed{f(x)=x^2-1 \forall x \in \mathbb{R}}$ and $\boxed{f(x)=-x-1 \forall x \in \mathbb{R}}$

Answers are $f(x)=-x-1,f(x)=2x-1,f(x)=x^2-1$. Let $f(x)=g(x)-1$. We have \[g(x+y)-g(x)-g(y)=g(xy)+2xy-g(x)g(y)\]Let $P(x,y)$ be the assertion of this equation. $P(x,0)$ yields $-g(0)=g(0)-g(x)g(0)$ or $g(0)(g(x)-2)=0$. Since $g$ is non-constant, we get that $g(0)=0$. \[P(1,-1): \ \ -g(1)-g(-1)=g(-1)-2-g(1)g(-1)\iff (g(1)-2)(g(-1)-1)=0\]Case $I$: If $g(1)=2$, then plug $P(x,1)$ to verify that \[g(x+1)-g(x)-2=g(x)+2x-g(x)\iff g(x+1)=2(x+1)\iff g(x)=2x\]We get the solution $f(x)=2x-1$ in this case. Case $II$: If $g(-1)=1$, subsituting $y=-1$ implies $g(x-1)-g(x)-1=g(-x)-2x-g(x)$ or $g(x-1)-g(-x)=1-2x$. Changing the sign of $x$ gives $g(-x-1)=g(x)+2x+1$. Plug $y=-x-1$ to get the equation \[1-g(x)-g(-1-x)=g(-x-x^2)-2x-2x^2-g(x)g(-1-x)\]Picking $x=-2$ yields $1-g(-2)-g(1)=g(-2)-4-g(-2)g(1)$ and since $g(-2)=g(1)+3$, we have \[-2-2g(1)=g(1)-1-g(1)(g(1)+3)\iff g(1)^2=1\]Case $II.I$: If $g(1)=1$, then plugging $y=1$ implies $g(x+1)-g(x)-1=g(x)+2x-g(x)$ or $g(x+1)=g(x)+2x+1$. Note that by induction, we get $g(n)=n^2$ for positive integers. Let $g(x)=x^2+s(x)$. \[(x+y)^2+s(x+y)-x^2-y^2-s(x)-s(y)=s(xy)+2xy-x^2s(y)-y^2s(x)-s(x)s(y)\]\[s(x+y)-s(x)-s(y)=s(xy)-x^2s(y)-y^2s(x)-s(x)s(y)\]Subsituting $y=1$ implies $s(x+1)-s(x)=s(x)-s(x)=0\iff s(x+1)=s(x)$ hence $s(x+n)=s(x)$. Compare $x,y$ with $x+1,y$. Left hand side doesn't change thus, \[s(xy+y)-(x+1)^2s(y)=s(xy)-x^2s(y)\iff s(xy+y)-s(xy)=(2x+1)s(y)\]Pick $x=\frac{n}{y}$ to observe that $0=s(y+n)-s(y)=(\frac{2n}{y}+1)s(y)$. So $s\equiv 0$ which gives the solution $f(x)=x^2-1$. Case $II.II$: If $g(1)=-1$, then $P(x,1)$ gives $g(x+1)-g(x)+1=g(x)+2x+g(x)$ or $g(x+1)=3g(x)+2x-1$. By induction with base cases $g(0)=0$ and $g(1)=-1$, we observe that $g(n)=-n$ for each integer $n$. Let $h(x)=g(x)+x$ with $h(n)=0$. \[h(x)+h(y)-h(x+y)=h(xy)-h(x)h(y)+xh(y)+yh(x)\]Plug $y=1$ to get $h(x)-h(x+1)=2h(x)\iff h(x)+h(x+1)=0$. Subsituting $y=-1$ implies $2h(x)=h(x)-h(x-1)=h(-x)-h(x)$ thus, $h(-x)=3h(x)$. However, changing the sign gives us $3h(-x)=h(x)$ which means $h(x)=0$ for all reals. We get the solution $f(x)=-1-x$ as desired.$\blacksquare$

Davron wrote: can anyone explain how he could get : Given $y=-1-x$ we have , $f(x)(f(x)+2x+1)=f(x^2+x-1)$ thanks He put $y=-1-x$ at original equation, then used $f(-x)=f(x-1)+2x-1$

The only solutions are $f(x) = -x - 1\forall x \in \mathbb R$, $f(x) = 2x - 1\forall x \in \mathbb R$, and $f(x) = x^2 - 1 \forall x \in \mathbb R$. We first check that these work (i was bored in airplane so i decided to type it out). If $f(x) = -x - 1$, the LHS becomes $-x - y - 1 + (x+1)(y+1) = xy$ and the RHS becomes $-xy - 1 + 2xy + 1 = xy$ also, so $f(x) = -x - 1$ works. If $f(x) = 2x - 1$, the LHS becomes $2x + 2y + (2x - 1)(2y - 1) = 4xy + 1$ and the RHS becomes $2xy + 2xy + 1 = 4xy + 1$, so $f(x) = 2x - 1$ works. If $f(x) = x^2 - 1$, the LHS becomes $x^2 + y^2 + 2xy - 1 + (x^2 - 1)(y^2 - 1) = x^2 y^2 + 2xy$ and the RHS becomes $x^2 y^2 - 1 + 2xy + 1 = x^2 y^2 + 2xy $, so $f(x) = x^2 - 1$ works. We now show these are the only solutions. Let $P(x,y)$ be the given assertion. $P(0,0): f(0) + f(0)^2 = f(0) + 1 \implies f(0) \in \{-1,1\}$. Claim: $f(0) = -1$. Proof: Assume not. $P(0,0): f(0) + f(0)^2 = f(0) + 1 \implies f(0) = 1$. $P(x,0): 2f(x) = 2$, so $f\equiv 1$, which does not work. $\square$ $P(x,-x): f(x) f(-x) - 1 = f(-x^2) - 2x^2 + 1$. $P(1,-1): f(1) f(-1) - 1 = f(-1) - 1$, so $f(1) f(-1) = f(-1)$. This implies either $f(-1) = 0$ or $f(1) = 1$. $P(2,2): f(2)^2 = 9\implies f(2) \in \{-3,3\}$. $P(1,1): f(2) + f(1)^2 = f(1) + 3$. Thus, $f(1) \in \{-2,-1,0,1\}$, the first two with $f(2) = -3$ and the second two with $f(2) = 3$. Case 1: $f(1) \ne 1$. We note that $f(1) \in \{-2,-1,0\}$ and $f(-1) = 0$. $P(x,-1): f(x-1) = f(-x) - 2x + 1$. Subcase 1.1: $f(1) = -2$. This implies $f(2) = -3$. $P(x,1): f(x+1) = 3f(x) + 2x + 1$. Thus, $f(x) = 3f(x-1) + 2x - 1 = 3f(-x) - 4x + 2$. We also have $f(-x) = 3f(x) + 4x + 3$. Thus, $f(x) = 3(3f(x) + 4x + 2) - 4x + 2 = 9f(x) + 8x + 8$, so $f(x) = -x - 1$. Subcase 1.2: $f(1) = -1$. We again have $f(-1) = 0, f(2) = -3$. $P(2,1): f(3) + 3 = 2$, so $f(3) = -1$. $P(2,-1): f(1) = f(-2) - 3$, so $f(-2) = f(1) + 3 = 2$. $P(4, -1): f(3) = f(-4) - 7$, so $f(-4) = f(3) + 7 = 6$. $P(2,-2): -7 = f(-4) - 7$, so $f(-4) = 0$, absurd. Subcase 1.3: $f(1) = 0$. $P(x,1): f(x+1) = f(x) + 2x + 1$. We also have \[ f(x) = f(x-1) + 2x - 1 = f(-x) ,\]so $f$ is even. This in fact implies (from $P(x,y)$ compared with $P(x,-y)$) that \[ f(x+y) - 2xy = f(x-y) + 2xy \implies f(x+y) - f(x-y) = 4xy\] So $f(x+2y) = f(x) + 4y^2 + 4xy$. Hence \[ f(x+y) = f(x) + y^2 + 2xy \]Thus, \[f(x+y)^2 - (x+y)^2 = f(x) - x^2,\]so $f(x) - x^2$ is constant. Since $f(0) = -1$, we have $f(x) = x^2 - 1 \forall x \in \mathbb R$. Case 2: $f(1) = 1$. $P(x-1,1): f(x) = 2x - 1 \forall x \in \mathbb R$.

Lets go (prolly different from other solutions): $P(x, 1)$ gives: $$f(x+1)=af(x)+2x+1(*)$$where $a=1-f(1)$. Now, notice that: $P(x, y+1)$ gives us: $$a(f(x+y)+f(x)f(y))=f(xy+x)+2xy+1-(2y+1)(f(x)+1).$$From the given equation: $f(x+y)+f(x)f(y)=f(xy)+2xy+1$, we have: $$a(f(xy)+2xy+1)=f(xy+x)+2xy+1-(2y+1)(f(x)+1).$$Set $x=2t, y=-1/2$, to get: $a(f(-t)-2t+1)=f(t)-2t+1$. Now, plugging $t \to -t$ and eliminating $f(-t)$ gives us: $f(t)(1-a^2)=2(1-a)^2 t + a^2-1$. If $a \neq \pm 1$, then: $f$ is clearly a linear function: with $f(t) = \frac{2(1-a)}{1+a}t-1$. Plugging $t=1$ with $f(1)=1-a$, we have $a=0$ or $a=3$. Both of them gives valid solutions $\boxed{f(x)=2x-1, -x-1}.$ If $a= \pm 1$, then consider two cases: Case-1: $a=-1$; Plugging back into $f(t)(1-a^2)=2(1-a)^2 t + a^2-1$ would end up $8t=0$ (which is absurd). Case-2: $a=1$. Subtracting $P(x, x)$ from $P(x, -x)$ gives: $f(2x)=4x^2+f(0)$. Putting $x=0$ in (*) gives us $f(1)=0$, thus: $f(0)=-1$. Plugging back: $f(2x)=4x^2-1$ or $\boxed{f(x)=x^2-1}$.