Any function which maps anything but $0$ into $0$ is a solution. Now let's find the remaining solutions (those for which $f(n)\neq 0$ for some $n\neq 0)$. Let's first prove that for any $x \in \mathbb Z$, there doesn't exist a positive integer $n$ such that $f^n(x)=x$. If $f^n(x)=x$, then the set $S=\{x, f(x), \ldots \}$ is finite. Let $z$ be the largest (by absolute value) element of $S$. Then, taking $(a,b)=(z, 0)$, we have $$f^{z^2}(z)=zf(z).$$Since the left hand side is an element of $S$, we must have $f(z)\in \{-1,0,1\}$ due to maximality of $z$. Therefore, it suffices to prove that $f^n(x)\neq x$ for every $n>0$ for $x \in \{-1,0,1\}$. Considering $(a,b)=(-1,0)$, we obtain $f(-1)=0$. Therefore, it suffices to prove that $f^n(0)\neq 0$ and $f^n(1)\neq 1$ for every $n>0$. Considering $(a,b)=(1,-1)$, we obtain $f(f(0))=f(1)$. Suppose that $f^m(0)=0$ for some $m>0$. Then consider $(a,b)=(x,-x)$ for some $x$ greater than every element in the set $\{0, f(0),\ldots f^{m-1}(0)\}$. We have $f^{2x^2}(0)=x(f(x)-f(-x))$. If $f(x)\neq f(-x)$, the right hand side is bigger than the left hand side. Therefore $f(x)=f(-x)$ and $f^{2x^2}(0)=0$. But then, doing the same for $x+1$, we also have $f^{2(x+1)^2}(0)=0$, which implies $f^{2\gcd(x^2,(x+1)^2}(0)=f(f(0))=0$. This means $f(1)=0$. Suppose $f(n)=0$ for all nonzero $n$ such that $|n|<k$. Considering $(a,b)=(k,-(k-1))$ and $(-k, k-1)$, we have $f(k)=0$ as well. Inductively, $f(n)=n$ for every nonzero integer $n$, which is a contradiction. Therefore, $f^m(0)\neq 0$ for every $m>0$. Suppose that $f^m(1)=1$ for some $m>1$. Note that for every $n>0$, $f^n(1)$ is not equal to $-1$ or $0$. Let $f^j(1)$ be the greatest element of $\{1,f(1),\ldots, f^{m-1}(1)\}$. Similarly as before, we conclude $f^{j+1}(1)=1$, but also $f^{j-1}(1)=1$. Therefore, $f(f(1))=1$. Taking $(a,b)=(-1,2)$, we have $f^5(1)=f(1)=2f(2)$, which means $f(1)$ is even. Taking $(a,b)=(x, 1-x)$ and $(x, f(1)-x)$, we obtain \begin{align*} f^{x^2+(1-x)^2}(1)&=xf(x)+(1-x)f(1-x), \\ f^{x^2+(f(1)-x)^2}(f(1))&=xf(x)+(f(1)-x)f(f(1)-x). \end{align*}However, the two left hand sides are equal since $x^2+(1-x)^2$ and $x^2+(f(1)-x)^2$ have different parity. Therefore, $(1-x)f(1-x)=(f(1)-x)f(f(1)-x).$, i.e. the function $x \mapsto xf(x)$ has period $f(1)-1$. Finally, taking $(a,b)=(x,x)$ and $(x+(f(1)-1), x-(f(1)-1))$, we obtain the equality $$f^{2x^2}(2x)=2xf(x)=(x+f(1)-1)f(x+f(1)-1)+(x-f(1)+1)f(x-f(1)+1)=f^{2x^2+2(f(1)-1)^2}(2x),$$but then $f^{2x^2}(2x)$ is a periodic point. Hence, $f^{2x^2}(2x)=2xf(x) \in \{1, f(1)\}$ for all $x$. However, this is impossible since $f(1)\neq 0$. Therefore, there are no cycles. Plugging in $(m,0)$ and $(m,-1)$, we obtain the equality $$f^{m^2}(m)=f^{m^2+1}(m-1).$$Now suppose $f(n)=n-k$ for some $k>0$. Let $S=n^2+(n-1)^2+\ldots+(n-k+1)^2$. Then, using the equality $f^{m^2}(m)=f^{m^2+1}(m-1)$ $k$ times, $$f^{S}(n)=f^{S+1}(n-1)=f^{S+2}(n-2)=\ldots=f^{S+k}(n-k)=f^{S+k+1}(n),$$but then $f^S(n)$ is in a cycle, a contradiction. Therefore, $f(n)>n$ for all $n$. Of course, this means $f$ is injective and then $f^{m^2}(m)=f^{m^2+1}(m-1)$ implies $f(m-1)=m$ for all $m \in \mathbb Z$. It's trivial to check that $x \mapsto x+1$ does satisfy the condition.

Deleted solution

Solved with nukelauncher We claim the only solutions are $f(x)=0$ and $f(x)=x+1$, both of which can be easily checked to work. Let $P(a,b)$ be the given assertion. $P(-1,0)$ gives that $f(-1)=0$, and $P(a,0)$ combined with $P(a,-1)$ give \[f^{a^2}(a)=af(a)=f^{a^2+1}(a-1).\]$P(a,-a)$ gives $f^{2a^2}(0)=a(f(a)-f(-a))$. Call an integer $n$ explosive if the sequence $f(n),f^2(n),f^3(n),f^4(n),\dots$ is unbounded. We distinguish two cases based on whether 0 is explosive. Case 1: If 0 is not explosive, then $f(a)=f(-a)$ for all sufficiently large $a$ (sufficiently large means sufficiently large in magnitude) from $f^{2a^2}(0)=a(f(a)-f(-a))$. On the other hand, we have \[-af(a)=-af(-a)=f^{a^2}(-a)=f^{a^2}(a)=af(a)\implies f(a)=0\]for such $a$, so $f(a)=0$ for all sufficiently large $a$. Then $f^{a^2-1}(0)=f^{a^2}(a)=af(a)=0$ for all sufficiently large $a$, implying that $f(0)=0$ (since no prime divides $a^2-1$ for all sufficiently large $a$). Finally, for any $b\neq 0$, we can take sufficiently large $a$ and get \[P(a,b)\implies 0=f^{a^2+b^2-1}(0)=f^{a^2+b^2}(a+b)=af(a)+bf(b)=bf(b)\implies f(b)=0,\]so the only solution in this case is $f(x)=0$. Case 2: Now suppose that 0 is explosive. Claim: For any $a$ and $k$ and all sufficiently large $N$ ($N\gg\max(a^2,(a-k)^2)$), $f^N(a)=f^{N+k}(a-k)$ Proof. From $f^{a^2}(a)=af(a)=f^{a^2+1}(a-1)$ we have that $f^N(a)=f^{N+1}(a-1)$ for all sufficiently large $N$. With repeated applications of this fact, we get $f^N(a)=f^{N+k}(a-k)$ for all sufficiently large $N$, as desired. $\blacksquare$ Claim: All integers are explosive. Proof. By the previous claim, $f^N(a)=f^{N+a}(0)$ for all sufficiently large $N$. Since $f^{N+a}(0),f^{N+a+1}(0),f^{N+a+2}(0),\dots$ is unbounded from 0 being explosive, then the sequence $f^N(a),f^{N+1}(a),f^{N+2}(a),\dots$ is unbounded as well, so $a$ is also explosive. This works for any $a$, so all integers are explosive, proving the claim. $\blacksquare$ Claim: $f(a)=a+1$ for all $a$. Proof. Suppose FSoC that some $a$ existed with $f(a)\neq a+1$. Then note that for all sufficiently large $N$ we have \[f^N(a+1)=f^{N+1}(a)=f^N(f(a))=f^{N+f(a)-a-1}(a+1).\]Since $N\neq N+f(a)-a-1$, this implies that the sequence $f(a+1),f^2(a+1),\dots$ eventually reaches a cycle, contradicting the fact that $a+1$ is explosive. $\blacksquare$ Therefore, the only solution in this case is $f(x)=x+1$. In conclusion, the only solutions of the FE are $f(x)=0$ and $f(x)=x+1$.

Pretty easy for its position. Let $P(a,b)$ denote the original equation. $P(-1,0)$ gives $f(-1)=0$. Comparing $P(a,0)$ and $P(a,-1)$, we get $$f^{a^2}(a)=af(a)=f^{a^2+1}(a-1)$$If we restrict $a$ to being a positive integer, applying the above repeatedly we get $$f^{a^2}(a)=f^{2a}(f^{(a-1)^2}(a-1))=f^{2a+2(a-1)}(f^{(a-2)^2}(a-2)) \cdots = f^{2a+2(a-1)+ \cdots 2}(0) $$$$\implies af(a)=f^{a^2}(a)=f^{a^2+a}(0)$$for all positive integers $a$. Call this $Q(a)$. We split the rest of the proof into two cases. Case I: $f^m(0)=f^n(0)$ for some integers $m>n \geq 0$. In this case, the sequence $f^i(0)$ for $i \geq 0$ is eventually periodic, and hence bounded. But all numbers of the $af(a)$ for positive integers $a$ lie in this sequence. Therefore, we must have $f(a)=0$ for all large positive integers $a$. Comparing $Q(a)$, $Q(a+1)$, and $Q(a+2)$ for large $a$ gives $$f^{a^2+a}(0)=f^{a^2+3a+2}(0)=f^{a^2+5a+6}(0)=0$$$$\implies f^{2a+2}(0)=f^{2a+4}(0)=0$$$$\implies f^2(0)=0$$Since $2 \mid a^2+a$ for all integers $a$, $Q(a)$ $\implies$ $f(a)=0$ for all $a>0$. Then $P(1,1)$ gives $f(0)=0$. Finally, $P(a,-a)$ for any $a>0$ gives $f(-a)=0$, and we get the solution: $$\boxed{f(x)=0 \ \ \forall x \in \mathbb{Z}}$$ Case II: All the terms of the sequence $f^i(0)$ for $i \geq 0$ are distinct. Let $k$ be any positive integer, and let $a=f^k(0)$. Then $Q(a)$ gives: $$f^{a^2+a}(0)=f^{a^2}(a)=f^{a^2+k}(0)$$$\implies$ $a^2+k=a^2+a$ by our assumption for this case $\implies$ $k=a$ $\implies$ $f^a(0)=a$ for all $a>0$. In particular, $f(0)=1$, and $$f(a)=f(f^a(0))=f^{a+1}(0)=a+1$$for all $a>0$. Finally, $P(a,-a)$ for any $a>0$ gives $f(-a)=-a+1$, and so we get the solution: $$\boxed{f(x)=x+1 \ \ \forall x \in \mathbb{Z}}$$

We claim that the only answers are $f(x)\equiv0$ and $f(x)\equiv x+1.$ It is easy to check that these work. Next, we will proceed to show that these are the only answers. Let $F(a,b)$ be the assertion $f^{a^2+b^2}(a+b)= af(a)+bf(b)$. $F(-1,0): \ f(-1)=-f(-1)$. Therefore, $f(-1)=0$. We will consider two separate cases. Case 1 $f(0)=0$ $F(a,-a): \ af(a)=af(-a)\ \forall a\in\mathbb{Z}$ so $f(a)=f(-a)\ \forall a\in\mathbb{Z}\setminus\{0\}.$ Including $f(0)$, we have $f(a)=f(-a) \ \forall a\in\mathbb{Z}.$ $F(a,-a-1):\ 0= af(a)-(a+1)f(-(a+1))\ \forall a\in\mathbb{Z}$. Therefore, $af(a)=(a+1)f(-(a+1))=(a+1)f(a+1)\ \forall a\in\mathbb{Z}$. Hence, $f(a)=\frac{a+1}{a}\cdot f(a+1)\ \forall a\in\mathbb{Z}\setminus\{0\}.$ From here, we can induct to show that $f(x)\equiv0$. Case 2 $f(0)\neq0$ Claim 1: For each $a\in\mathbb{N}_0$, we have $f^{a^2}(a)=f^{a^2+1}(a-1)=f^{a^2+2}(a-2)=\dots=f^{a^2+a}(0)=\dots=f^{a^2+2a+1}(-a-1)=f^{a^2+2a+2}(-a-2)$.

Now, since $f^{a^2}(a)=af(a)$, we have $af(a)=f^{a^2}(a)=f^{(a+1)^2}(-a-1)=-(a+1)f(-(a+1)) \ \forall a\in\mathbb{N}_0.$ We can check that this implies $af(a)=-(a+1)f(-(a+1)) \ \forall a\in\mathbb{Z}$. Claim 2: $f(x)=0$ if and only if $x=-1$.

Claim 3: $f$ is injective.

Now, recall that $f^{a^2}(a)=f^{a^2+1}(a-1)\ \forall a\in\mathbb{Z}.$ By Claim 3, $a=f(a-1)\ \forall a\in\mathbb{Z}$. Therefore, $f(x)\equiv x+1$ as desired.

solved with archp $b = 0$ gives $f^{a^2}(a) = af(a)$. Subbing in $a = -1$ to this gives $f(-1) = -f(-1)$ so $f(-1) = 0$. Then substituting $b = -1$ into the original equation gives $f^{a^2 + 1}(a - 1) = af(a) = f^{a^2}(a)$. Define $S_n = \{f^i(n): i \in \mathbb{Z}_{\geq 0}\}$. Note that by the above result, $S_n$ and $S_{n + 1}$ overlap in all but finitely many elements, so $S_m$ and $S_n$ overlap in all but finitely many elements for all pairs $(m, n)$, i.e. all such sets are finite, or all such sets are infinite. Case 1: All such sets are finite. Let $a$ have absolute value greater than the absolute value of every member of $S_0$. Substituting $(a, -a)$ into the original equation gives $f^{2a^2}(0) = a(f(a) - f(-a))$, but the left-hand side must be in $S_0$ so the right-hand side also is. But by assumption, since $a$ divides $a(f(a) - f(-a))$ and is larger than it, $f(a) = f(-a)$, so $f^{2a^2}(0) = 0$. Thus the minimal period of the sequence $0, f(0), f(f(0)), \dots$ (let it be $T$) must divide $2a^2$, but it analogously divides $2(a + 1)^2$ and therefore their gcd, which is 2. So $f(f(0)) = 0$ and $a(f(a) - f(-a)) = f^{2a^2}(0) = 0$ for all $a$ giving $f(a) = f(-a) \forall a \neq 0$. Comparing the substitutions of $(a, 0)$ and $(-a, 0)$ then gives that $f(a) = 0 \forall a \neq 0$. Thus since $0 = af(a) = f^{a^2}(a) = f^{a^2 - 1}(0)$ for such $a$, taking $a = 2, 3$ gives $f^3(0) = f^8(0) = 0$ so since $\gcd(3, 8) = 1$ we must have $f(0) = 0$ giving the solution $f(x) = 0$ for all $x$, which clearly works. Case 2: All such sets are infinite. Note that $f^{a^2}(a) = f^{a^2 + 1}(a - 1)$ implies that $f^k(a) = f^{k + 1}(a - 1)$ for sufficiently large $k$, and repeatedly using this gives $f^k(a) = f^{k + m}(a - m)$ for sufficiently large $k$ and given $m$. Then $f^k(a + 1) = f^{k + 1}(a) = f^k(f(a)) = f^{k + f(a) - a - 1}(a + 1)$, so if $k \neq k + f(a) - a - 1$ then the set $S_{a + 1}$ must be finite since the sequence $a + 1, f(a + 1), f(f(a + 1)), \dots$ is eventually periodic, contradiction. Thus $f(a) = a + 1$ and doing this for all $a$ gives the other solution $f(x) = x + 1$ for all $x$, which clearly works.

Eyed wrote: Find all functions $f : \mathbb{Z}\rightarrow \mathbb{Z}$ satisfying \[f^{a^{2} + b^{2}}(a+b) = af(a) +bf(b)\]for all integers $a$ and $b$ Cute problem. Let \(P(a,b)\) denote the assertion of the given functional equation. \(P(0,-1)\) gives us that \(f(-1)=0\) and comparing \(P(a,0)\), \(P(a,-1)\) gives that \[f^{a^2}(a)=af(a)=f^{a^2+1}(a-1)\]Inductively, we have that \[f^{a^2+a}(0)=af(a)\] We now take two cases, case 1 where \(f^{i}(0)\) are all not distinct and case 2 dealing the opposite of case 1. Case 1. There exist \(m\) and \(n\) with \(f^{m}(0)=f^{n}(0)\). This is the most difficult case of both cases. Note that \(f^{m}(0)=f^{n}(0)\) implies that \(f^{m^2+m}(0)=f^{n^2+n}(0)\) so \(mf(m)=nf(n)\). Analogously, \[(m+a)f(m+a)=(n+a)f(n+a)\]for all non-negative integers \(a\). Now, we have that \[nf(n)=(m+n-m)f(m+n-m)=(2n-m)f(2n-m)=(3n-2m)f(3n-2m)=\hdots=c\]but then \(c\) has infinitely many divisors, namely \(n, 2n-m, 3n-2m,\hdots\), so \(c\) must be \(0\), implying that \(f(n)=0\). Similarly, we get that \(f(n+a)=0\) for all \(a\geq0\) and it follows that \(f(x)=0\) for all \(x\geq\min{m,n}\). Since \[f^{m^2+m}(0)=0\]and \[f^{(m+1)^2+(m+1)}(0)=0\]we have that \(f^{2m+2}(0)=0\). Similarly, we get \(f^{2m+4}(0)=0\) by comparing \(f^{(m+1)^2+(m+1)}(0)\) and \(f^{(m+2)^2+(m+2)}(0)\). This implies that \(f(f(0))=0\). But since \(a^2+a\) is even, we have that \(af(a)=f^{a^2+a}(0)=0\) so \(f(a)=0\) for all \(a\neq0\), which is indeed a solution. Case 2. When all the \(f^{k}(0)\) are distinct. This case is relatively simple, let \(k\) be such that \(f^{k}(0)=a\). Then, \[f^{a^2+a}(0)=f^{a^2}(a)=f^{a^2+k}(0)\]so \(k=a\), implying that \(f^a(0)=a\). Therefore, \(f(a)=a+1\) for all non-negative integers \(a\). If \(a\) is positive, then since \[f^{a^2-a}(0)=-af(-a)\]we have that \(f(-a)=-a+1\), implying that \(f(x)=x+1\) for all integers \(x\).

First, don't forget to chug some trivial cases!!! $P(0,-1)$ gives $f(-1)= -f(-1)$, so $f(-1)=0$. Now, $P(a,0)$ gives $f^{a^2}(a)=af(a)$ and $P(a,-1)$ gives $f^{a^2+1}(a-1)=af(a)$. Therefore, it follows that $f^{a^2}(f(a-1))=f^{a^2}(a)$ for every integer $a$. (1) Now, if $f$ is injective, $f(x)\equiv x+1$ by (1). Otherwise, if $f(x)=f(y)$, we take $N>100x^4$ large enough such that $f^N(y)=f^N(x)=f^{N+1}(x-1) = \cdots = f^{N+x-y} (y) = f^N(y)$ so we have a cycle. I claim $f(x)\equiv 0$. Call $a_1\rightarrow a_2\rightarrow \cdots a_k\rightarrow a_1$ where $f(a_j)=a_{j+1}$ and $a_{k+1}=a_1$. If $0$ is in this cycle, then $f^k(0)$ is bounded, so for all sufficiently large $a$, $P(a,-a)$ gives $f^{2a^2}(0)=af(a)-af(-a)$. Since RHS is divisible by $a$ but the absolute value of LHS is less than $a$, it follows that $f(a)=f(-a)$ for all $a>C$ for some $C\in \mathbb{N}$. However, $P(0,a)-P(0,-a)$ gives $0=f^{a^2}(a)-f^{a^2}(-a)=af(a)- (-a)f(-a)=a(f(a)+f(-a))$. Since $f(a)=f(-a)$, it follows that $f(a)=f(-a)=0$. Going back, $f^{2a^2}(0)=0$ for all $a>C$, which implies $f^2(0)=0$. Now, $P(x,-x)$ for any $x$ implies $f(x)=f(-x)$ is an identity and we can deduce $f(x)=f(-x)=0$ for all $x\in \mathbb{N}$ via $P(0,x)-P(0,-x)$. Now, $P(a,0)$ gives $f^{a^2}(a)=f^{a^2-1}(0)=0$. Picking $a$ even proves $f(x)\equiv 0$. If this cycle has length 1 (i.e. $f(x)=x$ then $P(x,0)$ gives $x=f^{x^2}(x)=xf(x)=x^2$. Thus, $x\in \{0,1\}$). The $x=0$ case is processed above. If $x=1$ then $P(1,1)$ gives $f^2(2)=2$. Claim: If $f(x)$ is not identically 0, then a finite cycle of length at least 2 doesn't exist. First of all, since $0$ is not in the cycle and $f(-1)=0$, $-1,0$ must not be in the cycle. EXTREMAL PRINCIPLE: Let $k$ be the element with largest absolute value in the cycle. Then $P(k,0)$ gives $f^{k^2}(k)=kf(k)$. Since $f(k)\ne 0$ and $|f^{k^2}(k)|\le k$, it follows that $f(k)\in \{1,-1\}$. However -1 is not in the cycle so $f(k)=1$. Now, say $f(a)=k$. Then $P(a,0)$ gives $f^{a^2}(a)=ka$ but $|f^{a^2}(a)|\le k$, so it follows that $a=1$. Thus, the cycle is $1\to k\to 1$. Now, $P(k,1-k)$ gives $k=f^{k^2+(1-k)^2}(1)=kf(k)+(1-k)f(1-k)$ since $k^2+(1-k)^2$ is odd. This means $f(1-k)=0$. Now, $P(2k-1, 1-k)$ gives $f^{(2k-1)^2+(1-k)^2}(k) = (2k-1)f(2k-1) + (1-k)f(1-k) = (2k-1)f(2k-1)$ LHS is either $k$ or $1$. In any case, we have $|k|\ge |2k-1|$ (since 0 is not in a cycle), which means $k=1$. Therefore, $f(1)=1$. Now, $P(-1,1)$ gives $f^2(0)=1$, so $f^k(0)=1$ for all $k\ge 2$. Now, $P(2,-2)$ gives $1=2(f(2)+f(-2))$, contradiction.

Let $P(a,b)$ be the given assertion. Trivially, $f(-1)=0$ it follows $f^{a^2}(a)=f^{a^2+1}(a-1)$ from $P(0,a)$ and $P(-1,a).$ We do casework on the bounded-ness of the sequence $n,f(n),f^2(n),\dots, f^i(n).$ A. It is unbounded: Note that for large integer $M,$ $f^M(n)=f^{M+1}(n-1)$ and by induction $f^M(n)=f^{M+c}(n-c).$ From here if $f(x)=f(y)$ we can easily get contradiction with unbounded-ness if not $x=y.$ So we have the solution $f(x)=x+1$ satisfy. B. It is bounded: Then $P(n,-n)$ gives $f(n)=-n,$ for large $n.$ It follows that $f(n)=0$ from $P(n,1-n).$ Note that $f(0)=0.$ Now it is not hard to that show $f(x)=0$ for all $x,$ it satisfies.

What a nice problem! After a lot of sufferment, I was able to solve one 2020 ISL A. Call the problems relation \[f^{a^{2} + b^{2}}(a+b) = af(a) +bf(b) ~(*)\]We claim the only function that satisfies $(*)$ are $f \equiv 0$ and $f(a) = a+1, \forall a \in \mathbb{Z}$ which clearly satisfies it. Now we shall show these are the unique ones. At first, we only make some natural substitutions. $\bullet$ Plugging $(-1,0)$ at $(*)$ yelds \[ f^{1^2+ 0^2}(-1) = (-1)\cdot f(-1) + 0\cdot f(0) \]that is $f(-1) = 0~(1A).$ $\bullet$ Now, plugging $(a,-1)$ using $(1A)$ yelds $$f^{a^2+1}(a-1) = af(a), \forall a \in \mathbb{Z}~(2A)$$On the other hand, by plugging $(a,0),$ we get $$ f^{a^2}(a) = af(a), \forall a \in \mathbb{Z}$$Comparing the above equation with $(2A)$ yelds $$ f^{a^2}(a) = f^{a^2+1}(a-1) = af(a), \forall a \in \mathbb{Z}~(3A)$$The above relation suggest us that if we were able to show that either $f \equiv 0$ or $f$ is injective, then we are done. Then, the following claim will be the main step of the solution. Claim: If exists some $n \ge 1$ such that $f^n(0) = 0,$ then $f \equiv 0.$

Now assume that $f^n(0) \ne 0, \forall n\ge 1.$ Now we have two main steps to finish step 1: The sequnce ${ \{f^n(0) \} }_{j \ge 1}$ is injective that is $f^x(0) = f^y(0)$ implies $x= y$ for any $x,y \ge 1.$

step 2: $f$ is injective.

Now we are done! Indeed, since $f$ is injective, the relation $(3A)$ implies that $$ f(a-1) = a, \forall a \in \mathbb{Z} $$that is $f(a) = a+1, \forall a \in \mathbb{Z}.$

We claim the only solutions are $f \equiv 0$ and $f(n) = n+1$. If $f \equiv 0$, both sides vanish, and if $f(n) = n+1$, $f^{a^2+b^2}(a+b) = a^2 + a + b^2 + b = af(a) + bf(b)$. Now we show they are the only solutions. Let $P(a, b)$ denote the given FE. Part 1: Pre-requisites. $P(0, a): f^{a^2}(a) = af(a)$. Let this be $I(a)$. $P(-a, a): f^{2a^2}(0) = af(a) - af(-a)$. Let this be $II(a)$. $P(a-1, -a-1): f^{2a^2 + 2}(-2) = (a-1)f(a-1) + (-a-1)f(-a-1)$. Let this be $III(a)$. Note that $I(-1)$ gives $f(-1) = -f(-1) \Rightarrow f(-1) = 0$. Thus $P(-1, a)$ gives $f^{a^2+1}(a-1) = af(a) = f^{a^2}(a)$. Let this be $IV(a)$. $IV(-1)$ gives $f^2(-2) = f(-1) = 0$. Hence, $III(a)$ can be re-written as $f^{2a^2}(0) = f^{2a^2}(f^2(-2)) = (a-1)f(a-1) + (-a-1)f(-a-1)$. Claim 1.1: $nf(n) = (-1-n)f(-1-n)$ for all $n$. Proof: $I(a)$ and $III(a)$ give $af(a) + (-a)f(-a) = (a-1)f(a-1) + (-a-1)f(-a-1) \Rightarrow d(a) = d(a-1)$, where $d(n) = nf(n) - (-1-n)f(-1-n)$. Hence $d(n)$ is fixed for all $n$. Yet $d(0) = 0f(0) - (-1)f(-1) = 0$, so $d(n) = 0$. $\square$ Corollary 1.1.1: $n+1 \mid f(n)$. $P(a, -1-a): f^{2a^2 + 2a + 1}(-1) = af(a) + (-1-a)f(-1-a) \Rightarrow f^{2a^2 + 2a}(0) = 2af(a)$. Let this be $V(a)$. Part 2: Analysing the kernel. Case 1: $f(0) = 0$. Then by $II(a)$ and Claim 1.1, $af(a) = -af(-a) = (a-1)f(a-1)$. Inducting we get that $nf(n)$ is fixed for all $n$, yet $0f(0) = 0$. Thus, $nf(n) = 0$, and for all $n \neq 0$, $f(n) = 0$. Here we get $f \equiv 0$. Claim 2.1: If $f^3(0) = 0$, then $f \equiv 0$. Proof: If $a \equiv 0, 2 \pmod{3}$, then $2a^2 + 2a \equiv 0 \pmod 3$. Hence, $V(a)$ gives $0 = 2af(a)$ for all $a \equiv 0, 2 \pmod{3}$. This means $f(3) = 0$. $I(3)$ gives $f^9(3) = 0 \Rightarrow f^8(0) = 0$. Thus 3 and 8 are both periods for $f^i(0)$, so their gcd is too. Hence $f(0) = 0$ and we reduce down to Case 1. $\square$ Case 2: $f(1) = 0$. $1f(1) = -2f(-2) \Rightarrow f(-2) = 0$. $I(-2)$ gives $f^4(-2) = 0 \Rightarrow f^3(0) = 0$. By Claim 2.1, $f \equiv 0$. Case 3: $f(m) = 0$, where $|m| > 1$. Similar to above, $f(-1-m) = 0$ and $f^{m^2 - 1}(0) = 0$. Furthermore, $f^{(-1-m)^2-1}(0) = 0$, so $f^{m^2 - 2m}(0) = 0$. $\gcd(m^2 - 1, m^2 - 2m) = \gcd(m^2-1, 2m-1) = \gcd((m-1)(m+1), 2m-1)$. If $p \mid m-1$ and $p \mid 2m-1$, then $p \mid m$ yields a contradiction. Thus, $\gcd((m-1)(m+1), 2m-1) = \gcd(m+1, 2m-1) \mid 3$. So either $f^3(0) = 0$ or $f(0) = 0$, both cases yielding $f \equiv 0$. Part 3: Finding successors. Now we can assume $f(n) = 0 \Rightarrow n = -1$. First we do some base calculations. $f^2(-2) = 0 \Rightarrow f(-2) = -1$. Then Claim 1.1 gives $f(1) = 2$. $II(1)$ gives $f^2(0) = f(1) - f(-1) = 2$. If $f(n) = 2$, then by Corollary 1.1.1, $n = -3, -2, 0, 1$ are the only options. $f(0) = 2 \Rightarrow f(2) = 2$ yet $3 \mid f(2)$ so contradiction. $f(-2) = -1$ so $f(-2) \neq 2$. $f(-3) = 2 \Rightarrow f(2) = -3$, and so $I(2)$ gives $f^4(2) = 2f(2) \Rightarrow 2 = -6$ contradiction. Thus $f(n) = 2 \Rightarrow n = 1$, and since $f(f(0)) = 2$, we have $f(0) = 1$. Claim 3.1: For $n \neq -1$, $f(n) \neq -n-1$. Proof: Assume $f(n) = -n-1$. By Claim 1.1, $f(-n-1) = n$, so actually $f^2(n) = n$. Then $I(n)$ gives $f^{n^2}(n) = nf(n) \Rightarrow n(-n-1) = -n-1$ if $n$ is odd or $n(-n-1) = n$ if $n$ is even. In the former, either $n = -1$ or $n = 1$, and in the latter either $n = 0$ or $n = -2$. In these cases, we know only $n = -1$ satisfy $f(n) = -n-1$. $\square$ Claim 3.2: $|f(n)| - |n| \ge -1$. Proof: Since $n+1 \mid f(n)$, $|f(n)| \ge |n+1|$ or $f(n) = 0$. Thus $|f(n)| - |n| \ge |n+1| - |n| \ge -1$, or $n = -1$ and $|0| - |-1| = -1$. $\square$ Claim 3.3: Suppose $f^k(n) = n+k$, where $n \in \mathbb{N}$ and $k \le n/2 + 1$. Then $f(n+i) = n+i+1$ for $i = 0, 1, ..., k-1$. Proof: Consider $a_0 = n$, and $a_{i+1} = f(a_i)$. Then $a_k = n+k$. $a_i + 1 \mid a_{i+1}$, so either $|a_{i+1}| \ge 2|a_i + 1|$, $|a_{i+1}| = |a_i + 1|$ or $a_{i+1} = 0$. We proceed by induction to show $a_i = n + i$. Base case: $a_0 = n$. Inductive step: Assume $a_{i-1} = n + (i-1)$. Assume $i \ge 1$. Case 1: $|a_i| \ge 2|a_{i-1} + 1| = 2(a_{i-1} + 1) = 2(n + i)$. By Claim 2, $|a_j|$ decreases by at most 1 for each increase in $j$. Thus, $|a_k| \ge |a_i| - (k-i) \Rightarrow n + k \ge |a_i| - k + i \Rightarrow n + 2k - i \ge |a_i| \Rightarrow n + 2k - i \ge 2(n + i) \Rightarrow 2k \ge n + 3i$. This is a contradiction, since we get $n + 2 \ge 2k \ge n + 3i \ge n + 3$. Case 2: $a_i = 0$. But then $-1 = a_{i-1} = n + (i-1) > 0$ contradiction. Thus, $|a_i| = |a_{i-1} + 1|$, or equivalently $|f(a_{i-1})| = |a_{i-1} + 1|$. By Claim 3.1, since $a_{i-1} \neq -1$, we have $f(a_{i-1}) = a_{i-1} + 1$, or rather $a_i = a_{i-1} + 1 = n + i$. $\square$ Now we prove by induction that for all $n \in \mathbb{N}$, $f^n(0) = n$. Base cases: $f^1(0) = 1$, $f^2(0) = 2$. $P(1, 1)$ gives $f^2(2) = f(1) + f(1) \Rightarrow f^2(2) = 4$. By Claim 3.3, $f(2) = 3$ and $f(3) = 4$. $I(2)$ gives $f^4(2) = 6 \Rightarrow f^2(4) = 6$. Claim 3.3 gives $f(4) = 5, f(5) = 6$. $II(2)$ gives $f^8(0) = 2f(2) - 2f(-2) \Rightarrow f^2(6) = 8$. Claim 3.3 gives $f(6) = 7$, $f(7) = 8$. Inductive step: Assume for all $i \le 2(a-1)^2$, $f^i(0) = i$. Note that this means $f(i-1) = i$, and hence by Claim 1.1, $f(-i) = -i + 1$. Assume $a \ge 3$. For $a \ge 3$, $a-1, a \le 2(a-1)^2$. Hence $V(a-1)$ gives $f^{2(a-1)^2 + 2(a-1)}(0) = 2(a-1)f(a-1) \Rightarrow f^{2a^2 - 4a + 2 + 2a - 2}(0) = 2a(a-1) \Rightarrow f^{2a^2 - 2a}(0) = 2a^2 - 2a$. Furthermore, $I(a)$ gives us $f^{2a^2}(0) = af(a) + (a-1)f(a-1) \Rightarrow f^{2a^2}(0) = 2a^2$. Now we have $f^{2a^2 - 4a + 2}(0) = 2a^2 - 4a + 2$, $f^{2a^2 - 2a}(0) = 2a^2 - 2a$, and finally $f^{2a^2}(0) = 2a^2$. Thus $f^{2a-2}(2a^2 - 4a + 2) = f^{2a^2 - 2a}(0) = 2a^2 - 2a$ and $f^{2a}(2a^2 - 2a) = f^{2a^2}(0) = 2a^2$. $2a - 2 \le a^2 - 2a + 1 \iff a^2 - 4a + 3 \ge 0 \iff (a-3)(a-1) \ge 0$. Hence, for $a \ge 3$, $(2a - 2) \le (2a^2 - 4a + 2)/2$, so by Claim 3.3, $f^i(0) = i$ for $i \in \{2a^2 - 4a + 2, ..., 2a^2 - 2a\}$. Furthermore, $2a \le a^2 - a \iff a^2 - 3a \ge 0 \iff a(a-3) \ge 0$. Hence for $a \ge 3$, $(2a) \le (2a^2 - 2a)/2$ and by Claim 3.3, $f^i(0) = i$ for $i \in \{2a^2 - 2a, ..., 2a^2\}$. Hence, we have proved that for all $i \le 2a^2$, $f^i(0) = i$. Finally, we have $f(n) = n+1$ for all $n \in \mathbb{N}$, as well as $f(0) = 1, f(-1) = 0$. By Claim 1.1, we get $f(-1-n) = -n$ as well, so indeed $f(n) = n + 1$ for all integers $n$.

Let $P(a,b)$ denote the assertion, then \begin{align*} P(-1,0) &: f(-1)=-f(-1) \implies f(-1)=0 \\ P(a,0) &: f^{a^2}(a)= af(a) \\ P(a,-1) &: f^{(a^2+1)}(a-1) = af(a)-f(-1) = af(a)=f^{a^2}(a) \end{align*}From the third one we know that $f^{(N+1)}(a-1)=f^N(a)$ for all $N\ge a^2$. Therefore, we have $f^{N}(a)=f^{(N+a)}(0)$. This shows: \[f^N(a)=f^{(N+1)}(a-1)=f^N(f(a-1))=f^{(N+f(a-1))}(0)=f^{(N+f(a-1)-a)}(a)\]Therefore, if $f(a-1)\neq a$, then the sequence $a$, $f(a)$, $f^2(a)$, $\dots$ is eventually periodic. Note that $f(a)=a+1$ is definitely a solution to the original functional equation, so let us consider a single value $a$ such that the sequence $a$, $f(a)$, $f^2(a)$, $\dots$ is eventually periodic. Note that if $f^N(a)=f^{N+k}(a)$ then $f^{(N+a)}(0)=f^{(N+k+a)}(0)$, which means that the sequence $0$, $f(0)$, $f^2(0)$, $\dots$ is eventually periodic. Suppose that the maximum value that the absolute value of this sequence attains is $M\ge 0$. Then, $P(M+k,-(M+k))$ for positive values $k$ gives \[f^{2(M+k)^2}(0) = (M+k)(f(M+k) - f(-(M+k)))\]then we have \[-\frac{M}{M+k}\le f(M+k) - f(-(M+k))\le \frac{M}{M+k}\]so $f(M+k) = f(-(M+k))$. Thus, $f^{(M+k)^2}(M+k)=f^{(M+k)^2}(-(M+k))$, which implies that \[(M+k)f(M+k) = -(M+k)f(-(M+k))=-(M+k)f(M+k)\]This shows that $f(K) = f(-K) = 0$ for all sufficiently large $K$. Note that $f^{a^2+a}(0)=af(a)=0$ for sufficiently large $a$. Note that $\gcd(a(a+1),(a+1)(a+2),\dots)=2$ so the minimum period of the sequence $0$, $f(0)$, $f^2(0)$, $\dots$ is divisible by $2$. Regardless of size, $af(a)=f^{a^2+a}(0)=0$ for all $a$, due to the periodicity by $2$. Thus, $f(a)=0$ for all $a\neq 0$. If $f(0)\neq 0$ then $P(a,-a)$ is a contradiction. Thus, the solutions are $f(a)=0$ and $f(a)=a+1$, which both work.

The answer is $f \equiv 0$ and $f(x)=x+1$ only, which clearly work. I will now show that these are the only solutions; let $P(a,b)$ denote the given assertion. From $P(-1,0)$, we find that $f(-1)=-f(-1)$, hence $f(-1)=0$. Hence for any $a \in \mathbb{Z}$, by comparing $P(a,0)$ and $P(a,-1)$ we have $$f^{a^2}(a)=af(a)=f^{a^2+1}(a-1).$$If $a \geq 0$, by comparing $P(a-1,0)$ and $P(a-1,-1)$ we also find that $f^{a^2-2a+1}(a-1)=f^{a^2-2a+2}(a-2)$, hence $f^{a^2+1}(a-1)=f^{a^2+2}(a-2)$. Repeating this, we find that for any $a \geq 0$, we have $$f^{a^2}(a)=f^{a^2+1}(a-1)=\cdots=f^{a^2+a}(0)=f^{a^2+a+1}(-1)=af(a)\qquad(1).$$If $a<0$, then by comparing $P(a+1,0)$ and $P(a+1,-1)$, we have $f^{a^2+2a+1}(a+1)=f^{a^2+2a+2}(a) \implies f^{a^2-1}(a+1)=f^{a^2}(a)$, hence by repeating this we have $$f^{a^2+1}(a-1)=f^{a^2}(a)=f^{a^2-1}(a+1)=\cdots=f^{a^2+a+1}(-1)=f^{a^2+a}(0)=af(a)\qquad(2).$$Suppose that $f$ has a cycle, so there exists some $c$ such that $f^k(c)$ is bounded when $k$ ranges across $\mathbb{Z}^+$. If $c\geq -1$, then take $a$ massive and positive in $(1)$ to find that $f(a)=0$ for all sufficiently large positive $a$, since otherwise the absolute value of the RHS is at least $a$, which is unbounded, while $f^{a^2+\text{something}}(c)$ is bounded. If $c \leq 0$, then take $a$ massive and negative in $(2)$ to find that $f(a)=0$ for all sufficiently large negative $a$, for the same reason. Suppose that $c \geq -1$, so every large enough positive $a$ has $f(a)=0$. Then for all sufficiently large $a$, we also have $f^{a^2+a}(0)=0$, hence $0$ belongs to some cycle (this actually implies that $c \leq 0$ is also fine, and every small enough negative $a$ has $f(a)=0$, but that's not important). Furthermore, if the length of this cycle is $l$, then we must have $l \mid a^2+a$ for all sufficiently large $a$, hence $l \mid 2$, since otherwise we can pick $a \equiv 1 \pmod{l}$ to get a contradiction, hence $f^2(0)=0$. Then, for any integer $b \neq 0$, pick $a$ large and positive such that $f(a+b)=0$ and $a$ is the opposite parity of $b$, so $a^2+b^2$ is odd. Then $P(a,b)$ yields $$f^{\text{odd}}(a+b)=bf(b) \implies f^{\text{even}}(0)=bf(b) \implies bf(b)=0 \implies f(b)=0,$$so $f$ is zero everywhere, except for possibly at $0$. Now pick $a$ and $b$ of the same parity such that their sum is nonzero, so $P(a,b)$ yields $$f^{\text{even}}(a+b)=0 \implies f^{\text{odd}}(0)=0 \implies f(0)=0,$$so we extract the solution $f \equiv 0$. The case where $c \leq 0$ is similar. Therefore, suppose that no cycles exist. I claim that $f$ is injective. Indeed, suppose that there existed $p \neq q$ such that $f(p)=f(q)$. If $|p|<|q|$, then we have $$f^{p^2}(q)=f^{p^2}(p)=f^{p^2+p}(0) \implies f^{q^2}(q)=f^{q^2+p}(0),$$but we also have $f^{q^2}(q)=f^{q^2+q}(0)$, so $0$ is in a cycle, contradicting our assumption. Hence the only such $(p,q)$ that could possibly exist satisfy $p+q=0$, so by $P(p,q)$ we find that $f^{2p^2}(0)=(p+q)f(p)=0$, hence $0$ is in a cycle since $p \neq q \implies p \neq 0$: also a contradiction. From here, it follows from $f^{a^2}(a)=f^{a^2+1}(a-1)$ that $a=f(a-1)$, i.e. $f(x)=x+1$ for all $x \in \mathbb{Z}$, which is our other solution. $\blacksquare$ Remark: The main difficulty in this problem is that we are given an iterated FE, so we will probably want to consider the "arrows graph" of $f$ at some point, but the FE in its original form does not give us any real way of doing this, since multiplication and addition combined make it hard to relate $af(a)+bf(b)$ to anything in this arrows graph. Therefore, a key part of the problem is finding $f^{a^2}(a)=f^{a^2+1}(a-1)$, which is clearly something that relates very directly to the arrows graph, and the rest of the problem is basically using this to pin down the structure of $f$, although there are a lot of annoying details throughout (beginning with the fact that our chains in $(1)$ and $(2)$ stop at $-1$ and $0$, which I did not even realize was the case while originally doing the problem).

The answer is $f \equiv 0$ and $f(x)=x+1$, and they clearly work. Denote by $P(x, y)$ the assertion that the functional equation holds for $(a, b)=(x, y)$. Lemma: For each $0 \le k \le a$, \[ f^{a^2+a-k}(k) = af(a). \]Proof. First, $P(-1, 0)$ yields $f(-1)$, and applying transitivity on $P(a, 0)$ and $P(a, -1)$, we have \[ f^{a^2}(a)=f^{a^2+1}(a-1)=af(a). \]The lemma follows by applying this for smaller $a$ and inducting down. Now we look at the chain $\mathcal{C}:=(0, f(0), f(f(0)), \ldots)$. Case 1: All elements of $\mathcal{C}$ are distinct. Then, each element of $\mathcal{C}$ is a unique element of $\mathbb{Z}_{\ge 0}$, as all elements are nonnegative. Thus, for each $a$, there exists unique $k_a$ such that $f^{k_a}(0)=a$. By the lemma it follows that \[ f^{a^2}(a) = f^{a^2+a}(0) = f^{a^2}(f^a(0)), \]so $k_a=a$. Thus, $f(a)=a+1$ whenever $a \ge 0$. To extend this, note that \[ f^{a^2-a}(0)=-af(-a) \]by a similar argument as in the proof of the lemma, which finishes. Case 2: For distinct nonnegative $m$ and $n$ where $f^m(0)=f^n(0)$. Then, $f^{m^2+m}(0)=f^{n^2+n}(0)$, and thus $mf(m)=nf(n)$. Thus we can shift by $k \ge 0$ and obtain \[ (m+k)f(m+k)=(n+k)f(n+k). \]Call the above equality $K(m+k, n+k)$. This allows us to induct in the following funny way: WLOG $m<n$, and take $K(m, n)$, $K(n, 2n-m)$, $K(3n-2m, 4n-3m)$ and so on, with the integers in the $K$-statements forming an arithmetic progression $a_0, a_1, \ldots$. Now we have all $a_i$ dividing $mf(m)=nf(n)$, which implies $f(m)=f(n)=0$. Since this also holds for all $(m+k, n+k)$ as well, we have $f(x)=0$ whenever $x \ge m$. So it suffices to look at the small cases. Noting that $f(m)=f(m+1)=f(m+2)=0$, we readily have \[ f^{m^2+m}(0)=0, f^{m^2+3m+2}(0)=0, f^{m^2+5m+6}(0)=0. \]Thus $f^{2m+2}(0)=f^{2m+4}(0)=0$, so $f^2(0)=0$. Since $2 \mid a^2+a$, $af(a)=f^{a^2+a}(a)=0$, which implies $f(a)=0$ for all nonzero $a$. By $P(1, -1)$, $f(0)=0$, and we are done.

Sketch: 1. First get $f^{a^2}{(a)}=f^{a^2+1}(a-1)=af(a)$. Repeat this 2. then split into cases whether the arrow-graph of $f$ has cycles. 3. If it has cycles, conclude f is bounded, $0$ must be contained in some cycle, showing $f \equiv 0$ 4. If it does not have cycles, prove that $f$ is injective 5. Conclude by step 1.