Let $F:\mathbb R^{\mathbb R}\to\mathbb R^{\mathbb R}$ be a function (from the set of real-valued functions to itself) such that
$$F(F(f)\circ g+g)=f\circ F(g)+F(F(F(g)))$$for all $f,g:\mathbb R\to\mathbb R$.
Prove that there exists a function $\sigma:\mathbb R\to\mathbb R$ such that
$$F(f)=\sigma\circ f\circ\sigma$$for all $f:\mathbb R\to\mathbb R$.
1) $F$ is injective
This give $F(f_1)\circ g=F(f_2)\circ g \iff f_1\circ F(g)=f_2\circ F(g).$
2) If we let $S_a$ be the range of $F(A(x))$ where $A(x)$ is a constant function with range $\{a\}$,then
$$R(F(g))=\bigcup_{a\in R(g)}S_a$$Where $R(f)$ is the range of $f$ for any function $f.$
3)For any constant function $A,B$,
$$R(F(A))=R(F(B)) \text{ or } R(F(A))\cap R(F(B))=\phi$$And if $R(F(A))=R(F(B))$,then $f(a)=f(b)$ for all $f\in R(F)$ where $a,b$ is the value of function $A,B.$
4) For any constant function $A$,there exists exactly one function $B$ such that $F(B)=A$,and that function must be constant.
5)There exists $\sigma_1,\sigma_2$ such that $$F(f)=\sigma_1\circ f \circ \sigma_2$$for all function $f.$
6)$\sigma_1=\sigma_2$
Apology for a 3.5 years late response!
I accidentally found this post again recently and want to finally post a complete solution (mostly because I think that the problem is quite innovative).
I don't really remember the solution I came up with in mid-2018, so the present solution will probably be quite a bit different from the original one. (Here, I skip step 3 and 5 entirely.)
math90 wrote:
Let $F:\mathbb R^{\mathbb R}\to\mathbb R^{\mathbb R}$ be a function (from the set of real-valued functions to itself) such that
$$F(F(f)\circ g+g)=f\circ F(g)+F(F(F(g)))$$for all $f,g:\mathbb R\to\mathbb R$.
Prove that there exists a function $\sigma:\mathbb R\to\mathbb R$ such that
$$F(f)=\sigma\circ f\circ\sigma$$for all $f:\mathbb R\to\mathbb R$.
For simplicity, I denote by $c_a$ the constant function with value $a$ and domain $\mathbb{R}$. $R$ still remains the same as in #2. Moreover, any function mentioned will have the domain and codomain $\mathbb{R}$ unless specified otherwise.
We denote by $P(f,g)$ the equation.
The solution can be broke down into several parts:
ClaimFor all $a\in\mathbb{R}$, there exists a function $f$ and real number $x$ such that $F(f)(x)=a$.
Proof$P(c_{a-F(F(F(g)))(0)},c_0)$ implies the claim.
Claim$F$ is injective.
ProofAssume for the contrary that there exist two non-identical functions $f_1,f_2$ such that $F(f_1)=F(f_2)$. Note that since they are non-identical, there must exists a real number $k$ such that $f_1(k)\neq f_2(k)$.
In this case, we know that for any functions $g$ and real numbers $x$, we have $f_1(F(g)(x))=f_2(F(g)(x))$. However, since $F(g)(x)$ can take any real value (by part 1), we get a contradiction.
From injectivity, we get that $F(f_1)\circ g=F(f_2)\circ g \iff f_1\circ F(g)=f_2\circ F(g).$
ClaimFor any $a\in\mathbb{R}$, there exists $b\in\mathbb{R}$ such that $F(c_a)=c_b$.
ProofFrom the previous part and the first part, we know that there exists $r\in\mathbb{R}$ such that $a\in R(F(c_r))$.
Now, consider $P(c_0,g):\, F(F(c_0)\circ g+g) = F(F(F(g)))\implies F(F(g))=F(c_0)\circ g+g$. This implies that for any constant function $g$, $F(F(g))$ is also a constant function.
Consider $F(F(c_r))=c_s$ for some $s\in\mathbb{R}$. Since $a\in R(F(c_r))$, we have
$$R(F(c_a))\in \bigcup_{x\in R(F(c_r))}R(F(c_x)) =R(F(F(c_r)))=\{s\}\implies F(c_a)=c_s$$as desired.
Claim$F$ permutes the set of all constant functions.
ProofSince we already prove injectivity, we only need to prove that all constant functions are in the range of $F$. This can be done by noting that if for some $a\in\mathbb{R}$, $c_a$ is not in the range, then $a\not\in R(F(c_x))$ for any $x$ and thus
$$a\not\in \bigcup_{x\in R(g)}R(F(c_x)) = R(F(g))$$for any functions $g$, which contradicts the first part.
From the previous part, we can conclude that there exists a permutation $h$ of real numbers such that $F(c_a)=c_{h(a)}$
Claim$F,h$ are involutions such that $h(0)=0$.
Proof$P(c_0,c_0):\, F(c_{h(0)})=F(F(F(c_0)))\implies c_{h(0)}=c_{h(h(0))}\implies h(0)=h(h(0))\implies h(0)=0$. $P(c_0,g)$ then implies the involution property for both functions.
From the previous part, we can simplify $F(F(F(g)))$ to $F(g)$.
Claim$\sigma\equiv h$ works.
ProofFor any $a,b\in\mathbb{R}$, we have
$$P(c_{h(a)},c_b):\, c_{h(h(h(a))+b)}=c_{a+h(b)}\implies h(a+b)=h(a)+h(b)$$Thus, $h$ is an additive function.
Finally, for any real numbers $a$ and functions $f$, we have $P(f,c_a):\, c_{h(F(f)(a)+a)}=c_{f(h(a))+h(a)}\implies h(F(f)(a)+a)=f(h(a))+h(a)\implies h(F(f)(a))=f(h(a))\implies F(f)(a)=h(f(h(a)))$ and we are done.
For any additive function $h$ that's also an involution, $F(f)=h\circ f\circ h$ works.