Suppose to the contrary that there's another pair, called $(c,d)$ where $c>d$. Easy to see that $a,b,c,d$ are pairwise distinct. WLOG $a>b$. Observe that we've $f(f(a))=a$ and $f(f(b))=b$. Similarly, $f(f(c))=c$ and $f(f(d))=d$. So $a, b,c,d$ are four distinct roots of the equation $f(f(x))=x$ and there's no other root. Now consider the continuous polynomial function $g(x)=f(x)-x$. We've $g(a)<0$ and $g(b)>0$. So there exists real number $r\in (a,b)$ that $g(r)=f(r)-r=0\Rightarrow f(r)=r\Rightarrow f(f(r))=r$. Hence $r\in \{ a,b,c,d\}$. But we have $f(t)\neq t$ for all $t\in \{ a,b,c,d\}$, contradiction, done.

Another solution: Easy to show $f(x)=k(x-a)(x-b)-x+a+b$ where $k \neq 0$ $d=f(c)=k(c-a)(c-b)-c+a+b$ and $c=k(d-a)(d-b)-d+a+b$ so $k(d-a)(d-b)=k(c-a)(c-b)=(c+d)-(a+b) \to (d-c)(d+c-a-b)=0 \to a+b=d+c \to (c-a)(c-b)=0 \to $ contradiction

RagvaloD wrote: $f(x)$ is square polynomial and $a \neq b$ such that $f(a)=b,f(b)=a$. Prove that there is not other pair $(c,d)$ that $f(c)=d,f(d)=c$ Does square polynomial mean quadratic polynomial here?

madmathlover wrote: RagvaloD wrote: $f(x)$ is square polynomial and $a \neq b$ such that $f(a)=b,f(b)=a$. Prove that there is not other pair $(c,d)$ that $f(c)=d,f(d)=c$ Does square polynomial mean quadratic polynomial here? Yes