$\sqrt{2}<x<\sqrt{3}$ so $\{\frac{1}{x}\}=\frac{1}{x}$ and $\{x^2\}=x^2-2$. Hence $\frac{1}{x}=x^2-2$. So $x^3-2x-1=0=(x+1)(x^2-x-1)$. We deduce that $x=\frac{1+\sqrt{5}}{2}$. Hence $x(x^7-21)=13$. To calculate $x^8-21x$ you can use $x^2=x+1$.

joa51736 wrote: $\sqrt{2}<x<\sqrt{3}$ so $\{\frac{1}{x}\}=\frac{1}{x}$ and $\{x^2\}=x^2-2$. Hence $\frac{1}{x}=x^2-2$. So $x^3-2x-1=0=(x+1)(x^2-x-1)$. We deduce that $x=\frac{1+\sqrt{5}}{2}$. Hence $x(x^7-21)=13$. Why $\{\frac{1}{x}\}=\frac{1}{x}$ and $\{x^2\}=x^2-2$ ?

Because $0<\frac{1}{x}<1$ and $2<x^2<3$.

I understand now. Thanks joa51736.