Sketch: Make the trig substitution $x=\tan\left(\frac{_A}{^2}\right), y=\tan\left(\frac{_B}{^2}\right), z=\tan\left(\frac{_C}{^2}\right)$ with $A+B+C=180$ (since $xy+yz+zx=1$). Then plugging this into $x+\frac1x$ and simplifying gives $x+\frac1x=\frac{1}{\sin \left( \frac{A}{2} \right) \cos \left( \frac{A}{2} \right) }=\frac{2}{\sin A}$, and similarly for $y+\frac1y$ and $z+\frac1z$. This means that $\frac{\sin A}{\sin B}=\frac{5}{12}$ and $\frac{\sin A}{\sin C}=\frac{5}{13}$. As $A,B,C$ are the angles of a triangle, it follows by the law of sines that they are the angles of a triangle similar to a 5-12-13 triple. Therefore, $\tan A=\frac{5}{12},\tan B=\frac{12}{5}, \sin C=1$. So $z=\tan\left(\frac{C}{2}\right)=\tan 45=1$, and we can get the other two values by using double angle formulas for tangent, and we end up getting $x=\frac{1}{5},y=\frac{2}{3},z=1$.

Can you explain (prove) why that substitution works?

just remark $tan(\frac A 2)=\frac 1 { tan(\frac{B+C}2)}$ and the formula. Also, the negative values for $x,y,z$ work too obviously.

Shouldn't the equation be \(xy + yz + zx = 1\)?

5 12 13 whispers a geometric interpretation

There is a completely algebraic solution: Just homogenize the first equation using the second: \[x+\frac{1}{x}=\frac{x^2+1}{x}=\frac{x^2+xy+xz+yz}{x}=\frac{(x+y)(x+z)}{x}.\]So we get \[\frac{xy+xz}{5}=\frac{yx+yz}{12}=\frac{zx+zy}{13}\]from where we can easily solve for $xy=\frac{2}{15}, yz=\frac{2}{3}, zx=\frac{1}{5}$ and then easil find that $(x,y,z)=\left(\frac{1}{5}, \frac{2}{3}, 1\right)$ or $(x,y,z)=\left(-\frac{1}{5}, -\frac{2}{3}, -1\right)$. (Incidentally, the solution in #2 misses this second solution, which shows how careful one has to be with geometric substitutions.)

This is a full solution from #2's sketch. First of all, notice that the sign of $x$ also determines the sign of $y$ and $z$; thus if $(x, y, z)$ is a solution, then so is $(-x, -y, -z)$. Now use the same substitution as that of #2. Rewrite the first equation to be $\frac{x}{5(x^2 + 1)} = \frac{y}{12(y^2+1)} = \frac{z}{13(z^2+1)}$, where by the double-angle formula we obtain $\frac{x}{x^2+1}=\frac{\tan{\frac{A}{2}}}{\tan^2{\frac{A}{2}}+1}=\frac{\tan{\frac{A}{2}}}{\sec^2{\frac{A}{2}}}=\frac{\sin{A}}{2}$, and doing similar operations on $y$ and $z$ yield $\frac{\sin{A}}{5} = \frac{\sin{B}}{12} = \frac{\sin{C}}{13}$ where we obtain the information that the sides of the triangle ABC are similar to a 5-12-13 triangle. Thus $\frac{x}{x^2+1} = \frac{5}{26} \Longrightarrow x = 5 \wedge x = \frac{1}{5}$ which implies $y = \frac{2}{3} \wedge y = \frac{3}{2}$ and $z = 1$. Substituting $z = 1$ into the second equation clearly yields the only solution in positive reals as $\left(\frac{1}{5}, \frac{2}{3}, 1\right)$ and thus $\left(-\frac{1}{5}, -\frac{2}{3}, -1\right)$ is a solution too and both of them are indeed all the solutions.