Of course, it can be solved by standart descent method. Case 1. Suppose $\gcd(n,m)=1$. Then $mn$ is a perfect square, so $m$ and $n$ are perfect squares. Let $m=a^2$, $n=b^2$, $a$, $b$ are nonnegative integers. We have $a^2-3ab+b^2-5=0$. Assume WLOG $a>b$. It is clear $b>0$. If $b=1$ then $a=4$, if $b=2$ then $a$ is irattional. Assume now $b>2$ and consider pair $(a',b)$, where $a'=\frac{b^2-5}{a}<a$. This pair also satisfies the equation and $a'+b<a+b$. Therefore, step by step we will necessary come to pair $(4,1)$. Moreover, as we see, all solutions can be found in a reccurent sequence. Indeed, define $a_1=4$, $b_1=1$ and $a_{k+1}=\frac{a_k^2-5}{b_k}$, $b_{k+1}=a_k$. Case 2. Suppose $\gcd(n,m)>1$. Then it is necessary $\gcd(m,n)=5$. So we can write $m=5a^2$ and $n=5b^2$. Apply descent method again.

Assign $m=u+v,n=u-v$. Hence, $5(u+2)^2-9v^2=45$. So $u+2=3x,v=5y$, imply $x^2-y^2=1$. Therefore, $x=1,y=0$ so $m=n=1$.

analysis90 wrote: Assign $m=u+v,n=u-v$. It requires $m$ and $n$ to be the same parity.

ThAzN1 wrote: Find all pairs of nonnegative integers $(m,n)$ such that $(m+n-5)^2=9mn$. Write this as $(2m-7n-10)^2-45(n+2)^2+360=0$ So $2m-7n-10=15u$ and $(n+2)^2-5u^2=4$ Take then any solution of the Pell equation $x^2-5y^2=4$ and build a solution $(m,n)=(\frac{7x+15y}2-2,x-2)$ And the solutions of the Pell equaiton are quite classical. So, for example : $(x,y)=(7,3)\to(m,n)=(45,5)$ $(x,y)=(123,55)\to(m,n)=(841,121)$ $(x,y)=(2207,987)\to(m,n)=(15125,2205)$ $(x,y)=(39603,17711)\to(m,n)=(271441,39601)$ $(x,y)=(710647,317811)\to(m,n)=(4870845,710645)$ ...

Wow, how would one think of this factorisation $(2m-7n-10)^2-45(n+2)^2+180=0$?

Churent wrote: Wow, how would one think of this factorisation $(2m-7n-10)^2-45(n+2)^2+180=0$? Expand : $m^2-7mn+n^2-10m-10n+25=0$ Make the $-7mn$ disappear : $(m-\frac 72n)^2-\frac{45}4n^2-10m-10n+25=0$ Replace the remaining $m$ by the recently present $m-\frac 72n$ : $(m-\frac 72n)^2-10(m-\frac 72n)-\frac{45}4n^2-45n+25=0$ $(m-\frac 72n-5)^2-\frac{45}4n^2-45n=0$ Rest is obvious : $(m-\frac 72n-5)^2-\frac{45}4(n^2+4n)=0$ $(m-\frac 72n-5)^2-\frac{45}4(n+2)^2+45=0$ $(2m-7n-10)^2-45(n+2)^2+180=0$