For $a)$ we need to find the integral solutions to $$m(m-1)(m-2)=6n^2$$It's easy to see $m=pu^2,m-1=qv^2,m-2=rw^2$ for $p,q,r\in\{1,2,3\}$ and $p\neq q\neq r$. After consider the cases $3!=6$ cases. $4$ of them are easy to clear just modulo $3$ and $4$. The potential cases are $$2u^2=v^2+1=3w^2+2~\cdots~(\heartsuit)$$$$u^2=3v^2+1=2w^2+2~\cdots~(\bigstar)$$ For $(\heartsuit)$ we can consider the $\text{Pell-equations:}$ $v^2-2u^2=-1$ and $v^2-3w^2=1$. Easy computation give us $$(1+\sqrt{2})(3+2\sqrt{2})^k+(1-\sqrt{2})(3-2\sqrt{2})^k=(2+\sqrt{3})^m+(2-\sqrt{3})^m$$$$[(1+\frac{1}{\sqrt{2}})(3+2\sqrt{2})^k+(1-\frac{1}{\sqrt{2}})(3-2\sqrt{2})^k)]^2-4=\frac{1}{2}[(2+\sqrt{3})^m-(2-\sqrt{3})^m)]^2$$Don't be scared! Since this are general solutions must be integer just do variable changes and we can bash over $\mathbb Z$ and after conclude in the respective ring. Unfortunately $(\bigstar)$ doesn't work with some bash... maybe we need to consider more extensive ways!

For a) $m=50, n=140$ work. For b), use modulo 7 since we have sum of two squares. Obviously 7 doesn't divide $n^2+3^2$, so 7 doesn't divide the LHS too. Then check cases for the residues modulo 7 of the LHS.

a) The Diophantine equation $(1) \;\; {m \choose 3} = n^2$ is wellknown in the theory of figurate numbers. The solutions of equation (1) give us the numbers which are both square and tetrahedral (see https://mathworld.wolfram.com/TetrahedralNumber.html). The only solutions of equation (1) are $(m,n) = (3,1), \: (4,2), \: (50,140)$ (proven by Meyl in 1878). b) The Diophantine equation $(2) \;\; {m \choose 4} = n^2 + 9$ is equivalent to $24(n^2 + 9) = m(m - 1)(m - 2)(m - 3) = [(m(m - 3)][(m - 1)(m - 2)] = (m^2 - 3m)(m^2 - 3m + 2) = (m^2 - 3m + 1)^2 - 1$, i.e. $(3) \;\; (m^2 - 3m + 1)^2 = 24n^2 + 217$. Then combining the fact that $7 | 217$ with equation (3) we obtain $(4) \;\; (m^2 - 3m + 1)^2 \equiv 3n^2 \pmod{7}$. Congruence (4) implies $(3n^2/7) = (3/7) = -1$ since 3 is a quadratic nonresidue modulo 7. Conclusion: Equation (2) has no solution.

Can someone tell how to get the solution for (a)? thanks

Hi! Please tell me if my solution is incorrect but i think this problem is a bit too easy for the national round. The answer is no, there doesn't exist any n or m that satisfies the conditions. We want ${m \choose 4}-{m \choose 3}=9$ and ${m \choose 3}$ to be a square for a positive integer m. It is obvious that ${m \choose 4}\geq{m \choose 3}\implies m\geq7$ If ${m \choose 4}-{m \choose 3}=9$ then $\frac{m(m-1)(m-2)(m-3)}{24}-\frac{m(m-1)(m-2)}{6}=9$ and $m(m-1)(m-2)(m-7)=216.$ For $m>7$ the LHS is bigger than RHS and for $m=7$ the equation doesn't work.

Captain_Baran wrote: Hi! Please tell me if my solution is incorrect but i think this problem is a bit too easy for the national round. The answer is no, there doesn't exist any n or m that satisfies the conditions. We want ${m \choose 4}-{m \choose 3}=9$ and ${m \choose 3}$ to be a square for a positive integer m. It is obvious that ${m \choose 4}\geq{m \choose 3}\implies m\geq7$ If ${m \choose 4}-{m \choose 3}=9$ then $\frac{m(m-1)(m-2)(m-3)}{24}-\frac{m(m-1)(m-2)}{6}=9$ and $m(m-1)(m-2)(m-7)=216.$ For $m>7$ the LHS is bigger than RHS and for $m=7$ the equation doesn't work. Hi! I think there are two points of the problem here, not a system of equations.