Find all integers $n > 1$ for which it is possible to fill the cells of an $n \times n$ grid with the integers from $1$ to $n^2$, without repetition, such that the average of the $n$ numbers in each row and each column is an integer.