Nice Floor function Problem.
Part 01: $xy = 1$
\[ n - 1 = \lfloor x \lfloor ny \rfloor \rfloor > \lfloor x (ny - 1) \rfloor > x(ny - 1) - 1 = nxy - x - 1 \]So if $xy > 1$, we can take large $n$ such that $nxy - x > n $, in particular take $n$ such that $n > \frac{x}{xy - 1}$, and we get a contradiction.
Similarly,
\[ n - 1 = \lfloor x \lfloor ny \rfloor \rfloor \le \lfloor x ny \rfloor \le (xy) n\]If $xy < 1$, take $n$ such that $n > \frac{1}{1 - xy}$, and we are done.
Part 02: $y$ irrational
Hence, we conclude that $xy = 1$.
Now, suppose $y$ is a rational number. Then $x$ must be a rational number as well. Let $y = \frac{a}{b}$ and $x = \frac{b}{a}$, where $a,b \in \mathbb{N}$. Then we have for $n = b$, that $yn = a$, and hence
\[b = \lfloor b \rfloor = \lfloor xa \rfloor = \lfloor x \lfloor a \rfloor \rfloor = \lfloor x \lfloor ny \rfloor \rfloor = n - 1 = b - 1\]a contradiction
Part 03: $y > 1$
Notice that
\[ \lfloor x \lfloor yn \rfloor \rfloor = n - 1 \]\[ n - 1 \le \frac{1}{y} \lfloor yn \rfloor \rightarrow yn - y \le \lfloor yn \rfloor \rightarrow \{ yn \} \le y , \forall n \in \mathbb{N} \]Since $y$ is irrational, then the value of $\{ yn \}_{n \in \mathbb{N}}$ is dense over the real, and therefore we could take a value $n$ such that $\{ yn \} > y$ as $y < 1$, a contradiction.