A slightly more theoretical approach (of course boiling down to similar algebraic manipulations), showing why such things occur. Since we are told $\cos \alpha \not \in \mathbb{Q}$ and $\cos 2\alpha \in \mathbb{Q}$, it means that, taking $\cos 2\alpha = a \in \mathbb{Q}$, it follows $x = \cos \alpha$ is a root of the polynomial $f(X) = 2X^2 - 1 - a \in \mathbb{Q}[X]$, but not of a polynomial of degree $1$ from $\mathbb{Q}[X]$, thus $f(X)$ is its minimal polynomial. But we are also told $\cos 3\alpha = b \in \mathbb{Q}$, so $x = \cos \alpha$ is also a root of the polynomial $g(X) = 4X^2 - 3X - b \in \mathbb{Q}[X]$. This means we need have $f(X) \mid g(X)$, leading to $a=1/2$, $b=0$. Now everything is clear - we have $\cos 4\alpha = -1/2$, but $|\cos\alpha| = |\cos 5\alpha| = \sqrt{3}/2 \not \in \mathbb{Q}$, a simple example is $\alpha = \pi/6$.