Very nice problem ! The answer is $\boxed{\frac{2}{3}}$ It is attained for $n=3,a=b=1$ Now let's see why it is maximum. We have $2$ cases: Case 1: $(a,n)=d>1$ Then choose $k=\frac{n}{d}\cdot j$ where $1\le j \le d-1$ Obviously $\{\frac{ak}{n}\}=0$ and we want $\{\frac{bj}{d} \}\le \frac{2}{3}$.We choose $j$ such that $bj\equiv (b,d) \pmod d$ and so we want $\frac{(b,d)}{d}\le \frac{2}{3}$.If this is not true then $(b,d)=d$ and so $(a,n)\mid (b,n)$.But then choose $k=\frac{n}{d}$ and the fractions are both $0$ so we are done. $\blacksquare$ Case 2:$(ab,n)=1$ Letting $k\to ka^{-1}$ we can suppose WLOG $a=1$.So we want to prove the existence of a $k$ such that $$\frac{k}{n}+\{\frac{kb}{n} \}\le \frac{2}{3}$$Suppose not.Then for $k=1$ we get $b+1>\frac{2n}{3}$ Let $b=n-t$ where $2\le t<\frac{n}{3}+1$ and so we want $$\frac{k}{n}+\{\frac{-tk}{n} \}\le \frac{2}{3}$$But $\{\frac{-tk}{n} \}=1-\{\frac{tk}{n} \}$ and so the final conclusion is that we want $$\frac{k}{n}+\frac{1}{3}\le \{\frac{tk}{n}\}$$Let $k$ maximum such that $tk\le n-1$. we have $\{\frac{tk}{n}\}=\frac{tk}{n}\ge \frac{n-t}{n}$ and so if $k+t\le \frac{2n}{3}$ we are done,else$ k+t>\frac{2n}{3}$ so $k>\frac{n}{3}-1$.But since $k$ maximum such that $tk\le n-1$ then $t\in \{2,3\}$ so $b\in \{n-2,n-3\}$.Those cases are easy to check so we are finally done !! $\blacksquare$