The answer is $5$ achieved by $1,3,4,5,7$. To prove this we may wlog the smallest and largest numbers be $1$ and $A$. Now $\frac{A+1}{2}$ is also in the set. If a number $x$ is in the upper half of $[1,A]$ we must have $\frac{x+1}{2}$ in the set and for the other case $\frac{x+A}{2}$ we will call this number $x$'x friend. Starting from some $x$ the sequence of friends must be a cycle since it is finite hence at some point the friend of the friend of $x_0$ is already in the sequence. It can be easily seen that this must be $x_0$ since the sequence is monotone on each half of $[1,A]$ from $\frac{y-\frac{A+\frac{1+y}{2}}{2}}{\frac{A+\frac{1+y}{2}}{2}-\frac{A+\frac{1+\frac{A+\frac{1+y}{2}}{2}}{2}}{2}}=4$ but as $\frac{y -\frac{2A+1}{3}}{\frac{A+\frac{1+y}{2}}{2} - \frac{2A+1}{3}}=4$ we conclude that this only happens if $x = \frac{2A+1}{3}$ and we are done.