A square is divided into $16$ equal squares, obtaining the set of $25$ different vertices. What is the least number of vertices one must remove from this set, so that no $4$ points of the remaining set are the vertices of any square with sides parallel to the sides of the initial square?