You must remove all colored squares. You cannot remove single squares, only a group of at least 2 of the same colored squares.