Roosters being aggressive to people has nothing to do with how many hens they have and *almost* everything to do with their natural temperament.
I would not keep an aggressive rooster. If you can only have 4 chickens get 4 nice hens and get rid of the male.