I dont want to just go against everyone but these are my thoughts.
I grew up with chickens, and my dad normally had one rooster with about 4-6 hens at a given time.
I started my flock out with one rooster and 3 hens, then went to 5 hens.
Everything was great, save some bare spots on the hens back, but he calmed down after 6-8 months (or their skin just got tougher). The rooster was having a good time i suppose.
Anyway, i like having the rooster in my small flock. I feel like he keeps watch over my girls. You may be jumping the gun a little by getting rid of him.
My suggestion is to grab a glass of sweet tea, and a chair, and watch them for 30 minutes. See if he is constantly mounting, or chasing them. Then make you decision. I feel like a rooster has a lot to offer.
In my opinion, noting is as cute as seeing a broody hen hatch some eggs and raise them. The grandkids will love it if you can get them to become broody.