Sometimes I have to hold up the empty metal pie-plate as a shield to protect my feet and legs. He's never actually hurt me, but he has pecked at me and scratched me.
With the risk of infection I'd consider a scratch as trying to hurt you. If you happen to be bent over when he pecks your eyes are at risk. Children especially are at risk as their eyes are closer to him and they probably aren't that good at protecting themselves.
The 3 poor hens each have bare backs from his attentions. They don't seem to be very fond of him although I'm not sure if I would ever be able to tell.
Some people keep 3 hens with a rooster and do not have barebacked or over-mating issues. Some do. That may not always be the rooster's fault, some hens have brittle feathers that can easily break even if he does everything correctly. But if he is not there that should not be a problem.
I haven't kicked him out because I do feel responsible for him. I bought all four from the local Tractor Supply store 1 year ago.
That is a personal issue. I don't have a rooster and individual hens. I have a flock that consists of chickens that are sometimes replaced. I try to solve for the peace of the flock, not any one individual. Why make the rest of the flock suffer for the benefit on one individual? Your opinion may vary.
What are your goals as far as having that rooster? The only reason you need one is if you want fertile eggs. Anything else is personal preference. Nothing wrong with that, I have a few preferences myself. But those are a want, not a reason. If your goals mean you need a rooster or if your preferences are to have one for whatever reason you can try to keep him, especially if you feel responsible for him.
If you let him loose the two roosters will almost certainly fight. That may be a fight to the death, one may become seriously wounded, or they may reach an accommodation where they get along, probably by always staying far apart. You can try that and see how it turns out or you can build a separate coop/run to keep him in by himself. He won't be very happy about that at first but may settle in. The girls will probably miss him at first but you may be amazed at how quickly they adjust to him being gone.
My goals are different from yours. With me having been around human aggressive roosters before he would have been gone by now. My goals are not what are important, yours are. But I think you will find that experienced people on here will not put up with a human aggressive rooster. It takes most of the fun out of it and you and yours are at risk.