Agree with this.Really, you are the only one who can answer this question. How are your hens? Are they stressed? Being harassed by the roosters? Do they get to eat, drink and dust bathe without being chased around by several roosters wanting to breed them? Are the roosters fighting? How old are the "roosters"? Have the reached breeding age yet? If they haven't that might be why they're all getting along. When they start breeding, that could change. I'm glad you're getting rid of the one that's rough on the hens. That will help reduce stress on them. Hens don't lay well when stressed. Ridgerunner - a very respected member of the BYC community - suggests keeping as few roosters as you need to meet your goals. It doesn't guarantee you will have no problems, but it does lessen the chances of problems.
I have two cockerels (roosters younger than a year old) and 20-some hens and pullets. The boys get along just fine - they've had a few skirmishes but have worked things out. I will be separating them and the girls this winter. (Not enough space in one coop for them to be one big happy family when cabin fever hits) Once that happens, they will not be out at the same time again. I'm afraid they will actually to harm-causing battle with one another when they're more mature.
I've had one rooster with two hens, and everyone has been happy, productive, and glossy feathered, no bare backs or resistance to mating.
I've had a free range rooster with his choice of two dozen hens stalk and terrorize two specific hens.
And lots inbetween.
Mostly, I agree with keeping as few roosters as possible. Folks tend to think just because they got these Oops birds, they're obligated to them for live. Not so, feel free to sell/butcher/give away ANY bird that does not fit in with your flock. Look at each bird and decide if he has a place in your flock, not just "I have to keep him cause I wound up with him".
No matter how many birds you have, you need to constantly monitor and evaluate your flock to see if what you have is working.
