A month ago we had 3 roosters (though at the time, only 1 was confirmed). We are now down to 1. As the two younger ones, who were raised together, began to mature, and once they were introduced to a new coop and some new hens, problems began to arise. We gave away the older one, because he was picking on both the younger two (which weren't confirmed at that point), and then once we knew the other two were both roos, we understood why things were so stressful. We took the meanest of the two, who's aggression really bloomed once the first roo was removed, and gave him away, leaving us with one, maybe 5 month old roo, to 9 hens (another 4 at about 13 weeks getting ready to be introduced). With the removal of those 2 roos, the flock has been awesome together. Literally, in 8 days, we went from total chaos to peace and tranquility.
My suggestion, is that you react to the circumstances. If you start to have a problem, find the main aggressor and remove him. If your heart is really set on keeping him, keep him sequestered for a bit, and put him back with the flock. Could be the removal reset his place in the pecking order. But I think chances are you'll be better off giving him away or serving him for dinner
Do this with your roos until you have a happy flock. I felt a little bad about giving away our gold sebright. He was a really cool little roo. But when I walk out there now and see how happy everyone is, the guilt goes away.