From what I have found, it greatly depends on the particular roo- and his attitude towards life, people and hens...
I have kept 10 or 12 roos together in a pen by themselves before, with no problem. of course there are exceptions.
However, I've had roos penned seperately, who want to fight through the fences.
I have kept several roos together before in the same pen, with my general population of egg layers, with no problems as well. Seems like I've had 5 roos and 25 hens together at certain times of the year. These were free range birds though that would seperate off into their own little clans in the day time, and really the only time spent together would be at night when roosting. I would do this practice during the winter time, when I didn't have intentions to hatch eggs, and my birds really weren't laying up to par anyways. This would get them out of the breeder pens, and free ranging again. It also helped keep the henhouse warmer with more birds in it.
Now, if you have on particular bird, that's a problem, solitary confinement might be his only option. you could use something like the plastic barrel method used by show fanciers. It works real well to keep a large number of bird penned by them self, in a small ammount of space.
Now what you have to ask yourself, is do you really NEED to keep all these roosters? Unless you are mating and breeding birds for certain criteria to sell as seedstock-- I see no reason to have extra roos running around. Think of it this way-- roos are slouchers, they eat, crap, and pester everything.. and in return give nothing. I've seen one roo successfully cover 10-15- and maybe 20 hens just fine. If you are just raising chickens for eggs and meat, why not take advantage of crossbreeding or heterosis? Pick out your best roo, or two, and leave them with the hens full time...