Will the roosters go mean or aggressive? Maybe, but not because they are in pens. It is an individual personality thing.
I don't know what your set-up is. If you keep all the roosters in one pen by themselves, they will probably get along OK together. They will have pecking order issues, but so will hens kept together. It is possible there will be dominance fights, but not a guarantee of this. Many people do what I think you are talking about. You might do a search on bachelor pen to find some of those threads.
I don't do this, so I have no direct experience with this. I'm basing this on what I have read on here and just general experience. And of course, this depends on your set-up and how you want to manage them. There is no one right answer for all of us.
I'd keep all roosters together in one pen. Do not allow any rooster with the hens so you don't have to worry about waiting until the sperm from the old rooster has gone away. That can take over three weeks. Then when you want hatching eggs, put the rooster you want in a pen of no more than 10 hens. Start checking the eggs for fertility and when most of them are showing fertile, start collecting eggs for hatching. I'd expect it to take five to seven days for most to show fertile, but each flock is different. When that breeding program is over, put him back in the bachelor pen. I'd expect some pecking order issues there, but maybe no dominance fights. I'd also expect some pecking order issues with you putting thise hens back in the general population, but there is a reasonable chance none of these will be that big an issue. Of course, these could be very severe in either group, roosters or hens. There is just no way of knowing for sure in advance.
This is just one way. If your goals or set-up is different than I am assuming, then you could do something totally different.
Good luck!