Little roosters are going to fight. That is their way of obtaining dominance. Dominance = breeding rights as far as the hens are concerned. I have 5 little OEGB Silver Duck Winged bantams that my hens hatched this past summer. Two are small, overall, in size than the other three but I've caught one of the smaller ones doing his best to beat up one of the bigger ones the other day. Nothing came of it other than a good laugh but it's all fun and games until somebody gets hurt and in this case there is no exception.
Separating them is going to only make things worse when you try to merge the one you've pulled back into the flock. Roosters will look at it as being a newcomer that must be conquered.
If you want to keep your boys, a rooster pen where they are not seeing the pullets or better yet, even hearing them will settle them down. Still they will squabble for dominance. Once the pecking order is obtained tho, things will settle down.
If you plan to rehome extra cockerels now might be the time to consider it. Otherwise you will be supervising the little devils day and night until the hormones settle down,.....in like three years if I'm any judge of my roosters's behavior.