I used to always have just a rooster pen. You will have less fighting by keeping all your roos together away from hens than if you kept them together with hen.
If all 4 of your roosters are raised together, it should not be too bad. Problems arise when you introduce a new roo to an already...