You'll feel worse if you wait around for something bad to inevitably happen with 4 roosters in your relatively small flock.
The best time to move them is while they're all calm.
I have 2 bachelor pads with 2 roosters (and 1 with 3 (was 4)).
One of the two-pads has D'Uccle's which are pretty easy-going birds to begin with. The other is a wyandotte and an araucana x.
4 out of 5 of my breeding flocks have more than 1 rooster.