Yes I think is the answer.
Pull all the roosters now. Roosters tend to out grow hens, and you should be beginning to see a size difference by now. Generally speaking, if you separate these boys to the outbuilding, they should continue to get along, and grow out. I did this myself last year...