You should be fine. --You could always get more hens, though! These chickens are a slippery slope.
Just pay attention to how the roo is treating the ladies. They're all a bit different. I've had to rehome a few roos, and have found them all very nice new families (that didn't eat them).