4 females is really the MINIMUM, more is always better and often needed. But like you said, the problem is he's picked a favorite, not that he's hurting all of them and doesnt have enough.
Since he's not displaying aggression with the other 3 girls, you might be able to get away with pulling...