Chickens can read human body language. Human females are often more meek, so often roosters will pick up on that and attack them first. I carry myself in a dominant manner around my birds. It's important to stand square, walk with confidence, make birds move away from you, and never give your back to a rooster who may be eyeballing you.
Men often already move in a dominant manner. A real confident rooster will go after the men as well eventually.
Next time don't handle the males, and make sure they move away from you, never towards. Setting up good manners in male chicks will carry through to adulthood.