Think about the difference between the guys and you. They likely walk around that rooster like he doesn't matter, walk right through him probably and don't give him a second's thought. The rooster knows the difference between calm assertive nature and someone who is more timid, more submissive. Standing up to him as he attacks is not going to work, because you've let him take the initiative in the whole scenario...that leaves you on the defensive and makes you a lesser creature in his eyes. He knows you are no threat to him and he is going to press his advantage.
Stop and think a moment. Why are you scared of him? Afraid he will hurt you with his talons, obviously. Mostly they don't even do much with those when they attack a human, mostly it's a lot of wing flapping and movement, but if you are wearing pants, he can't do much damage at all. He's a 2 ft. tall animal, if he's lucky, and big. Can't really do much in the way of hurt right now as he hasn't even gotten his spurs.
Now that we've established he is small, you are large, he only has toenails and a beak, you have the advantage of height, the ability to use tools and you have higher thought processes...meaning you have all the advantages on your side, you just don't stop to think about them when you are being attacked. Why not think about them BEFORE you are attacked? This works...it really works if you work it.
Go on the offensive instead of defensive...use his own tactics against him. Get yourself a long and limber cane/switch and prepare where you will teach him his first lesson in what it means to be a woman on this Earth. We don't get back, we get even.
Take away the feed and feed by meals for a few days so that all chickens have to come to the feeding area at feeding time. Then take your cane and not let him in to the feed by tapping his head, his back, his tail....get him on the run away from you...in fact, use your body and the cane to get him running and keep him running. Don't worry about the hens...at first they'll freak out but soon they realize they are not the target and will go back to eating.
Then...let him to the feeder, let him relax for a few seconds. Then take your cane and give him a swift smack to the fluffy butt, hard enough to startle him and make him jump, then keep tapping him on the back and head as he runs away.
Take your time, enjoy the process. Trust me...this will be fun! This won't take much of your time, but it will be time well spent. Drive him completely out of the coop or away from the flock and don't let him back in.....face him at all times and advance, never back up. If he advances towards you, give him a thwack with your cane and keep walking towards him, make him RUN.
The object of the lesson is to teach him you are the predator in this situation, you have the power to control his movements because he is most obviously the prey and smaller..and weaker. The final goal is to have him looking over his shoulder at all times to see where you are and him moving away from you as you get near.
The next day...do it again! I'm betting you'll see quicker response times on his part, more alertness to when you walk into the coop or run...only he won't be moving towards you, but away.
After that, keep your cane leaning near the coop and tune him up a little every now and again, just for funsies. More important....don't be timid when you walk into that flock. You won't make them mad at you or fear you too much if you walk with confident, wide strides, walking through them and not around them. Walk towards him intentionally every day to make sure he retreats when you do so...that's the desired response. He should walk a wide berth around any humans because we rule the roost, not him. He's food.
If you follow this and follow through on your own behavior, I'm betting you'll have yourself a good bird there. He already recognizes authority, which is why he doesn't bother your menfolk...you just need to let him know you have your own authority and it's called woman.