Before you assume your rooster is just mean, check out his imagination. He may just be very creative and imaginative.
I have found that when I wear certain colors and styles of clothing, my rooster goes berserk.
He's totally gentle and tame when I wear loose, khaki and green pants, but when I wear forest brown or anything red, it drives him mad and he turns into a hen protector. I truly believes he imagines me to be a dog, fox, or wolverine in my red and dark brown clothing.
But if I wear baggy khaki or raggedy denim, he is calm and not a bit interested in attacking me.
Also, I think your shoes matter a lot. My roo does not like my faux crocodile/snakeskin boots. If I wear my beat-up black loafers, he is just fine with those.
It may be that I give off a more relaxed air in my comfy clothes. They are much easier to walk in, and I don't worry about snagging them, etc., unlike in my dressier clothes. It is only when I am in my dressy clothes that he has pecked.
Also, my dress pants are straight-legged, not baggy. I think my roo doesn't like seeing my skinny legs. They look more predator-like than my legs do hidden in baggy pants.
I know this sounds wild, but I have documented this behavior for some time now, and I think I am right.
Worth a try, anyway.