My best guess is it is a combination of how reactive he is to dominance challenges and your behavior. If so, you would get less reaction if you either clearly establish dominance over him or don't present a challenge.
I would go with the don't present a challenge route. So, don't squarely face him as you move toward him, don't talk to him as you approach, and, (especially) don't do both at the same time. There are more things like that - your stance, and maybe how you move your feet. I don't have a rooster, so haven't studied this much. This can get hard to do because people tend to do just the opposite of the things they should do as the rooster gets their attention or startles them or makes the suspicious or afraid.
If he isn't toooo reactive, I think you can be a part of the scenery rather than either a challenge or a subject. It will be harder to get there after he has been targeting you, though.