Your roosters sound like my guy. He's full of bravado as long as the fence is between us, but when I step through the gate he keeps his distance. Strange part is, he'll walk around peaceably if I'm in the 4'x12' run with him, or walk down the ramp within mere inches of me in the mornings when I open the coop door. Go figure!Day or night my birds (both chickens and turkeys) know the difference between me and a coyote or dog or possum or coon. A reassuring word brings contented clucks.
I haven't had a mean rooster around since I was a kid, and he "accidentally" got in the way of the tractor mower deck. My roos retreat to the far side of the coop or yard when I enter. The tom turkeys don't, they walk nearby, but if I turn and face them they retreat.