I think you definitely have room to work with him! Ours used to act exactly how you're describing toward my fiance, but after they spent time around each other, the rooster saw he wasn't a threat. He rarely shows aggression toward me, and when he does, it's only because I'm doing something to make one of his ladies uncomfortable, so he's trying to protect her, which we praise since that is his job. If you're able to catch him right before he "attacks" you, a swift kick to the chest (but not hard enough to hurt him!) has worked wonders for us. It helps show him that you aren't afraid of him, and that you're Alpha. I gave him a few love taps, and he's been great ever since! I think there's hope for you!