I find that even when the rooster is nice to his flock and older hens had pecked him into line, he can still be a terror to you. It's about having confidence in the coop and also not letting the big boy push you around. As I said in an earlier comment, I pick up my rooster and carry him around, talking to him and handling him so he starts understanding that I am not going to hurt him or the hens.
On another note, what are thoughts on removing spurs? In an experiment one day, I began filing down my older roosters spurs, and they popped off. There was just a little blood, and the cockerel was fine. I'd started filing down his spurs because I had introduced another younger rooster and did not want them to hurt one another. All was fine in the end, but I had no idea you could just pop off the spurs. It made me rest easier when young people would go collect eggs, not that there was ever an issue, my roosters are handled often and respect people.