My opinion is yes, he's feeling his hormones, and possibly being protective, but may also just be showing off, in rooster speak saying, "Hey, look at me!" Admittedly, that can be just a few steps away from challenging you, but not necessarily.
I'm not big on getting a broom or a boot out after them either. I mean, I would in self defense, but it'd have to be an actual attack that I had no choice about and really I like to think I'm smarter than my roo and with some forethought, I can avoid that. Getting physical with him just encourages him IMO. I've done the picking up and carrying around thing, and it really does work on many roos, even most, but I also find this article SUPER helpful... it's not so much about what specific actions to take, but more how to view things and prevent trouble which is how I prefer to act. Oh, and also, I try not to expect affection from my roosters, though if they'll tolerate it I'm as nice to them as I am to the hens. Mostly I just let them be the flock guards they're meant to be, they go their way and I go mine, but anyway, see if this helps at all!
http://www.plamondon.com/faq_roosters.html
I like the attitude of this article, this guy makes sense to me, though there are other equally non-violent methods that can and do work. Keep in mind though, a lot of this depend on you, your temperament, your rooster, his temperament, and the set up of your coop, pens and or yard. What works perfectly on one rooster, for one person, may not work for you, or me, or Joe Schmo from Kokomo.