Inexperienced people often times do not recognize the warning signs that a rooster gives off OR makes excuses for their behavior, as in it is something the human did to provoke that behavior.
It bothers me that he is puffing up at you. It might get better, but it might not. All chickens have a brain the size of a half a walnut. So I am just not a great believer in the theory that it is a person's fault for chicken behavior. First of all, very few people live in their coops. Second of all chickens forget a life long flock mate in a matter of days. Thirdly chickens do not have real advanced reasoning capabilities. 4th - Roosters are highly controlled by hormones, and the pecking order of the flock.
I would do something with the spurs. There are ways to remove them or reduce the sharpness. Do be proactive. Being flogged or spured is not nice. If you have small children under the age of 6, do not keep him. His puffing up at you, is a warning, and they tend to attack children first.
Often times, people say don't be scared, which I think is meant, pretend to be brave. My advice is, if you are scared, a lot of the times, there is something to be scared about. Don't be scared by removing the rooster from your flock. NOT ALL ROOSTERS WORK, and roosters have ruined the whole chicken hobby for a lot of people.
A rooster IMO should be the first one to see you as you approach the flock. Within days, he will position himself so that most of the hens are behind him away from you, but he himself should move away from you, about 5-6 feet. If he starts sneaking behind you - not a good sign. If he gives the stink eye, or is bold not a good sign. He needs to be slightly wary of you. Not terrified, but not bold. You need to always be aware of how your rooster is acting.
Roosters are a crap shoot, always have a plan B for roosters, always have it ready to go. A fish net on a 5 foot handle, can subdue a pretty upset rooster, just have it handy. You may never need it. Have a sharp knife handy. You may never need it, but please do not try and keep a rooster, blaming yourself for his aggression.
Mrs K