I think a lot of people get caught off guard by bucks. Their first year rut isn't intense, and they barely smell. Then come the NEXT year, whew, they can knock over a vulture with their smell. Their 'fragrance' is both a mixture of urine that they like to get all over their face and legs, and scent glands that are located behind the horns (or where the horns would be, if polled or disbudded). Sometimes disbudding can burn away those scent glands, if someone aims for them, reducing buck smell.
Also, it kinda depends on the goat. I've met some goats in full rut, that are mature adults (2 years old and greater), who don't smell too bad. Then others, they peel paint with their aroma. I happen to own one of those ultra stanky boys. He is SUCH a sweet, mild mannered lad, but wow, he has a powerful smell to him.
Also, just curious, when you said you banded them right away, did you band them within a few days of birth? A lot of breeders like to wait at least a month, preferably two, until making them wethers. It gives a chance for their urethra to grow and mature, reducing the chance of a urinary blockage later in life. But also, someone I know who owns goats says it is easier to make sure both testicles palpable and below the band because they are somewhat bigger. If you band and 'lose' a testicle above the band, it retreats into the abdominal cavity, and you still have a potentially fertile animal. Less fertile than an animal with two intact danglies, but still able to impregnate.