In terms of cattle, in our case castrating isn't done to improve tenderness. Steers gain weight better than bulls as they are not fixated with fighting and looking for cows to breed. They are also a little easier to handle.
I raise goats, but I've never eaten goat. What I have eaten is deer, and a lot of it....but I've yet to eat a 'wethered' whitetail. It's all good, provided it's handled right.
Some folks will tell you that bucks and does have a different taste, but I've come to the conclusion that about 99% of the table experience depends on what happens in the relatively short span of time between when the animal hits the ground and when the meat hits the freezer. If you kill a big buck, leave him in the back of the truck for a half a day with his innards puffing up, cut a gut dressing him, get hair all over everything skinning him, fail to leave the meat to age properly, cut through a bunch of bone processing him, leave the big nasty green glands in the shoulders and hams -- or WORSE -- grind them up with the meat...or grind in a bunch of tallow...well, yeah.
Unfortunately, those are just the sorts of things a lot of people do. Then they give the meat to an unsuspecting someone who wants to try deer, and put them off to it FOREVER.
Sorry.. I didn't mean to get off on a venison rant, but you get what I'm saying. Whether it's got nads, or doesn't, probably won't make nearly as much difference as you can make by processing and handling the meat properly.
I've eaten plenty of goat, but we always castrate the boys so I don't know if leaving them intact changes the flavor. I'd definitely suggest butchering him before late summer, at least, unless you LIKE the smell of buck. ;D It sounds like your goat's in the perfect window to butcher right now, anyway. Go for it! At worst, you'll have to find a family with a raw-fed dog.