I raise goats, but I've never eaten goat. What I have eaten is deer, and a lot of it....but I've yet to eat a 'wethered' whitetail. It's all good, provided it's handled right.
Some folks will tell you that bucks and does have a different taste, but I've come to the conclusion that about 99% of the table experience depends on what happens in the relatively short span of time between when the animal hits the ground and when the meat hits the freezer. If you kill a big buck, leave him in the back of the truck for a half a day with his innards puffing up, cut a gut dressing him, get hair all over everything skinning him, fail to leave the meat to age properly, cut through a bunch of bone processing him, leave the big nasty green glands in the shoulders and hams -- or WORSE -- grind them up with the meat...or grind in a bunch of tallow...well, yeah.
Unfortunately, those are just the sorts of things a lot of people do. Then they give the meat to an unsuspecting someone who wants to try deer, and put them off to it FOREVER.
Sorry.. I didn't mean to get off on a venison rant, but you get what I'm saying. Whether it's got nads, or doesn't, probably won't make nearly as much difference as you can make by processing and handling the meat properly.