You should be fine, I have bucks, does, and wethers and none are aggressive, and all can live together comfortably.
The only thing that "might" happen, if the wether was castrated late he may let you know when the does come in (even some castrated early), it won't be a problem (and can even be a help if you plan to breed), he'll just act alittle bucky w/ the does. I have 3 wethers and one of them likes to flirt w/ the does.
And because I love posting pics, here is my flirty boy Grim,
My other two wethers are nubians,
This is Riply,
and Rip Swimming,
Couldn't find a pic of Onyx on here, he's solid black.
I love my wethers, I have another one reserved if the does cooperate and have a boy.