If I were your neighbor I would probably threaten you too. If your neighbor doesn't want your birds on their property it is your responsibility to keep them off, otherwise they have every right to deal with the problem how they see fit. Your animals are harassing their animals. If you don't keep your ducks out the neighbor could use deadly force to protect his ducks.
You could try to make them their own pond on your property. Though that doesn't guarantee they won't still try to go to the neighbor's pond. It's likely your only option is erecting a fence or enclosure for your ducks.
