If you have birds, predators are always going to be a problem. It's your yard and your property as far as other humans are concerned, but those walls are going to be ignored by the feral cats, dogs, hawks, coyotes, foxes, raccoons or owls who notice you have food wandering around. If you catch a predator killing your birds, you're justified in defending your fowl, but the neighbor's cats aren't killing your ducks, they're just mingling.
I'm kind of sympathizing with the neighbor. What are you expecting them to do? Tell the cat not to do that? Their only option is to keep the cat indoors, and who is going to change their lives and their pets' lives that much for a stranger? Once a cat is an outdoor cat, it does not easily become an indoor cat. Our cats are indoor/outdoor cats, and if we keep them from going out, they drive us insane. After three days of being trapped in the house, our cats would become so irritating that I'd shoot them myself. (Kidding.)
If the cat gets in your yard again, your best bet is to spray it with water. If you do that enough times, the cat will decide it doesn't want to be there. (How about a motion-detecting sprinkler system...hmmm...is that possible?)
But cats don't necessarily have to be the problem you're making it out to be. Our cats are very interested in our chickens, so a couple of times we put them in the coop with the hens. The cats quickly learned that full-grown hens are not to be messed with. Unless the cats are feral, or unusually aggressive, they probably won't hurt your birds.