The way I see it, you have two choices.  
1.  Build the fence.  If any of them fly over into your yard, catch them up and trim their wings.  Then they won't be able to get over...at least until their flight feathers grow back, then trim them again!
2.  Catch them up and take them to the pound.  Either your neighbor will have to pay to go bail them out (something which he will tire of eventually), or he'll leave them there and they'll be gone.  If he gets more, and they come into your yard, take them to the pound too.  If it gets to the point where it hurts his bank account, he'll be more willing to keep the birds on his property.