If my hens to down the hill to my neighbor's property and their dog gets them, I have no grounds to complain. My fence is for my chickens' protection and to keep roaming dogs from just running across the property. As was stated, it doenst have to be an expensive fence. My 4 and 5 ft welded wire fences work just fine to keep my girls in (I have standards, not banties, except for one).
Even though you think you've been a good neighbor and I'm sure you have, your neighbor may not think that you should allow your birds to come onto his place, even when you have been generous with your eggs, etc. It's just best to provide a barrier for the safety of your birds and to avoid unpleasant neighbor relations. I'd not tell them you are putting up a fence since it is your right on your own property, aside from any covenants preventing it. Just do it and if they question it, the explanation is that you are keeping your girls home so they dont bother the neighbor.(never mind that it also keeps that neighbor off your property, LOL)