Your neighbors are jerks, but they sound like TOTLLY average people to me, that's just the way the world is now, it's horrible but true... I honestly think it's because of kids growing up in the city in small cramped places instead of on farms. I think seeing animals eat, give substinance (Milk, meat, eggs) die, give birth etc really helps children to learn to nurture and sympathize, plus it keeps them from growing up to be anti social, selfish, lazy adults...
As per some of the replies to the OP... People are certainly self centered, but people are also WAY too sensitive! Who cares who owns the tree for crying out loud! if you're (un)fortunate enough to buy a place w/ a tree you should be obligated to go rake up all the leaves? thats ludacris! The tree could just as easily be planted on public property, what then? Leaves, debris and garbage will end up finding it's way to your property, people need to deal with it! Thats life, it sucks, but if you get wound up about every little thing you'll be a nervous wreck! 
If you don't like it you better put up a huge fence... I just think it's rediculous to expect everyone with a tree to pick up leaves that blow around, it's an act of nature... I can see if your trash blows over or your dog happens to wonder to the neighbors and poops (Our dog did that on rare occassions when we were getting out of the car, distracted by kids he'd tromp to neighbors and mark/deposit.) We always cleaned things like that up and always helped neighbors, especially the elderly, but never because we felt obligated to clean up after our tree! hahahahaha