It's not money down the drain, it's money to give you comfort and privacy. It's probably cheaper than moving to somewhere more remote. I know that this must be very difficult for you, change is never easy, but it is to some extent inevitable. It helps to try to be at peace with those things that you can't change, be thoughtful in the changes you can make, and be wise in figuring out the borders of your influence.
Again, I wish you peace. In your circumstances, building a fence is probably the wise choice, for the comfort of you and your new neighbors.
It saddens me the level of vitriol in this thread against people, potential neighbors, and the assumptions everyone is making about people that they don't even know. There are certainly evil people out there, but there are also a lot of wonderful people. Even some of the people who don't seem so wonderful have often become that way through trials that you may not even be able to imagine.
It seems fashionable these days to ascribe all sorts of horrible motives to people we disagree with. I wish we could disagree with equanimity and respect. It's not always easy.
I don't know that my thoughts are helpful to anyone but myself at this point, so I'll leave the thread now.