You can pick your friends, but you can't pick your family OR your neighbors. I'm sorry that you're having to deal with someone with such a nasty attitude, but seeing as you are well within your rights to do what you're doing just as he is within his to do what he is doing, I would only make a few minor changes to your property; cameras.
Game cameras or some fancier security camera systems, it doesn't matter, but since you're alone on 20 acres and have had threats made against your livestock, I'd do it as soon as you can. We had our house broken into five years ago, not two months after we first moved in, and it was during the day. We live on 2/3 of an acre and have four neighbors within 200 feet of our home (albeit in the woods, so in the summertime you can't clearly see any of the homes). The cops came, they sympathized, but they couldn't do anything. There was nothing to trace and they told us that our area is far enough from a police barracks that unless there is a car patrolling nearby, it was at least 10-15 minutes for them to reach our home; plenty of time for someone to pack up and run. And with all of the trees and most people working during the day, it was an area that burglars would like to case if they could. They told us to get cameras, because if we could show them a face, they could put it behind bars. So we did...and they cover our coop/run, too.
Also, always report any/all threats to your person and property (including livestock). That way, if something does happen, there's a trail that they can start on.