I would find some scrap lumber..doesn't really matter what it looks like...and build a very tall, very ugly fence on that side of your garden~stain it something pretty on your side. That way you won't have to look at him looking at you and he won't have to "see" all that you are doing on your land. I'd also invest in some electric fencing between the property and get a nice, big GP or Anatolian LGD to guard your sheep and your roosters.
Your land, your stuff, your life. It doesn't sound like this is a person who cares about "nice" and you really can't deal with those kind except ignore them as best you can and keep the moments of direct exposure to a minimum.