When we were looking to buy land we made sure to introduce ourselves to all of our prospective neighbors. I'm so glad we did!
The folks to the east of us are super nice, even mowed our front lawn when our lawn mower broke down one weekend. Their daughter is in high school and babysits for us from time to time. Our neighbors to the west of us have three boys close in age to our two sons. The dad is a big hunter (he killed a momma fox with two kits last year so they wouldn't come over and "git" my chickens - thoughtful, but I'd have rather lost a hen or two). Otherwise, he's nice as could be, his wife if quiet but "normal", and the mother-in-law who lives with them is always waving and smiling when she drives by. Finally, our neighbors to the north of us have become our friends. We go over to their place for cookouts, and their kids come over here after school sometimes when there's nobody home to meet them off the bus.
Had we moved next to the couple you're describing I would never feel at "home" - I would always feel a level of tension about everything they do and would never feel comfortable enjoying my property if I knew they were around or watching me.
I say, offer to sell them the land at a much higher price than you paid. If they decline you can sell to someone else. If they accept, then in a sense YOU will have "won"!!
Good luck!