Are the roosters what’s upsetting him ?
4 is more than you need to protect that many hens. Is there a lot of crowing and that’s what he’s bothered by? I have just a tiny family flock and 1 Roo and get worried about him sometimes if he gets rowdy 🤷🏻♀️
I’d call the town (or go to town hall) and ask all those questions myself to lay it to rest mentally (I’d also see about giving them his name on a report to establish this challenging touchy neighbor that you’d prefer im sure to have an amicable relationship with), so then you can let him know...