No crime in peeking over a fence. I'd invite him over and introduce him to the birds, get his opinions on housing ideas for them, ask him whether they've bothered or inconvenienced him at all. If so, make an offer to move the birds further away from his property. Essentially do whatever you can to be a good neighbor.
hey if the ladder is still up in a way you can see it, I say snap a picture and if someone raises any ruckus you claim that he caught a glimpse of you sunbathing in your backyard and treat him as a creep he is so acting out to be
maybe that would be crossing the line
buuuut
"Officer he looked over our fence and peeped at our girls!"