I would set boundaries. He can come to the property if he is nice to the animals -- doesn't chase them etc. Tell them you are concerned about the animals, but also about the kid, because animals are unpredictable when they are scared.
But then my neighbors think I am weird for not letting...