Put up a fence. Even if your kids like to visit with the neighbor's other animals, this can and likely will lead to problems:
1. Spread of disease from neighbor's animals to yours.
2. With your hens around, the neighbor's roosters might become agressive towards your children.
3. Your neighbor's roosters, if they have long spurs, can seriously tear up your hens.
4. Your children should only be playing with your neighbor's animals when your neighbor is there to supervise. Children get hurt, animals get hurt. You love your kids, presumably he places value on his livestock.
5. Combine 1&4. If your hens contract an illness or disease, they can spread it to the neighbor's chickens. YOU will be liable if your flock strays to his--and if you free range them, they will.
I'm sure it's lots of fun for your kids to play with the neighbors livestock, but this is where you have to be the grownup. We live in a world where people who can make trouble will. Your neighbor sounds like someone who is only interested in their own convenience. A neighbor who will take advantage of your limited knowledge of livestock (getting a hen knocked up?) is a neighbor who will take advantage of you in other ways.
Put up a fence and continue to educate yourself.