That is one of the rudest things-inviting yourself into someone's back yard. Not only that, if a grandchild get hurt on your property, they can SUE your homeowner's insurance, then of course, your homeowner's will drop you like a hot potato.
My back yard is my "Private Idaho". If I accept a visit, its at the front door. My neighbor sees me in my backyard and every time he does, he waves and wants to chat, however, he only speaks Spanish, even tho he's been here for years, and I was muddling thru my 8th grade Spanish, but now refuse to be an enabler. He can see me on the patio right up to the patio door. It's awful. This year we are planting fast growing hedges, and putting privacy panels on one side of the patio. I work 12 hour nights, and spend alot of time on the patio in my pajamas. I really don't want to chat every time he sees me. On top of that, he always "needs" something. Now, I just say I don't understand what he wants. He sure knows the english word for what he wants!
I would absolutely put a stop to this "visiting". Next thing is that the grandchildren will be visiting without grandma, maybe when you're not home. Put a lock on the gate. It's your house, you don't owe anyone an explanation for locking up your property. Karen