So good to hear that you were gracious to the kids and conscious of setting a good example.
That said, there are quite a few issues here that could be addressed, beginning with "Do not go onto somebody else's property unannounced and make yourself at home with their stuff."
Years ago, I had a neighbor whose grandson started coming around when I'd be outside feeding the chickens. (I had extensive pens then, breeding several varieties.) I knew enough about the kid to know that he'd had a hard life, and that's how he ended up being raised by his grandmother, so I thought his interest in chickens was a good idea.
But eventually, he started coming around any time he wanted. One day I made a remark like, "Just don't be coming over and opening the pens, etc., when there's nobody home here." I explained about legal liability in case something happened, etc. The kid seemed to take it OK, but a few minutes later I got a call from his grandmother screeching and cussing.
Eventually, she had half our block involved in a "feud." I have no way of knowing whether it would have been any better for me to tell the kid to get lost the first time he came over, but I don't see how it could have turned out worse.