I also understand the need of tact with the neighbors, as there is nothing worse than living next to an enemy! So I do think you need to be careful about how you handle it.
I would just say (with a big, friendly smile) "Sorry guys, this just isn't a good time! Why don't you give me a call next time so I can make sure its a good time?" (big, cheery smile)
Seriously, they may think that you guys are "like family" and don't mind at all. Or they may be horrible people. Either way, you don't want to live with contention!! (Especially if they ARE horrible!)
The day we moved into our new home (we were in a neighborhood that had JUST been built, so everyone was moving in the same week or two), my stepdad was hauling a trailer of all of our stuff behind his big truck. He must have hit the corner light/end of our neighbors car with the back of the trailer and didn't notice it. They came over later, after they noticed their back end messed up and saw the trailer also showing some damage. My stepfather denied it was him (he didn't feel anything), even though his trailer had a missing light on the side that was found at the neighbors car! It was obvious to everyone that he did it.
Instead of saying "I'm so sorry, I had no idea!" and handing over his insurance info, he denied it and had a big screaming fight with our neighbor! The people I am supposed to be living next to for the next however-many-years!!!
I went over later and explained that he has always been a grouch, and that he was currently undergoing treatment for cancer (in fact, he was attached to a chemotherapy drip that day!) and was even *more* unlike his usual self. I assured them that WE were not mad at them, and would have done exactly what they did in the situation (talk to the person responsible, call the police when they denied it, require insurance info, etc). We have lived here for 5 years now, and while we don't "hang out" together, we watch each other's animals when we are gone and are VERY friendly to each other!!
Be nice! Be firm! IT ALL COMES BACK TO YOU!!!