Ahh, that's too bad. I'm lucky that I've never had neighbors like that, but I know plenty of people who do. I can understand why you're nervous!
Maybe when you are explaining what happened, tell them you saw exactly what your chickens did. Ask them to go out in their yard WITH you so you can specifically point to any damages your birds are responsible for. That way your neighbors won't have the luxury of perusing the whole yard when you're not there and claiming extra stuff after the fact. If they do point to other things you will be able to challenge questionable things ("sorry, no, that damage looks a lot older than this..." etc.). Might be tense and uncomfortable, but better than giving them free reign to blame you for whatever they want.
If they are the vindictive types them I'd also try to take some of the power out of their hands. You don't want them blaming you for every little thing in the future, just because of this one incident. Do something to show you're on top of fixing the problem. Put up a fence (even a temporary one), or keep the chickens in a run for a while. That way when rabbits get in their garden they won't be able to automatically blame you. You can point to the fixes and confidently say, "No, sorry, not my chickens this time!"
Best of luck!