Honestly, the more I think about it. I'm going to say that you should not go. Your going to make yourself sick worrying about it. Your going to be watching him constantly to make sure that he doesn't so much as glance in her direction. You will end up getting mad at him and probably end up in an argument over HER and SHE is not worth one second of your time.
But, my Mom taught me one thing. You can not break up a happy home. If you are truly happy and in Love with your partner, then there will be no feelings for anyone else, no matter how attractive that someone else is.
My husband's Ex has tried for the last 10 years to split us up. Were happy, and nothing she has done or said has worked. She has made up alot of lies about me too just to get him to leave me. He just laughs at her. I use to get mad, but not anymore. She isn't worth it. It's best to steer clear of people like that.
I'm sure that your DH is not upset with you, he just isn't sure how to go about telling them that he will not be there.
Hope everything works out for the best.