What would you do??

Not sure if this is the right place to post this, but I wanted a little insight.

We travel to see my DH's family(brothers, sister and parents) twice a year usually for a two or three days(staying with my mother and father in law). They visit here about once a year.
About 6 months ago, DH's grandmother died and we traveled to the general area where his family lives for the funeral. We drove up the evening before the funeral. MIL and FIL were hanging out at SIL house. My DH has a brother who lives far away who came into town and was also hanging out at SIL's house that evening. My DH hadn't seen him in about 3 years and was excited to see him.
On the drive there DH called SIL's house several times to make plans to see his brother and everyone else. When we got into town and at our motel my DH called again to BIL about getting together and BIL said "No", so my DH told him it felt like maybe they didn't want to see us. BIL cussed him out and said that my SIL didn't want us at her house because she thought our kids were going to mess up the house or break her things and BIL said he wouldn't want us at his house either. We offered to meet somewhere else(bowling alley, which in the end we did). It really hurt my feelings and I don't feel welcome at their houses with my children.
I have never been to BIL house and we've been to SIL house twice before this(they always came to MIL's to visit) and both times the kids didn't mess up anything or break anything nor did she act like there was anything wrong. I understand that some people just don't like kids in there home, fine.
But....
Now her daughter is turning 1 and has invited the whole family for a Halloween party and I really don't feel comfortable going there now that I know she doesn't want us in her home. I feel like it is a courtesy invite and she will be relieved if we don't come. I told my DH there is no way I want to go.
My DH says we should go anyway(3 hour drive) because she extended the invitation and it would be the right thing to do and I needed to get over it.
Maybe I do need to get over it....what do you think?
I don't mind meeting or seeing them somewhere else, just don't feel comfortable at her house.
Am I being unreasonable??

 

I'm glad that I'm not the only one then...I was starting to wonder if I was just being petty.
Yeah, I don't like to be where I'm not wanted and I definitely don't feel wanted there.
Thanks for the responses!