My neighbors have a shop/garage right next to the property line that has an attic area the just stands wide open (picture a barn loft with a big opening in the side) there is a whole family of raccoons that life up there. I see them coming and going all the time. The neighbors do not care but it sure worries me. Even when I am gone during the daytime I worry that somehow they will want a snack and come over and get one of my girls so I know how you feel..I would worry too.