I'm the same way. Even if I go somewhere all I do is wonder if the animals are ok, is the garden ok, etc.Vacations almost cause me more stress than just staying home. I worry about my animals so much!!
And I chose "no" even though I technically do travel to see old friends, but the way we handle it is hubby and I take trips separately. So someone stays at home to mind the animals and garden. The last time we traveled together was some 3 or 4 years ago, for 5 days, and the neighbors came over to pick up eggs and lock up the chickens at night. Before that, even without chickens, we hadn't traveled together since attending a funeral about 7 years prior.