Ugh, yes. I was homeschooled and if we were ever out in public on a school day we would often have strangers ask us if it was a holiday. When we answered that we were homeschooled, my mom would get quizzed on her expertise, as far as whether she was an actual teacher or not, or people would start asking us kids random math questions. RUDE!

I plan on homeschooling my kids and I will be pretty aggressive if anyone insinuates that my kids are getting a less than stellar education.
As you mentioned, I love that homeschooling is very individualized. People learn in different ways and also have different interests, and there's so much flexibility offered with homeschooling.