I wonder about this myself. Especially because I think a person could easily keep a few silkies, bantam cochins, or seramas in your house and no one would even know they're there. Aside from maybe large free-roaming roosters, I really don't think chickens are more disruptive, disease-spreading, or anything else really than dogs, cats, parrots, etc. They're pretty nice animals to be around, all-in-all...
I think a big part of it is that most people don't spend much time with chickens and therefore don't know what keeping them is actually like. They associate them with smelly farms and assume the chickens are going to be roaming all over pooping and crowing and chasing people. Sadly, I don't think the average person has had enough contact with chickens to know much about them other than stereotypes and examples of extremely bad poultry management.