From reading all the responses it appears we have become a bunch of germaphobes. I guess that's alright, whatever floats your boat.
I try not to get all phobic about things and just wash my hands before handling our food and after handling something nasty/or public. Rarely get sick with viral or bacterial infections.
I have seen more than a few parents that were all freaky about keeping their kids hands, toys and other kid paraphernalia clean. Those kids grew up to have the worst allergies. The immune system has a job to do and it needs to be introduced to germs to do its job.
Plus, as someone already pointed out, few germs can survive for long on toilet seats, flush handles, etc.
Having said all that, I think a good handwashing after using the restroom is enough.