Not in everyday maintenance do I wear a mask, but if I'm doing a big cleanup and it's dusty, I'll wear a disposable or a bandana. When I used DE in the coop I always wore a mask.
You and your chickens can get a lot of things, but that's no reason to freakout. Chickens can get so many illnesses (and humans, too) it's amazing we're not always sick or that we're not always freaking out about what we could get.
I think taking reasonable precautions and using a bit of common sense will get you a long way.