Slightly dirty is not a problem. Large clumps of mud may have compromised the bloom, even if it is dry now. If it is dry I don't see any benefit to getting it off, even if it is a clump. So I'd let them be.
Pay attention to smell. If you start smelling the rotten egg smell sniff them...