Oyster shell is pretty cheap, so it may not be worth the effort of cleaning it.
Spreading it in the sun to dry for a while might get rid of the mildew, but I'm not sure.
If I really wanted to clean it well, I would put the oyster shell in a jar with water and some bleach (chlorine bleach)...