Thanks for the replies!
Just checked the eggs again this morning. They don’t look too dirty, I think I got most of it out. I replaced the bedding material. Attaching a picture. I checked the hen and she seems clean as well.
I have 3 broody chickens all taking turns siting on the eggs but...