If you don't wash them, then you don't have to refrigerate them (this is what many in europe do). But once washed, bacteria can get in, so they need to be kept cold to prevent spoiling.
I keep my nestboxes clean and try to make sure the eggs never get dirty. If an egg is a little dirty I'll wash it, if it is very dirty then it is fed to the dogs or chickens. If one of the girls poops in the nest and soils all the eggs, then they all get fed to the pets. I've had a few eggs where the poop stains are inside the egg even after washing (the egg was basically sitting in poop), so anything with more than a smear is not for human consumption.