Never had this with chickens, but my ducks do this all the freaking time. They take their sweet time getting out of the eggs and then the gunk dries to them in the process. I just fill a shallow bowl with warm water and soak them in it for a few minutes (keep their head above water!), rubbing over their feathers with my thumb until the water is dirty or they are clean. If the former comes first, I dump and refill with warm water and repeat. If they start getting cold, it's into a damp towel with their head out, and under the heat lamp directly til they warm up. It's a pain, but if you don't get that stuff off fast, they lose way too much heat way too quickly even in the brooder, because their feathers can't fluff to keep them warm.
Best of luck!