Answer: Yes! Your coop isn't apparently warm enough at night. Here is what we do.
The coop has two separate parts, mainly because it's big. If you have a small coop, it's not necessary. I keep an eye on the weather, and pretty much when it gets below the twentys, I have a heat lamp that hangs about two feet above the roost, that I turn on. Works great! Haven't had any frostbite at all. There is a wood door that works as the divider, and there is a hole just large enough for the chickens to go through, to keep the heat in there. I don't know what you keep our Chickens in, but even if they are in cages, just put the cages in a place where you can turn a light on. I understand a lot of coops don't have electricity in them, so in that case, you could either run a cord, or do big project and insulate. Our chicken coop isn't insulated.