My situation is different than yours. I have a big brooder in the coop. I only heat one end and let the rest cool off as it will. While the heated end was nice and cozy, the far end had frost in it a few mornings this past February when I had chicks in it. The chicks were just a few days old. They stayed by the heat.
The point I’m trying to make is that my chicks get acclimated. They go play in the cooler end, not so much when it is below freezing, but they do play in pretty cold temperatures. They get used to cooler temperatures and it helps them feather out faster.
I have moved chicks to my unheated grow-out coop at 5 weeks of age when the overnight lows were in the mid 40’s. I’ve had chicks go through the mid 20’s in that unheated coop before they were six weeks old.
I don’t know how old yours are right now, but if you put a heat source in one corner of your coop, they should be fine even if they are just a couple of weeks old. Mine go into something like that straight from the incubator.