We are down in Augusta, Ga and I have been having the same concerns with our first time flock. The mid 20s the last 2 nights had/has me concerned.
My 10 chickens survived last winter here in northern Minnesota with temps as low as -35F at night, without any heat in their coop. I built my coop with a couple sliding windows (North and South) for cross ventilation and I also have a couple vents at the top of the rafter peeks (East and West). If it gets really windy/stormy in the winter, I will close the north window.
My understanding is that chickens need a well vented coop, but not a drafty coop. A vented coop lets the warm moist air escape the coop and you do not build up that ammonia smell. A drafty coop is not good for the chickens if the draft comes up from underneath their roost, which cools down the chickens. So I have my roost on the opposite end of the windows, which allows cross ventilation but does not really create any drafts under their roost.
Also, for our cold climate, I built my roost out of a 2X4 laid flat. This allows the chickens to roost on the 2X4 with their feet entirely on the board and they lay their body on their feet - which keeps them warmer.
I don't know how much of my experience is relevant to winter in GA, but take what works for you.