With those gone is that too much air flow during winter season? We can get really low temperatures at times. I do not know exactly how low of temperatures chickens can handle.
I don't think there is a specific temperature where it goes from OK for the chicken to where it is in trouble. That's going to depend on other conditions. Your main risk from cold is frostbite. Technically frostbite can happen anytime the temperature is below freezing yet you and I can walk outside in temperatures well below freezing and be OK. Some of the factors involved are how cold is it, how long are we exposed to that temperature and how are we dressed. Wet and/or windy can get us in trouble fast even if the temperature isn't that cold..
Chickens wear a down coat. I've seen them walking in snow for hours without having problems with their feet. When they roost, they squat down and fluff their feathers to cover their feet, whether they are roosting on tree branches, 2x4's on end, or 2x4's flat. Wood is a good insulator. Metal or plastic are good conductors. If your roosts are wood they should be OK from that aspect, if they are plastic or metal you could see frost-bitten feet.
In general if you can keep the coop dry and them out of a direct breeze they are not in danger of frostbite. You can't keep the inside of the coop drier than the outside by passive means like good ventilation. I am not putting a dehumidifier out there. That should be good enough though. Wind chill is real so you don't want them in a direct breeze so roost placement is important. Other than being able to walk in there without bumping our head that is another reason to have a little height in the coop. If any openings are above their heads when they are on the roosts and breezes should pass harmlessly over their heads. That will stir up a little gentle turbulence in there which is good. That helps get the moisture from their breathing, their poop before it freezes, and any thawed water out of the coop.
If you have openings below where they are roosting you might get a breeze through the roosts. In the summer when it is hot that is a good thing but not in the winter. Your pop door will be open during the day when they are not roosting but you probably want to close it at night for this reason, let alone predators.
There is another important aspect to the strength of the breeze. Their down coats work by trapping air in tiny voids in the feathers. It is not the feather material that actually insulates them it is those voids filled with air. If a breeze ruffles the feathers it can release those air bubbles wrecking the insulating qualities. A gentle air movement is OK, a breeze strong enough to ruffle their feathers can quickly get them in trouble.
For what it is worth when I grew up in Tennessee I saw chickens sleeping in trees when the daytime temperature never got above zero Fahrenheit for four straight days and nights. These chickens were not on a bare branch overlooking a bluff squawking defiantly in the teeth of a blizzard. They were in a sheltered valley, they could move around the tree trunk to get out of any breeze if they needed to but the air was pretty much dead calm. You can't get any better ventilation. There were no frostbite issues.
I don't know if that board you have the holes drilled into is a 2x4, 2x8 or something else. I don't know the size of those holes but the amount of ventilation they allow will be a small fraction of the ventilation with that board removed. From a physics aspect it has to do with the friction of the air moving along the edges of those holes. The bigger the hole the less the relative friction, there are engineering formulas to calculate that. Instead of going to all that trouble to drill those holes I'd leave the entire board out and cover the opening with hardware cloth to keep predators out. If you are really worried about it being too big you can cover part of it with a smaller board, but remember those chickens sleeping in trees and the great ventilation they had.