They still need air. Don't cover with plastic.
There are lots of things you can cover the windows with that will still allow air infiltration.
By windows, I figure that means openings. All mine have hardware cloth rather than glass but they are still windows. Coop windows should always allow...
In a dire smoky situation, the only choice is to get them out of that air. Is your coop big enough that you can keep them in a few days and filter the air coming in?