Is less insulation ok for winter? As in, closing some of it without adding additional?
Am also considering just covering the back window with plywood. Any harm in that?
If I cover those but make a few small windows where the sides meet the roof, I wonder if that's enough ventilation? (because I have no space from what I can see to add a bigger window that won't blow cold air on them)
I think you meant "less ventilation" in the first post?
The answer is no, ventilation is just as critical in the winter as in the summer because you still have to move warm, moist, ammonia-laden air out and bring fresh air in. Chickens, like all birds, have very delicate respiratory systems.
You want your airflow to look like this:
Or like this:
If your ventilation is adequate the temperature and humidity in the coop will be just about the same as it is outdoors -- but completely free of wind.
You can test for drafts by standing next to the roost on a windy day holding a light ribbon to see if it blows around.
A gentle waft of air is OK. What you want to block is wind strong enough to ruffle the birds' feathers.

If I use a furnace filter....what do you cover that with to prevent them from eating it and it from getting wet?
I haven't used it because I don't need it, but the installations I've seen are on the outside of the hardware cloth and away from anything a chicken could sit on to reach the window. I *think* it doesn't matter if it gets wet because the roof overhangs and/or awnings that keep rain out of the windows normally protect it and a little bit of blown snow isn't an issue.