Bubblewrap over the windows (taped on with doublesided tape for the winter, same as you would use for that plastic film you can put on the insdie of your house's windows) will help a bit with heat loss and frosting-up. It is only R1 or a little better but compared to the bare windowglass it does help.
Fire-hazard-wise, remember that the biggest problem with heat lamps is their high wattage and you quite often really needn't use that big a bulb. Quite often, a regular ol' 100w bulb (or even smaller) hung above the roost can give the chickens a nice little pool of warmth without being an egregiously large fire risk (still, make sure the bulbs have a guard over them so chickens can't whop into 'em, and dust them off before turning on). If you need a larger area, mount several bulbs a coupla feet adjacent to each other. Several small is safer than one large!
If you want a bulb without light, either rig a barrier in front of it to cast 'shade' on the chickens and most of the coop (metal is good b/c it will still radiate heat in the chicknes' direction but whatever you do you need to be very thoughtful and cautious about things getting hot or falling down or etc), or try one of those ceramic infrared heat emitters they sell for reptiles (kept well-dusted, and in a ceramic heat-lamp fitting not a regular lightbubl socket)
If you want something electric but safer, those oil-filled radiators are pretty coop-safe at least for temporary use. Although, they are high wattage (most of the ones I've seen are like 700-1500w) and you would need to make REAL SURE your coop wiring, or whatever else you're plugging into, and the rest of its circuit in the house, can handle that load. Ahem, seriously.
If you want smaller electric appliances, some people buy those flat panel heaters for just a few chickens in a small coop (I have no personal experience of them and am skeptical, but some like them) or use WELL PROTECTED pet heating pads (ditto, and be careful of safety).
Large covered buckets of hot water brought from the house every evening, e.g. those buckets cat litter comes in that have the nifty lids that snap back on
, are a good no-electricity solution. You can pile shavings against the wall side and the top, so that only the 'chicken side' is exposed if it is a smallish coop with fewish chickens, and they can snuggle against it if they wish.
W/r/t running a car-heater accessory off a battery, it is of course a free country and if some people are happy doing it then that is fine, but I would point out that running something with a bare-wire type heat source is rather dicey in a dusty coop as is running something all night that is not really intended for constant use. I would not do it myself, in a million years.
Good luck, have fun,
Pat