You don't have to be concerned until -20F. If you get prolonged periods of time where it's that cold and you don't have winter hardy breeds you may need to do some things.
Temperature is not your problem, wind/drafts and moisture are your potential problems. Moisture leads to frostbite and...