Correct but I'm trying to avoid the frozen water issue.
Then heat the water, not the whole coop.
You can buy a heated base to sit the waterer on, or put a light bulb or heat lamp directly over the waterer, or various other solutions depending on the style of waterer.
Depending on your weather, and the temperature that naturally happens inside your coop, you might just need to bring fresh water first thing in the morning. Chickens don't drink while they're sleeping in the dark, so it doesn't matter if the water freezes then.