Thanks chicksalot and spookyevilone for the links to the heat grabber. There was a similar article in Mother Earth News a couple years ago, about a larger design that hung on the outside of the house. These look smaller and more doable... I have just a few home improvement projects on the list!
My chicken coop is built into the south-west corner of my barn and typically right around the winter solstice (sun least intense), on a sunny day with outside temp of 15°F, inside will be just above freezing, maybe 35-40°F.
Patandchickens is right about thermal mass, and I do need to add some. The thing about thermal mass is that it works both directions. If you have stone cold -- stones -- or even frozen water in jugs, it will tend to keep the coop cool. Once it is warmed up, then yes it will help keep the coop warmer overnight. Patandchickens is also right that having the coop stay cooler in the daytime can be helpful once the temps start getting a little higher in the spring (or if you live in a southern clime, even now). But, here in Vermont, I will take every BTU of thermal gain I can get on a sunny January day!
One solution to prevent thermal mass-cooling would be to have 2 sets of jugs, and to figure out a way to warm up the second set without simply bringing inside your house (which would warm the jugs OK, but at the expense of your home heating bill).
Start by filling both sets up on a sunny day.
Put one set inside the coop, and the other in a sunny location where they will heat up as much as possible. Before the sun goes down, bring the second set inside your house.
Overnight, the ones inside the coop will dissipate heat. In the morning, they may be frozen or very cold. Take them out of the coop, and replace them with the ones that have been in your house overnight.
Put the frozen jugs in a sunny place where they will warm up, hopefully to room temperature, and then bring them into your house before the sun goes down.
The key would be to have a place for the second set of jugs to warm up, that is not inside the chicken coop. Inside a cold frame would work great -- too bad my cold frame is under 2.5' of snow!