Honestly, I would say your whole coop is too small... Recommended minimum coop space is 4 sq. feet of floor space per bird, and recommended minimum run space is 10 sq. feet per bird.
This could be part of the problem. That coop doesn't look big enough for 5 full-grown chickens. You don't want your chickens getting in the habit of sleeping in the nest boxes. That will result in dirty, poop covered eggs. Recommended roost space is 1 ft. per bird, recommended coop space is 4...
Could be the heat lamp is too warm for them. Could be a pecking order issue. Could be a space thing. As was asked in your post in another thread, how big is your coop in feet by feet?
Honestly, unless you have a BUNCH of them, I don't think they'd put off that much heat to make enough of a difference. Maybe.... I guess you won't know until you try it. Of course, when I think "coop", I think of my coops. The one they're in this winter is an 8x16' shed, with an 8' roof. It...
The concern I would have is the humidity that would be given off when the water is cooling down. If it's a dry cold, it's far better for them to deal with the temperatures, than humidity and cold.
If they are dry and out of the wind, they will be fine without supplemental heat. It's -15 right now where I live, and my chickens are doing great. It was -20 at 8 am. Do you have songbirds where you live? I do. They're out and about, eating from the feeders, drinking from the heated waterer...