its really hard to regulate the temp in a homemade incubator like this. Now, a thermostat is supposed to achieve a fixed temp, so in theory, it should cool the heat source if it goes above the desired temperature.
What is the temperature difference? I use a thermostat with my exotic invertebrates, and it is able to be set a set limit. So if I want it to be 37.5 degrees, I can give it a 0.5 diff, so that it wont go below 37, or above 38.
Do you have this setting?
And it is important to know where the probe is located, so that you know it is able to do its job.