By three weeks I usually have them outside, and they run around in all weather, but they can go back under the lamp when they choose/need to.
If it's 80ish in your brooder without the lamp then yes, I'd let them do without for a few hours, unless they start piling on each other. They may have...