It does vary with the weather. I've been as early as 3-1/2 weeks and as late as 5. I've had broody hens totally wean their chicks in the summer at 3 weeks, so I have proof they can do OK that young.
I keep a fairly large brooder in the coop. I only keep one area heated and allow the rest to cool off as it will, with the far corners coming down to ambient, even in the winter. That way I don't have to worry about keeping the whole thing the right temperature. They will self-regulate. Plus they get acclimated to cooler temperatures. I think that helps them feather out properly.
This year, I had 5-1/2 week olds in the grow-out coop with the overnight low in the mid-20's Fahrenheit. The coop was well ventilated but no breezes were blowing directly on them. There were 14 of them, so they could help keep each other warm if they needed to.
In my opinion, there are a few variables involved. I don't think there is one right answer for all of us in all kinds of situations. But I do think they are tougher than many people think.