A broody hen can raise chicks outside even when it is below freezing. The chicks go out to eat, drink and play a bit but run back under her to warm up when they need to. Many people are amazed at how much time a young chick can spend out in cold weather before it has to warm up. But they do need a place to go warm up.
I put my chicks in my brooder in the coop straight from the incubator even when it is below freezing. My brooder is 3' x 6 feet and can get frost in a far corner some mornings but I keep one end toasty. As long as they have a place to go to warm up when they need it they can be outside.
I've had chicks a bit younger than yours (5-1/2 weeks) go through nights in the mid 20's Fahrenheit. I had about 20 chicks so they could huddle up to keep each other warm and the coop they were in had great wind protection down where they were and good ventilation up high.
There is nothing wrong with providing heat or taking them in at night as long as you don't overheat them. It will not hurt them and will probably make you feel a lot better. There is value in you feeling better.
Many people take them out for afternoons like you plan. The chicks usually love it. It gives you a chance to observe them and see how they react. That should help boost your confidence much more than what some stranger over the internet like me says. Have fun with them.