She remained with the flock up until they hatched then I put her and babies in a new coop.
I let mine hatch with the flock and don't separate them at all, the hen raises the chicks with the flock from Day 1. I've never had an issue with integration, though once the hen weans them the chicks are on their own with the flock. The chicks manage that by avoiding the adults until the chicks mature enough to force their way into the pecking order. I have the room where that isn't a problem. The hen has never had an issue reentering the flock when she weans her chicks.
Lots of people do something similar to what Aart does, keep the hen and chicks separated a few days before letting them out with the flock. Plenty do something similar to what you did, isolate the hen and chicks for a relatively long time. For whatever reasons we each do things our own way.
The way I see it you have three entities to deal with. You have your regular flock, a single hen that has weaned her chicks and is on her own as far as being integrated into that regular flock, and a group of 5 week old chicks totally on their own as far as integrating into the regular flock. The hen has weaned them, she will no longer protect them. Don't think of integrating the hen and chicks as a unit, they are not a unit.
For the chicks, treat them as you would if they were brooder-raised. People do that all the time on here. I suspect the "flock" could see them in that separate coop as they were free ranging. The chicks should be in the habit of sleeping in that separate coop. The way I'd approach the chicks is let them free range with the flock during the day and let them continue to sleep in that separate coop until they prove they can get along with the rest. That's very close to what I do. I'm in no hurry, I let them roam in the same space for a month before I move them to the main coop to sleep. I don't know how big your main coop is or what it looks like inside, that may be a bottleneck for you.
I'd do the same for that single hen but I suspect she'll want to return to the main coop to sleep and rejoin the main flock during the day. If that hen is being attacked by a specific hen then isolate the aggressor for a week or so, then let her out to rejoin the others. Sometimes that changes their behaviors. I've had success and failures trying something like that.
I’ve also seen some people separate their coop with wire and make a pop hole in the wire that just the hen and the babies can fit through.
Think about that one a minute. That is a common technique when integrating small chicks, but how do you cut a hole the broody hen can fit through that the rest of the hens her size can't? That's a great technique if the chicks are small enough but doesn't work with adults.
I agree with you on free ranging. That's a great way of getting them used to each other while giving the chicks room to get away and avoid the adults. I do that and it sure makes my life easier.