It does take some work, but you can change the dynamic. I agree with Katharinad. In our flock, the ducks love peas as treats, and that can entice them. When we first moved them outside, I fed them peas in their little house. I tried to associate good things with being there.
While they were in temporary fencing as I built their day pen, they would sometimes find a "hole" in the fence and pop out. To get them back in, I would walk around behind them and hold my arms out and walk them back toward the pen or the house. It took me a few minutes to learn how to best guide them, and sometimes they would end up in a corner and I would need to pick them up and place them on the other side of the fence, but it was a low-anxiety adventure.
Now the temporary fence is set up around garden areas, and yes, sometimes they find my mistakes and pop out. But I just walk behind them, raise my arms out to my sides, and walk them back into the pen.
I have worked for months with these ducks, so that they will eat out of my hand and not be afraid of me. It takes time but they seem happy to have a routine, to know where to go when I give certain signals. They even let me know when I am not doing it right!
Perhaps if you begin earlier in the day, you can reduce the feeling of being rushed as you retrain the ducks about the evening routine.
May it work out well and soon.