Whenever I came back from a two-week vacation with my parents, my ducks seemed to remember me but were not as calm as before. They weren't as distrustful as they would have been towards complete stranger, but they definetely acted not as tame as usual and it took me a few days until they were again. But when it happened for the first time I already had my ducks for two or three months so they were already used to me as the harmless food dispenser who likes to stand around and watch.
Since you got your ducklings just yesterday, they might not have bonded with you yet. But if they have, they are imprinted on you and see you as their mother. In that case you cannot let them alone for they will call for you until they almost keel over. Maybe you should make a test. leave them alone and see what happens. If they start calling ( one could almost call it crying) and cannot be simply calmed down by another person, they are imprinted on you and should stay with you.
Can't you take them with you, anyhow? Ducklings can easily be kept indoors and don't make a mess like grown ups.