I would have to agree that ducks are skittish by nature, in general. Especially compared to geese, from what I've seen so far.
My ducklings did used to like to be held and petted, once caught. We handled and cuddled each one every day. One of them would lie along our hand and dangle his head off and fall asleep (Kupo - he's the one who attacks us daily now!).
When they became teenagers, they had no use for us anymore. Now, they will get right underfoot hoping for food, or else trying to herd us away from the direction we want to walk in (the drakes do this). If I am holding a feed cup, there are 3 that don't mind terribly being touched, but they don't enjoy it either. The 4th would usually rather starve than be touched. She'll stand near us, but doesn't want hands to touch her body.
None of them actually likes to be petted, and they are very indignant if they are picked up. We don't touch the shy one just because she hates it so much, and she trusts us more if we don't ever attempt it.
Hehe, the baby gooses on the other hand always come up to us, rub their beaks on us, nibble our hair, want to be petted and cuddled, etc. (As long as it doesn't interfere with their eating ... but even then they want to be near us and "talk" to us as they stuff their little faces.)
Good luck with your baby ducks!
trish