It is common for ducks to not enjoy being touched. My ducks and I have had a great relationship for 6.5 years. They follow me around, come when I call, and gaze adoringly into my eyes. One of them even hops right into my lap. But they do not enjoy being touched. I respect that, and I don't push them. When I want to cuddle I go inside and find my cat.
I recommend that you don't force your ducks because they will associate you with a negative experience. Just give them as many positive experiences with you as possible, and don't be scary. They might decide touching is ok, but if not appreciate them for what they are.