If there is a food source then there is always a possibility you will keep finding mice. You already know that. Ducks are very messy eaters. You already know that too.
 
Here's what I do.
 
I changed from a hanging feeder to a 1 gal bucket. They make a tiny mess but a whole lot less than before because they can scoop up a mouthful easier than with the hanging feeder. All (well, most anyway) spills fall right back into the bucket. On nice days I put the bucket in their run and on rainy days I put the bucket in their house so it doesn't get wet. Then when I lock them up at night I bring the bucket into the house (I don't give them food or water in their house at night).
 
You can also put a tray under the hanging feeder to catch spills and remove it every evening to clean up any access to free rodent food. When there is no more access to food the rodents will move elsewhere.
 
Just my suggestions, other members will have good ideas too.