We've done this for years and for the same reason. We just leave the dairy out of the regular recipes and have found there is little difference in the outcome.
Like mashed potatoes? We just put in some olive oil instead of milk and they come out creamy and tastey. Omelets that usually have...