We used to have a separate heat pump for a sun room on the back of the house, and the duct work for the sun room ran underneath it in an unheated crawl space. In the fall, condensation would form in the duct, and that in turn caused mildew to form. It was so bad one year that when we turned the heat pump back on, we had black stains forming on the vent cover on the wall. We ended up having to take the ducting down to scrub it out. Eventually, we re-ran the ducts, and we leave the fan running even if we are not heating or cooling.
I'd suggest scrubbing out your duct work, as you may have mildew. When spring and fall hit, and you're not running the heat or the AC, keep the fan running so that you don't have still air allowing condensation to form. The fan costs money for the electricity, but it's better than blowing mold or mildew into your house.