I've heard the quality of hay affects the taste of the milk. A friend of mine only feeds the goats that are milking really high quality hay. She feeds high quality, sweet smelling, (usually alfalfa) hay to the ones in milk and then the lower quality to the rest.
Since your hay is the only thing...