Trust me... it's no fun trying to fix oversized bantams. Doing that will probably just wreck the diversity all over again.Would it be possible to breed the Bantam Lav. males over the LF Lav. hen's, and then keep the resulting pullet's to breed back to the Bantam Lav. males to increase your genetic diversity? That way you wouldn't loose your color.

Keeping a healthy gene pool should solve problems, not cause more of them.
Breeders of bantam Buckeyes all over have been trying to reduce them by about ten ounces and it just isn't happening because of a tiny gene pool. Improving other traits that need improvement always causes size to be pushed back.
I'm just finding this out this year.