The problem I see with this, is when you pull a rooster, and then put him back. A lot of commotion and fighting for all of them. Some rooster do live together peacefully, but a lot do not. It is one of those deals, it might work, but there is a very good chance that it won't work.
And all of your roosters would be old at the same time. Often times, as a rooster gets older, he covers less hens, or even if he covers them, they may not be fertilized.
Maybe I am missing the why you want 4 roosters? Most chickens are not that long lived, and you are going to be feeding 3 roosters all of that time for really no reason? And the older roosters get, the less viable they breed. In your situation, all of your roosters would be old all at the same time.
I think a better strategy would be to pick either the Buff or Lavender rooster- breed and raise chicks, and two years later, raise up underneath your flock a replacement of the other breed. You could either get some straight run chicks of the other breed, or some hatching eggs, bound to be a rooster in there. Raise him up in your flock, and then remove the old boy. You get young vigor, you tend to get a much nicer rooster when raised in a multi-generational flock and you don't have the mess of trying to keep a bunch of roosters, with crowing contests, aggression, extra coop and a feed bill.
No real need for 4 roosters to cover 12 hens. And you can have way more than 6 pure bred chicks. You could just breed a single hen, collect her eggs for several days, until you get as many as you want, and then hatch that clutch. That is what professional breeders do, they pick their BEST hen, and BEST rooster, and then only hatch those eggs. Not every hen's eggs in the flock are hatched, most are not.
Mrs K