A lot of it is due to space. Simply put, to really improve breeds you need numbers. It's a bit of conventional wisdom that for every 10 you hatch you'll get 1 worth using (although some say less than that), but if we take that as a minimum, and say we want to have at least four keepers (based on having one pair and a backup for each bird). That means at an absolute minimum you need to hatch at least 40 birds and raise them to an age where you can feel confident about making selections. That's an absolute minimum, yet I see "breeders" talking about hatching much less. And that's per breed and per variety. Want to do two breeds? Now you're at absolute minimum of 80. Six breeds? Now your minimum is 240. When dealing with those low of numbers progress will be slow, but steady, but you can see how fast it adds up.
Yes time and knowledge are factors too. It takes time to really know the ins and outs of a breed, and for a newbie especially to develop their eye and all that. But one thing that doesn't care about your knowledge or experience is amount of space, and economics of feed bills. I think that is truly the inhibitor barring most for being able to work successfully with multiple breeds or varieties.