This is a good idea, as compared to selecting for chick down color. If you cull at birth for chick down color, you have no idea what you are throwing away as far as size of bird at maturity, time to reach maturity, egg production, health of birds, etc.Edit to add another idea: you could ignore color for now, and stabilize the other traits you want. Then see what colors you have at that point. T
With this type of project, there is no way to avoid raising a lot of birds to maturity and then selecting the best ones for your purposes. You can, however, be keeping notes on which chick down colors grow to be which adult colors, and use that information when it comes time to weigh which birds you will continue to breed from.
Sounds like a fun project. Good luck!