A lot depends on your goals. Even if you are not selling hatching eggs or chicks, what do you want your flock to look like and what traits do you want them to have? That's more than feather color or pattern. That could be size, egg production, egg color, comb type, whether or not they go broody, and many other things. That could affect which hens or roosters you allow to breed and which eggs you hatch or which you decide to keep in your flock and which you get rid of.
I suggest you be ruthless if you see defects in the chicks you hatch or even the parent flock. Crossbeak and crooked toes immediately come to mind but there can be many other traits you do not want in your flock. I eliminate birds that have traits I don't want. I do not breed hens that go barebacked as that can be an inherited trait. I had a pullet that laid from the roost for a month when she first started laying. I don't mind if they do that for a week or so as it can take a while for them to get that straightened out, but a month was too long. There was something wrong with her.
Most people can go 4 or 5 generations before genetic diversity gets so bad that you need to bring in a new rooster to reset it. If recessive traits show up with most of them you might need to act sooner or even start over, but the long term problems of losing genetic diversity is the fertility of your flock can drop, production (egg laying) can drop, or they may become susceptible to certain diseases. So be selective in which ones you allow to breed. Breed your best and eat or get rid of the rest.