No, it isn't that easy. And sadly many people do think it is, which is why it keeps popping up.
Some of the others may just have one muff/beard allele, which is what causes it. Think of it like BBS breeding -
Mb/mb (one allele) x Mb/mb (one allele) - 50% birds with one allele, 25% clean faced birds, 25% fully muffed and bearded birds.
So if you get rid of the clean faced ones, it's like getting rid of splash birds from a flock - You still have the blues. (in this case, birds that have muffs and beard but only one allele)
The key in the beginning is to look for the chicks with the biggest, fullest muffs and beard. Then, breed them and see what your rates are of small or large muffs, and if any clean faced birds pop up. Do this over several birds, and see who does and who doesn't throw clean faced birds. If a pairing throws clean faced birds, BOTH parents need to be culled. If a pairing throws 100% fully bearded, big muffed chicks - they're okay. If they throw a variety, one of the parents has only one allele. So from there, you pair her with a bird you know has only one allele. If they throw clean faced chicks, then you know that one too has only one allele, and needs to be culled.
Here's a photo of comparison. One of these has both alleles, one has none (clean faced) and one has only one. Can you tell the difference?
The one to keep is in the middle. Here's a better photo of her and her enormous muffs. She's now my favorite hen, and has never given me any clean faced birds nor birds with small muffs/beard.
As an adult, here's another good comparing note.
Her.
Her sister - only one allele. Notice the small muffs that press back against the neck. It may be hard to tell though, perhaps I need better photos.
Now to me, this judgment shouldn't be put on other colors though. I don't hear of clean faced birds coming from small muffs in other colors like I know in BBS.