You may have a hen in either of your flocks that is too closely related to your rooster. And you may be getting a genetic throwback of some sort.... I'm sorry I know that's not the "scientific" word to use but that's the best I can do.....
A few questions.
How long have you been hatching...