My main breed is Cubalaya in various colors. My wife has an Olive Egger/Easter Egger project going right now. I hatch 200-400 chicks a year depending how quickly I want a project to progress and the amount of space I have.
It's okay to use a slightly smaller rooster, just be sure to breed him to your biggest hens and then from that breeding you keep a large cockerel to continue. I really like breeding projects so this gets my interest.
It is a normal phenomenon. I have sold hundreds of roosters and it happens almost every time that I sell some that I get the same feeling that you expressed. You're not alone.