I'm not sure what breeds you have tried, there are several different breeds that can come in Buff and RR could mean a few different things. It sounds like they are dual purpose though, not broilers.
Dual purpose birds are not going to grow nearly as fast as the broilers (Cornish X, Rangers, etc). At four months two pounds dressed weight sounds a little low for cockerels but not that bad. It would probably be good for pullets. Some of that depends on how and what you are feeding them.
As chickens get older the meat gains texture and flavor. The Cornish X you buy at the store was probably butchered somewhere around 6 to 8 weeks old. You can fry, grill, or cook that any way you wish. But by 4 months the meat has changed, especially with cockerels. You need to change how you cook it. There are different ways to cook older birds and get a really nice meal, they generally involve lower temperatures, longer cooking times, and more moisture.
Aging is a tested method of tenderizing meat. I don't age mine but a lot of people do. I put mine in the freezer pretty soon after I butcher, but I don't butcher a lot at a time. I put the frozen bird in the fridge on Sunday so it can thaw and cook by Thursday, maybe that counts as aging. There are different ways to approach these things.
I normally butcher mine between 5 and 6 months of age, there is more meat and it has even more flavor and texture. I cut it into serving pieces and coat it with herbs, then cook it in a tightly covered baking dish at 250 degrees F for 3-1/2 hours. I've served this to friends and taken some to gatherings. People rave over it. There are plenty of other ways to cook these slightly older birds, this is just the way I settled on.
When i was a kid Mom would tell me she wanted a chicken so I'd deliver one to her freshly plucked and gutted. She cooked it immediately, no time for rigor to set in. Depending on the bird she might fry it or make Chicken 'n Dumplings.
So what are you doing wrong? Maybe you have the wrong expectations as to how big they should be, especially if you are raising hatchery dual purpose birds. Those are not bred to put on a lot of meat early. But if it is tough, you are not cooking it right. Don't feel bad about that at all. We all have to start somewhere and you are trying and asking. You'll do better next time.