I suppose this is one of the debates that will never end.
I think you are exactly right. The only legitimate study I've seen on roosters eating Layer was a Canadian study from several years ago and I do not have a link. They were fed a diet where the only thing they ate was a feed with the calcium equivalent of Layer. The conclusion was if the only thing the roosters eat is layer then some can develop internal problems.
There are some concepts included in that that some people find hard to understand. If Layer is not the only thing they eat then the amount of calcium they eat in a day can be reduced, maybe substantially. It's not how much calcium is in one bite, it's how many total grams of calcium they eat in a day, and even that is sort of averaged over several days. If they forage or you feed low-calcium treats then the total amount of calcium they eat in a day could be substantially less. The same thing is true with growing chicks. If a broody keeps them out most of the day foraging instead if them only eating Layer feed the total amount of calcium they eat in a day may not be that much, even if the only chicken feed available is Layer.
The other part of this is that "some" develop problems. Some doesn't mean each and every one without fail 100% of the time. That's a concept some people seem to have a problem with.
The way they did that study was not just to see which ones fell over dead. Of course they tracked that and mortality was slightly higher or some were not as thrifty as the control group. As I remember a few more had fertility issues. But most did not have any symptoms you could observe. They cut some of them open to look at the internal organs. On "some" of those roosters internal organs were damaged. That "might" make them more susceptible to problems further down the road, maybe years down the road, even if it was not enough to immediately cause problems.
An example of this is that I have liver damage after having hepatitis over 3 decades ago. There are certain medicines my doctors will not prescribe for me. With some medicines they watch liver panels pretty closely. There are certain things I should not consume, especially in quantity. It's not that I'm going to immediately die if I do but the potential for life changing events are there. You cannot tell that I have liver damage just by looking at me, but it is there. It's something I want to manage.
As I have growing chicks or other non-laying chickens (a rooster, broody hens, molting hens) with the flock all the time I feed a low calcium feed to all of them with oyster shell on the side for the hens that need it for shells. With the way mine forage and the kitchen scraps, stuff from the garden and the orchard they get I'd probably be OK feeding Layer but I never do. To me the worry free option is to not feed Layer.