I really would not worry about it. There is nothing you can do to turn back the clock.
I just looked up a site to see what the difference is, grower is 14-16, and layer 15-18 percent protein. Starter is 20-22. If your feed is similar ratios, then you've actually been giving them higher protein, not less.
As for the calcium, if there are no signs of effects of having had too much, I wouldn't worry, but if you are concerned, I'd get the contents list of the feed that I'd been giving to see exactly what its proportions are. It may be that it isn't that high in calcium to worry. There is also a big difference between excess that causes organ damage, and excess to requirements that can be excreted normally. I strongly suspect that manufacturers aren't loading up any of their feed with an amount that would actively harm anybody - people will often be feeding the "wrong" thing without the intensive research that those of us who join forums do. Also, if your feed is organic, they might not have been loading it up with added "anything".
If you were trying to decide what to do before you do it, then researching all of this would be time well spent. As the "horse has already bolted" it really won't achieve anything other than to possibly alleviate your worry and it could just make you more worried/confused/upset.
If it makes you feel any better, the boy at the chicken farm said he though that grower feed was a lark for manufacturers to make you buy 3 lots of food and not just 2....
It's quite common for sizes (of every animal, including people) to vary and it's quite common for there to be a "runt" (if that's what your small one even is, maybe he's just small).
I'd just determine an appropriate food for 14 week old birds and feed them that. Carry on. They look healthy and happy, that's probably exactly what they are.