The extra calcium in layer feed is potentially damaging to the liver of non laying (Roos, hens during molt) chickens, and birds in development. They can't excrete it, and it builds up.
But because its one of those time and dosage things, how much damage it may do, and what effect it may have on them over the course of their lives is quite variable. Your hatchlings getting into layer feed a couple times isn't going to kill them.
Feeding layer exclusively, from hatching on up, will likely result in shorter lifespans for your birds, particularly your roosters. How much shorter will depend in part on how much they eat, and on how frequently your hens lay eggs.
Its complicated.
but as you've noted, its hard to maintain separate feed programs in a mixed age flock, and its a hassle to keep multiple feeds on hand. Much easier for the typical backyard flock owner to go for an "all flock/flock raiser" feed and free choice oyster shell on the side for birds wanting extra calcium.