No, lodged in the gizzard is a good thing. It is important to understand how the avian digestive system works.
Chickens have no teeth, just a beak and a tongue. The tongue moves food into the esophagus, passes to and gets collected in the crop whole. Saliva, mucous and the enzyme amylase have been added to moisten the food at this point but up to now little enzyme action has affected the food.
The crop in avians is similar in some respects to the adaptation of ruminants. What I mean is that for prey animals, they can gorge on food and go hide for a while. Ruminants fill the first compartment of their stomach where the contents ferment and at their leisure, cough up and chew their cud which is then re-swallowed and goes through the other 3 compartments on the way to the intestines.
After filling the crop, chickens can go dust bathe and rest in the shade for a while till the crop empties and they get hungry again.
From there it moves into the proventriculus, also known as true or glandular stomach. Hydrochloric acid, enzymes and hormones are added to the mix here but still the food is whole and not broken down as chewed food would be in mammals.
From there it moves into the gizzard, sometimes referred to as the mechanical stomach. It contains extremely strong muscles that contract pressing the food against the lining which is a thick, rather hard ridged membrane. This action grinds and mixes the food with acids and enzymes to aid digestion. However, this is not extremely effective without the teeth that are the relatively sharp insoluble stones lodged in this organ.
Not having these stones present is analogous to pulling all a person's teeth and expecting them to have good digestion and nutrient uptake.
That lodged grit stays until the acidic environment and constant grinding eventually wear it down to the point it can exit into the small intestine.
You'll get a better understanding by reading the following.
http://www.poultryhub.org/physiology/body-systems/digestive-system/
For the rooster, ideally he should not eat a high calcium diet. You have a couple choices. You can feed all birds a grower or all flock feed that only has about 1% calcium and provide oyster shell in a separate container for the hens to consume when they feel the need. That's what most people do.
However, the protein in those feeds is also too high for an ideal level for mature roosters. They are better off with a range from 13-15% protein. Another option, if you can find a lower protein feed, is to put that in a hanging feeder too high for the hens to reach and put the layer feed in a feeder with openings too small for the rooster to get his head in.
Still another option is to house the roosters separately and just give them conjugal visits a couple times a week.