I offer very small quantities of grit right away, because my starter feed is a whole grain mash (whole grains milled into small enough pieces for the chicks to eat). The key is limited quantities, if you do offer it early, because they are still learning what is food and what is not. The biggest concern with offering very young chicks grit is that they will over eat it and become impacted. Gradually they will learn not to over-eat it, but at first one must take great care. Since I am raising them in a brooder instead of under a hen, I offer each chick just a few pieces of chick-sized grit individually so I can monitor how much they get.
If you are feeding a starter crumble then grit isn't necessary (since crumble just turns to mush when it mixes with liquids in their crop/GI tract) until they eat something that needs "chewing", so to speak. Most people do this when they start to offer treats.