If they have access to the ground when you put them outside, they are getting their own grit. If you have bedding or something that keeps them from getting to the ground, they may have trouble finding it.
I personally give mine grit on Day 3, as soon as they have figured out what their main food is. I don't give them a tremendous amount to where they can fill up on it, but some. I usually just take some dirt from the run or some very small gravel and sand from my driveway and give that to them until I let them out of the brooder.
A broody hen leads her chicks to grit as soon as they are off the nest. It is never too early to give them grit. If all they eat is the chick feed, they don't need it. But if they are eating creepy crawlies or grass and weeds, they do need it. In your case, they are probably already getting it for themselves.
