Personally, I don't feed my chicks any scraps while they are in the brooder. I do however start to throw clumps of grass, dandelions, clover, etc. into their box around a few weeks old. Dirt and roots attached. The chicks love it. Also, my kids love to find them little worms to feed them. I provide them with chick grit as well.
Tossing dirt in with sparing greens is fine provided that you are prepared to treat them for cocci. I wouldn't let the kids toss any worms from outside to them unless you are prepared to deal with the parasites that may be introduced by them.