The risk gets lower as they get larger and older. Risk being that they consume an imbalanced diet during a period of fast growth. For example, if your table scrap is low in protein (most veggies), they would be consuming a low overall percentage of protein and this would impact their development. That is not even to mention the vitamin and minerals and amino acids they need.
You can start as young as you want, but the amount needs to be scaled to their size and weight. Small chicks = extremely small amounts of table scraps.
Also, they need chick sized grit as soon as you start feeding anything other than commercial chick food. Note I said "grit", not oyster shell/calcium. Some people get the two confused. They are not the same.