There's nothing really wrong with only feeding him a layer feed, however he shouldn't have been fed it from hatch that could be a reason he has stunted growth. Pellets are a complete diet and don't need anything added.
He can just be especially small, sometimes people's birds are smaller due to...