Check his crop in the morning to see if it is empty. If it is not empty, could be sour crop/impacted crop problems I have read.
Also, some vets will do a fecal float test for worms for a small fee (false negatives are possible) if you are afraid to worm him. Skinny and lethargic could mean...