Yes, you should be worried. If he isn't doing well he needs to be separated from the others. They may be bullying him away from food and water and he is too weak to do anything about it. If he keeps cheeping, he is probably too cold.
A newly hatched chick needs to be kept away from drafts and cold. He should have a heat lamp over half of his brooder / whatever so that the temp. at floor level is 95 degrees.. Every week you can lower it by 5 degrees until he doesn't need extra heat (when he is fully feathered out). Always leave half of his area without the heat so if he is too hot he can move away. Make sure family pets and youngsters don't accidentally hurt him.
Give him water in a swallow dish and put some clean marbles or pebbles in it, so he won't accidentally fall in and drown. That can happen even to the healthiest chicks. You could put his feed on the brooder floor on a paper plate so it's easier for him to reach it. Maybe what you feed the chickens is too coarse for him and he needs it grinded a bit to be smaller bits.
If you hope to save him - give him a little stuffed animal to cuddle with. Maybe a small mirror so he thinks he has another chick with him. Tempt him to eat some hard boiled finely mashed egg yolk , moistened with a little plain yogurt.
Get him some Polyvisol infant drops (at
Walmart, etc. pharmacy). He can have 2 drops 3 times a day, this will give him a boost. You can put the drops just inside his beak, so if he doesn't want to swallow he can let it run out and NOT choke on it. If he is eating anything put the drops on a piece of food. He may like little pieces of grape - which would give him a little liquid as well. Put a bit of sugar in his water - you want him to drink so he doesn't dehydrate. It's really better to tempt him to eat than force feed him. The struggle will really stress him and he needs peace and quiet to recuperate.
I hope it isn't too late to save this precious little chick. When his needs are met, he should start feeling better and eventually start growing.