If it is over four batches, yes something is wrong. You say some survive. About how many die and how many survive? What are their symptoms when they get weak? How long do they last after they get weak?
Photos of your set-up, the chicks, and the poop could help. What are you feeding them and is the brooder dry? How much time between the different batches?
It could be a disease. Coccidiosis comes to mind but it doesn't really sound like the way it develops. Still you can't discount it.
To me it sort of sounds like poisoning, maybe a slow poisoning. Chickens have delicate respiratory systems so fumes can cause problems. Is there some source for poisonous fumes, maybe an open flame nearby? Or are chemicals stored in the area? Are you feeding them the same feed, is it contaminated? I remember something like this came up a few years back. It turned out the chicks were being fed contaminated gerbil food as a treat.
I see you are in South Carolina, thanks for that info. You might call your county extension office and see what it would take to get a necropsy. That's where an expert cuts up a dead chicken to try to determine what caused the death. Each state does it differently, some pretty inexpensive, others not so much. So ask about the cost, where you take the chick, and how you handle it. They usually want it refrigerated or packed in an ice chest but not frozen. You might consider it worth it to try to figure it out.
