How are you approaching the chicks? If you have your brooder on the floor and are reaching down into the box to pick up the chicks, they will be as terrified as if you were a hawk diving at them from the sky. It's instinct. As long as you continue to do it this way, they will become more scared of you trying to touch them as time goes by, rather than become more used to you.
You need to reach toward the chicks slowly from the side, ideally sliding your hand, sideways toward their little feet. Rest the side of your hand against their toes, and they will soon trust you enough to step onto your hand. At first, after the chick steps onto your hand, either the palm or backside, raise your hand up and down a few inches, coming back to rest on the floor of the brooder. Soon chicks will compete to get a "ride"on your hand. Raise them up a foot or more. They'll love it! By this time, they are fully trusting you, and you can then bring your hand toward you and hold the chick. The taming process doesn't take long when approached in this manner.
This is much easier with a brooder placed up on a table, with a side access, instead of a brooder placed on the floor with a top access.
All, and I do mean ALL, of my chicks raised in this manner have turned into lap chickens. My hens are as affectionate as puppy dogs, jumping into my lap to be cuddled whenever I sit down. When I'm in the run, on my knees raking poop from under the coop, sometimes I'll have as many as seven hens crowding in close to be group-hugged. It's so funny to back up and they'll all remain in tight "formation" like a racked set of balls on a pool table.
Chickens are capable of being as tame and affectionate as you are willing for them to be.