I handle my chicks from the time they are fluffed out and moved to the brooder. I have only had chicks trust me more for it. Try cupping them on their backs and rubbing their bellies. It puts my seramas right to a comfy sleep. :)
The chicks I have messed with less frequently early on also...