I sat with mine and read for hours at a time. Just sitting with them gets them very used to you. They are the nicest little (big) chickens now; follow me around and eat out of my hands...
I've never had a problem. I should have picked them up more when they were little, but they still turned out great.
Short answer: be with them. It takes chickens time to warm up to anything, including people. Try to handle them, feed them from your palm, and speak gently to them. Over time, they will become familiar with you and accept you as their caregiver.