At 4 weeks you are an evil monster, so they are not going to want to be picked up.
Mine are 4 weeks and until yesterday, they'd still seek out hanging out next to me and if I picked up Dill, Jasmine wanted to find a way up onto my lap.... without my help. Helping resulted in scratching the crud out of me. Since I was picking up to check out wings last night, neither are too fond of me right now. Peas - and they'll come around again.
Handle them as much as you can. When mine were little, I'd sit and watch tv at night with them. Now I just plop one into my lap after they've pooped and the other wants to join as well.
You figure with feathers coming in, it can't be too comfy to be touched, so I just hang with them and when I do put them on my lap, I let them decide what they are doing, I don't really pet them. Usually they walk on me until they find my hand and then sit in it.