Ummm, I wouldn't count on them not picking on him. I have a silkie hen or two who will pick on just about any roo they encounter, especially a young one. BTW, I have never had a mean silkie roo. I have heard of them, but have never encountered one myself.
It may be at least partially that I do not expect them to be mean. I have a friend who is terrified of one of her roos. I walked into the coop, reached down and picked him up with absolutely no problem as she stood watching in amazement. I handled him and looked him over and put him down and not once did he act at all aggressive. I cannot recall whether I handled any of his hens, though.
I have a rooster who dashes over to protect his hens if I pick one up and she squawks. Then looks up at me and seems to say, "OH! It's just you--carry on," and walks away.