I've never had a problem with dual purpose breeds picking on the cornish x's. I've done silkies, RIR, sexlink, BR, wyandotte, and at 3-6 weeks, open the door and let them free range with the adults. Not that the cornish wander more than a few feet from food and water though. I do however have lots of free range space and so the birds aren't interested in each other. I bet the older layers with chicks walk by and say "look kids, that is a pig."
But, if you do raise them together, keep an eye on it all and be prepared to separate if there are any problems. And of course, if you do small numbers, I think everything would be fine.