It's not about size, silkies and polish and some other exotic looking breeds get picked on for a couple of reasons 1) with the big crests and beards obscuring their vision they don't see well enough to fight back. If you aren't going to show - carefully clip back the beards and crests and you will see silkies aren't slow, stupid or cowardly.
reason 2) birds tend to peck at them because they look so different. A Mix of breeds raised together generally are accepting of the silkies - because they were there from the start. When you add them to an established flock, you get the usual integration problems - any new bird would have.
Some silkie roosters end up being the Alpha male and can kick bottoms of larger breeds & end up with the girls for themselves.