Some people do say that males raised together grow up to be more accepting of eachother. I haven't had the chance to figure that out yet, so I can't really say if I've found it to be true, only that it does make some sense.
The most important question for me, would be what are your keeping arrangements. Are they free ranging? If so, they are likely to split into groups/tribes, so there won't be much of an issue. Free ranging helps you get away with keeping more males. It all depends on each individual rooster how often and how rough they are with mating, of course, but more space makes it easier for the females to get away from a male they don't particularly like.
I have not found any direct correlation between "facing off" early-on in life and aggression (or lack thereof). Both the pullets and the cockerels face off here, and it's not usually a sign of who is more (or less) aggressive. The one who "wins" most face offs might be the top bird amongst the youngsters, but it rarely translates to actual aggression.
As far as silkie, and generally bantam males go, whether or not they don't aggravate the LF males as much as other LF males really depends each individual male, once again. A general truth I have found is that the older a rooster gets, the more sensible he becomes.
A mature, established rooster would realise that a much smaller rooster might not only be less of a concern, but he tends to attract hens of the same size, and therefore hens not best suited for the LF. Same goes for the small guy. If he's also a mature male, he will understand that there is no particular advantage in fighting for the LF's hens. Of course all this is applicable if both males already have a couple of females under their wing. If not, no matter the age, a male will try and get some females under his protection. In all actuality though, size doesn't really matter. Mature males of any size (assuming they already have hens) will only fight as a last resort, and much prefer to resolve territorial (or otherwise) disputes without a fight.
Young roosters and cockerels are much more hot-headed than that, however. They might try and fight/challenge eachother with less provocation. This is pretty standard behaviour for birds that free range. In the case of birds kept in a coop and run 24/7 (or in the rarer change of a free range group not splitting), your males are likely to form some kind of hierarchy.
The problem that might arise with this is that at any given point, one of the males might decide that he doesn't like his place, or, that he doesn't like sharing. Though the latter is unlikely to happen with the number of females you have, the first is still a possibility. The fight that will follow could be to the death, even though it rarely is. In your situation, it is likely that the LF will establish himself as the head rooster, though not a guarantee.
If this is your first go at raising chickens, I'd think about maybe letting go of some of the males. Keeping the ones your females like best seems like the best idea, that's what I've always found to be best, at least.
I hope my post was helpful, rather than confusing, seeing now that it has dragged on for quite a bit!
