Based on the picture, it's not a matter of anything the other birds are doing. I would let the others do their thing. This is something physical happening to just this rooster.
One thing I would check for is parasites. Check by his vent, under the wings etc. and make sure his inactive...
When you say he can't keep up - is that a physical thing? i.e. he gets tired or is there lameness-- or is it (as suggested above) that the other rooster has handily pushed him out of the flock and now he keeps his distance? How old are they?