Chickens have a limited memory -and separating them often causes more problems.
What you are trying to do, is add a 'new' single bird, a very hard integration, everyone else is afraid of this bird, and KNOWS this bird does not belong. So attack.
What you need to do, is not to add a single bird, but rather a pair or trio. Put a single bird with this guy, there will be a bit of bluster - but one bird to one bird, not 4 birds to one.
Wait 3 days, then add one of the adult birds to this pair, let that settle for three days, then add the trio back to the flock.
Clutter in the run is important - places to get out of sight, to escape.
I am a bit worried about the brother. VERY OFTEN brothers, even birds raised together, (and now your birds have not been) do not get along. They can fight once, settle it, fight again later. If you have a small set up, and generally when you have 5 birds, you do have a small set up, then I strongly recommend removing at least one of the roosters. And if you are new to chickens, I would remove both. Roosters are a crap shoot, the more roosters you have the greater the chance of it going wrong.
Roosters take experience, and depending on the age of these birds, it might be impossible to add this rooster back to an established flock. Rooster do not understand sharing, and will fight to defend their territory and hens. It might just not work no matter what you do. They do not call it cockfighting for nothing. Inexperienced people often vastly underestimate the violence of roosters.
Mrs K