This is a tough one. This spring, when two of my Bantam chicks turned out to be roosters, I intended to keep both. But the one who was supposed to be a Cochin and turned out to be something else, was a little hellion. He chased the girls, me, and the other boy, who was clearly more subordinate.
Once Pauly was rehomed, the remaining Cochin boy, Billie, totally changed. He has moved up in status and now takes care of his girls, keeping them close and making sure they get treats before he eats.
Maybe the leghorn's behavior with the girls and theirs with him will change if you rehome the brown one? I don't have enough experience with multiple roosters to offer a solid opinion, just a possibility.
My only other rooster is permanently cooped with a couple of hens because he wanted to breed EVERYONE all the time and refused to come inside at night. He and Billie get into crowing contests, but neither has shown any aggression toward the other.
Let us know what you decide. Good luck deciding.