I'd get rid of the three who have already shown human aggression, then let some time go by and see how the behavior of the other two may change or mellow out.
You can't go by breed for anything, really. It's all individual. My roosters are Delaware, Barred Rock and Del x BR. They are chosen for temperament, which comes from the 6 year old Delaware rooster, who is father of one and grandfather of the other male. The Del rooster in my avatar is the one I'm talking about and he comes from a line chosen for temperament, which is generally heritable.
Most all roosters will do for watching out for the hens--they should never "protect" the hens against you. That is a no-no here. You want the one with the "smart genes", the one who knows who is top dog (you are) and knows you will not harm the hens. I'm not talking about some young guy boundary testing bites, I'm talking about all-out flogging and attacks. Biting can usually be stopped with some aversion therapy, I've found, IF the male is not genetically prone to human aggression anyway.
That said, the barred rooster will pass barring to all his progeny. So you'll have plenty of barred chicks. if you don't want that, that might influence your decision.