I enjoy my roosters. We have three right now in a flock of 49 hens. I've only ever had one that tried to dominate me, and he left the flock very quickly. Since then, we've had six different boys, and none have been aggressive. Your older hens will harass the young boy until he hits maturity, then he'll all of a sudden come into his own and put them in their places.
My advice is this: Keep one of the boys, but DON'T make a pet out of him. Always let him be a little bit cautious around you and move away from you if you walk directly at him. If he's getting snuggles and sitting on your shoulder while he's immature, then right around sexual maturity he's likely to think you're an easy target. I've seen so many heartbreaking stories of roosters that were a BYC Member's best friend until they matured, and then they became little demons.