I will give I've you my honest to God opinion on the subject, if he challenges you, you really can't always back away, you have to feed water collect eggs and care for your hens in other ways, clean coop out whatever, how are you going to do all these chores if continually backing down to your rooster who if he is aggressive will likely begin attacking at the slightest sign of a person. My advice is kill the darn thing and make a nice pot of soup, I have never seen a mean rooster be cured of being mean, which may be because we simply don't tolerate them, and even if you are able to calm him down some , you will be keeping him around and he will be passing on his nasty genetics to following generations.
There are plenty of non human aggressive roosters out there I would get a new one and honestly I would stop knocking them off hens and trying to dominate them, if you have a bird that isn't being nasty there is no reason for that sort of thing and I think it will backfire and make him eventually challenge you. That sort of dominance training I feel is more aimed at trying to break a bird who is already being nasty, which honestly I wouldn't bother with for the above mentioned reason.
If you get a new one and it ends up being a nice bird go ahead and hatch some out and maybe you will be able to get your own line of calm roosters going to replace them when you need to.