I had a buck that I raised from a little kid. When the vet banded him they missed a testicle and he became insanely aggressive. Used to charge me and slam me into walls, knock me over, etc. He actually hurt me pretty good a few times.
I let it go for a while because I loved him a lot and I had raised him but it isn't worth it. They can seriously injure or even gore you if they catch you right (if they have their horns). If you can't get him to back off, consider banding him or selling him.
So sorry you're going through this!