I understand. My turkeys are the same, they are pets, and I think of their spot in my home like a dog or cat. My Tom's were all babied, and I've had no aggressive behavior aimed at me. The worst I had was my favorite tom going after my mom and trying to tread my legs while I sat down, and sadly after a year I decided it was best for him and us to rehome him. I always kept the offer open to buy him back if it didnt work out. I still worry about him all the time. He saw me as another turkey, but I think he saw me as a hen and not a competitor tom.
I think the best solution for you would to be to separate him. I dont know how to curb aggression towards you, since my boy only was aggressive to everyone but me. I hope you can find something that works.