Hi, welcome to BYC!
You should NOT be scared... but you should know what you are facing.
He is trying to dominate you and if you don't put him in check immediately it will escalate into an attack... possibly when you aren't looking. A common problem with "first roosters" raised since egg. Even if you make the effort now, it will depend on how hormonally stupid he is already. Some boys will reform while others continue to escalate until they meat their fate.

Teach him to fear and avoid you. In the chicken world fear equals respect.
No, I wouldn't worry... I would make a plan.

Teach him to get with the program, rehome him, or send him to freezer camp. You can research rooster training techniques. And probably get lots of input here. How old is he now? Spring time means hormones are raging.