Ohhhh no, I would be worried too. Even though cats hate me... I don't know why.
Do you stare at them? Staring at cats makes them uneasy. Try the slow blink. People that are afraid of cats tend to stare at them (not saying you are afraid) and that makes them wary of you.