One person talking trash? Kiddo, I'm sorry that he got to you. You are a sensitive person and I hate when decent people are snubbed.
But the thing to do is as Joe recommends. As long as he is all talk, then it is one person; so give him the benefit of your foot up his backside (figuratively) and get on with life. If you are right and there is nothing that can be laid at your feet, then move away from this turd bird.
This next advice may ring harsh, but it is meant to succor:
Grow a skin, my dear friend. A thick one that can shed a bit of trash and bile. The world is full of such people, and the sooner you can let it roll off, the better. Think of the duck in the storm.
Pray for the doofus, then get back to what matters. I'm reminded here of Mr. Miyagi, in the movie "The Karate Kid":
"Better learn balance. Balance is key. Balance good, karate good. Everything good. Balance bad, better pack up, go home. Understand?"
These words are spoken by Mr. Miyagi to Daniel Russo, played by Ralph Macchio. Daniel is wanting to do more advanced techniques and Mr. Miyagi is reminding him that in order to improve, a good foundation of the basics must first be in place. Mr. Miyagi wasn't just speaking about karate, he was speaking about life.