Thanks everyone.
Sometimes I want to be blunt and outright with him, but I also don't want to make a big deal. Right now he's only annoying, but if it increases to more than that I will be blunt about things. For example, the other morning I intentionally got to class early to study and he kept staring at my notebook and reading what I had written. He was completely in my space, and continued to do it even when I moved away. I can't stand when people hover like that! He then went on and on and on about how I should have taken this other professor because they are so much better than the one I have. I just wanted to smack him! He'll also brag about how smart he is and how he is in all these programs, but I know he is lying about a lot of those things. He also talks about himself like he is a musical genius and that bugs the heck out of me. Before valentine's day he was complaining about how he doesn't have a valentine and I just sort of shrugged at him and asked why that was such a big deal. He has asked me to study with him, but I told him that I prefer to study alone. Every once in awhile he will make a strange joke or sexual reference and one time I did get angry at him for this. He hasn't done it since, thank goodness.
He's mostly just very awkward in social situations and he thinks he has has to have a girlfriend to be accepted, or something like that. Sometimes I feel bad for him, but not bad enough to be his friend. I think he's really just looking for attention, but he comes off as a creepy guy. I'm not really worried about him taking it too far, but it is a small concern in the back of my mind. I am pretty good at taking care of myself if I need to (That punching story was great, BTW!) He doesn't know where I live on campus or anything like that. He did get my phone number freshmen year when we were all new and looking for friends, but surprisingly he will only text every once in awhile and its never anything creepy. I just don't reply to him. I have him blocked everywhere else, like facebook, etc.
Thanks for all the good advice though, and keep it coming. I'm just hoping the end of the semester comes quickly, he transfers, and I never have to see him again!