I would also be furious. Obviously this guy is completely lost in social interaction, so I don't think that ignoring will even register with him. I would certainly call the police and just tell them everything that is going on. You can make it clear that you don't want to hurt your neighborhood relations (and you still are going to have to live next to this guy), but the police need to be aware of what is going on and have a complaint on file in case he does something serious. I am not trying to be dramatic or scary, but I wonder what is stopping him from walking in the front door when you don't answer right away. I would think that the police can have a rational conversation with him about property rights and trespassing that shouldn't harm your relations too much. After all, the guy is clueless and may not even take offense.