Having been a teacher in an inner city school for more than 25 years, I can tell you that more often than not, when some one is unreasonably upset/angry with me, it usually has more to do with things going on in their lives than with anything I did. Reasonable people know how to hold diverse opinions, and find compromises. He seems very unreasonable, and sounds like he is trying to provoke you. The best way I've found to handle such situations is to let the person vent, stay calm, then ask them what their goal is and try to find a fair way for them to achieve it. He probably knows he should put up a privacy fence like he said he was going to, maybe even feels guilty about not doing it. Perhaps he can't afford it since he's out of work. Maybe you could pay for materials, and he could do the labor to the fence up? With you both having children, I would try to find a way to live peacefully - your children are going to go to the same schools and activities, maybe be friends! Sometimes it's more important to find a solution than to be right - you'll have to decide whether to escalate, de-escalate, or ignore the problem.