To give you advice, we really need to hear the whole story. Otherwise, there is no way to really say. Normally, as long as the seller upheld their end of the bargain, NO- deposits should not be refundable. You admit you didn't uphold your end of the deal though. The reasons for that are somewhat irrelevant. You say you are two weeks behind on the delivery. What are the rest of the details? How long will you be late total? How long did you tell the buyer it would take? If you are two weeks late on say a six month deal, it is not as big an issue as if you were two weeks late on a one week deal for example (but really, it is the buyer's money, not ours, we can't really say *why* he is so upset or judge his reasons not knowing his side).
It just all depends on the entirety of the situation. In general, I would say deposits are not refundable, he has no right to harass you, and he should be understanding (if he is in a position to be flexible). That said, you should be as well. If you promised delivery within a certain time frame and you are already two weeks late, you might want to just refund the guy's money. It may not be your fault per se that you are late, but it certainly isn't the buyer's fault either. At the end up the day, which party in the transaction did not uphold their side of the deal?