Most cases the suit is more expensive than the deposit. Offer a discount or a partial refund, if he refuses and sues without a contract I hardly think he has a chance. Your word against his, what would hurt you is if the date you gave him was a firm date or just a estimate of time.