Wow, I just love all these stories.
My guess is, since he hesitated, that your son said "no" because he was a bit worried you wouldn't believe him, as most kids would be when asked about paranormal things. Even at young ages, kids realize that it's not "normal" to see and hear ghosts, unless they've specifically been taught otherwise.
If he's in the closet and sounds like he's talking to a friend, I wouldn't worry about anything. Only if he doesn't want to sleep in the room would I be concerned. (And I agree that I wouldn't make him sleep there if he's scared to. But you didn't say that, so you know what I mean.)
Lots of good suggestions in here. Another is for you to talk to the closet. Yes, out loud. Tell the "person" there that you don't mind him/her being there, but that the opening of the closet door has to stop because it's scaring you and your son. A lot of ghosts/spirits don't realize that they're being scary and will stop the behavior when you ask. You might have to repeat the asking if it starts up again.
If you notice a pattern of the door being opened only at night, maybe putting in a battery-powered nightlight might help, too. It's possible that the culprit is a child that's scared of the dark. I'd do this after asking for the behavior to stop; and if it doesn't, I'd tell the child that I'm putting the light in there and will turn it on at night so that he/she isn't scared, so that the door can remain closed.
Please, continue with updates, and everyone post more stories.