Little sweetheart, there are lots and lots of us. I've been there and lived through it. Let me relate two things that I think will help you.
Tell him how you feel without blaming. You are very articulate so I'll bet you can do it really well. Should go something like; " when I'm not able...