holding hands should work. or putting your arm through his. it's a practical solution.
there's two parts to this...
1) you have different natural paces
2) you don't tell him when you hurt.
not suggesting you should make a big deal out of whining, but if you're in pain a lot and don't mention it, and only say something *after* you've way over the edge of what you can tollerate, he probably doesn't gauge your level of pain or ability correctly... you have to teach him what that really is, and communicate what you can reasonably do.
he offered to drop you off, even though he knew it would mean missing half the game. that was considerate of him. you elected to walk instead. that was considerate of you.
on the way back, given that he probably isn't gauging your ability to exert yourself correctly, it would have helped if you spoke up and said, "I'm really sore from the walk up here, I think the walk back will be a bit much for me. would you mind getting the car and coming to pick me up?" he'd probably have been considerate again and done it. you wouldn't have hurt yourself, and there wouldn't have been a situation where you were pushed over the edge with pain and frustration, and where you - once again - were angry with him over the same old failing.
in a way, it was a bit of a set-up - although I'm sure not intentional. you wanted him to know how you felt, and wanted him to act to prevent your pain... that would show consideration and caring. but he probably doesn't know enough about when you're in pain, or how much pain you're in, because you don't always tell him. and much as we would like, guys, even ones who love us a lot, often don't read our minds well.
if this were me and my hubby (and I do have some similar limitations) he'd be calling me on the carpet, and rightly so, for expecting him to read my mind and not giving him enough information. I'd have been the one with the information that I shouldn't do the walk back, so I'd be resonsible for sharing that. if he didn't respond appropriatly at that point, then I'd have a legitimate gripe.
I'd love it if my hubby always knew what I needed, always did what I wanted, always remembered everything that bugged me and never did any of it. but hey, he's human and we're not identical. if I'm not getting what I need, in the moment, it's my job to speak up.
in addition, it means I'm not accumulating gripes when they could be reasonably avoided, and can enjoy our time together more.
I've been on the other side of this too. my ex thought if I loved him, I would know what he needed and wanted. if he had to tell me, somehow it spoiled it for him... because it meant I didn't love him enough to know. needless to say there were a LOT of things he needed and wanted that I didn't know, and since he wouldn't tell, he went without. made him feel unloved and made me feel set up. that's a big part of why he's an ex.
just tell him, in the moment. and hold hands.