For a temporary fix you could tack some burlap or stuff furnace filter material (I think that's what @aart has suggested in the past?) into the open vents to help buffer or blunt drafts while still allowing air flow.
I agree having vents up even higher, like maybe some louvered gable vents in...