Push him out of your space. Teach him to yield his hindquarters (pull his head toward you, walk briskly toward his butt, and tap it vigorously until he moves it out of your way). Do that both ways. Get him backing up when you say "back" - no arguments, you said back, he goes back. Yielding the forehand is a little trickier, but bumping the shoulder or waving the whip beside his face to make him want to move away can work. When he does what you want, even if it's just one step, drop the pressure and tell him "good boy!" Rub him with the whip - it's pressure, not the enemy; you want him to respect it, not fear it (remember, a horse bites another horse and grooms another horse with the same mouth - they understand these things!)
It sounds to me as if you are unconsciously yielding space - he comes toward you, and you back up to keep him from stepping on your toes. You need learn to stand your ground and get him to move around you, not toward you unless you invite him to move into your space intentionally.