I don't know if you are babying Eli or not, but it seems to me that, if you have been handling a stallion for 6 years and he hasn't chewed you up yet, you must be doing
something right.
Horses are flight animals; their lives are governed by fear. Where fear ends, trust begins; as Clinton Anderson put it, "I don't want my horse to fear either me or my tools." People who don't understand horses can get all bent out of shape about using a whip on a horse, but a whip is just a tool; used correctly, it can be extremely useful when working with a horse of any size.
A whip is not just a tool of correction, though it can be. Some horse people talk about the "three second rule" with horses - when a horse misbehaves, he must be corrected immediately, or he learns nothing. If a horse kicks up at me when in a round pen, he needs to be corrected
when he kicks up; it does no good to wait until I catch him to read him the riot act or whale the stuffing out of him. Beating a horse does nothing except teach him that very unpleasant things may happen if he lets me catch him, so if he's smart, he won't let me catch him - not a lesson that any smart horseman wants his horse to learn!
A whip, used correctly, should be more or less an extension of your arm. If you can get your horse to move with just your voice and body language, great! You should always aim for the least amount of pressure that it takes to get the animal to do what you want. If you watch horses in a pasture, they are constantly moving each other around; they usually do it by the use of body language. They generally only get physical when the lower-ranking animal doesn't move out of the way of the dominant animal. As long as the subordinate horse shows respect for the dominant horse, they get along peacefully.
Particularly with minis, there can be a very fine line between getting respect and creating fear. Sometimes, with a disrespectful animal, you need to go into the fear zone, just to make sure he knows where the boundaries are. But knowing how hard to push, and when to back off, takes a certain amount of experience and finesse. You want a horse to be thinking about what you are asking and what it is doing, not just reacting on blind instinct. It sounds like your family member came on very heavy-handed, which pushed Eli over the edge into "fight or flight" mode. It sounds like that really accomplished nothing beyond upsetting both the horse and you.
At some point, you will need to work on getting Eli over this fear of whips. You might say, "but, I'm never going to use a whip on my horse; if he's scared of it, won't it just teach him to be scared of me, if I expose him to it?" Actually, avoiding things that you know your horse is scared of just makes him more scared. Getting him to stay calm and think around "scary" things that you can control, makes it more likely that he will continue to think when something you
can't control happens. The horse learns to stay calm, and trust you to do the thinking, rather than panicking and trying to run away.