In my experience, horses are better about leaving others than they are about being left. So, what this means is that if someone canters off from you, likely, your horse will try to go with, also, they don't like being left in the barn alone. But, if you take the horse out of the barn and leave the others inside, they tend to be ok with that.
The way I look at it, and this has drawn some heat on here so take it as you wish, is that my horses are never alone, they are always with another horse or they are with a human. If I am with my horse, the horse is not alone. So, you need to get the horses respect so that in the absence of another horse, your horse respects and trusts you to pull the shots so that they can be relaxed with just you on a ride the same way they would be relaxed following another horse. Now, with a young or green horse, it still helps to have a pasture buddy lead through anything scary the first few times so that they can see it didn't eat the other horse, but that is just because you are likely with the horse for 1-2 hours and pasture buddy is with him for the other 22-23 hours, so obviously they will trust the other horse more and follow their lead better since they are more familiar with them, but for something not super scary, you should have enough respect from the horse that they will know that if you tell them to go somewhere that you won't allow them to get hurt.
Does this make sense?