(Most) bugs don't have the best vision. You're just another part of the landscape to a wasp or hornet!
Certain colors and scents attract them so they might hover around you for a while trying to figure out what you are, and at that point you can just gently wave them away (it might take a few tries until they get the picture). I don't mind it though, and I'll let them land on me to smell/taste around until they realize I don't have anything to offer them.
If you're nervous they will pick up on it. Mostly that's because of physical changes you're exhibiting that you might not even pick up on, like jerkier movements and heavier breathing. If you keep yourself calm and respect their space there's no reason they should think of you as a threat
In fact a lot of stings that come "out of nowhere" are due to proximity to the nest. Wasps and hornets are very proactive about protecting their queens, eggs and larvae. If you accidentally brush up against a nest that you don't see, and they take that as a threat, they're going to do what they can to drive off the threat!