The main concern would be biosecurity. Even if the flocks are kept completely separate being that close together likely means whatever your flock has, the neighbor's will too, and vice versa. Since your chickens have already gone over there and you've been on their property too, it's probably a moot point now.
If you still want to keep the flocks separate a fence is realistically the way to go. Otherwise you could just let them mingle, some people do have shared flocks, but then you'll need to decide how to divide up the eggs and cost of feed and that might just be a big hassle.