I first thought this, too, but you could get around this problem by creating a trap (one way) door specifically for the pigeons and teaching them where it is. There are several types of these – I have a
bob door.
I can let my pigeons out the front door of their aviary, but they will always go around to the back to try to get in because either I leave that door open for them or there is a trap door up high on the back for them. It’s actually kind of funny; if both trap and back human door are closed, but the front door is wide open, they’ll just hang out by the back desperate to get in come suppertime, rather than check out the front door! They are clearly trained to go in that way.
But if the pigeons can get out you would have to have pigeons right from the start who were not set to another location. (Is that the right word? Grrr, I feel like there’s a different word just trying to get out of my brain but I can’t grab it.) Youngsters who had never flown before would want to come back to this home, but an adult who had flown from a different loft may try to return there at least in their early months at the OP’s home.
All that said, having seen my chickens chase the pigeons when they are out in the yard together and get close, I like having my pigeons in their own quarters. It’s OK when they’re all out free in the yard , but in the close quarters of aviary (6’x16’) or covered run (8’x16’) I like them to be able to feel safe.