It's not just that he flew away and may not want to come back, it's also that he might have wanted to come back but met with an untimely end, since there are many predators of pigeons.
My suggestion is like others have said before -- don't let out any birds that you can't afford to lose. This includes during a vital time like nesting or taking care of babies.
However, having said that, I have had a pigeon that took up to 3 days to return, on two different occasions. One was a particularly bad stormy day, and the other pigeons came in from the storm but he didn't. I thought for sure he was a goner what with having to be outside exposed to the bad weather but he came back home a few days later. My guess is that he really did just go try to find a place to be safe and had to home back from wherever he managed to hole up.
Another time was when my couple of birds got scared off by some bird predator. Most of them returned by the evening but one bird was not there. I thought he didn't make it and got taken but he came back after a few days. I like to think that perhaps he was the one that the predator was chasing and he lost his way for a few days as he led the predator on a wild pigeon chase, but he did make it back as well.
So it could still take a few days for the bird to get home, don't lose hope yet!