Good thing your building them a pond because four goldfish I don't think would survive in a 20 gallon.
Yeah, I definitely wouldn't keep them in there permanently! If I wasn't building a pond I would buy a bigger tank or a stock tank and might still do thar for winter but I've always wanted a pond.