I never paid much attention in the past other than to just wait the 4 weeks to be sure.
I just did a recent pen change and monitored the eggs closely to see what would happen (if the rooster started mating right away and fertility remained or if the rooster took time to adjust and fertility went away). Now 3 weeks later, I am seeing infertile eggs from the girls that were moved. They were fertile before being moved (from the previous roo). I know the new roo they're with now takes an adjustment period before making his move (at least that's been his way in the past). So based on this I would say 3 weeks is a safe bet, but 4 is better to be sure.