sucks, but the complainers have every right to do so.
this is a good reason why you don't answer your door for people you don't know and/or aren't expecting. it could save your life.
i'd have told the city workers to pound sand. they want to see my operation they can get a judge to issue an order or warrant.
judges are hesitant when it comes to invasion of privacy (especially when no one's life is in danger) because if they're wrong they look really bad and it can (and should) come back to bite them come election-time.
even if you aren't doing anything wrong, don't willfully let gov't officials invade your privacy...it only emboldens them and encourages them.
we already live in a society where the government operates extra-constitutionally, we don't need the same garbage from local munis.