First, we'll assume you are in the city and fall under that code. Then let's ignore the rabbits as they are house pets. You have 4 poultry; you do not need a permit with that few. At least I do not see that in the city's code. If he insists, ask him to give you the exact section in the code.
If you can figure out the setbacks in your zoning, you can determine whether there is space available for a coop that is 50' from all houses. If you have to encroach. make it toward your own house. That said, you do not have to allow him into your yard. Simply tell him that the birds are within code, and politely ask him to leave. If he insists on seeing them, ask to see his search warrant.