Usually even with structures that don't need a permit, you must honor the setback (the amount of distance a structure can be from the fenceline/property line. My Dads a contractor/designs houses. If someone complains, the setback could be an issue even if you don't need a permit. That said, most people ignore setbacks.