That stinks. Someone must have complained. Do your neighbors have larger properties? Not having structures in front of the line of your primary residence is a common building ordinance policy. Also, most have rear line rules as well (so many feet from back line, so many feet from front line, coops also will have special ordinances, like 300 feet from another persons residence, etc...). If you have a moveable structure (like a chicken tractor or a coop on skids (that you can actually move) it will often fall outside of the building code requirements since it is not a permanent structure. That does not stop anyone from complaining about it being obtrusive.
Right to farm was established to protect farmers from nuisance complaints and prohibitive ordinances. Each state has its own version of this national act and to be qualified under the act in NJ you need 5 acres and a $2,500 in farm products annually. If under 5 acres you need $50,000 in products annually (not sure how you could bring in that kind of money with less than 5 acres, which is, I imagine, the reason they set it at that amount).
http://www.nj.gov/agriculture/sadc/rtfprogram/rtfact/index.html So, you not only need 5 acres, but you have to actually sell farm products and you would also probably qualify for farmland assessment. If you do not qualify for right to farm you will be regulated by your townships ordinances regarding land usage. Here is the Vernon ordinance:
http://clerkshq.com/default.ashx?clientsite=Vernon-nj
I have to say though, I looked through the ordinance and I don't see anything prohibiting the keeping of livestock. The animal section deals only with dogs, cats, and bears.