The reasoning iss usually that the property is "zoned" to a particular purpose, and the building rules are in support of that pourpose. On properties intended primarily residential, zoning encourages the building of residences (and residence-like buildings) to which it often permits ancilliary uses - construction of a "pool house", a "garden shed", a one room detached "home office" or detached garage - but as the purpose of the zone is residential, code usually prohibits construction of more than one or two accesory buildings, and usually dictates that they be considerably smaller than the house.Hm is the reasoning for that something like if I plant a garden, but Bobs building a bigger barn then it’ll block the sun? As an example.
Doing otherwise, it is claimed, "destroys" the nature of the community.
One would not expect to see a bunch of quarter acre lots with 3k to 5k sq ft homes, then a one room mobile "house" on the next property with a detached 20 car garage or three story 10,000 sq foot barn... Zoning is intended to stop such things, rightly or wrongly.