LOL! Short answer... to keep the neighbors happy. If I didn't live on a 10,000 sq. foot lot, in an established neighborhood, with finicky neighbors, I'd be keeping my poo, too! I have neighbors who absolutely hate the fact that I have a compost bin and refuse to remove it. I'm dreading the day they discover we have chickens. Legally, I can have chickens on our lot, but that doesn't mean the neighbors are going to like it. One of our neighbors had rats living under his shed. They don't keep the area around the shed clean. It is weedy and junk is piled around it. But, we got blamed for the rats because we have a compost bin.
I encourage you to read what you wrote again.
Who is dictating how you live your life?
There are no do overs in life.
You are dreading they find out you are EXERCISING your RIGHT to have chickens? Huh?
Ask yourself if you could go ahead in time to ten years from now knowing the whole time you missed out on one of the great fantastic benefits of owning chickens would you still feel this way?
All this time you threw those benefits away because you are letting neighbors control you missing out on better flowers, vegetables, more soil.
IMO it's way past time to tell your neighbors you know a good real estate agent they can use if they want to keep on whining.