You are right about cleaning out the flower beds. I would suggest that the brooder material be removed from the flower bed, and put it in a compost bin to let it decompose properly first.
Fresh chicken poop can be too "hot" (too high in nitrogen) to use directly around plants as fertilizer, it can burn the flowers and contaminate the vegetables. When chicken poop gets wet, it smell really bad, potentially spreading disease organisms and attract flies. There needs to be a proper balance between the carbon (leaves, wood shaving) and nitrogen (chicken poop) ratio + time, to let the chicken poop decompose into a rich garden soil before using it.
There are many discussions on BYC on how to compost chicken poop. Using the deep litter method can reduce the frequency of cleaning out the bedding to twice a year. Many people set up a compost bin on the side to do hot or cold compost, maybe you can create a compost bin to manage the "black gold". Here is a quick explanation:
http://www.seattletilth.org/learn/resources-1/city-chickens/compostingchickenmanure
I personally use two different closed top compost drums with separate compartments. When the chicken poop is removed daily from the poop board, it goes into one compost compartment. When the bin is full, move on to fill the other compartment. Every 6 months, I clean out all the deep litter bedding in the run, the materials goes into the large compost drum. These drums are mixed and rotated frequently. It is smells, just add more dry leaves as the carbon material to balance the composition. After several months to a year, the manure started to break down, I empty it out onto my next door neighbor's yard. I am very lucky that my next door neighbor loves chickens and gardening. She has a large pile of wood chips from a down tree last year, the wood chips have turned into a compost pile. We mix our materials together to create a community compost pile to use in the gardens.
Even with understanding neighbors, I have to be careful in managing the compost bins and the chicken coop to make sure that there is minimum smell. Noise and smell would be the first things to trigger complaints.