It sounds like your chickens have a good lifestyle and enjoy being with you. Mine follow me around the property and like to sit with us, too. It's obvious you love them very much.
Whether the number of chickens or other animals a person has is a problem isn't usually as much about numbers as it is about the conditions they're kept in. As long as they're well taken care of, that's the most important thing. Do they get enough food and have clean water? Is the housing clean? Do they have enough space? Do they have any medical problems that aren't being treated? Do they have any internal or external parasites they aren't being treated for? Are you starting to feel overwhelmed by the amount of work it takes to care for them? If they're in good shape and you're happy, that's a good sign.
In the case of your neighbors, they may just be the type of people that think everyone should live their lives the exact same way they do. Some people are impossible to please. Some people don't like the idea of chickens, even just a few. If you have any neighbors like that, I'm sure they're having a fit.
I would ask myself honestly, though, if they have any legitimate complaints. Especially if you're all living on fairly small pieces of land close together. You have 5 roosters with 25 hens. Are they crowing occasionally or is it all day long, as they compete with each other? Are you having any odor problems? How about when the weather is wet and the run is wet? Is your property in good repair? Do things look pretty good, if people are close enough that they have to look at your property when ever they're in their yards? If any of these things are an issue, I would try to work on the problem. If they aren't an issue, I wouldn't worry about it.
Are your hens sitting on large clutches? If you have a couple batches of chicks added to what you have now, I'm a little concerned about the number of roosters you're going to end up with. You might want to start thinking about removing some of the cockerels or roosters from the flock, after you end up with more males. You've already got a rooster for every 5 hens. A 50/50 pullet/cockerel hatch will make that ratio even worse. How are your roosters getting along now?