If no one has complained and he's not bothering you then don't borrow trouble.
When I had my in-town flock with a rooster the neighbor on one side liked to hear him because he'd grown up on a farm. The other neighbor figured that they didn't mind our rooster if we didn't mind their muscle car...