Yes, it can be irresponsible. However, it's not just the cat who is at stake. Cats have decimated many of our wild bird populations. The popular collars with bells on don't help, birds don't associate bells with danger and aren't wary. All they do is make the cat owner feel better. If you have an old cat who is no longer a killer, it's probably fine. But if you have a cat who you know loves to chase and kill birds, you are impacting more than just yourself with your choice to let it stay out.
If it's strictly a barn cat and only killing rodents, that's fine too. But really, how many outdoor cats are? Most are suburban.
I can't garden without finding cat poop every day, yes, right next to the tomatoes. I have plants die regularly from cat urine. There are loud cat fights every night outside my window. I love cats, but to think that they aren't ever a nuisance to others is wishful thinking.
My Charlie is an indoor cat, he started out as an indoor/outdoor cat. When we kept him in, it took a couple of months for him to get used to it. It's been years now and he's perfectly happy.