2 cents worth coming right up...
My personal feeling is that if you recognize the dog as a neighor's you should bust its butt with a marble from a slingshot or with a paintball gun. No real damage done, but some serious pain and fear inflicted. After that the dog probably won't be back. And you won't have started a feud with your neighbor.
If you have to shoot an animal do it humanely. If you just try to "wing" it, you may kill the animal anyway, but in a way that causes it to suffer. That's just wrong, and should be avoided.
I know you are safety conscious, but others who read this may not be. Remember safety! If there are houses, barns, vehicles, animals, people, etc. beyond your target, then don't shoot. A .22 will kill somebody several hundred yards away. A shotgun could still blind somebody's horse or their kid, 300 yards away if you are shooting buck shot, and maybe 100 yards away with birdshot.
A shotgun within 30-40 yards is very deadly and humane. Beyond that you need a rifle, at least a .22. And a shotgun makes a lot of noise. You fire that baby out your back door and everybody in the neighborhood is going to hear it.
A handgun isn't much good for this kind of work. It doesn't have much range, and it's poor accuracy makes it harder to kill things humanely, than would be possible with a rifle or shotgun. I have killed snakes with .357/38 bird shot, and it works great. But I wouldn't expect a dog to feel it beyond 20 feet or so. Besides being tiny, the shot is being swirled out into a broad pattern by the rifling in the barrel of your rifle or handgun.
The one gun that every homestead should have is a .22 rifle. It is cheap and fun to shoot and very useful for home and livestock defence. Where the .22 rifle comes into its element is for fairly close shots (within 50 yards) at small things like rabbits, skunks, dogs, coyotes, raccoons, foxes, etc. where it's best to keep it quiet. But I have seen 1000 lb steers drop, dead before they hit the ground, when hit in the forehead with a .22 rifle. Tho deadly up close, it doesn't have the range or killing power of a high powered rifle like a .223, 30-30, 30-06 etc, so it is much safer to shoot around civilization. But you still have to be careful what's beyond your target.
The 20 ga Remington 870 sounds like a must see. You can't beat an 870 for ruggedness and reliability. And they feel and point great too. Being chambered for 3" magnums would be ideal. The maximum length of shell will be stamped on the barrel. The recoil of 20 gauge 2 3/4" shells is a lot less than a 12 ga, and if you put 3" mag shells in it you're close to to a 12 ga in performance (and recoil). With the proper ammo you can kill anything from mice to bears with it.