This is my own personal preference for my handgun: I like a nice auto 25 cal. The one I have will actually fit in the palm of my hand, so it is pretty much un-noticeable. Great for concealment and for fitting it into places (handbag, purse, pocket). I know a 25 cal. is not a heavy duty shot, but a couple police friends explained to me that the bullet will enter the body and instead of through the body shot (exit the other side), this is light enough that it sort of bounces around inside the body ( bouncing off of ribs, bones, etc, causing damage to what it hits during it's bouncing around).
For varmint shooting: take into account of what you are going to be shooting at and your ability or marksmanship. With a rifle, you have a much more narrow field of scope for hitting the animal. Imagine throwing a singe rock at something and hitting the object. Are you good enough to hit it with only one projectile? Now imagine throwing a handful of rocks at something. You are a lot more likely that at least one of those rocks it going to hit the target. But another thing to take into account; with a single projectile (from a rifle), you have a smaller area of dispersal and less likely to hit something that may also be in the area of the predator (say a chicken, pet or anything else) than something that is going to spread out in a larger pattern (shotgun). You are going to have to figure out on your own of how good of a shot you are: can you hit something with just one projectile or do you need the spreading pattern of a shotgun to hit your target? And figure in how much room do you have for your shot: narrow (rifle that is not going to spread out), cause of other animals and such around or plenty of room (so you could use the shotgun and not hit something else)