My DH is in the army right now, he is a 19D serving with 4/4 CAV . He just bought a Hipoint 380 for me and a Hipoint 45 for himself. They are not so pretty , but they do their job. If you have to put down a large mammal its not all about the size of the round. Its also what kind you use and about placement of the bullet. A hollow point will expand and do more damage as it travels through tissue.THey are less like ly to ricochet and/or overpenetrate
A full metal jacket round on the other hand has advantages too:
Because the bullets do not expand, they are more effective at piercing armor. 
They are more likely to kill since they more reliably penetrate the body and reach vital organs. 
Their rounded tips facilitate proper transit up the feed ramp, whereas the usage of hollow point bullets in some weapons may increase the chance of a failure to feed. Most modern weapon designs, however, will feed reliably with both types of bullet. 
My gun has hollow points right now while DH fires "regular" looking bullets.
A handgun will be enough. You dont want to risk shooting anything but a fox that might have entered your coop. If you shoot a shotgun in this situation there is a good chance that you might hit your chickens or other animals.