Place the trap up against the side of the coop, with the trigger mechanism (which usually is a wire bar that runs on the outside of the cage) up against the wall. This prevents the critter from snapping the trap shut from the outside. If your pets are not able to get to the trap, you want to bait it with something meaty and smelly. If your animals can get to it, try a cob of dry corn. Some people go for peanut butter/marshmallows, but dogs like PB. Little pieces of stuff (like cat food or corn kernels) aren't the best choice b/c mice can get in there and trip the trigger.
If you find that the bait is missing and the trap is snapped, sometimes a varmint bigger than the trap has gotten in, and the door wouldn't shut all the way, allowing it to back out.