The problems with mice are:
if there is one...there ARE more
they are fast and hard to catch, if you release them outside, they will come back
Get some traps and place a tiny bit of peanut butter on it, you will get him. Wrap a plastic bag over your hand, grab trap with mouse still caught...