Watch the temperament. You don't want a wild, crazy, fearful animal that you can't handle.
Look for clear nose, eyes, ears - shouldn't be any crusties, milkiness, or other abnormalities.
Excrement should be normal not runny or off-colored
Check the structure, make sure the goat is sound, no limping, hooves are neat and/or trimmed, etc.
Make sure other goats in the building/area are healthy and well-kept.