Absolutely, I enjoy goat's milk much more than cow's milk. Nigerian Dwarves. And raw (unpasturized).
There is nothing wrong with drinking unpasturized milk as long as you keep everything incredibly clean while milking. Keep your hands clean, use sanitized stainless steel to milk into, sanitize the udder before milking (keeping the goat's udder clipped free of hair reduces hair in the milk), sanitize udder after milking (use different cloths/wipes, obviously), filter milk, and store milk in sanitized glass jars.
Friendly hint: the goat's milk sold in stores tastes NOTHING like fresh goats milk. It is rank and nasty. Actually, the taste reminds me of how an unaltered goat buck smells. So if you've tried that and are disgusted, just know it isn't how fresh tastes. The fresh milk is delicious, sweet, and rich.
Actually my husband insisted that we try goats milk before he will take my "dairy goat dream" seriously and the milk we got from Trader Joes is delicious! I posted about it on Facebook and a friend kept insisting that every goat milk she's tasted was gross and "goaty" flavored or like "grass" and that she had tried several kinds. Idk?? Ours tastes similar to cows milk, but it is pasteurized so I'm guessing that changes the flavor a little bit. My husband is SO picky so her comments made me nervous.