I agree with ChatyChick and birdnutz. If the foal is past the bottle stages and is drinking water on his own, milk isn't something as desired by him anymore and he will be fine if he doesn't drink it. Also, he may be nervous. Horses by nature are herd animals and in times of danger they tend to band together. Most foals nowadays are imprinted by birth and come to picture their human handlers as part of their herd and thus forms a bond between them. The foal may be slightly more timid around you if he is not use to you. According to him, his herd members left him and he's not sure if you are a fellow "horse" that will take him in or a predator.
I would reinstate the "friend factor" rub him along his crest (the part of his neck where the mane grows from)... most mares groom their foals along there so it'll bring him some comfort.
I'm sure he'll be just fine.
We lost a mare about 2 summers ago at our barn... she died 2 days after she foaled and we had to take care of the colt too. First the milk in the bottle, later transfer the milk to a bucket so he get used to drinking out of that, and finally just water. He's doing just fine and has started training.
Good luck.