Horses definitely remember; the bad as well as the good. Sometimes, their reactions can make you cringe . . .
A few years ago, I was doing the evening feeding at the barn where I worked, and a couple of other people were cleaning out a storage area at one end of the barn. This space had gone through a stage where it was individual slots about 2 1/2 feet wide, and one side was made up of a row of little doors. To simplify the organization process, the guys had opened all of the doors. I don't know how many horses I walked past that; most were vaguely curious about things they didn't usually get faced with but otherwise showed no reaction. Then I led in Hitman, an Off-the-Track-Thoroughbred. He took one look and refused point-blank to get anywhere near the storage area. I walked him around, tried again - nothin' doin'! We finally had to close all of the doors before he would come past, prancing and blowing and acting like he thought the devil himself was going to jump out at him. He'd been walking past that area for weeks if not months at that point; the only thing we could think was that with all those doors open and two people moving around in it, the storage space looked like a starting gate to him. We learned to close the doors and get people out of the area when Hitman was coming in, otherwise, his brain fell out. Offhand, I don't think he had much fun as a racehorse . . . .