You only have to file if you make a certain amount of money.
I really don't know WHY I know my SSN, I've just known it for WAY longer than you have been alive. Could be that before all the issues with identity theft, your SSN was used for a ton of stuff so maybe you just used it enough to memorize it. I'd bet that neither of my girls (23 and 25) could give their SSN.
When I was in college it was used as our ID number. That is when I memorized mine. I had to give it out a jillion times.