I know exactly what you mean, though my story is different.
I started gaining weight when I was 14. I discovered food, especially sweets. By the time I graduated from high school, I was about 60 lbs overweight.
I discovered I liked the company of my boyfriend better than a pound of M&Ms (which, yes, I COULD eat in one sitting!), and I slowly lost the weight. I have kept it off since then, though there are a few pounds that creep on/get dieted off on a somewhat recurring basis.
People who didn't know me when I was overweight said, "Oh, you don't have to watch what you eat! You're so thin, it doesn't matter!" Ah, yes, it does, very much. And yes, I lost the weight by putting down the fork, eating less $hit food, and getting more exercise. There is no magic. There is just doing the work involved to lose weight and keep it off.
Deal with it.