or were they too tender when I was trying to harden them outside?
I had excellent results starting my tomato plants inside the house this year, 8 weeks before my last frost date, which was something like the 29th of May. On nice days, I would start to take my plants outside for hardening off during the daytime.
I watched a YouTube that discussed what happens if you try to set out your tomato plants too early. If they don't killed by a late frost, they might get severely stunted in their growth. Bottom line, the recommendation was just to wait until your average last frost date has passed and then transplant your starts.
Also, probably in that same video, it mentioned that peppers should not be put out into the garden until 2 weeks after your average last frost date. So, this year I waited until mid-June to transplant my peppers. It was the best year ever for my peppers, but I think a lot of that also had to do with my new raised beds with fresh topsoil and compost.
All lessons learned...trying again next year.

