I feel your pain. I still strongly remember a gift exchange at school as a child back in the mid 70s. We were dirt poor, but Mom somehow came up with a new toy for my person (I think back then the cap was $2.00...lol). I ended up with a cheap, hastily wrapped ball point ink pen (not in a package or anything). I was honestly crushed, and barely kept from crying. Years later I realized that the child probably did not bring a gift, and that the teacher probably pulled that pen out of her desk so that I would have something to open. Because of that, I've never done gift exchanges with my school kids; it's just so unfair to some.
My DH's family does a gift exchange. It helps that each person makes a list of the kinds of items that they like, but we have still opted out this year due to our budget. At $25 a gift, that's $75 extra to spend at this time of year.
I think Secret Santa-ing can be fun if it's among a small, tight-knit group of people who know one another pretty well. But when it's among people you really don't know well, it loses something for me...