Unless your kids are using the playset (and thus the structural damage could result in an injured child) I'd leave them alone.
Every year they try to infest the frames of my decks, and this year they've been nosing around the coops and run. Any other wood around the property I leave them alone...