I haven't tried this, because  have perfectly behaved hens that never waste a single bit of hay, and carefully make their bed after they leave the nestbox, BUT, what if you cut a hunk of wire mesh to lay on top of the straw so they get used to seeing and smelling it there, but would have to work really really hard to get any of it out, and then take it out after a few weeks, or add a bit of hay on top the wire to test them first? Sooner or later they get used to it, but then never quit messing with it. Even "good" hens will scratch around, move it here and there with their beaks, place bits on their backs, bending it and breaking it down as they go. If you don't replace it they will turn it into strawdust finer than wood shavings over time. Me, I'd just throw out a ton of it and let them be bored of it. Also, maybe hang a curtain to make it dark in the nest box and less fun to play there.