The drawbacks are that sawdust is poor insulation (low R value) compared to fiberglass batts or rigid foamboard, and it will settle (i.e. become worse and worse insulation value as time goes on), and when it gets damp it will be even *worse* insulation and encourage rot.
If it is sawdust or nothing, I'd go with sawdust, but if you are in a climate where insulation really matters I'd pick something with a better R value.
At the very least, if you use sawdust do your UTMOST to prevent moisture from being able to get in there, and maybe make the top of the inside wall panels a separate piece so you can top up the sawdust next year after it's started to settle.
Good luck, have fun,
Pat