Maybe. It [vermiculite] has asbestos sometimes.
Rice Hulls.
Link to info
I found a paper on how rice hulls are not as effective as vermiculite or perlite but it was put out by suppliers of perlite.
Link.
Their main point was rice hulls are flatter so don't provide as much benefit to soil structure as the round perlite or more angular vermiculite.
So I think buckwheat hulls may be a little better than rice hulls but haven't found info on it. Buckwheat hulls might be hard to find as only hulls but buckwheat seeds are inexpensive, more available in the north than rice hulls, and include the hull. I think you could bake the seeds to keep them from sprouting in your potting mix. I would be tempted to try them viable and either use mulch or weed them as they came up.
The link to the perlite company has a nice chart on the second page about the properties of several options.
I know the bags of vermiculite I purchased years ago had a banner claiming it was asbestos free. So, I was not concerned about that - assuming they test the vermiculite, and it passed the testing to be asbestos free.
I had not considered rice hulls. Probably because I have never seen any sold around here. However, I looked online and my local big box stores do not carry rice hulls. At any rate, that second link to "properties of several options" was interesting. Just yesterday I was sweeping up lots of saw dust and thinking to myself that sawdust might be useful for something, so I put it into a pail. Turns out, at least on the link you provided, saw dust has about the same properties as rice hulls.
I only have about half an ice cream pail full of sawdust at this time, but I might as well blend that in with any homemade potting soil I mix. I don't think it would hurt.
Also, a couple years ago when I was loading up free wood chips at our local county landfill, there was a big pile of saw dust there as well. I had no idea why anyone would want to take saw dust, unless maybe they use it as some kind of organic absorbent material. But I think I'll check out the landfill again this spring and maybe pick up a big tote of free saw dust if I find mixing it in a potting mix blend works for me. Free is good!
Looking at the chart, it convinces me once again that vermiculite was probably the best option for blending into a potting soil mix. I mainly wanted something that would retain water, but I was surprised to see that vermiculite also adds K & Mg to the potting soil mix.
Finally, let me mention that I have all kinds of chicken run compost that is fantastic in my raised garden beds. But I don't want to bring that compost inside the house because it is full of life and I am certain that my house would be full of bugs in no time. But maybe I could use more compost in my potting mix for outside planters on the deck.

I am still learning about this stuff, and just when I think I have a few answers, I can no longer get the stuff (inexpensive vermiculite) that worked so well for me.
Again, thank for the links and good info.