As I suspected after Googling myself, there is nothing wrong with giving them the vanilla bean. It is not toxic but should only be given as a treat. Considering you aren't adding this into their diet long-term, there's no reason you can't let them have some if you want to do so.