With horses there is considerable disagreement among horsemen (and among vets even) whether eating black walnut leaves or nuts or husks can be a problem... basically what it boils down to is that some horses are out there nibblin' on 'em with no obvious problems but others develop problems that are awfully suggestive of walnut being the cause. (Walnut shavings in bedding *definitely* cause laminitis/founder in horses, there is no dispute about *that*).
With poisonous plants for sheep/goats it is in general a similar situation overall -- some people have animals get sick or dead from eating <plant X> and others have had their stock eating it for years with nary a problem.
Basically in my opinion what it boils down to is something like this: it's YOUR call. Plants with a history of possible-poisoning of stock are probably more of a crapshoot than plants that everyone agrees are entirely safe. That does not mean that they are *necessarily* going to cause your animals a problem, only that there is probably a higher *chance* than for something like, I dunno, orchardgrass or birdsfoot trefoil.
Whether you choose to avoid the risk by avoiding the plant, or whether you choose to just keep a close eye on the animals and see what happens and hope it's nothing bad, is totally a personal choice and I do not think there IS a one-size-fits-all correct answer.
JMHO,
Pat