Your case is very dependent on your zoning laws. Here we must have our animal fencing at least 100 ft from the property lines so not to be intrusive on the neighbors. If your fence is right on the property line and they are sticking their heads through the fencing into his yard, technically your animals are trespassing on his property. If he had poisonous plants on his side of the fence and your animals ate them he would not be liable, you would be at fault.
There may be some legal room if the millet that he is spreading on his side of the fence is poisoned, that is unlawful as any animal, whether yours or another neighbors, could be harmed, in addition, any young child could as easily be poisoned. This is where you may be able to get him legally, not through the poisoning of your animals, but the potential poisoning of your children.
Good Luck.