I don't know about fruit trees, but if you have any cats, you NEED to keep it away from them. Since cats lick themselves clean, they could get very sick if they ingest the permithrin.
I've used it for lice, worked like a charm! Wear protective gear (gloves and mask, maybe even long sleeves)...