Yes, the damage is caused by an insect like the plum curculio:
http://en.wikipedia.org/wiki/Plum_curculio
There are several methods you can use to prevent.
I use wooden balls (actually wooden eggs) painted bright glossy red and coated with sticky trap goop. Any insect which lays its eggs on fruit are attracted to shiny colored spheres and can't take off after contacting the goo.
I also use old Clorox/style jugs hung up in the tree with a baseball size opening cut into the top and a pint of apple cider with a teaspoon of molasses in the jug. Insects smell the fermenting odor and fly in and drown.
You can have your fruit trees in the barnyard with calves and chickens, the calves rubbing against the tree will cause the Plum curculio to drop off and the chickens will eat them.
Or you can get advice from the agriculture salesman, get a insect fogger, fog your orchard once a week killing everything that walks, crawls, or flies whether its a good or bad insect, poisoning the air, water, and soil. ( and yourself and family)