I don't buy soil if I don't have to. I certainly don't have the best soil in our yard, but I've had to regrade parts of the yard, clear old flower beds, and dug out a pond. That left me with a lot of dirt to relocate. Nearly all of it is in our garden beds and raised rows. That said, I've added several yards of homemade compost over the past few years. I also pile on leaves in the Fall.
At the old house, I had beautiful beds. They were smaller as we lived in the city instead of the suburbs (my lot now is at least 3 times the size of the old one), but very productive. I filled those with some logs from a small tree I cut down and then cheap top soil, mushroom compost, and composted steer manure. All of this was the cheapest possible from a big box store. Those beds grew very well for us. Herbs, tomatoes, cucumbers, the best harvest I've ever had of zucchini and yellow squash...
At the end of the day, top soil is top soil. It's dug up, dried out and dead. It should be loose, but even that is typically wrong about it because a little bit of clay always seems to find it's way in. Garden soil is that dead, crappy top soil (I really hate the top soil...) amended with organic matter to give it structure and life (bacteria and fungi).
I say take you dirt from wherever you can get it, and mix it plenty of organic matter. Don't go pure compost (too much of a good thing is a bad thing as already discussed), but put as much compost as you can into your soil and your plants will be happy for it.
As a side note on this ranting though, pay attention to the needs of the plants as the grow and understand what you are growing. Fertilize as necessary with sources of calcium (egg shells, lime, epsom salt, compost*), phosphorous (bone meal, urine, shellfish scraps, some manures, egg shells, compost*), potassium (kelp/seaweed, wood ash, banana peals, compost*), nitrogen (compost, manure, compost/manure tea, urine, coffee grounds, nitrogen fixing plants, etc) and magnesium (compost, epsom salt, lime). Note that all these key elements can come form compost, with the only catch being the compost was made from matter containing the elements to begin with. Kitchen scraps and chicken manure take care of that for us. However, you may still need to give an extra boost to plants as they grow by putting appropriate nutrient sources around them.
Just my overall $.02, take it or leave it.
