That's kind of what I'm doing. I have cinderblocks all the way around, when I get it built up enough to be near the tops of them I will add another layer of blocks to a height of around 16 inches above natural level.A suggestion worth pondering..
Consider making your garden in raised beds.
Advantages; easier to control weeds. The perimeter will control the erosion of soil inside the contained area.
You also can use those horse tennis balls to enhance the nutrition source for the garden. Amish do it all the time.
I just don't want to fork over the money to fill a 4x8 bed from scratch with potting soil so I'm adding a bale or so of peat moss and whatever compost I generate each year until it is built up. I will be gathering manure this winter and letting it rot down until next fall and adding it to the beds then.