What kind of startup costs would I be looking at to raise a small herd of goats?

Honestly, the best thing for you to do is to price it out and do your research. My guess would be at least a couple thousand, but it depends on what you're doing for material, how much you can get of used materials, if you're going to buy new, what type of goats, how much that type goes for in your area, etc.
It may be an idea to keep an eye out for someone getting out of goats - selling their herd. Usually they're discounted a bit to get them gone or to get only a buck and two does and build your own herd since it's a lot easier and cheaper to replace a buck than it is to buy does.

 

The price will depend on the quality and kind of goats you want -plain ol' weed eating goats, boar type meat goats, dairy goats, etc. You could start out with two bred female (nanny) goats. Those goats will probably/possibly give birth to two kids each, then you will have six goats. Keep all the females you want and feed out all the rest to eat or sell. Then by the time those kids are born have a nice male (billy) goat either bought or borrowed and breed your nanny goats back when they are ready.
Since you don't own the land I would buy 2 rolls of the Premier brand electric netting and a solar charger. I don't own any of it but friends say it is easy to move around to new pasture when needed or you could pack it up and take it with you when you need to. Lots of people use it for chickens.
For housing about anything is good if you will just keep them dry and sheltered in the winter. I would buy a used dogpen, put a cattle panel roof on it and cover it in something like good tarps.
If the neighbor's place is getting overgrown you ought to be able to strike up a good deal with him. Goats are wonderful for clearing land. They love to eat weeds, briers, saplings, bushes, etc. For feed, anything will do such as a tub or pan of some kind. For water, again, anything. Just make sure it's not deep and they can reach the water and the babies not drown in it by falling in and can't get out.
If you had to buy all of that I would guess at maybe $1,000.

 

If you were to fence off one acre is a perfect square shape, the perimeter would be 834 feet. If it were a rectangle shape, it would be closer to 1100 feet, and a circle would be less. So... let's figure 900ft on the perimeter length. For a six wire fence, you would then need 5,400 ft of wire to do all the fence.

14 guage wire, 1/2 mile spool = $50 x 2 = $100
14 guage wire, 1/4 mile spool = $25
Total Wire cost: $125

If you set your t-posts around 8ft apart (that's what we did and it made a very sturdy fence that the goats only went through if the electricity was off) you would need : 900/8 = 112.5 t posts.

115 (for good measure) x $4 (roughly) = $460

You need 6 insulators per post (one for each wire) which would be 115 x 6 = 690 insulators.
They generally come 25 per bag, so 690/25 = 27.6 bags. You can't buy a partial bag, so you would need 28 bags of insulators

28 bags x $4per bag = $112

You also need a fencer.... plan to spend around $200 on the fencer, battery, and/or solar charger. Do you have electric close by or will you need to use solar?

Do you have a post driver or a tractor of some sort to drive the posts? What about fence pliers? I'm not going to figure those costs in, but if you might need to.

Will you need to build a gate? If so, you will need to figure in a few more dollars for gate hardware.

To get a basic 6 strand 1 acre electric fence up, you're looking at:

Wire: $125
T-Posts: $260
Insulators: $112
Fencer: $200
Total Fence Cost: $697


Ok let's talk about shelter... If you're wanting portable, I would purchase a polydome calf hut. Either 1 large or 2 small ones for 5 goats. They would run you about $250-$350 in cost.
If you are going to build a shelter yourself, I'm not real sure on price. For now I'm going to put shelter at $250-$750

Feeders and Waters.... You will need some sort of water tub. A nice sturdy one will run you $50-$75. Hay feeder... if you can build it yourself I'd say around $100. If you purchase it, I would double that price.

Goats.... I HIGHLY (and I cannot stress that enough) recommend you don't just go buy whatever you come across at the auction. I can't give you an exact answer on price on goats until I know what kind you want. Registered dairy goats? "Just some goats"? Do you want ones you can also milk? A ball park estimate would be $500 - $1500. Do you plan to keep a buck? If so, you will need a separate pen for him, as it is not good practice to run a buck with your does all year round. If you are willing to buy bred does, kid them out, and keep a couple of their doe kids back you could get by a little cheaper. Just please whatever you do, buy from a reputable breeder. They will cost a little more to start with, but they won't come with all the possible diseases that you will more than likely get with auction goats.

So a rough estimate:

Fencing: $697
Shelter $250 - $750
Waterers/Feeders: $150-$275
Goats: $500-$1500

Totals: $1597 - $3222

This number does not account for things like a milk stand if you plan to milk, a basic first aid/medicine/vaccines/wormer stock to have on hand, buckets, hay, grain (if necessary) minerals (100% necessary) mineral feeder, etc. It also doesn't account for being able to find some of these items used.

Goats are wonderful, and I highly highly recommend them. I just don't want you to be surprised at the hidden costs associated with them and have to sell out if something goes wrong. Good luck.