Goats are trouble. That's a fact. You need to accept that to keep goats whether it's 2 or 20. I can nearly guarantee that your goats will continue to test your patience, and their enclosure. That's what goats do. I'm not being negative, it's just the nature of the critter. They are smart, and like monkeys. Always testing stuff and always putting stuff in their mouths. They also like to rough house and bash each other about.
So decide if you can deal with that. More goats means they will need to figure out the pecking order. Eventually they will bond as a herd. Goats can make good pets, but in the end they are livestock and need to be treated as such. That's just me opinion though.
Gotta decide whether you really enjoy goats or not. They certainly aren't for everyone. Every goat is different. Some make more troubles than others. Most eventually settle down and into routine once you get it figured out. A happy goat is a good goat, usually.