I know a person who built her goat pen with many trees inside. Mostly cedar. These goats don't have horns, but they have removed all the bark that they can reach. They girdled the trees within a year. If you are fond of the tree, I'd suggest building elsewhere. If not, go for it!
Also, if you build the fence with the tree inside, please do not keep collars on your goats. I don't recommend keeping a collar on an unattended goat anyway, but another goat keeper I know had a boer doe who died because she got hung up on her collar in a tree....