I've known some really nice intact breeding boars, but they were constantly interacted with. I have met only a handful of intact potbellies and they were rather docile as well. That said, being an intact anything comes with some potential behavior issues and it can be more difficult to castrate a pig the older they get. He may be fine but he may not and you just need to factor that into your decision.
If you are not planning to breed him, I would probably just go ahead and castrate him. That way if you ever decide to get a female in the future you won't need to worry. Otherwise, I recommend chatting with some local breeders and see what their recommendations are.