Only doing weekend, once-in-a-while chores at other people's farms and at my aunt's house.
They stink. A lot. I would put the pigpen area as far from any human habitation as possible. Unlike chickens, which only smell when they are dirty and not being kept properly, pigs smell all the time no matter what. There is nothing you can do about this either--pigs don't sweat, they need a mucky area to wallow so they can cool off in the summer. The wallowing area will stink and have flies. Their pen will stink even if you are pretty diligent about scooping poop. They poop a LOT. I mean, A LOT. You will never, ever keep on top of it, you just have to kind of resign yourself to the fact that when they are bacon, they will smell better.
They bite. Sure, they are pretty smart, they can be trained to be somewhat gentle and come when they are called for food. If you raise them from piglets to be gentle with humans, they might not bite. But if you are not handling them much, or don't want to be too attached to a food animal, they can get bitey. For this reason, you will want a sturdy fence and be assured that the neighbor kids aren't going to want to climb said fence to pet the cute piggy.
On the plus side, they eat everything. Kitchen scraps, acorns, grass clippings, bugs, other pigs...Obviously not an ideal diet, but you can feed them just about all your table scraps and garden waste, and they will love it. Downside to that is obviously, they eat a LOT.
They are not hard to butcher yourself, if you've got the stomach for that. The skin separates from the meat really easy, compared to a deer, because of the fat layers.