I don't blame you for being scared. Pig bites hurt and pigs, even "mini" are still formidable creatures with lots of power/strength and a good amount of speed behind them.
In truth, they can sense your mistrust/fear (energy) and will feed off of it themselves.. so whatever you can do to set your mind in a different place will help immensely. Get a different picture in your mind and focus on that, so it's what you're projecting.. like how sweet they were when you met them or how you want and expect the interaction to go (friendly) rather than what you don't want.
Try to let the past be the past with regards to the undesirable behavior and otherwise scary experience. Do what you can to trust him, while still using the techniques to keep yourself safe.
The following link (down below) has some great suggestions that I would agree with.. Spend some time out there, just sitting and being present so they get familiar with you (on the other side of the fence if inside the pen is too scary, then let it progress to inside the pen as you become more comfortable).. talking to them in a calm voice is helpful and they will get familiar with hearing YOUR voice and having it become comforting. Maybe say their names and tell them they are good piggies, see their tails wag! (as they start to calm down and enjoy your presence) Bring snacks, within reason.. an apple slice, cucumber slice, baby (size) carrot, strawberry, etc.. something big enough that it's easy for them to find when you toss slightly near you.. initially at a distance that's comfortable for you both (even from outside the fence if need be) and start associating good things with this new person and environment.
https://americanminipigassociation.com/interactions-with-pigs/gaining-trust/
Additionally if the female is still enjoying your company and allowing interaction.. let him see you petting her and when he sees that she can trust you, he will feed off of that also. Keep your sorting board still, in case he initially feels the need to protect her.. and keep in mind that he's actually being defensive not aggressive as it may seem. I'm not saying feel sorry for him, just reminding that he isn't a demon that's out to get you. He is just as scared as you are.
Use your words.. a stern low and toned "no" when that's what you mean. I use the word "easy" sometimes, or "gentile".. whatever it is that communicates your intent and that you can remember. They will learn to communicate with you and vice versa.
When you initially start to make contact for petting or belly scratches.. using something like a back scratcher that has an extended reach may help to calm your fears. My husband used to use the dogs' ball tosser as while he was working before retirement was very distant and a stranger to the pigs. It's pretty incredible to see the transformation that's taken place.. they now greet him at the fence, follow him around, and even let him pet them with his hands.
Another technique I use if they're inside their hut and can't see me approaching is grunting to them in a contented grunt sound, which is kinda just a low "uh, uh" sound. I also do it when I see them out on pasture. Here is a link that has some general sounds and meanings associated with them (recordings at the bottom of the page)..
https://www.minipiginfo.com/mini-pig-sounds-101.html
I hope this tips are helpful!
ETA: also remember to move somewhat slow and deliberate. And try to keep a routine so they know what to expect.
