Is there a washer between the head of the pivot bolt and the wood?.....it doesn't look like it.
That's part of the problem.
The pivot lever is not tight enough, that's why it's wobbling....wheel is probably wobbling also, so the misalignment is exponentially worse.
You really need to use bushings in the pivot lever where the bolt goes thru so you can tighten the bolt but not impede the pivot action, probably have the same problem where the 'axle' of the wheel is mounted.
That's part of the problem.
The pivot lever is not tight enough, that's why it's wobbling....wheel is probably wobbling also, so the misalignment is exponentially worse.
You really need to use bushings in the pivot lever where the bolt goes thru so you can tighten the bolt but not impede the pivot action, probably have the same problem where the 'axle' of the wheel is mounted.