Ok, this is idea came from @onaharley.
It just slides in from the top and goes in and out easily. The barn wall isn't square so it doesn't fit exactly which I didn't take into account while building the divider. Not perfect but it serves the purpose.