Yep, with the picture I can now see clear as day your drill bit is the problem. It's dull and basically tearing through the wood instead of cutting it cleanly. Get yourself a nice sharp new 10mm bit, clamp the piece of wood down securely with a sacrificial board behind it (helps eliminate tearout on the back side), and don't go at it too hard or fast. Let the bit do the work. 118 degree point on the bit works the best in wood... the 135 degree ones can be a bit too aggressive in wood (more suited to metal). It looks like either pine or spruce which you are using, so with a good bit you shouldn't have any problems as it's soft wood.This is how I have been trying to make my fronts.
View attachment 1901184
And this is how they should look fitted. Unpainted.
View attachment 1901185 View attachment 1901186
But this is what's happening.
View attachment 1901187
Either the dowls are too big or the wood on the top and bottom isn't very good. Idk.
That would work great. I'm not sure exactly how big that is but you can get proper fronts for pigeon nest boxes that just attach to a plastic clip that is screwed in.