With proper management you can get away with it. Make sure you've plenty of roosting space, may have to add another pole and make them the same hieght. To add some space to your roosting area you could mount a nesting box on the outside of it and take out the nesting area inside. I guessing your nest are inside.
I've a considered crowded area too, my birds are unstressed and produce exceptionally. The only time there was some pecking was when I was ill, asked one of the kids to open the coop before going to school only to get out of bed at 10 am to find they were still locked inside. That's what I mean by extra management, they can't be locked in coop/roosting area much after daylight.
And be prepared for a harder time acclimating new comers to a smaller coop like that. The others may not like them in the coop until they are better accainted. I introduced 2 to my flock of 5 (small coop like yours and even smaller run). We had to allow free ranging every day. It took 2 weeks before the new comers went into coop at night with the others and would have to allow out free ranging all day for a month or pecking would commence.