The one who is by himself will be lonely- that is natural, and so I am happy that he can see the others.
I only integrate chicks that are brand new with those 2 weeks older, not three (have tried it unsuccessfully). So, I'd give the new one about a couple more weeks to grow and then try it. The reason I say this is that at age three weeks, chicks come into the ability to find their way in and out of my shed coops to their heat lamps.
Before three weeks of age, if I allow them to go outside they stand out there and freeze...I have to rescue them. There is a little light bulb that goes on in their heads at three weeks of age and they become much faster - would be more likely to keep up with the others. Now if you do see him getting hurt as you integrate pull him out and give it more time. They will peck each other of course, but what I mean is if the younger one isn't able to get away or isn't standing up for himself and looks like he might be killed. That isn't good.
Another thing you can do is take one of the older ones and put him in with the lone chick, in a week and a half or so, and see how that goes (watch carefully that the little one doesn't get hurt). Then maybe they will be friends by the time you remove the divider.
There are many right answers to this question you have posed...everyone does things a little differently. If the breeds you have are very docile you may find that they can be integrated a little earlier. If they are aggressive then it might have to wait until longer. Individual hens also are different. So it makes it hard to say exactly. But this is what I'd do given what you said.
My three week old Leghorns are kickboxing the others in the brooder right now on occasion. My Cochin chicks are so sweet and docile, running around minding their own business.