Diseases and parasites can be spread by them coming into contact with each other, eating or drinking from common containers, pecking at the ground where others have pooped, by you carrying stuff on your clothes (especially shoes), by various insects (including mosquitoes that fly), or just carried on the wind. It can be very difficult to truly quarantine them.
It is possible that you could infect your flock but if the other flock has not been in contact with other chickens for a while they are essentially in quarantine now. That's if you'd trust the people to notice if something is wrong and are honest enough to tell you about it if they see something.
Everything in life is a risk. If you stay in bed to avoid risks you are at risk of getting bedsores. It is your flock, you are on the ground looking at the situation. It has to be your decision. I think your risk is pretty low but I can't tell you what to do.
Size has very little to do with them being picked on. It is not unusual for bantams to dominate full-sized fowl. Different maturity levels are your biggest risk. If one group is more mature you may need to look at them as two separate flocks, coexisting but not mingling until the younger mature. But if both groups are mature they should sort out the pecking order and merge.
