Yes, you should be concerned about going from someone else's coop/flock to yours.
Illness can be spread on shoes, clothes, equipment, dust/dander, etc.
Have a bio-security plan in place. The best thing is to have a pair of shoes and clothes for your neighbors flock only while taking care of...