I'd wait until they're bigger so they can hold their own. My neighbor's cat batted at my smallest hen and the other seven literally chased the cat off. He hasn't been back since. It bears mentioning that my hens are standard size. When you do allow the silkies out, I'd keep a close eye on them until you know how it will play out.
Is the cat actually your neighbors pet or is it a stray that has been hanging around or something? If your chickens are legal and their cat is trespassing on your property and threatening your birds, they really need to contain the cat. Mixing a cat that hunts and six week old silkies would worry me. I would definitely keep them in their coop for awhile longer, and if you do leave them out keep a close eye on them.