Don't worry about him. I've only had two male birds that ever hurt babies--and I've had A LOT of boys here. At one point, there were nine full grown roosters and numerous cockerels here, and one buff Brahma would go after "teenaged" pullets. My young tom went after and tried to mate some baby turkeys this spring at the height of their breeding season and accidentally killed a couple (son't know which were done in by him and which were the fault of a psychotic neighbor child, but I did see him on one occasion try to have his way with one that way too young, so I know it can happen).
Lessons learned:
1) If the species has a dedicated time to be breeding, immature specimens should maybe be placed in after peak hormone period. Chickens aren't generally so much worse at one time (depends on breed mostly, but Silkies aren't one of those bad breeds about it). Turkeys and Muscovy ducks are about the worst offenders for this type of seasonal activity that I've seen, concerning aggression/accidents. Haven't seen muscovy drakes go after immature hens, however. I've heard game breeds and Sumatra are pretty bad in the spring, too, but they're generally nicer to their girls and harder on other guys, from what I've seen.
2) Roosters and drakes, when significantly crowded and facing enormous competition from other males for a limited supply of ladies are far harder on the hens and are more likely to target girls which are too young and inexperienced. This isn't guaranteed to happen, but it is more likely when lots of boys are around and there isn't enough space or ladies for them. Rape-y behavior is more likely in this circumstance too because lower rung roosters feel like they have to get it quick before head roosters beat the crap out of them.
Silkies aren't generally bad with babies, though his lack of adult company might be the problem point. I've known a number of Silkies to be very good with babies and really appreciate the company. Some are great "dads" and act almost like broody hens in raising babies. Even with stiff competition, my Silkie rooster was the sweetest thing and good with his girl. It might depend on your boy, but I'd go ahead and let him see them through a partition so he can get used to the babies without being able to harm them. Supervise visits until the babies are old enough to take whatever punishment his affection might deal them, should he get amorous, but I wouldn't worry excessively about it. Silkies aren't such enormous birds that they would inflict the kinds of damage heavier, gnarlier species/breeds might (like turkeys. oh my goodness, the carnage that can happen when their mating goes wrong). Your boy is undoubtedly lonely (like my tom was) so he needs to be able to see some other chickens around or he could even get sick and die or develop strange personality traits. Any company is better than nothing at this point, so just get him some friends.
