I think that's probably ridiculous. Practice good hand washing and keep the baby away from the brooder and the birds and you will be fine. Besides, babies love watching animals and there is nothing quite as entertaining as chickens.
If it makes you feel any better, I have three kids, 10, 3 (almost 4), and 18 months. We have done just fine and no one has gotten sick.
I have two girls, a 2 y/o and a 5 m/o. my 2 y/o helps me in the coop all the time and she was always playing with them in the brooder. my 5 m/o never touched but she does like to look @ them including peeking into the brooder when the hens were in it.
I don't know what to tell you, my wife is a germophobe and has never been to worried.