Why just infertile eggs? Having fertile ones don't change the way the eggs taste at all. If you want infertile ones keep the hens away from the roosters.
As for your questions yes, the hens can stay together, the roosters on the other hand, really depends on the bird. Some roosters can be housed together, heck i have four roosters that were housed together with no fights at all. But on the other hand some roosters will try and kill each other when they are in the same area.
How many birds do you have?
How many roosters?