In my experience with chooks I've bred and raised, and chooks others have bred and raised that I've gotten as adults, yes, it is a problem waiting to happen. Just seeing eachother but not interacting freely has only a limited amount of social value. Attitudes will change when there are no longer any barriers.
If you keep your different gender and age and breed birds separate as a rule for a good few generations, you risk developing a strain that loses their ability to tolerate and safely interact with birds of different ages/genders and appearances/breeds. You can breed a rooster who doesn't know what a hen is, and hens that don't know what roosters or babies are. You can even breed roosters who are attracted to roosters only and think hens are juveniles or a different species. And mating is distressing for some hens who don't have a clue what a rooster is.
There's no guarantee that will happen but I have had all those problems from birds others raised in artificial environments. Interestingly, non broody birds can learn from watching, and clueless roosters can also learn from watching. But that generally takes about a year.
I'm a devotee of the as natural as possible social and lifestyle method of keeping them. You can have a peaceful flock of all ages, breeds, genders, species, etc, it's not impossible by any means.