Oh, goody! One of my favorite topics!
There is not a one size fits all answer to this. In order to decide, you need to ask yourself some questions.
1) Do you have a source for high-quality birds that are free from defects and generally performing and appearing as you want them to? (It is expected you would want to continue to improve and fine-tune any birds you acquire.)
2) Does the breeder of those birds have an established line with no recent outcrosses to other lines?
If the answer to both of these is YES, absolutely DO NOT outcross to another line. Crossing established lines is as good as mixing breeds, you'll create mongrels with no consistency.
If you only have access to inferior stock with issues, day one is as good a time as any to find your parts birds. If you can find only one good bird, get it and line breed the best of the worse to them. (Male or female.)
I want to emphasize this again because the "you should add some genetic diversity" myth I see perpetuated is RUINING good stock—crossing established lines is as good as outcrossing to other breeds and breeding for mutts. It can take years and years of linebreeding and family breeding after a single outcross to get back to consistency so you can really dial in your birds. Good breeders establish their flock (preferably by acquiring stock from an established line) and work on it for decades without adding new blood. The higher the inbreeding coefficiency the better when you are working with sound birds.