If you want tender meat, and as much of it as possible, then yes you have to picky about when you butcher. But they are edible at any age and any size, as
@Ridgerunner said.
Young ones will be more tender than old ones, while old ones are more likely to be bigger. Either way can be better, depending on your personal goals.
You can also consider whether one time is more convenient for you than another. You might be extra-busy one week, or want them gone before they crow, or have some other reason to just butcher them at a certain time regardless of size.