We've had lots of bun's. Three of them were rescues that were found running loose. My husbands old shop used to back up to a bad neighborhood and loose rabbits were common. One arrived at the shop one day, came right to him and sat on his boot! Another was purchased at a younger age from a pet store. Our current two were bought from a breeder at 8 weeks of age.
All of these rabbits became very nice, friendly pets. Two of the adults that found were running loose were a little scared at first but they settled right in and were completely friendly. The two from the breeder are the same, very sweet, very friendly, you can do anything with them. All have been does except for one but the buck was friendly too.
So I think you really can have a perfectly nice pet no matter what age you start out with unless the bun is a nasty, grouchy one to begine with. There's bad apples in every group but generally, with consistent good care and handling, you should have yourself a nice pet.