It depends on the type of appointment. They are probably trying to multi-task.
Example:
Patient A is there for a regular checkup
Patient B is there with Swine Flu
Patient C is there for a follow-up on their latest heart medication adjustment
Patient A can have their weight, blood pressure and current history done by the nurse or physician assistant, then can be put in a room to undress and get the paper gown on. That will take several minutes. Meanwhile, a med tech can take a blood sample from Patient C. While the nurse is handing out healthy-eating pamphlets and the med tech is drawing blood, the doctor can write Patient B a note for work and recommend Mucinex before sending Patient B on her way. As the physician's assistant is reviewing Patient A's history, the doctor chats with Patient C about his symptoms and hands him a pamphlet about low sodium diets. By the time Patient C is out the door, ideally Patient A is ready for her official exam.
If Patient B suddenly barfs on the floor, or if Patient C turns out to have some especially awful complication that requires more attention and an immediate referral to a specialist, you can see how Patient A may be stuck in a freezing exam room wearing only a tissue-paper gown for a while.
Some doctors have switched to what's called "open access scheduling" where they only schedule a few real appointments per day, no more than a couple of weeks in advance, and they just assume that the rest of the day will fill up with not-quite-emergencies--and it does, without fail. This leaves them time to spend with the scheduled patients (because they are not overbooked) while at the same time ensuring that minor injuries and ailments are treated while they are still small problems. My doctor does this, and it is freakin' AWESOME. Maybe you could suggest it to your doc?