I used to have a humming bird feeder and every morning around six, a pair of them would be drinking, and flying around. Then land in a tree near by, then come back. They did that over, and over again. Then later on in the afternoon, and evening I'd see little finches on the feeder. I don't think their beaks could fit through the holes to get the sweets though. The best time I could catch the humming birds was early morning.
You have to use a type of feeder with deep holes in it. I have the same problem with the house finches raiding the hummer feeders. Look around in the stores and see if you can find some deep holed feeders. And ones that have no perches or places for finches to stand on. They can't cling to angles and such on the feeder.