My bread recipes say to put the sugar in with the yeast and water. You did not mention you did that might be the difference? It is sweet bread so I am sure there was sugar. I think the sugar help the yeast bubble they react to each other in some way.