It sounds like Carpenter bees not bumble bees, google them and you will find several ways to get rid of them. The dusting the holes at night (so sting risk is lowered and they are in the hole itself) and sealing the holes with putty/caulking worked for us. Some people spray them in the air with wasp killer but my concern with that was my chickens might eat a fallen one. Our Carpenter bees were on our shed also, lots of them and we had to replace a length of 4x4 because of the damage they had done. First I tried putting tape over the holes thinking they would leave if the holes were not visible but they chewed right out or back into those holes! At the local garden store they told us which dust to buy and about sealing the holes. A friend who had them said to use wd-40 and push that long tip right into the hole to spray and then seal so we did both, wd-40, dust then seal. We have not had a problem since and we felt bad about it but the darn things were turning our shed into swiss cheese and were attacking anyone who went near the shed.
Anyway that was our experience with the things others here might have a better solution for you, good luck.