When I was helping my mother breed rabbits we just let the buck and doe live together to make their nice little family. If you have nice bunnies and plenty of room to run or move around in their hutch then aggression issues shouldn't develop. But if he trampled the babies, or was pushing them out of the nesting box - which they always had as a place for them to go hide, feel cozy, and safe - than that was the only time that we would ever remove the boys. But understand, that was very rare, only one or two of them ever did that out of say... 20 bucks?
But if your hutch is only so small for the one rabbit and her babies so that they have to live separately, then you only need to leave the rabbits in together for a couple of hours. If you see them do it once or twice before your personal time frame of leaving them together is up then it should be safe to remove them. All it takes is just once, which is why they have the saying "breed like rabbits". If you don't notice any pregnant behavior such as growing a little fat, maybe attitude changes, collecting nesting material in their mouths, digging in their nesting boxes or removing fur anytime soon then you can always just put the male back in with her again and wait a little longer.
When the babies get to be about 6 weeks, or as soon as they are weaned, then is when you should take the time to make sure separate them to avoid any inbreeding as the babies will develop vary fast and be able to breed with their siblings/ mother. Even though they may still look like babies.
