It is common for does to lose their first litter. Some does won't mind if you handle the babies, others will be more territorial. You know your rabbit best, so if she is getting aggressive towards you then slow down on the handling for a few weeks. Otherwise, it won't harm anything.