If the kits are eating plenty of pellets and hay and drinking water on their own, and the doe seems fed up with them, then I would wean them now. I've raised two litters of rabbits before (with another on the way), and both times weaned the kits at 4-5 weeks of age. By that age, mine at least (and it sounds like yours, too) are hardly drinking any milk and their mother is miserable. I didn't lose a single kit weaning like this, and the does had no problem with mastitis.
I wouldn't necessarily recommend removing the mother and putting her back again. The kits are past, or nearly past, the age where they need to drink milk. The doe should be virtually dried off. If you remove her, she may not recognize her babies and may get annoyed with them or even attack them. If I were you, I would remove the doe permanently. Check her nipples every day for the next few days after removing her just to make sure there isn't a ton of milk or signs of mastitis there, but the kits and her should be fine.