IMO either would be fine - those who use vinyl usually add it with adhesive like a kitchen floor and when need to clean, remove the litter and wash the flooring.
But Rubber Mats without holes could be removed to clean - the only drawback I can think of is that the mats would leave a hiding space for bugs and pests under the mats - roaches love to hang out and breed in dark areas. So, the vinyl would be the better of the two.
I have just a plywood floor, but use sand in the coop - remove and replace the sand once a year. Add PDZ if sand gets damp areas. And the flooring has thus far remained clean.
I used a single sheet of heavy duty foamed back vinyl flooring, no glue, goes up wall 3-8 inches and is tacked in place. I tested several kinds to make sure they wouldn't crack when bent before buying, and the thicker vinyl with foam backing fared much better than the thinner ones with paper backing. It's held up well for 2 years now.
Really depends on what your floor is made of and how you plan to manage manure and 'clean'.
I use 4 x 8 foot black stall mats, they are heavy, but these are about 3/8s inch thick and have a slight rib on one side and a bubble rise on the other. I take them out and wash them in good weather and just clean in place in bad. I love them, my old knees appreciate the bit of give when I need to get down and it helps me with a insulator between the frozen ground and the not frozen feet of my birds. I think the reference to 'heavy' mats is directed toward the 3/4 inch solid ones, I have one in the back of my truck, they are very heavy.