As long as the oyster shell is free choice and not mixed into the other feed, it won't do any harm to have it available, even when hens are eating layer feed. They will just ignore it if they don't need it.You’re feeding layer feed and oyster shell? That’s a lot of calcium for older hens. I would second switching to higher protein feed.
But I agree about switching to a feed with higher protein and lower calcium. I would probably buy the other kind when the current bag runs out, not throw away the current feed.