Since I am EVER the questioner, here...and like to occasionally play Devil's Advocate...
I can understand expecting a man to remove a top hat. After all, it could block people's vision. I can understand removing a cowboy hat. It definitely blocks people's view...of whatever there is to see. If it's a trail worn hat, then it might be dusty & dirty. That's not very pleasant to have at the table. Hats used to be worn to help protect a person from the weather, dirt, dust, etc.  I can see removing hats that would be dirty, dusty, or block vision.
However, a CLEAN baseball cap is not something I'm going to concern myself over.  I view a dirty baseball cap in a matter akin to dirty clothing.  If it's a fast food place, and these guys have been out working & are just grabbing a quick lunch, then I'm not going to worry about it. However, it's not appropriate for a restaurant where clean clothes are required.  A clean baseball cap, though, in any place where jeans & a t-shirt are okay?  Why not?  It's not blocking your view of anything, so long as it isn't pulled way down over the eyes. 
Women's hats  - many of these have been worn as a fashion statement for a long time. They could also conceal dirty, grimy hair in a period when hair got washed once a week -- or once a month. I suspect that as much as anything, this is why it was acceptable for women to wear them inside.  A scarf or hat covered up stringy, greasy hair.  Obviously, a scarf or warm hat could also serve to protect from weather, dust, etc.  and sometimes has religious reasons.  
I see little difference today in continuing out of date traditions.  Unless it's a religious observance, what difference is there between genders when it comes to wearing a hat inside/outside? So long as it isn't blocking vision or creating a hygiene issue, I don't care either way.
Then again, I see no sense in arguing over who holds the door, either.  I hold the door for people - men or women -- and expect others to do the same as a courtesy for me, at least if my hands are full. Letting the door shut in someone's face is rude regardless of your gender.  (It's not like we are trying to hold and manipulate big hoop skirts through a narrow doorway anymore...)
Cultural traditions change. This is just another area that is shifting...and probably one that doesn't matter all that much in the course of world history.
Edited for typo