I thought the same thing about people who have not read the books. My husband was able to follow along fairly well since the rest of us in the house are huge HP nerds, but he agreed that it would have been kind of confusing if he hadn't heard us talking about it all the time. I just asked him about the mirror shard and he thought it was maybe a piece of the mirror of erised, so that definitely needed to be explained further. I do wish they spent more time on wandlore as well.
The changes I didn't care for were:
I wish they hadn't changed the part where Wormtail gets strangled by his silver hand.
I didn't like that they had Harry, Ron, and Hermione walking around instead of disapparating.
How are they going to explain how Snape knew where to put the sword and send the doe since they didn't have Phineas Nigellus' portrait with them.
I thought a great part of the book was when Harry realizes what Kreacher has been through and they become friendly, I wish that had been in the movie.
All in all though, I liked it much better than the last one. I thought the last movie was awful, this one was a nice improvement.