It's probably him, but that doesn't necessarily mean he's the problem. Chicken sex isn't people sex - it's basically an exchange - the rooster gets to mate with the hens in exchange for helping them find food, watching the sky, etc. If they don't let him mate, they don't get those services - which means they get killed, chased out of the flock, or the rooster keeps attempting until he succeeds.
By any chance is this an older hen - maybe one who was high in the pecking order before you got the rooster? They usually are the ones that seem to have a problem coming to grips with the new pecking order - and sometimes they never get it. Sometimes its a hen problem, and not a rooster problem.
That being said - if he's attacking you, that's a different thing, and its not something I'd put up, or want to allow to mate.