Yes, it's hard to tell, if not impossible at that age.
We got our chickens as chicks at 2 weeks old, and we couldn't tell until 5-6 weeks if they were truly male or female. You just have to wait for comb+waddle development, I guess. That was the quickest way we could tell.