There is no real difference in fertile versus unfertile eggs. They taste the same, have the same nutrients, cook the same, and look the same unless you look real close and know exactly what to look for. You can get some development if they are stored for a while at temperatures in the lower 80's Fahrenheit or above, but not for the first day or so and unless you hit close to incubation temperatures they probably won't develop enough for you to notice. That potential to develop is really the only difference in fertile versus unfertile eggs. If you gather them once a day and store them where they are not too warm, that is not a problem.
For thousands of years, people have kept flocks of free ranging chickens and eaten fertile eggs without being harmed in any way.