I think that people avoid it for ALL reasons mentioned above, and your answer will vary depending on who you talk to.
Now, I as a person do not like soy. That said, I don't think soy is
evil! I just think that there is a big difference between feeding it as a major part of their food ration every day is much different than eating it on occasion. In general, soy is in chicken feed to provide protein and sometimes in the forms of other by-products, fillers, etc. If you can provide a feed with quality protein without the soy, you aren't missing out on anything. Essentially what I am trying to say is that beyond protein (and, as a disclaimer, proteins can be complicated in the forms they come in and the way the body uses them), according to my research, soy provides no benefit.
I also avoid soy for environmental reasons, however I should be clear that even if I did not, I'd still prefer to avoid it for the above reasons.
GMO concerns are another issue entirely, and are a completely personal choice!
Feeding, and food, are complicated things and are deeply personal. As long as you research as best you are able, and do what you feel is right, then don't let anyone get you down no matter what your choice.