It is a fairly reliable rule that if you find mites or lice on one bird, they are all affected, you're correct.
http://www.the-chicken-chick.com/2012/08/poultry-lice-and-mites-identification.html
That article is very informative and helps you figure out exactly what you're dealing with. Mites are generally more dangerous/severe than lice. Mites suck blood, lice survive on the feathers (mostly) of the bird.
I personally fought a lice infestation for a long time, one hen in particular had a particular problem with them.
I treated by getting Sevin dust (for plant pests) and a poultry spray from
Tractor Supply (I'm not sure of the brand). I cleaned the entire coop out, then sprayed and dusted all the surfaces. I then put in new shavings with more dust.
Caution: Be careful and let the dust settle before allowing chickens back in, it's not good for them (or you) to inhale it. It is safe to use on their skin though.
I then caught all the birds one by one, and sprayed the areas where mites tend to hang out and lay eggs. Under the vent and wings, and in the neck feathers. Then I dusted their entire body with Sevin dust, being careful around their heads. If you put the Sevin in an old pantyhose or thin sock, you can pat around their body and get dust well distributed.
The hen with the worst mites was attacked by a dog about a year ago and it hindered her ability to clean her vent area. She had tons of eggs at the base of her feathers back there. The eggs are "glued" on and incredibly hard to wash off, so I just clipped a lot of the egg laden feathers, sprayed LIBERALLY and dusted.
Repeat the process to get all the newly hatched mites or any that may have survived Round 1. To prevent a re-infestation, provide them with a place to dust bathe and remove pests naturally, make the coop cleaning ritual permanent (spraying and dusting), and keep an eye on the situation with each bird.