How do I get rid of the smell?

I would think with only four birds that it won't get bad at all.  So do your chickens have access to the entire yard?  About how small is your yard?

If you move the coop every few days, it shouldn't smell at all.  Especially since you are raking, in fact I'd think you could even leave it longer if you are raking after moving.  If for some reason it does, it helps a lot to keep it dry.  If it rains, try to keep the outdoor pen area dry by covering with a tarp or something.  Stall dry or food grade diatomaceous earth (DE) can help dry things up.  Stall dry looks like kitty litter, it is absorbant volcanic material and I think it has some DE in it.

Chicken poop doesn't smell when it's dry. Do you have any wet bedding in there?

The other thing I can think of is that chickens do tend to have a chickeny odor to their bodies when they're inside an enclosed space. If your coop has good ventilation, that will help let this odor disperse.

I switched from a soy based to a nonsoy based chicken feed and noticed this chicken odor disappeared from my chicks. It was a welcome thing because they were still in a brooder in the corner of our den at the time.

I just have five girls, and the only time I smell poop is when I go in there to scrape the dropping board in the morning, because it's all piled right there.  I don't smell anything until the next morning (unless one is sitting right beside me and drops a smelly bomb...lol).  So I would include scraping the ladder you mentioned.  Also, you might consider blocking the nests at night, because they shouldn't be sleeping and pooping in there.  It makes clean up much easier if they are sleeping on the roost.  I agree with elmo that it's most likely NOT the poop you're smelling, but something damp/wet, although I've found that the Dumar brand of feed results in stinkier droppings for my birds, so I use the Purina.

Your yard is plenty big for 4 chickens to NOT stink it up

I have 9 chickens in non-portable coop that I don't scoop out (I deep litter and just add bedding as needed) and I just went out to lock up and scritch my girls on the roost. Guess what? The only smell that I noticed was the smell of slightly damp feathers because it was sprinkling at dusk.

My guess is that the grass clippings are your source of odor.  They can get quite nasty and slimy.  And with only four birds, I can' t imagine there being enough poop to stink up the coop area.  I agree with the other posts here that it doesn't sounds like a poop problem.  Just a thought.