^^^^ ditto.  3KillerBees is the expert.
You can't have deep litter too deep.  You *CAN* deep litter on boards/wo ground contact, but it takes a lot longer to get started - just as you can put a mulch pile on top of a tarp and, eventually, it will start decomposing as it should.  Ground contact is absolutely best, but as my recent cleanout demonstrates, you can get composting started even on a "floor" of concrete board suspended 3' in the air.
Your dimensions are boardline, based on the thumb rules, for your quantity of birds - and the fact that you can smell ammonia confirms that some part of your system (because a hen house is a system that needs to work together) isn't working as it should.  Either the litter, or the ventilation, most likely.  With 15" of litter, and only four widows, I strongly suspect the ventilation is the issue - not uncommon with converted sheds.
Pictures would help.
As to what to fill it with, particularly in a large coop???  I use straw in my nesting boxes.  Moderately cheap, dry, free of sticks, good padding.  WAY too expensive for large areas, can mat down, and trap moisture.  My coops, underneath them, and even in the run, I pile leaf litter from the woods surrounding my property (no Chip drop here).  The birdsa aren't nesting in it, so the sticks don't matter, and the irregular size and shape keeps any straw from the nesting boxes from matting together.
Free, except for my labor.  Seems like I dump 7 gorilla cart loads+ (about 2 cu yd) into each of my hen houses quarterly, and scrape out the raised hen house twice a year, to join the deep bedding on the ground level (where my ducks nest).  They don't mind the sticks and twigs, and their shells are SO SO much harder than a chicken's, there is no risk of breakage.