They are/were "roosting" - they just weren't doing so on the roost. They will roost on just about any surface they can, including the tops of nest boxes, on shelves, tops of doors, etc. Whether this location is acceptable to you or not is for you to decide. They do expel a LOT of waste during the roosted hours, so if you are okay with the accumulation of waste and/or the amount of cleaning the area will need if you permit the use of the box top for roosting it isn't necessarily a "bad" habit....sleeping *in* the boxes would be a different story as the waste would foul the nests and lead to dirty eggs. Any area you don't want them to use as a roosting spot needs to be made unattractive as a roosting spot - for the tops of nest boxes it can easily be done by installing a "roof" that slants at a 45+ degree angle instead of having the flat, open shelf top that it sounds like you currently have. Also, keep in mind that they are drawn to roost in the highest spot they can.
Does the top of the nest box set higher than the highest roost you have in the coop?