Well I can't say I 'fear' them, but I do carry a sidearm every time I go out to the barn(s), especially at night as I'm not silly enough to believe any are fully nocturnal... In my area, all spring and early summer when they have extra mouths to feed and during harsh winters when food is slim, many of those 'nocturnal' predators extend their hunting hours well past sunrise, and get an early start at night well before sunset...
I have seen more than my fair share of coons, possum, skunks, and coyotes snooping around my livestock buildings in full daylight as well... I even had coyotes walking all over my property at all hours of the day last winter because of the harsh weather...
For me the concern is far from limited to nighttime housing, it extends into their outdoor time as well...