The starter will still rise even in the fridge, so while the warmth speeds it up, no warmth wont hurt it, it just slow it down a bit so it takes longer to get to the same place as a warm one.
I just keep my starter in the fridge, take it out weekly and discard, feed/water stir and place it back in the fridge. Thus far ( over 2 years now I think) have had no problems with it.