I wonder if he has any clue as to how much it costs to put to kids in daycare? I would think that you both need to sit down at some point and really figure out how much it will cost to move into the city. (Where the cost of living will be higher.) You will need to figure out the cost of clothing and parking, and not to mention all the bits that come with a mother working. he sounds scared that he will loose his job and not be able to get another one. Still there is always hidden costs to consider.
It concerns me that he dismisses your feelings by calling them crazy and or out of control. That yo me shows a lack of respect for you as a person, much less a partner in life.