You can choose between two things. One you can take them out when they are all fluffy OR you cant wait untill the other three hatch and then take them all out. Have the other three pipped yet? If so then yes you need to take the chicks out so they dont hurt the lil'ones while they are hatching.
I just wait until they can walk reasonably well. With the humdity I have to keep in my bator, they really don't fluff out until they are in the brooder. Just make sure to account for the moisture you will lose by opening the bator.