I would say it depends on what those variables are. If they are in your hypothesis, or you are controlling for them, because previous literature has said they are important/there will be a relationship then it is important to leave them in.
However, if you are adding in a load of control variables e.g. age, gender etc. that are not identified as having any link with the outcome in ther literature, or there is no real reason you are doing it (apart from to show off you can do hierarchical regression - like my students seem to think is necessary), then I'd take them out.
Of course its really important to check for multicollinearity - to make sure they are truely non-significant and you're not missing a potentially significant effect because of supression.