My supervisor to be told me that sometimes you are best staying where you are - you know the staff, how the department runs etc. If your interests lie in an area that fits in with the interests of the dept you are in then why go somewhere else that might not suit your research? Hopefully it shouldn't affect job prospects as long as the uni you are at has a good reputation.
Good luck! xx