I think there are three things to consider
1) To what extent do the referees' research interests match your own (they'll also be more flattered that you're pursuing 'their' subjects when you ask them)?
2) How senior in the department and well-respected in their field are they?
3) How well do they know you (for instance, did they simply give lectures for a module you took or have you taken a class with them, or worked on a project with them as your supervisor?)?
I had one referee who was my personal tutor in my department, and one who was a specialist. The best thing to do is arrange an appointment to chat to them to remind them who you are and check they'd be happy to write a reference for you. They won't say no unless they're about to leave the country or something: it's part of their job.