As has already been said, you are unlikely to be 'sacked' from your PhD - removing anyone from a university is a long winded process. As you have implied he may well retort 'if you don't like it you are free to leave', but as a rule most academics will go out of their way to avoid losing a PhD student.
If you are worried about confronting your supervisor with your concerns you should speak to your postgraduate advisor (these people are there for precisely this problem).
If your department does not use a postgraduate advisor system then speak to the departmental postgraduate tutor, the head of postgraduate research or the head of department. You are in the best position to judge the potential reaction of your supervisor to you concerns, but remember it is polite to air your grievances directly to your supervisor before complaining to his colleagues (particularly senior ones).