You should absolutely find out the reasons behind it, otherwise, as you say, you won't be able to let it go. I don't think you should worry too much about how it looks, it's a valid question, and I can't see anyone disagreeing with that. This is assuming you simply ask, and not go in accusing people or being aggressive/defensive (which I'm sure you won't).
Personally, I would go for supervisor in person or lecturer by email. I would do supervisor first, then if no luck, go for lecturer. If you really are worried about how it looks, then highlight the fact you are only concerned because you want to know how you can improve your teaching, if this is the reason. I have to say I doubt it is the reason - much more likely is they want you to spend more time on your research this year and want to give someone else a chance. They have just gone about it in a really crap way.