If you approach this reasonably and calmly, and highlight the problem honestly (without brutal honesty) then I don't see why it should negatively impact your reputation. If you are lucky, it will solidify your reputation as being upfront and honest, characteristics most people find admirable.
Suffice it to say, if you are having a problem then you need to speak with the person. In some cases, that will allow you to solve the problem, in others, it might mean a change of supervisor.