Well, it seems like you've made up your mind about this and decided to take this step. Therefore I don't think you need to worry about the consequences: of course every action cause a reaction but in your case you have the chance to make a new beginning with a new supervisor. After all, people may try to make life hell and your reputation within the department may suffer for a certain period of time. Nevertheless it is up to you to make an impact in your new relationship with your new supervisor - in other words, if you manager to impress your new supervisor through hard work and committment, the bad reputation may only continue through gossip of a few lost souls who see no other purpose in life than gossiping all the time. They are not important anyway and remember: all you want to do is to complete a PhD, so as long as you manage to do that it is up to you to leave your departement which might shed a bit of hope on your current situation. Respect for your decision, though!