I also agree with everyone else. I was in a similar situation once (in a non-academic job environment) and in the end I couldn't return to work there because I felt so ashamed about the whole thing (what with my boss being the other person's brother). When the line was crossed the relationship changed from flirting and having a bit of fun at work to feeling completely sordid and embarrassed. It was a mistake on both our parts. And this was only a part-time job. The fallout of this occurring during a PhD would surely be much worse. I definitely echo everyone's advice to get the relationship back on an even and professional footing. 

I am not as adamently negative as some people here regarding this situation. You need to consider what you want and what are the consequences of each course of action.
If your emotional attachment is strong, do you envisage a long term relationship with him that transcends the PhD. This is bound to involve some trauma for somebody (i.e. the wife).
If you are happy to keep it at flirtation, be aware that men generally think differently to women. He is more likely to try his luck to extend the flirtation to something physical. How would you react to that. What would be the effect. Despite your feeling that you are buddies, if the relationship became physical the dynamics would change. It is unlikely that you can simply go back to being buddies. You might be able to discuss this first. It would be a superficial win-win if you could both agree to simply relieve the tension with a quick affair, but that might be unrealistic.
There are men who are highly focused on a conquest, and if he got physical with you, then he might also get tired of you. Would his professional interest then wane. This really brings up the question of how important the PhD is to you, in contrast to your emotional, sexual and love life. I am not one for suggesting that the academic side to your life should get absolute precedence. I have made the same mistake in the past, downplaying relationships. However, in three years time you might have to ask yourself how your life changed as a consequence of this cross-road. Perhaps you will say, the affair was fun, but my PhD suffered, or became untenable.
I've got no answers for you. I will say, you need to think of the long term and of the risks involved.
Perhaps the easiest thing is to get yourself another boyfriend.