Hi Jonathan
If you know that you want to do a PhD and the topic area, then I'd say just go ahead and apply. Obviously, your application would have been stronger if those three years of work were relevant to your chosen topic, but the longer you leave it the harder you may find it pyschologically to leave your job, get back into the groove of studying, used to the lower income, etc. Three years is not such a long time.
Good luck!