I worry about this all the time.
I try and treat it like a full time job, although if I'm honest I usually don't quite make it to 35 hours a week, more like 32-33. I take odd days, but not too many, just like you would with a regular full time job. I keep a vague track of these odd days and was aiming, with my summer holiday and the time I took over christmas, for 6-7 weeks in full. But due to unforseen circumstances (which my supervisor is aware of) it looks like it is going to end up being a little bit more than that.
Sounds like you are on track and you know where you're going with your work so I wouldn't worry about it.