Hi all

Both of my reviewers' have issues with my statistical approach (although they don't really state why - a preference thing possibly)... and both suggest (different) alternatives. The things is... I don't have a problem with what I've done, and (maybe more importantly?!) a recently published article in the same journal, doing a very similar investigation, also does the same thing as I do... (to be more specific I have done a set of preliminary analyses and then on the basis of no significant effects there excluded a number of potential confounding variables from my main analyses - both reviewers are questioning this and want to see more variables in the main analysis - but I see no need and don't want to underpower my main analyses by including loads of variables unnecessarily).

Could I refer to that other paper (that does the same thing I've done) in my response to the reviewers' comments?!

Thanks
Tudor. (whose head is a mess after messing around with different stats all day)

What you describe is how I have always approached the decision to include or not include co-variates in the main analysis... in three different settings under different academics/supervisors. However, sometimes you do include a particular variable of interest even if it is not significant in the preliminary analysis, because it still may have an effect on your main variables. So, I don't think you're doing something clearly wrong, but without knowing the specifics of your study, the reviewers *could* have a point.

I would start by re-running the analysis according to the two ways that have been suggested. If you can find a way that doesn't change the actual pattern of results, you may as well just update the numbers in the results section and give them what they want.

If both approaches do change things a lot, I would consider the nature of the rest of the reviewer comments. If they are minor and it seems the paper has a good chance of getting accepted, I would (politely) justify why you are using your current approach. Definitely cite that paper which does the same - in fact you can probably find several that do. You can also mention the fact that there is discrepancy between two reviewer's suggestions, which highlights there is no clear correct approach you have missed.

There's no rule to say you have to adhere to all the reviewer's suggestions, and many papers have been accepted despite not adopting reviewer's suggestions. However, if you feel like the other comments are pretty major and you don't want to add another thing that might be the tipping point for an editor deciding to reject your paper, maybe you do need to consider doing what one of them says, or including some but not all the variables they want, or still saying no and being prepared to submit to another journal (if you feel very strongly and aren't desperate for publications!)

Hope this helps!

Hi teegs90 -

Thanks for the response - it helps a lot!

After thinking TOO much about it I've even began to question why I am looking for effects of potential other IVs/covariates at all, given that it is a counterbalanced, within-subjects experimental design, which controls for effects of factors such as order of condition, age, and gender simply by having the same participants undergo both conditions and counterbalancing.

I guess it is to simply see IF anything is going in the data that might otherwise go undetected... like if the effect of the experimental condition on males or older participants is greater (for example)... rather than for purposes of trying to control other factors.

Would you agree with this? (in principle)

Thanks!
T.

Hmmm, I'm not the best at stats, but I agree in principle that for a within-subjects, counterbalanced study you shouldn't be too worried about covariates.

Another option that I have used before but slipped my mind... you say you did run preliminary analyses to check whether there were any unexpected effects, but there weren't any. You could include a table showing this analysis in your response to reviewer's comments to justify your approach, and then add a sentence to your methods/results with something like:"[whatever test you used for your preliminary analysis] were conducted to test associations between [potential covariates] and the dependent variable (data not shown). No significant associations were found, so no covariates were included in the main analysis". Then,hopefully, the reviewers/editor are happy that you haven't outright rejected their suggestion but you keep your main analysis.

Good luck, like I said, you don't need to follow every single thing a reviewer says, and unless there are other bigger problems with the paper I'm sure it will be accepted whichever way you go!

Thanks again! I had got to the point where I was starting to second guess myself! I have justified why I am sticking to preliminary analysis then main analysis (with no covariates) in my response to their comments. I may actually include the null results tests for them to see as well - hadn't thought of that.

Thanks!