Dear Dr. Sorkin,
It is hard to tell for sure, since I don't know what problem you are
working on; and much depends on context and on the inferential goals
of the study. However, I'm inclined to guess that this is a situation
where I would not engage a reviewer in this sort of debate at all.
I would probably have avoided significance tests of treatment groups
versus control; rather, I would just calculate a 95%-confidence
interval for each difference between treatment and control, using a
nonsimultaneous t criterion, with appropriate df (depending on whether
or not it makes sense to pool error estimates across the two treatments
and the control group). The two confidence intervals would be pretty wide,
I assume, and would make clear that, while there is evidence for some
difference between treatments and control, there is hardly any evidence
as to whether the differences are negligible, middling, or large.
Making the intervals simultaneous might or might not change whether a
difference of 0 is included in one or the other or both; but it should
not materially affect your scientific conclusions: the data suggest some
differences, but much more work is needed to confirm that the differences
are non-negligible, and to and pin their sizes accurately.
Dave Krantz
|