In a scientific study, post hoc analysis (from Latin post hoc, "after this") consists of statistical analyses that were specified after the data were seen. This typically creates a multiple testing problem because each potential analysis is effectively a statistical test. Multiple testing procedures are sometimes used to compensate, but that is often difficult or impossible to do precisely. Post hoc analysis that is conducted and interpreted without adequate consideration of this problem is sometimes called data dredging by critics because the statistical associations that it finds are often spurious.
Causes
Sometimes the temptation to engage in post hoc analysis is motivated by a desire to produce positive results or see a project as successful. In the case of pharmaceutical research, there may be significant financial consequences to a failed trial, although the US Food and Drug Administration does not accept post hoc analysis.[1]
In some cases, additional subgroup analysis may be requested by scientific peers or the editors of academic journals. In one such incident, journal editors demanded that the statistician Richard Peto provide a post hoc analysis of subgroups for the use of aspirin as secondary prevention for people who had experienced heart attacks. He refused the request as being statistically unsound and likely to lead to nonsensical results. When they persisted, he provided the editors with a subgroup analysis that evaluated the supposed response based upon the patients' astrological signs.[1][2]
See also
- Testing hypotheses suggested by the data
- Nemenyi test
References
- ^ a b Mukherjee, Siddhartha (2017-11-28). "A Failure to Heal". The New York Times. ISSN 0362-4331. Retrieved 2017-12-02.
- ^ Richard Peto, "Current misconception 3: that subgroup-specific trial mortality results often provide a good basis for individualising patient care", Br J Cancer, 104(7), pages 1057-1058 (2011). doi:10.1038/bjc.2011.79