Statistical significance: Difference between revisions
Pat Palmer (talk | contribs) (duplicate ref definitions) |
Pat Palmer (talk | contribs) m (ref errors) |
||
| Line 26: | Line 26: | ||
==Philosophical approaches to error testing== | ==Philosophical approaches to error testing== | ||
===Frequentist method=== | ===Frequentist method=== | ||
This approach uses mathematical formulas to calculate deductive probabilities (p-value) of an experimental result.<ref name="pmid10383371" | This approach uses mathematical formulas to calculate deductive probabilities (p-value) of an experimental result.<ref name="pmid10383371"/> This approach can generate [[confidence interval]]s. | ||
A problem with the [[frequentist]] analyses of p-values is that they may overstate "statistical significance".<ref name="pmid10383371" | A problem with the [[frequentist]] analyses of p-values is that they may overstate "statistical significance".<ref name="pmid10383371"/><ref name="pmid10383350">{{cite journal | author = Goodman SN | title = Toward evidence-based medical statistics. 2: The Bayes factor. | journal = Ann Intern Med | volume = 130 | issue = 12 | pages = 1005–13 | year = 1999|url=http://www.annals.org/cgi/content/full/130/12/1005 | pmid = 10383350}}</ref> | ||
===Likelihood or Bayesian method=== | ===Likelihood or Bayesian method=== | ||
Revision as of 13:37, 11 September 2024
In statistics, statistical significance is a "term indicating that the results obtained in an analysis of study data are unlikely to have occurred by chance, and the null hypothesis is rejected. When statistically significant, the probability of the observed results, given the null hypothesis, falls below a specified level of probability (most often P < 0.05)."[2] The P-value, which is used to represent the likelihood the observed results are due to chance, is defined at "the probability, under the assumption of no effect or no difference (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed."[3]
Hypothesis testing
Usually, the null hypothesis is the there is no difference between two samples in regard to the factor being studied.[4]
Choosing a statistical method
The choice of statistical method to use in an analysis is determined by:[5][6][7]
- Type of data, for example: continuous, categorical, dichotomous
- Whether the data is normally distributed. Various normality tests such as the Shapiro-Wilk[8] are available.[9]
- Whether the samples are independent or paired
- Number of samples to compare
Statistical errors
Two errors can occur in assessing the probability that the null hypothesis is true:
Type I error (alpha error)
Type I error, also called alpha error, is the the rejection of a correct null hypothesis. The probability of this is usually expressed by the p-value. Usually the null hypothesis is rejected if the p-value, or the chance of a type I error, is less than 5%. However, this threshold may be adjusted when multiple hypotheses are tested.[10]
Type II error (beta error)
Type II error, also called beta error, is the acceptance of an incorrect null hypothesis. This error may occur when the sample size was insufficient to have power to detect a statistically significant difference.[11][12][13]
Philosophical approaches to error testing
Frequentist method
This approach uses mathematical formulas to calculate deductive probabilities (p-value) of an experimental result.[3] This approach can generate confidence intervals.
A problem with the frequentist analyses of p-values is that they may overstate "statistical significance".[3][14]
Likelihood or Bayesian method
Some argue that the P-value should be interpreted with Bayes Theorem, or in other words, in light of how plausible is the hypothesis based on the totality of prior research and physiologic knowledge.[15][3][14][16][17] This approach can generate Bayesian 95% credibility intervals.[18] Details of Bayesian calculations have been reviewed.[19]
Bayesian inference:[14]
The Bayesian analysis creates a Bayes Factor. Unlike the traditional P-value, the Bayes factor is not a probability of rejecting the null hypothesis, but is a ratio of probabilities. The Bayes Factor is a likelihood ratio. A value greater than 1 supports the null hypotheses, whereas a value less than 1 supports the alternative hypothesis. The equation for the Bayes Factor is:[14]
Example of a coin flip that comes up heads in one of four tosses, is the coin unbiased or is the chance of heads in each toss only 1/4?:
Goodman gives the following three methods of interpreting an example Bayes Factor of 1/2:[14]
- Objective probability: "The observed results are half as probable under the null hypothesis as they are under the alternative."
- Inductive evidence: "The evidence supports the null hypothesis half as strongly as it does the alternative."
- Subjective probability: "The odds of the null hypothesis relative to the alternative hypothesis after the experiment are half what they were before the experiment."
The Minimum Bayes Factor is proposed by Goodman as another way to help readers make Bayesian interpretations if they are accustomed to p-values:[14]
Note that the Minimum Bayes Factor when p = 0.05, or Z= 1.96, is 0.15. It corresponds to the likelihood ratio for comparing the hypotheses that the mean is zero of a normally distributed variable Z with variance one, with the hypothesis that the mean is 1.96, when we observe the variable Z take the observed value z=1.96. Under classical (frequentist) hypothesis testing, we would just reject the null hypothesis that the mean is zero against the alternative that it is not zero, when testing with significance level 0.05, if we observed this value of Z. From the Bayesian point of view, this Bayes Factor leads to a posterior probability of 13%, if the prior odds were 50-50.[14] However it is suspect to be computing the Bayes factor or the posterior odds for a hypothesis, mean is 1.96, which is actually suggested by the data itself, z=1.06.
| Bayes Factor (B) |
Interpretation of support for the alternative hypothesis |
|---|---|
| > 1.00 | reduces the odds of the null hypothesis |
| 0.32–1.00 | "not worth more than a bare mention" |
| 0.100–0.320 | "substantial support" |
| 0.032–0.100 | "strong support" |
| 0.010–0.032 | "very strong support" |
| < 0.010 | "decisive support" |
An empiric comparison of frequentist and Bayesian approaches to the analyses of randomized controlled trials has been done.[21]
Applications of Bayesian approach
- Interim analyses of randomized controlled trials.
A Bayesian approach to interim analysis may help reduce bias and adjust the estimate of effect in randomized controlled trials.[22]
- Subgroup analyses of randomized controlled trials.
Bayesian analyses provide an alternative to Bonferroni adjustments when testing significance of multiple comparisons.[23]
- Adaptive trial designs.
The Bayesian method has been proposed for adaptive trial designs for comparative effectiveness research.[24] In the United States, Medicare's Centers for Medicare and Medicaid Services (CMS) is investigating this role.[25]
- Interpretation of clinical practice guidelines.
Diamond has proposed combining a Bayesian statistical approach with an evidentiary classification for the interpretation of clinical practice guidelines.[26]
References
- ↑ Anonymous (2006). “Normal Distribution”, NIST/SEMATECH e-Handbook of Statistical Methods. Gaithersburg, MD: National Institute of Standards and Technology. Retrieved on 2009-02-10.
- ↑ Anonymous. JAMAevidence Glossary. American Medical Association. Retrieved on 2009-02-10.
- ↑ 3.0 3.1 3.2 3.3 Goodman SN (1999). "Toward evidence-based medical statistics. 1: The P value fallacy". Ann Intern Med 130: 995–1004. PMID 10383371. [e]
- ↑ Mosteller, Frederick; Bailar, John Christian (1992). Medical uses of statistics. Boston, Mass: NEJM Books. ISBN 0-910133-36-0. Google Books
- ↑ Ludford PJ. An Overview: Choosing the Correct Statistical Test. University of Minnesota.
- ↑ How to choose a statistical test. GraphPad.
- ↑ Dinov I. Choosing the Right Statistical Test. University of California at Los Angeles.
- ↑ Shapiro, S. S. and Wilk, M. B. (1965). "An analysis of variance test for normality (complete samples)", Biometrika, Vol. 52, No. 3/4, pages 591–611. DOI:10.1093/biomet/52.3-4.591
- ↑ Stephens, M. A. (1974). "EDF Statistics for Goodness of Fit and Some Comparisons". Journal of the American Statistical Association 69: 730–737. DOI:10.2307/2286009. Research Blogging.
- ↑ Hochberg, Yosef (1988-12-01). "A sharper Bonferroni procedure for multiple tests of significance". Biometrika 75 (4): 800-802. DOI:10.1093/biomet/75.4.800. Retrieved on 2008-10-15. Research Blogging.
- ↑ Altman DG, Bland JM (August 1995). "Absence of evidence is not evidence of absence". BMJ (Clinical research ed.) 311 (7003): 485. PMID 7647644. PMC 2550545. [e]
- ↑ Detsky AS, Sackett DL (1985). "When was a "negative" clinical trial big enough? How many patients you needed depends on what you found.". Arch Intern Med 145 (4): 709-12. DOI:10.1001/archinte.1985.00360040141030. PMID 3985731. Research Blogging.
- ↑ Young MJ, Bresnitz EA, Strom BL (August 1983). "Sample size nomograms for interpreting negative clinical studies". Annals of internal medicine 99 (2): 248–51. PMID 6881780. [e]
- ↑ 14.0 14.1 14.2 14.3 14.4 14.5 14.6 Goodman SN (1999). "Toward evidence-based medical statistics. 2: The Bayes factor.". Ann Intern Med 130 (12): 1005–13. PMID 10383350.
- ↑ Browner WS, Newman TB (1987). "Are all significant P values created equal? The analogy between diagnostic tests and clinical research". JAMA 257: 2459–63. PMID 3573245. [e]
- ↑ Diamond GA, Kaul S (June 2004). "Prior convictions: Bayesian approaches to the analysis and interpretation of clinical megatrials". J. Am. Coll. Cardiol. 43 (11): 1929–39. DOI:10.1016/j.jacc.2004.01.035. PMID 15172393. Research Blogging.
- ↑ Ioannidis JP (August 2008). "Effect of formal statistical significance on the credibility of observational associations". Am. J. Epidemiol. 168 (4): 374–83; discussion 384–90. DOI:10.1093/aje/kwn156. PMID 18611956. Research Blogging.
- ↑ Gelfand, Alan E.; Sudipto Banerjee; Carlin, Bradley P. (2003). Hierarchical Modeling and Analysis for Spatial Data (Monographs on Statistics and Applied Probability). Boca Raton: Chapman & Hall/CRC. LCC QA278.2 .B36. ISBN 1-58488-410-X.
- ↑ Greenland S (June 2006). "Bayesian perspectives for epidemiological research: I. Foundations and basic methods". Int J Epidemiol 35 (3): 765–75. DOI:10.1093/ije/dyi312. PMID 16446352. Research Blogging.
- ↑ Jeffreys, Harold [1961] (1998). Theory of probability. Oxford: Clarendon Press. ISBN 0-19-850368-7.
- ↑ Wijeysundera DN, Austin PC, Hux JE, Beattie WS, Laupacis A (2009). "Bayesian statistical inference enhances the interpretation of contemporary randomized controlled trials.". J Clin Epidemiol 62 (1): 13-21.e5. DOI:10.1016/j.jclinepi.2008.07.006. PMID 18947971. Research Blogging.
- ↑ Goodman SN (June 2007). "Stopping at nothing? Some dilemmas of data monitoring in clinical trials". Ann. Intern. Med. 146 (12): 882–7. PMID 17577008. [e]
- ↑ Greenland S (June 2008). "Multiple comparisons and association selection in general epidemiology". Int J Epidemiol 37 (3): 430–4. DOI:10.1093/ije/dyn064. PMID 18453632. Research Blogging.
- ↑ Luce BR, Kramer JM, Goodman SN, et al. (June 2009). "Rethinking Randomized Clinical Trials for Comparative Effectiveness Research: The Need for Transformational Change". Ann. Intern. Med. 151 (3). PMID 19567619. [e]
- ↑ Anonymous. Health Care: Technology Assessment Subdirectory Page (English). Agency for Healthcare Research and Quality. Retrieved on 2009-08-03.
- ↑ Diamond GA, Kaul S (2009). "Bayesian classification of clinical practice guidelines.". Arch Intern Med 169 (15): 1431-5. DOI:10.1001/archinternmed.2009.235. PMID 19667308. Research Blogging.