Teaching evidence-based medicine: Difference between revisions

From Citizendium
Jump to navigation Jump to search
imported>Robert Badgett
(resolving more ref errors)
 
(23 intermediate revisions by 5 users not shown)
Line 1: Line 1:
{{subpages}}  
{{subpages}}
Teaching [[evidence based medicine]] (EBM) is difficult for several reasons including lack of consensus on what skills physicians need to have and also due to physicians' possibly being over confident in their  EBM skills.<ref name="pmid16807169">{{cite journal |author=Caspi O, McKnight P, Kruse L, Cunningham V, Figueredo AJ, Sechrest L |title=Evidence-based medicine: discrepancy between perceived competence and actual performance among graduating medical students |journal=Med Teach |volume=28 |issue=4 |pages=318–25 |year=2006 |pmid=16807169 |doi=10.1080/01421590600624422 |issn=}}</ref>
{{TOC|right}}
 
'''Teaching [[evidence based medicine]]''' (EBM) is difficult for several reasons including lack of consensus on what skills physicians need to have and also due to physicians' possibly being over confident in their  EBM skills.<ref name="pmid16807169">{{cite journal |author=Caspi O, McKnight P, Kruse L, Cunningham V, Figueredo AJ, Sechrest L |title=Evidence-based medicine: discrepancy between perceived competence and actual performance among graduating medical students |journal=Med Teach |volume=28 |issue=4 |pages=318–25 |year=2006 |pmid=16807169 |doi=10.1080/01421590600624422 |issn=}}</ref>


Most of this article is about teaching [[evidence-based individual decision making]] and not about evidence-based medicine at the institutional level. This first part of this article covers studies or reports of methods in [[Evidence based medicine|EBM]]. The second part reviews studies on the effectiveness of teaching evidence-based medicine. Both sections may guide the teacher of [[Evidence based medicine|EBM]] in selecting strategies, but the teacher should note that only ideas on the second section have empiric data on effectiveness.
Most of this article is about teaching [[evidence-based individual decision making]] and not about evidence-based medicine at the institutional level. This first part of this article covers studies or reports of methods in [[Evidence based medicine|EBM]]. The second part reviews studies on the effectiveness of teaching evidence-based medicine. Both sections may guide the teacher of [[Evidence based medicine|EBM]] in selecting strategies, but the teacher should note that only ideas on the second section have empiric data on effectiveness.


==Methods for teaching EBM==
==Methods for teaching EBM==
A multinational European group has used the [[Delphi method]] to design a comprehensive EBM course based on five steps of EBM.<ref name="pmid18042271">{{cite journal |author=Coppus SF, Emparanza JI, Hadley J, ''et al'' |title=A clinically integrated Curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project |journal=BMC Med Educ |volume=7 |issue=1 |pages=46 |year=2007 |pmid=18042271 |doi=10.1186/1472-6920-7-46 |issn=}}</ref> The effectiveness of the course has not been studied. Their proposed  [http://www.biomedcentral.com/1472-6920/7/46/table/T1 learning objectives] are available online:
A multinational European group has used the [[Delphi method]] to design a comprehensive EBM course based on five steps of EBM.<ref name="pmid18042271">{{cite journal |author=Coppus SF, Emparanza JI, Hadley J, ''et al'' |title=A clinically integrated Curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project |journal=BMC Med Educ |volume=7 |issue=1 |pages=46 |year=2007 |pmid=18042271 |doi=10.1186/1472-6920-7-46 |issn=}}</ref> The course is clinically integrated. Its effectiveness is at least as good as traditional teaching.<ref name="pmid19435520">Kulier R, Coppus SF, Zamora J, Hadley J, Malick S, Das K, Weinbrenner S, Meyerrose B, Decsi T, Horvath AR, Nagy E, Emparanza JI, Arvanitis TN, Burls A, Cabello JB, Kaczor M, Zanrei G, Pierer K, Stawiarz K, Kunz R, Mol BW, Khan KS. The effectiveness of a clinically integrated e-learning course in evidence-based medicine: a cluster randomised controlled trial. BMC Med Educ. 2009 May 12;9:21. PMID 19435520</ref> Their proposed  [http://www.biomedcentral.com/1472-6920/7/46/table/T1 learning objectives] are available online:
* 'generate structured questions arising from clinical problems in practice'
* 'generate structured questions arising from clinical problems in practice'
* 'search relevant literature, identifying systematic reviews wherever possible'
* 'search relevant literature, identifying systematic reviews wherever possible'
Line 20: Line 20:
# '''Tone'''. Create a respectful and non-threatening tone in the class.
# '''Tone'''. Create a respectful and non-threatening tone in the class.
# '''Take home messages'''. End with take home messages.
# '''Take home messages'''. End with take home messages.
===Formulation of the clinical question ('PICO')===
The 'PICO' format, which is mnemonic for:<ref name="pmid7582737">{{cite journal |author=Richardson WS, Wilson MC, Nishikawa J, Hayward RS |title=The well-built clinical question: a key to evidence-based decisions |journal=ACP J. Club |volume=123 |issue=3 |pages=A12–3 |year=1995 |pmid=7582737 |doi= |url=http://www.mclibrary.duke.edu/training/pdaformat/articlepda.html |issn=}}</ref>
*'''P''' - patients, problems, or population
*'''I''' - intervention, exposure, or test
*'''C''' - comparison
*'''O''' - outcome
Although PICO cannot represent all clinical question, especially questions that are not about interventions<ref name="pmid17238363">{{cite journal |author=Huang X, Lin J, Demner-Fushman D |title=Evaluation of PICO as a knowledge representation for clinical questions |journal=AMIA Annu Symp Proc |volume= |issue= |pages=359–63 |year=2006 |pmid=17238363 |pmc=1839740 |doi= |url= |issn=}}</ref>, use of PICO may improve the precision of searches<ref name="pmid17573961">{{cite journal |author=Schardt C, Adams MB, Owens T, Keitz S, Fontelo P |title=Utilization of the PICO framework to improve searching PubMed for clinical questions |journal=BMC Med Inform Decis Mak |volume=7 |issue= |pages=16 |year=2007 |pmid=17573961 |pmc=1904193 |doi=10.1186/1472-6947-7-16 |url=http://www.biomedcentral.com/1472-6947/7/16 |issn=}}</ref>.


===Search strategies===
===Search strategies===
Line 30: Line 39:
# "''''R'''eview the quality of evidence' for the information found"
# "''''R'''eview the quality of evidence' for the information found"
# "Discuss ''''L'''essons of the search.'"
# "Discuss ''''L'''essons of the search.'"
A subsequent negative study has been published about the effect of a single session of learning to search.<ref name="pmid22879808">{{cite journal| author=Ilic D, Tepper K, Misso M| title=Teaching evidence-based medicine literature searching skills to medical students during the clinical years: a randomized controlled trial. | journal=J Med Libr Assoc | year= 2012 | volume= 100 | issue= 3 | pages= 190-6 | pmid=22879808 | doi=10.3163/1536-5050.100.3.009 | pmc=PMC3411272 | url=http://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&tool=sumsearch.org/cite&retmode=ref&cmd=prlinks&id=22879808  }} </ref>


===Critical appraisal===
===Critical appraisal===
Line 37: Line 48:


===Clinical reasoning===
===Clinical reasoning===
There are various methods of clinical reasoning include probabilistic (Bayesian), causal (physiologic), and deterministic (rule-based).<ref name="pmid2655522">{{cite journal | author = Kassirer JP | title = Diagnostic reasoning | journal = Ann Intern Med | volume = 110 | pages = 893–900 | year = 1989 | pmid = 2655522 | doi = }}</ref>  In addition, medical experts rely more on pattern recognition which is faster and less prone to error<ref name="pmid7503827">{{cite journal | author = Leape LL | title = Error in medicine | journal = JAMA | volume = 272 | pages = 1851–7 | year = 1994 | pmid = 7503827 | doi = }}</ref>; however, clinical experts seem flexible and may use whichever method of reasoning most easily represents and solves a given problem.<ref name="pmid17124025">{{cite journal | author = Norman G | title = Building on experience--the development of clinical reasoning | journal = N Engl J Med | volume = 355  | pages = 2251–2 | year = 2006 | pmid = 17124025 | doi = 10.1056/NEJMe068134}}</ref> Scales to measure clinical reasoning have been proposed.<ref name="pmid9231115">{{cite journal | author = Boshuizen HP ''et al.''| title = Measuring knowledge and clinical reasoning skills in a problem-based curriculum | journal = Medical education | volume = 31 | pages = 115–21 | year = 1997 | pmid = 9231115 | doi = }}</ref> Explicit Bayesian thinking with precise numbers is rarely done.<ref name="pmid3277516">{{cite journal | author = Moskowitz AJ ''et al.''| title = Dealing with uncertainty, risks, and tradeoffs in clinical decisions. A cognitive science approach | journal = Ann. Intern. Med. | volume = 108| pages = 435–49 | year = 1988 | pmid = 3277516 | doi = }}</ref><ref name="pmid9576412">{{cite journal |author=Reid MC, Lane DA, Feinstein AR |title=Academic calculations versus clinical judgments: practicing physicians' use of quantitative measures of test accuracy |journal=Am J Med |volume=104 |pages=374–80 |year=1998 |pmid=9576412 |doi=}}</ref> Basic science knowledge is probably "encapsulated" into clinical knowledge.<ref name="pmid16043534">{{cite journal | author = de Bruin AB ''et al.'' | title = The role of basic science knowledge and clinical knowledge in diagnostic reasoning: a structural equation modeling approach | journal = Academic Medicine  | volume = 80 |  pages = 765–73 | year = 2005 | pmid = 16043534 | doi = }}</ref>
There are various methods of clinical reasoning include probabilistic (Bayesian), causal (physiologic), and deterministic (rule-based).<ref name="pmid2655522">{{cite journal | author = Kassirer JP | title = Diagnostic reasoning | journal = Ann Intern Med | volume = 110 | pages = 893–900 | year = 1989 | pmid = 2655522 | doi = }}</ref>  In addition, medical experts rely more on pattern recognition which is faster<ref name="pmid7503827">{{cite journal | author = Leape LL | title = Error in medicine | journal = JAMA | volume = 272 | pages = 1851–7 | year = 1994 | pmid = 7503827 | doi = }}</ref>; however, clinical experts seem flexible and may use whichever method of reasoning most easily represents and solves a given problem.<ref name="pmid17124025">{{cite journal | author = Norman G | title = Building on experience--the development of clinical reasoning | journal = N Engl J Med | volume = 355  | pages = 2251–2 | year = 2006 | pmid = 17124025 | doi = 10.1056/NEJMe068134}}</ref> When confronted with non-challenging cases physicians may use 'nonanalytical reasoning' such as pattern recognition; however, during more difficult cases physicians may switch to 'reflective reasoning'.<ref>Mamede S, et al. (2008) Influence of Perceived Difficulty of Cases on Physicians' Diagnostic Reasoning. Academic Medicine. 83(12):1210-1216. {{doi|10.1097/ACM.0b013e31818c71d7}}</ref> Reflective reasoning may especially help complex cases.<ref name="pmid18412886">{{cite journal |author=Mamede S, Schmidt HG, Penaforte JC |title=Effects of reflective practice on the accuracy of medical diagnoses |journal=Med Educ |volume=42 |issue=5 |pages=468–75 |year=2008 |month=May |pmid=18412886 |doi=10.1111/j.1365-2923.2008.03030.x |url=http://www3.interscience.wiley.com/resolve/openurl?genre=article&sid=nlm:pubmed&issn=0308-0110&date=2008&volume=42&issue=5&spage=468 |issn=}}</ref> Explicit Bayesian thinking with precise numbers is rarely done.<ref name="pmid3277516">{{cite journal | author = Moskowitz AJ ''et al.''| title = Dealing with uncertainty, risks, and tradeoffs in clinical decisions. A cognitive science approach | journal = Ann. Intern. Med. | volume = 108| pages = 435–49 | year = 1988 | pmid = 3277516 | doi = }}</ref><ref name="pmid9576412">{{cite journal |author=Reid MC, Lane DA, Feinstein AR |title=Academic calculations versus clinical judgments: practicing physicians' use of quantitative measures of test accuracy |journal=Am J Med |volume=104 |pages=374–80 |year=1998 |pmid=9576412 |doi=}}</ref> Basic science knowledge is probably "encapsulated" into clinical knowledge.<ref name="pmid16043534">{{cite journal | author = de Bruin AB ''et al.'' | title = The role of basic science knowledge and clinical knowledge in diagnostic reasoning: a structural equation modeling approach | journal = Academic Medicine  | volume = 80 |  pages = 765–73 | year = 2005 | pmid = 16043534 | doi = }}</ref>
 
Successfully distilling complex information into a short summary, perhaps using semantic qualifiers, may help diagnostic accuracy.<ref name="pmid9795669">{{cite journal| author=Chang RW, Bordage G, Connell KJ| title=The importance of early problem representation during case presentations. | journal=Acad Med | year= 1998 | volume= 73 | issue= 10 Suppl | pages= S109-11 | pmid=9795669
| url=http://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&retmode=ref&cmd=prlinks&id=9795669 }} </ref><ref name="pmid1930535">{{cite journal| author=Bordage G, Lemieux M| title=Semantic structures and diagnostic thinking of experts and novices. | journal=Acad Med | year= 1991 | volume= 66 | issue= 9 Suppl | pages= S70-2 | pmid=1930535
| url=http://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&retmode=ref&cmd=prlinks&id=1930535 }} </ref><ref name="pmid12191059">{{cite journal| author=Nendaz MR, Bordage G| title=Promoting diagnostic problem representation. | journal=Med Educ | year= 2002 | volume= 36 | issue= 8 | pages= 760-6 | pmid=12191059
| url=http://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&retmode=ref&cmd=prlinks&id=12191059 }} </ref><ref name="pmid15099125">{{cite journal| author=Kulatunga-Moruzi C, Brooks LR, Norman GR| title=Using comprehensive feature lists to bias medical diagnosis. | journal=J Exp Psychol Learn Mem Cogn | year= 2004 | volume= 30 | issue= 3 | pages= 563-72 | pmid=15099125
| url=http://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&retmode=ref&cmd=prlinks&id=15099125 | doi=10.1037/0278-7393.30.3.563 }} </ref>


{| align="right" border="1"
{| align="right" border="1"
Line 49: Line 66:
| Colspan="3" | The most important missing information is cell B
| Colspan="3" | The most important missing information is cell B
|}
|}
Possible strategies to improve clinical reasoning have been reviewed<ref name="pmid17124019">{{cite journal |author=Bowen JL |title=Educational strategies to promote clinical diagnostic reasoning |journal=N Engl J Med |volume=355  |pages=2217–25 |year=2006 |pmid=17124019 |doi=10.1056/NEJMra054782}}</ref><ref name="pmid12377672">{{cite journal |author=Graber M ''et al.'' |title=Reducing diagnostic errors in medicine: what's the goal? |journal=Academic Medicine |volume=77  |pages=981–92 |year=2002 |pmid=12377672 |doi=}}</ref> and using problem-based learning<ref name="pmid12377672"/>, include teaching appropriate problem representation creating a one-sentence summary of a case<ref name="pmid17124019"/>, standardized patients<ref name="pmid16423099">{{cite journal |author=Windish DM ''et al.'' |title=Teaching medical students the important connection between communication and clinical reasoning |journal=J Gen Intern Med |volume=20 |pages=1108–13 |year=2005 |pmid=16423099 |doi=10.1111/j.1525-1497.2005.0244.x}}</ref>, teaching hypothetico-deductive reasoning<ref name="pmid11893348">{{cite journal |author=Wiese J ''et al.''|title=Improving oral presentation skills with a clinical reasoning curriculum: a prospective controlled study |journal=Am J Med |volume=112 |pages=212–8 |year=2002 |pmid=11893348 |doi=}}</ref><ref name="pmid7070446">{{cite journal |author=Eddy DM, Clanton CH |title=The art of diagnosis: solving the clinicopathological exercise |journal=N Engl J Med |volume=306 |pages=1263–8 |year=1982 |pmid=7070446 |doi=}}</ref>, cognitive forcing strategies<ref name="pmid11073470">{{cite journal |author=Croskerry P |title=The cognitive imperative: thinking about how we think |journal=Academic Emergency Medicine |volume=7 |pages=1223–31 |year=2000 |pmid=11073470 |doi=}}</ref><ref name="pmid12414468">{{cite journal |author=Croskerry P |title=Achieving quality in clinical decision making: cognitive strategies and detection of bias |journal=Academic Emergency Medicine |volume=9 |pages=1184–204 |year=2002 |pmid=12414468 |doi=}}</ref> to avoid premature closure<ref name="pmid3736379">{{cite journal |author=Dubeau CE ''et al.''|title=Premature conclusions in the diagnosis of iron-deficiency anemia: cause and effect |journal=Medical Decision Making  |volume=6  |pages=169–73 |year=1986 |pmid=3736379 |doi=}}</ref>, teaching the competing-hypotheses heuristic<ref name="pmid3385753">{{cite journal | author = Wolf FM ''et al.'' | title = Use of the competing-hypotheses heuristic to reduce 'pseudodiagnosticity' | journal = J Med Educ | volume = 63 | pages = 548–54 | year = 1988 | pmid = 3385753 | doi = }}</ref>, using fuzzy-trace theory<ref name="pmid11251760">{{cite journal |author=Lloyd FJ, Reyna VF |title=A web exercise in evidence-based medicine using cognitive theory |journal=J Gen Intern Med|volume=16 |pages=94–9 |year=2001 |pmid=11251760 |doi=}} [http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pubmed&pubmedid=11251760 PubMed Central]</ref> and mixed-methods interventions<ref name="pmid10667884">{{cite journal |author=Windish DM |title=Teaching medical students clinical reasoning skills |journal=Acad Med |volume=75 |issue=1 |pages=90 |year=2000 |pmid=10667884 |doi=}}</ref><ref name="pmid16024422">{{cite journal |author=Struyf E, Beullens J, Van Damme B, Janssen P, Jaspaert H |title=A new methodology for teaching clinical reasoning skills: problem solving clinical seminars |journal=Med Teach |volume=27 |issue=4 |pages=364–8 |year=2005 |pmid=16024422 |doi=10.1080/01421590500046411}}</ref><ref name="pmid10667884">{{cite journal |author=Windish DM |title=Teaching medical students clinical reasoning skills |journal=Acad Med |volume=75 |issue=1 |pages=90 |year=2000 |pmid=10667884 |doi=}}</ref>. Studies are unclear about  teaching logic.<ref name="pmid3742999">{{cite journal |author=Cheng PW ''et al.'' |title=Pragmatic versus syntactic approaches to training deductive reasoning |journal=Cognitive Psychology |volume=18 |pages=293–328 |year=1986 |pmid=3742999 |doi=10.1016/0010-0285(86)90002-2}}</ref><ref name="pmid16907682">{{cite journal | author = Jenicek M | title = The hard art of soft science: Evidence-Based Medicine, Reasoned Medicine or both? | journal = Journal of Evaluation in Clinical Practice | volume = 12  | pages = 410–9 | year = 2006 | pmid = 16907682 | doi = 10.1111/j.1365-2753.2006.00718.x}}</ref> Regarding hypothetico-deductive reasoning, an observational study on the methods used by experts solving clinicopathological exercises reported that these experts use the following six steps:<ref name="pmid7070446">{{cite journal |author=Eddy DM, Clanton CH |title=The art of diagnosis: solving the clinicopathological exercise |journal=N Engl J Med |volume=306 |pages=1263–8 |year=1982 |pmid=7070446 |doi=}}</ref>
Possible strategies to improve clinical reasoning have been reviewed<ref name="pmid17124019">{{cite journal |author=Bowen JL |title=Educational strategies to promote clinical diagnostic reasoning |journal=N Engl J Med |volume=355  |pages=2217–25 |year=2006 |pmid=17124019 |doi=10.1056/NEJMra054782}}</ref><ref name="pmid12377672">{{cite journal |author=Graber M ''et al.'' |title=Reducing diagnostic errors in medicine: what's the goal? |journal=Academic Medicine |volume=77  |pages=981–92 |year=2002 |pmid=12377672 |doi=}}</ref> and using problem-based learning<ref name="pmid12377672"/>, include teaching appropriate problem representation creating a one-sentence summary of a case<ref name="pmid17124019"/>, standardized patients<ref name="pmid16423099">{{cite journal |author=Windish DM ''et al.'' |title=Teaching medical students the important connection between communication and clinical reasoning |journal=J Gen Intern Med |volume=20 |pages=1108–13 |year=2005 |pmid=16423099 |doi=10.1111/j.1525-1497.2005.0244.x}}</ref>, teaching hypothetico-deductive reasoning<ref name="pmid11893348">{{cite journal |author=Wiese J ''et al.''|title=Improving oral presentation skills with a clinical reasoning curriculum: a prospective controlled study |journal=Am J Med |volume=112 |pages=212–8 |year=2002 |pmid=11893348 |doi=}}</ref><ref name="pmid7070446">{{cite journal |author=Eddy DM, Clanton CH |title=The art of diagnosis: solving the clinicopathological exercise |journal=N Engl J Med |volume=306 |pages=1263–8 |year=1982 |pmid=7070446 |doi=}}</ref>, cognitive forcing strategies<ref name="pmid11073470">{{cite journal |author=Croskerry P |title=The cognitive imperative: thinking about how we think |journal=Academic Emergency Medicine |volume=7 |pages=1223–31 |year=2000 |pmid=11073470 |doi=}}</ref><ref name="pmid12414468">{{cite journal |author=Croskerry P |title=Achieving quality in clinical decision making: cognitive strategies and detection of bias |journal=Academic Emergency Medicine |volume=9 |pages=1184–204 |year=2002 |pmid=12414468 |doi=}}</ref><ref name="pmid12514691">{{cite journal| author=Croskerry P| title=Cognitive forcing strategies in clinical decisionmaking. | journal=Ann Emerg Med | year= 2003 | volume= 41 | issue= 1 | pages= 110-20 | pmid=12514691
| url=http://www.ncbi.nlm.nih.gov/entrez/eutils/elink.fcgi?dbfrom=pubmed&retmode=ref&cmd=prlinks&id=12514691 | doi=10.1067/mem.2003.22 }} </ref> to avoid premature closure<ref name="pmid3736379">{{cite journal |author=Dubeau CE ''et al.''|title=Premature conclusions in the diagnosis of iron-deficiency anemia: cause and effect |journal=Medical Decision Making  |volume=6  |pages=169–73 |year=1986 |pmid=3736379 |doi=}}</ref>, teaching the competing-hypotheses heuristic<ref name="pmid3385753">{{cite journal | author = Wolf FM ''et al.'' | title = Use of the competing-hypotheses heuristic to reduce 'pseudodiagnosticity' | journal = J Med Educ | volume = 63 | pages = 548–54 | year = 1988 | pmid = 3385753 | doi = }}</ref>, using fuzzy-trace theory<ref name="pmid11251760">{{cite journal |author=Lloyd FJ, Reyna VF |title=A web exercise in evidence-based medicine using cognitive theory |journal=J Gen Intern Med|volume=16 |pages=94–9 |year=2001 |pmid=11251760 |doi=}} [http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pubmed&pubmedid=11251760 PubMed Central]</ref> and mixed-methods interventions<ref name="pmid10667884">{{cite journal |author=Windish DM |title=Teaching medical students clinical reasoning skills |journal=Acad Med |volume=75 |issue=1 |pages=90 |year=2000 |pmid=10667884 |doi=}}</ref><ref name="pmid16024422">{{cite journal |author=Struyf E, Beullens J, Van Damme B, Janssen P, Jaspaert H |title=A new methodology for teaching clinical reasoning skills: problem solving clinical seminars |journal=Med Teach |volume=27 |issue=4 |pages=364–8 |year=2005 |pmid=16024422 |doi=10.1080/01421590500046411}}</ref><ref name="pmid10667884">{{cite journal |author=Windish DM |title=Teaching medical students clinical reasoning skills |journal=Acad Med |volume=75 |issue=1 |pages=90 |year=2000 |pmid=10667884 |doi=}}</ref>. Studies are unclear about  teaching logic.<ref name="pmid3742999">{{cite journal |author=Cheng PW ''et al.'' |title=Pragmatic versus syntactic approaches to training deductive reasoning |journal=Cognitive Psychology |volume=18 |pages=293–328 |year=1986 |pmid=3742999 |doi=10.1016/0010-0285(86)90002-2}}</ref><ref name="pmid16907682">{{cite journal | author = Jenicek M | title = The hard art of soft science: Evidence-Based Medicine, Reasoned Medicine or both? | journal = Journal of Evaluation in Clinical Practice | volume = 12  | pages = 410–9 | year = 2006 | pmid = 16907682 | doi = 10.1111/j.1365-2753.2006.00718.x}}</ref> Regarding hypothetico-deductive reasoning, an observational study on the methods used by experts solving clinicopathological exercises reported that these experts use the following six steps:<ref name="pmid7070446">{{cite journal |author=Eddy DM, Clanton CH |title=The art of diagnosis: solving the clinicopathological exercise |journal=N Engl J Med |volume=306 |pages=1263–8 |year=1982 |pmid=7070446 |doi=}}</ref>
# aggregation of groups of findings into patterns
# aggregation of groups of findings into patterns
# selection of a "pivot" or key finding
# selection of a "pivot" or key finding
Line 56: Line 74:
# selection of a diagnosis
# selection of a diagnosis
# validation of the diagnosis
# validation of the diagnosis
Scales to measure clinical reasoning have been proposed.<ref name="pmid9231115">{{cite journal | author = Boshuizen HP ''et al.''| title = Measuring knowledge and clinical reasoning skills in a problem-based curriculum | journal = Medical education | volume = 31 | pages = 115–21 | year = 1997 | pmid = 9231115 | doi = }}</ref>


====Numeracy====
====Numeracy====
Line 61: Line 81:


Studies are inconclusive on using cognitive feedback.<ref name="pmid7898300">{{cite journal |author=Poses RM ''et al.''|title=You can lead a horse to water--improving physicians' knowledge of probabilities may not affect their decisions |journal=Medical Decision Making |volume=15  |pages=65–75 |year=1995 |pmid=7898300 |doi=}}</ref>.
Studies are inconclusive on using cognitive feedback.<ref name="pmid7898300">{{cite journal |author=Poses RM ''et al.''|title=You can lead a horse to water--improving physicians' knowledge of probabilities may not affect their decisions |journal=Medical Decision Making |volume=15  |pages=65–75 |year=1995 |pmid=7898300 |doi=}}</ref>.
Framing bias is best avoided by using [[numeracy]] with absolute measures of efficacy.<ref name="pmid21792695">{{cite journal| author=Perneger TV, Agoritsas T| title=Doctors and Patients' Susceptibility to Framing Bias: A Randomized Trial. | journal=J Gen Intern Med | year= 2011 | volume=  | issue=  | pages=  | pmid=21792695 | doi=10.1007/s11606-011-1810-x | pmc= | url= }} </ref>
==Assessing the teaching of evidence-based medicine==
Two [[systematic review]]s of EBM provide the framework below for measuring outcomes.<ref name="pmid16954491">{{cite journal |author=Shaneyfelt  T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, Whelan C, Green M  |title=Instruments for evaluating education in evidence-based practice:  a systematic review |journal=JAMA |volume=296  |pages=1116–27  |year=2006 |pmid=16954491 |doi=10.1001/jama.296.9.1116|url=http://jama.ama-assn.org/cgi/content/full/296/9/1116}}</ref><ref name="pmid15514352">{{cite journal |author=Straus  SE, Green ML, Bell DS, Badgett R, Davis D, Gerrity M, Ortiz E,  Shaneyfelt TM, Whelan C, Mangrulkar R; Society of General Internal  Medicine Evidence-Based Medicine Task Force |title=Evaluating the teaching of evidence based medicine: conceptual framework |journal=BMJ |volume=329  |pages=1029–32 |year=2004 |pmid=15514352 |doi=10.1136/bmj.329.7473.1029}}</ref> Two instruments, the Berlin questionnaire<ref name="pmid12468485">{{cite journal| author=Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R| title=Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. | journal=BMJ | year= 2002 | volume= 325 | issue= 7376 | pages= 1338-41 | pmid=12468485 | doi= | pmc=PMC137813 | url= }} </ref> and the [http://www.bmj.com/content/suppl/2003/02/10/326.7384.319.DC1 Fresno test]<ref name="pmid12574047">{{cite journal| author=Ramos KD, Schafer S, Tracz SM| title=Validation of the Fresno test of competence in evidence based medicine. | journal=BMJ | year= 2003 | volume= 326 | issue= 7384 | pages= 319-21 | pmid=12574047 | doi= | pmc=PMC143529 | url= }} </ref> are the most validated.<ref name="pmid16954491"/><ref name="pmid15514352"/> These questionnaires have been used in diverse settings.<ref name="pmid21686291">{{cite journal| author=Kunz R, Wegscheider K, Fritsche L, Schünemann HJ, Moyer V, Miller D et al.| title=Determinants of knowledge gain in evidence-based medicine short courses: an international assessment. | journal=Open Med | year= 2010 | volume= 4 | issue= 1 | pages= e3-e10 | pmid=21686291 | doi= | pmc=PMC3116678 | url= }} </ref><ref name="pmid21286836">{{cite journal| author=West CP, Jaeger TM, McDonald FS| title=Extended evaluation of a longitudinal medical school evidence-based medicine curriculum. | journal=J Gen Intern Med | year= 2011 | volume= 26 | issue= 6 | pages= 611-5 | pmid=21286836 | doi=10.1007/s11606-011-1642-8 | pmc=PMC3101983 | url= }} </ref> The Fresno test is more thorough but the Berlin Questionnaire is easier to administer.<ref name="pmid16954491"/>


==Studies of the effectiveness of teaching evidence-based medicine==
==Studies of the effectiveness of teaching evidence-based medicine==
A systematic review of the effectiveness of teaching EBM concluded "standalone teaching improved knowledge but not skills, attitudes or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes and behaviour."<ref name="pmid15514348">{{cite journal |author=Coomarasamy A, Khan KS |title=What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review |journal=BMJ |volume=329 |pages=1017 |year=2004 |pmid=15514348 |doi=10.1136/bmj.329.7473.1017}}</ref> A second review concluded improvements in unvalidated measures of "knowledge, skills, attitudes or behavior."<ref name="pmid17655743">{{cite journal |author=Flores-Mateo G, Argimon JM |title=Evidence based practice in postgraduate healthcare education: a systematic review |journal=BMC Health Services Research |volume=7 |pages=119 |year=2007 |pmid=17655743 |doi=10.1186/1472-6963-7-119}}</ref> Neither review examined improvements in clinical care.
A systematic review of the effectiveness of teaching EBM concluded "standalone teaching improved knowledge but not skills, attitudes or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes and behaviour."<ref name="pmid15514348">{{cite journal |author=Coomarasamy A, Khan KS |title=What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review |journal=BMJ |volume=329 |pages=1017 |year=2004 |pmid=15514348 |doi=10.1136/bmj.329.7473.1017}}</ref> A second review concluded improvements in unvalidated measures of "knowledge, skills, attitudes or behavior."<ref name="pmid17655743">{{cite journal |author=Flores-Mateo G, Argimon JM |title=Evidence based practice in postgraduate healthcare education: a systematic review |journal=BMC Health Services Research |volume=7 |pages=119 |year=2007 |pmid=17655743 |doi=10.1186/1472-6963-7-119}}</ref> Neither review examined improvements in clinical care.
Two [[systematic review]]s of EBM provide the framework below for measuring outcomes.<ref name="pmid16954491">{{cite journal |author=Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, Whelan C, Green M |title=Instruments for evaluating education in evidence-based practice: a systematic review |journal=JAMA |volume=296  |pages=1116–27 |year=2006 |pmid=16954491 |doi=10.1001/jama.296.9.1116|url=http://jama.ama-assn.org/cgi/content/full/296/9/1116}}</ref><ref name="pmid15514352">{{cite journal |author=Straus SE, Green ML, Bell DS, Badgett R, Davis D, Gerrity M, Ortiz E, Shaneyfelt TM, Whelan C, Mangrulkar R; Society of General Internal Medicine Evidence-Based Medicine Task Force |title=Evaluating the teaching of evidence based medicine: conceptual framework |journal=BMJ |volume=329  |pages=1029–32 |year=2004 |pmid=15514352 |doi=10.1136/bmj.329.7473.1029}}</ref> Two good instruments for evaluating learners of EBM are the Fresno Test<ref name="pmid12574047">{{cite journal |author=Ramos KD, Schafer S, Tracz SM |title=Validation of the Fresno test of competence in evidence based medicine |journal=BMJ |volume=326 |issue=7384 |pages=319–21 |year=2003 |month=February |pmid=12574047 |pmc=143529 |doi= |url=http://bmj.com/cgi/pmidlookup?view=long&pmid=12574047 |issn=|quote=Test items are listed at http://www.bmj.com/cgi/content/full/326/7384/319/DC1 }}</ref> and Berlin Questionnaire<ref name="pmid12468485">{{cite journal |author=Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R |title=Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine |journal=BMJ |volume=325 |issue=7376 |pages=1338–41 |year=2002 |month=December |pmid=12468485 |pmc=137813 |doi= |url=http://bmj.com/cgi/pmidlookup?view=long&pmid=12468485 |issn=}}</ref>. The Fresno test is more thorough but the Berlin Questionnaire is easier to administer.<ref name="pmid16954491"/>


===Information retrieval===
===Information retrieval===
Line 71: Line 94:
A [[randomized controlled trial]] of volunteer senior medical students found that access to information portal on a handheld computer increased self-reported use of information.<ref name="pmid14604933">{{cite journal |author=Leung GM ''et al.'' |title=Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students |journal=BMJ |volume=327 |pages=1090 |year=2003 |pmid=14604933 |doi=10.1136/bmj.327.7423.1090}}</ref> The information portal contained multiple pre-appraised resources, including a textbook and drug resource, and would best resemble the "user" mode. The study was not able to isolate which resources in the portal had increased use. It is possible that the benefit was solely due to the textbook or drug resource.
A [[randomized controlled trial]] of volunteer senior medical students found that access to information portal on a handheld computer increased self-reported use of information.<ref name="pmid14604933">{{cite journal |author=Leung GM ''et al.'' |title=Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students |journal=BMJ |volume=327 |pages=1090 |year=2003 |pmid=14604933 |doi=10.1136/bmj.327.7423.1090}}</ref> The information portal contained multiple pre-appraised resources, including a textbook and drug resource, and would best resemble the "user" mode. The study was not able to isolate which resources in the portal had increased use. It is possible that the benefit was solely due to the textbook or drug resource.


A randomized controlled trial of teaching and encouraging use of [[MEDLINE]] by medical resident physicians showed increased searching for evidence during 6-8 weeks of observation.<ref name="pmid11903763">{{cite journal |author=Cabell CH ''et al.''|title=Resident utilization of information technology |journal=J Gen Intern Med |volume=16 |pages=838–44 |year=2001 |pmid=11903763 |doi=}}</ref> Based on the median number of searches and hours spent searching, each search averaged 22 minutes, which may not be sustainable over the long term.
A randomized controlled trial of teaching and encouraging use of [[MEDLINE]] by medical resident physicians showed increased searching for evidence during 6-8 weeks of observation.<ref name="pmid11903763"/> Based on the median number of searches and hours spent searching, each search averaged 22 minutes, which may not be sustainable over the long term.


A [[randomized controlled trial]] studied the effect of a searching tutorial that met for up to 6 1-hour small group sessions to search for answers to questions about their hospitalized patients. The intervention did not improve their success or speed in completing an Objective Structured Searching Evaluation (OSSE).<ref name="pmid17922170">{{cite journal |author=Stark R, Helenius IM, Schimming LM, Takahara N, Kronish I, Korenstein D |title=Real-time EBM: from bed board to keyboard and back |journal=J Gen Intern Med |volume=22 |issue=12 |pages=1656–60 |year=2007 |pmid=17922170 |doi=10.1007/s11606-007-0387-x |issn=}}</ref>
A [[randomized controlled trial]] studied the effect of a searching tutorial that met for up to 6 1-hour small group sessions to search for answers to questions about their hospitalized patients. The intervention did not improve their success or speed in completing an Objective Structured Searching Evaluation (OSSE).<ref name="pmid17922170">{{cite journal |author=Stark R, Helenius IM, Schimming LM, Takahara N, Kronish I, Korenstein D |title=Real-time EBM: from bed board to keyboard and back |journal=J Gen Intern Med |volume=22 |issue=12 |pages=1656–60 |year=2007 |pmid=17922170 |doi=10.1007/s11606-007-0387-x |issn=}}</ref>
An evidence-based medicine course consisting of six 2-hour workshop sessions shifted the types of resources learners used during a patient vignette.<ref name="pmid18769979">{{cite journal |author=Kim S, Willett LR, Murphy DJ, O'Rourke K, Sharma R, Shea JA |title=Impact of an evidence-based medicine curriculum on resident use of electronic resources: a randomized controlled study |journal=J Gen Intern Med |volume=23 |issue=11 |pages=1804–8 |year=2008 |month=November |pmid=18769979 |doi=10.1007/s11606-008-0766-y |url=http://dx.doi.org/10.1007/s11606-008-0766-y |issn=}}</ref>


====Improving clinical care====
====Improving clinical care====
Teaching "''user''" mode only using syntheses and synopses, without summaries, has not shown benefit in two studies. A controlled trial of teaching the "user" mode (see above) was negative.<ref name="pmid17683300">{{cite journal |author=Shuval K ''et al.'' |title=Evaluating the impact of an evidence-based medicine educational intervention on primary care doctors' attitudes, knowledge and clinical behaviour: a controlled trial and before and after study |journal=Journal of Evaluation in Clinical Practice |volume=13 |pages=581–98 |year=2007 |pmid=17683300 |doi=10.1111/j.1365-2753.2007.00859.x}}</ref> However, this study encouraged the use of syntheses and synopses and did not encourage the more practical "summaries" (evidence-based textbooks) of the "5S" search strategy.<ref name="pmid17080967">{{cite journal |author=Haynes RB |title=Of studies, syntheses, synopses, summaries, and systems: the "5S" evolution of information services for evidence-based health care decisions |journal=ACP J Club |volume=145 |pages=A8 |year=2006 |pmid=17080967 |doi=}}</ref> A quasi-randomized, controlled investigation of teaching medical students the use of studies, syntheses, and synopses using an automated search engine was negative.<ref name="pmid11532204">{{cite journal |author=Badgett RG ''et al.''|title=Teaching clinical informatics to third-year medical students: negative results from two controlled trials |journal=BMC Medical Education |volume=1 |pages=3 |year=2001 |pmid=11532204 |doi=}}[http://www.pubmedcentral.nih.gov/articlerender.fcgi?tool=pubmed&pubmedid=11532204 PubMed Central]</ref>
Teaching "''user''" mode only using syntheses and synopses, without summaries, has not shown benefit in two studies. A controlled trial of teaching the "user" mode (see above) was negative.<ref name="pmid17683300"/> However, this study encouraged the use of syntheses and synopses and did not encourage the more practical "summaries" (evidence-based textbooks) of the "5S" search strategy.<ref name="pmid17080967">{{cite journal |author=Haynes RB |title=Of studies, syntheses, synopses, summaries, and systems: the "5S" evolution of information services for evidence-based health care decisions |journal=ACP J Club |volume=145 |pages=A8 |year=2006 |pmid=17080967 |doi=}}</ref> A quasi-randomized, controlled investigation of teaching medical students the use of studies, syntheses, and synopses using an automated search engine was negative.<ref name="pmid11532204"/>


===Information awareness===
===Information awareness===
Line 86: Line 111:


===Clinical reasoning===
===Clinical reasoning===
Teaching Bayesian understanding of diagnostic testing improved actual test ordering in a controlled trial.<ref name="pmid2492066">{{cite journal |author=Davidoff F, Goodspeed R, Clive J |title=Changing test ordering behavior. A randomized controlled trial comparing probabilistic reasoning with cost-containment education |journal=Med Care |volume=27 |issue=1 |pages=45–58 |year=1989 |pmid=2492066 |doi=}}</ref>
Teaching Bayesian understanding of diagnostic testing improved actual test ordering in a controlled trial.<ref name="pmid2492066"/>
 
Improving physicians' probability judgments did not lead to improvements in clinical decision making in a controlled before and after study.<ref name="pmid7898300"/>


Improving physicians' probability judgments did not lead to improvements in clinical decision making in a controlled before and after study.<ref name="pmid7898300">{{cite journal |author=Poses RM, Cebul RD, Wigton RS |title=You can lead a horse to water--improving physicians' knowledge of probabilities may not affect their decisions |journal=Med Decis Making |volume=15 |issue=1 |pages=65–75 |year=1995 |pmid=7898300 |doi=}}</ref>
Encouraging the use of reflecting reasoning by suggesting to diagnosticians that a case is especially difficult may improve diagnositc accuracy.<ref>Mamede S, et al. (2008) Influence of Perceived Difficulty of Cases on Physicians' Diagnostic Reasoning. Academic Medicine. 83(12):1210-1216. {{doi|10.1097/ACM.0b013e31818c71d7}}</ref>
 
Teaching the SNAPPS method of student and preceptor interaction may help:<ref>Wolpaw, T, MHPE; Papp, KK; Bordage, G. Using SNAPPS to Facilitate the Expression of Clinical Reasoning and Uncertainties: A Randomized Comparison Group Trial. Academic Medicine April 2009;84(4):517-524 {{DOI|10.1097/ACM.0b013e31819a8cbf}}
</ref>
* '''S'''ummarize history and findings
* '''N'''arrow the differential
* '''A'''nalyze the differential
* '''P'''robe preceptor about uncertainties
* '''P'''lan management
* '''S'''elect case-related issues for self-study


===Critical appraisal===
===Critical appraisal===
There are no studies that teaching critical appraisal (including journal clubs), in isolation, improves clinical care. A systematic review was inconclusive whether journal clubs improved information habits.<ref name="pmid8448709">{{cite journal |author=Audet N, Gagnon R, Ladouceur R, Marcil M |title=[How effective is the teaching of critical analysis of scientific publications? Review of studies and their methodological quality] |language=French |journal=CMAJ : Canadian Medical Association journal &#61; journal de l'Association medicale canadienne |volume=148 |issue=6 |pages=945–52 |year=1993 |pmid=8448709 |doi= |issn=}}</ref>
There are no studies that teaching critical appraisal (including journal clubs), in isolation, improves clinical care. A systematic review was inconclusive whether journal clubs improved information habits.<ref name="pmid8448709">{{cite journal |author=Audet N, Gagnon R, Ladouceur R, Marcil M |title=[How effective is the teaching of critical analysis of scientific publications? Review of studies and their methodological quality] |language=French |journal=CMAJ : Canadian Medical Association journal &#61; journal de l'Association medicale canadienne |volume=148 |issue=6 |pages=945–52 |year=1993 |pmid=8448709 |doi= |issn=}}</ref>


More recently, a [[randomized controlled trial]] found that the READER model (Relevance, Education, Applicability, Discrimination, overall Evaluation) increased the critical appraisal skills of learners.<ref name="pmid9552953">{{cite journal |author=MacAuley D, McCrum E, Brown C |title=Randomised controlled trial of the READER method of critical appraisal in general practice |journal=BMJ |volume=316 |issue=7138 |pages=1134–7 |year=1998 |pmid=9552953 |doi= |issn=}}</ref>
However, [[randomized controlled trial]]s have found that the READER model (Relevance, Education, Applicability, Discrimination, overall Evaluation)<ref name="pmid9552953">{{cite journal |author=MacAuley D, McCrum E, Brown C |title=Randomised controlled trial of the READER method of critical appraisal in general practice |journal=BMJ |volume=316 |issue=7138 |pages=1134–7 |year=1998 |pmid=9552953 |doi= |issn=}}</ref> and other methods<ref name="pmid19707728">{{cite journal| author=Harewood GC, Hendrick LM| title=Prospective, controlled assessment of the impact of formal evidence-based medicine teaching workshop on ability to appraise the medical literature. | journal=Ir J Med Sci | year= 2010 | volume= 179 | issue= 1 | pages= 91-4 | pmid=19707728 | doi=10.1007/s11845-009-0411-8 | pmc= | url= }} </ref> increase the critical appraisal skills of learners.


If being a successful journal club is defined as one that has high attendance or is long-standing, then successful attributes are "associated with smaller residency programs, making attendance  mandatory, promoting a journal club independent of faculty,  providing formal teaching of critical appraisal skills, making food available,  and emphasizing original research articles."<ref name="pmid7763125">{{cite journal |author=Sidorov J |title=How are internal medicine residency journal clubs organized, and what makes them successful? |journal=Arch. Intern. Med. |volume=155 |issue=11 |pages=1193–7 |year=1995 |pmid=7763125 |doi= |issn=}}</ref>
If being a successful journal club is defined as one that has high attendance or is long-standing, then successful attributes are "associated with smaller residency programs, making attendance  mandatory, promoting a journal club independent of faculty,  providing formal teaching of critical appraisal skills, making food available,  and emphasizing original research articles."<ref name="pmid7763125">{{cite journal |author=Sidorov J |title=How are internal medicine residency journal clubs organized, and what makes them successful? |journal=Arch. Intern. Med. |volume=155 |issue=11 |pages=1193–7 |year=1995 |pmid=7763125 |doi= |issn=}}</ref>
==References==
<references/>


==See also==
==See also==
Line 105: Line 138:
* [[Sensitivity and specificity]]
* [[Sensitivity and specificity]]


==References==
{{reflist|2}}


[[Category:CZ Live]] [[Category:Health Sciences Workgroup]]
[[Category:Suggestion Bot Tag]]

Latest revision as of 08:18, 30 October 2024

This article is developing and not approved.
 Main Article
 Discussion
Related Articles  [?]
Bibliography  [?]
External Links  [?]
Citable Version  [?]
 
https://en.citizendium.org/wiki/index.php?title=Teaching_evidence-based_medicine&printable=yes
This editable Main Article is under development and subject to a disclaimer.

Teaching evidence based medicine (EBM) is difficult for several reasons including lack of consensus on what skills physicians need to have and also due to physicians' possibly being over confident in their EBM skills.[1]

Most of this article is about teaching evidence-based individual decision making and not about evidence-based medicine at the institutional level. This first part of this article covers studies or reports of methods in EBM. The second part reviews studies on the effectiveness of teaching evidence-based medicine. Both sections may guide the teacher of EBM in selecting strategies, but the teacher should note that only ideas on the second section have empiric data on effectiveness.

Methods for teaching EBM

A multinational European group has used the Delphi method to design a comprehensive EBM course based on five steps of EBM.[2] The course is clinically integrated. Its effectiveness is at least as good as traditional teaching.[3] Their proposed learning objectives are available online:

  • 'generate structured questions arising from clinical problems in practice'
  • 'search relevant literature, identifying systematic reviews wherever possible'
  • 'assess the quality (validity) of systematic reviews and primary research included within them'
  • 'assess the applicability of research findings in clinical practice'
  • 'effectively implement the output from above activities into clinical practice'

The 6Ts Teaching Tips have been recommended by experienced teachers of evidence-based medicine:[4]

  1. Time management. Plan in advance how class time will be used.
  2. Teamwork. Encourage all of the class to participate.
  3. Tools. Use audio, visual, and multimedia to improve learning.
  4. Triage. Be prepared to triage class content and to eliminate content if time is short.
  5. Tone. Create a respectful and non-threatening tone in the class.
  6. Take home messages. End with take home messages.

Formulation of the clinical question ('PICO')

The 'PICO' format, which is mnemonic for:[5]

  • P - patients, problems, or population
  • I - intervention, exposure, or test
  • C - comparison
  • O - outcome

Although PICO cannot represent all clinical question, especially questions that are not about interventions[6], use of PICO may improve the precision of searches[7].

Search strategies

A strategy for information retrieval similar to the 5S strategy should be taught for use when the searcher has limited time available during clinical care. This is based on one positive study of its use[8] and two negative studies[9][10] of teaching the use using secondary and primary publications. In addition, indirect evidence on the time needed to search also supports the emphasis on using tertiary publications. Doctors may have two minutes available to search[11], whereas using MEDLINE may take 20 minutes or more.[12][13]

Teaching MEDLINE searching would be appropriate for Doers who might be willing to invest time in searching MEDLINE when not hurried by clinical care. Based on studies of common errors in searching MEDLINE, learners should be taught Medical Subject Headings (MeSH) terms and their explosion, appropriate limits, and best evidence to search for.[14] The mnemonic PEARL may guide how to each.[15] PEARL stands for:

  1. "Choose a 'Preplanned search intervention'"
  2. "Allow learners to 'Execute the search,' thus committing themselves"
  3. "'Allow learners to teach other learners' about their search process
  4. "'Review the quality of evidence' for the information found"
  5. "Discuss 'Lessons of the search.'"

A subsequent negative study has been published about the effect of a single session of learning to search.[16]

Critical appraisal

It is very common to read only the abstract of an article.[17] However, many abstracts contain errors compared to the body of the article.[18] Fortunately, these are usually errors of omission rather than contradiction.

One strategy for teaching critical appraisal has been to make a goal of a journal club include the goal "composing, editing, and submitting a [group] letter to the editor[19]

Clinical reasoning

There are various methods of clinical reasoning include probabilistic (Bayesian), causal (physiologic), and deterministic (rule-based).[20] In addition, medical experts rely more on pattern recognition which is faster[21]; however, clinical experts seem flexible and may use whichever method of reasoning most easily represents and solves a given problem.[22] When confronted with non-challenging cases physicians may use 'nonanalytical reasoning' such as pattern recognition; however, during more difficult cases physicians may switch to 'reflective reasoning'.[23] Reflective reasoning may especially help complex cases.[24] Explicit Bayesian thinking with precise numbers is rarely done.[25][26] Basic science knowledge is probably "encapsulated" into clinical knowledge.[27]

Successfully distilling complex information into a short summary, perhaps using semantic qualifiers, may help diagnostic accuracy.[28][29][30][31]

Competing-hypotheses heuristic[32]
Finding Disease A Disease B
Fever 66% cell B
Rash cell C cell D
The most important missing information is cell B

Possible strategies to improve clinical reasoning have been reviewed[33][34] and using problem-based learning[34], include teaching appropriate problem representation creating a one-sentence summary of a case[33], standardized patients[35], teaching hypothetico-deductive reasoning[36][37], cognitive forcing strategies[38][39][40] to avoid premature closure[41], teaching the competing-hypotheses heuristic[32], using fuzzy-trace theory[42] and mixed-methods interventions[43][44][43]. Studies are unclear about teaching logic.[45][46] Regarding hypothetico-deductive reasoning, an observational study on the methods used by experts solving clinicopathological exercises reported that these experts use the following six steps:[37]

  1. aggregation of groups of findings into patterns
  2. selection of a "pivot" or key finding
  3. generation of a cause list
  4. pruning of the cause list
  5. selection of a diagnosis
  6. validation of the diagnosis

Scales to measure clinical reasoning have been proposed.[47]

Numeracy

Various methods have been proposed for improving quantitative literacy. For diagnosis, likelihood ratios[48] or sensitivity and specificity are two methods. It is unclear which method is better according to the results of a controlled trial.[49]

Studies are inconclusive on using cognitive feedback.[50].

Framing bias is best avoided by using numeracy with absolute measures of efficacy.[51]

Assessing the teaching of evidence-based medicine

Two systematic reviews of EBM provide the framework below for measuring outcomes.[52][53] Two instruments, the Berlin questionnaire[54] and the Fresno test[55] are the most validated.[52][53] These questionnaires have been used in diverse settings.[56][57] The Fresno test is more thorough but the Berlin Questionnaire is easier to administer.[52]

Studies of the effectiveness of teaching evidence-based medicine

A systematic review of the effectiveness of teaching EBM concluded "standalone teaching improved knowledge but not skills, attitudes or behaviour. Clinically integrated teaching improved knowledge, skills, attitudes and behaviour."[58] A second review concluded improvements in unvalidated measures of "knowledge, skills, attitudes or behavior."[59] Neither review examined improvements in clinical care.

Information retrieval

Increasing use of information

A randomized controlled trial of volunteer senior medical students found that access to information portal on a handheld computer increased self-reported use of information.[60] The information portal contained multiple pre-appraised resources, including a textbook and drug resource, and would best resemble the "user" mode. The study was not able to isolate which resources in the portal had increased use. It is possible that the benefit was solely due to the textbook or drug resource.

A randomized controlled trial of teaching and encouraging use of MEDLINE by medical resident physicians showed increased searching for evidence during 6-8 weeks of observation.[13] Based on the median number of searches and hours spent searching, each search averaged 22 minutes, which may not be sustainable over the long term.

A randomized controlled trial studied the effect of a searching tutorial that met for up to 6 1-hour small group sessions to search for answers to questions about their hospitalized patients. The intervention did not improve their success or speed in completing an Objective Structured Searching Evaluation (OSSE).[61]

An evidence-based medicine course consisting of six 2-hour workshop sessions shifted the types of resources learners used during a patient vignette.[62]

Improving clinical care

Teaching "user" mode only using syntheses and synopses, without summaries, has not shown benefit in two studies. A controlled trial of teaching the "user" mode (see above) was negative.[9] However, this study encouraged the use of syntheses and synopses and did not encourage the more practical "summaries" (evidence-based textbooks) of the "5S" search strategy.[63] A quasi-randomized, controlled investigation of teaching medical students the use of studies, syntheses, and synopses using an automated search engine was negative.[10]

Information awareness

A cluster randomized trial of McMaster Premium LiteratUre Service (PLUS) led to " increased the utilization of evidence-based information from a digital library by practicing physicians."[64]

No controlled studies have addressed improving clinical care by use of information awareness strategies.

A controlled trial of teaching Bayes Theorem (probabilistic reasoning) "improves the efficiency of test ordering."[65]

Clinical reasoning

Teaching Bayesian understanding of diagnostic testing improved actual test ordering in a controlled trial.[65]

Improving physicians' probability judgments did not lead to improvements in clinical decision making in a controlled before and after study.[50]

Encouraging the use of reflecting reasoning by suggesting to diagnosticians that a case is especially difficult may improve diagnositc accuracy.[66]

Teaching the SNAPPS method of student and preceptor interaction may help:[67]

  • Summarize history and findings
  • Narrow the differential
  • Analyze the differential
  • Probe preceptor about uncertainties
  • Plan management
  • Select case-related issues for self-study

Critical appraisal

There are no studies that teaching critical appraisal (including journal clubs), in isolation, improves clinical care. A systematic review was inconclusive whether journal clubs improved information habits.[68]

However, randomized controlled trials have found that the READER model (Relevance, Education, Applicability, Discrimination, overall Evaluation)[69] and other methods[70] increase the critical appraisal skills of learners.

If being a successful journal club is defined as one that has high attendance or is long-standing, then successful attributes are "associated with smaller residency programs, making attendance mandatory, promoting a journal club independent of faculty, providing formal teaching of critical appraisal skills, making food available, and emphasizing original research articles."[71]

See also

References

  1. ↑ Caspi O, McKnight P, Kruse L, Cunningham V, Figueredo AJ, Sechrest L (2006). "Evidence-based medicine: discrepancy between perceived competence and actual performance among graduating medical students". Med Teach 28 (4): 318–25. DOI:10.1080/01421590600624422. PMID 16807169. Research Blogging.
  2. ↑ Coppus SF, Emparanza JI, Hadley J, et al (2007). "A clinically integrated Curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project". BMC Med Educ 7 (1): 46. DOI:10.1186/1472-6920-7-46. PMID 18042271. Research Blogging.
  3. ↑ Kulier R, Coppus SF, Zamora J, Hadley J, Malick S, Das K, Weinbrenner S, Meyerrose B, Decsi T, Horvath AR, Nagy E, Emparanza JI, Arvanitis TN, Burls A, Cabello JB, Kaczor M, Zanrei G, Pierer K, Stawiarz K, Kunz R, Mol BW, Khan KS. The effectiveness of a clinically integrated e-learning course in evidence-based medicine: a cluster randomised controlled trial. BMC Med Educ. 2009 May 12;9:21. PMID 19435520
  4. ↑ Cook D, Figurski J, Patel R, Burneo J, Langlands S, Keitz S (2007). "6Ts teaching tips for evidence-based practitioners". Evid Based Med 12 (4): 100–1. DOI:10.1136/ebm.12.4.100. PMID 17885149. Research Blogging.
  5. ↑ Richardson WS, Wilson MC, Nishikawa J, Hayward RS (1995). "The well-built clinical question: a key to evidence-based decisions". ACP J. Club 123 (3): A12–3. PMID 7582737. [e]
  6. ↑ Huang X, Lin J, Demner-Fushman D (2006). "Evaluation of PICO as a knowledge representation for clinical questions". AMIA Annu Symp Proc: 359–63. PMID 17238363. PMC 1839740. [e]
  7. ↑ Schardt C, Adams MB, Owens T, Keitz S, Fontelo P (2007). "Utilization of the PICO framework to improve searching PubMed for clinical questions". BMC Med Inform Decis Mak 7: 16. DOI:10.1186/1472-6947-7-16. PMID 17573961. PMC 1904193. Research Blogging.
  8. ↑ Patel MR et al. (2006). "Randomized trial for answers to clinical questions: evaluating a pre-appraised versus a MEDLINE search protocol". JMLA 94: 382–7. PMID 17082828. [e]
  9. ↑ 9.0 9.1 Shuval K, Berkovits E, Netzer D, et al (2007). "Evaluating the impact of an evidence-based medicine educational intervention on primary care doctors' attitudes, knowledge and clinical behaviour: a controlled trial and before and after study". Journal of evaluation in clinical practice 13 (4): 581–98. DOI:10.1111/j.1365-2753.2007.00859.x. PMID 17683300. Research Blogging.
  10. ↑ 10.0 10.1 Badgett RG, Paukert JL, Levy LS (2001). "Teaching clinical informatics to third-year medical students: negative results from two controlled trials". BMC medical education 1: 3. PMID 11532204. [e]
  11. ↑ Ely JW et al. (1999). "Analysis of questions asked by family doctors regarding patient care". BMJ 319: 358–61. PMID 10435959. [e]
  12. ↑ Chambliss ML, Conley J (1996). "Answering clinical questions". The Journal of Family Practice 43: 140–4. PMID 8708623. [e]
  13. ↑ 13.0 13.1 Cabell CH et al. (2001). "Resident utilization of information technology". J Gen Intern Med 16: 838–44. PMID 11903763. [e]
  14. ↑ Gruppen LD et al. (2005). "A controlled comparison study of the efficacy of training medical students in evidence-based medicine literature searching skills". Academic medicine 80: 940–4. PMID 16186614. [e]
  15. ↑ Silk H et al. (2006). "A new way to integrate clinically relevant technology into small-group teaching". Academic Medicine 81: 239–44. PMID 16501264. [e]
  16. ↑ Ilic D, Tepper K, Misso M (2012). "Teaching evidence-based medicine literature searching skills to medical students during the clinical years: a randomized controlled trial.". J Med Libr Assoc 100 (3): 190-6. DOI:10.3163/1536-5050.100.3.009. PMID 22879808. PMC PMC3411272. Research Blogging.
  17. ↑ Saint S, Christakis DA, Saha S, et al (2000). "Journal reading habits of internists". J Gen Intern Med 15 (12): 881–4. PMID 11119185. [e]
  18. ↑ Pitkin RM, Branagan MA, Burmeister LF (1999). "Accuracy of data in abstracts of published research articles". JAMA 281 (12): 1110–1. PMID 10188662. [e]
  19. ↑ Kallen AJ, Wilson CT, Russell MA, et al (2006). "Group writing of letters to the editor as the goal of journal club". JAMA 296 (9): 1053–4. DOI:10.1001/jama.296.9.1053. PMID 16954483. Research Blogging.
  20. ↑ Kassirer JP (1989). "Diagnostic reasoning". Ann Intern Med 110: 893–900. PMID 2655522. [e]
  21. ↑ Leape LL (1994). "Error in medicine". JAMA 272: 1851–7. PMID 7503827. [e]
  22. ↑ Norman G (2006). "Building on experience--the development of clinical reasoning". N Engl J Med 355: 2251–2. DOI:10.1056/NEJMe068134. PMID 17124025. Research Blogging.
  23. ↑ Mamede S, et al. (2008) Influence of Perceived Difficulty of Cases on Physicians' Diagnostic Reasoning. Academic Medicine. 83(12):1210-1216. DOI:10.1097/ACM.0b013e31818c71d7
  24. ↑ Mamede S, Schmidt HG, Penaforte JC (May 2008). "Effects of reflective practice on the accuracy of medical diagnoses". Med Educ 42 (5): 468–75. DOI:10.1111/j.1365-2923.2008.03030.x. PMID 18412886. Research Blogging.
  25. ↑ Moskowitz AJ et al. (1988). "Dealing with uncertainty, risks, and tradeoffs in clinical decisions. A cognitive science approach". Ann. Intern. Med. 108: 435–49. PMID 3277516. [e]
  26. ↑ Reid MC, Lane DA, Feinstein AR (1998). "Academic calculations versus clinical judgments: practicing physicians' use of quantitative measures of test accuracy". Am J Med 104: 374–80. PMID 9576412. [e]
  27. ↑ de Bruin AB et al. (2005). "The role of basic science knowledge and clinical knowledge in diagnostic reasoning: a structural equation modeling approach". Academic Medicine 80: 765–73. PMID 16043534. [e]
  28. ↑ Chang RW, Bordage G, Connell KJ (1998). "The importance of early problem representation during case presentations.". Acad Med 73 (10 Suppl): S109-11. PMID 9795669.
  29. ↑ Bordage G, Lemieux M (1991). "Semantic structures and diagnostic thinking of experts and novices.". Acad Med 66 (9 Suppl): S70-2. PMID 1930535.
  30. ↑ Nendaz MR, Bordage G (2002). "Promoting diagnostic problem representation.". Med Educ 36 (8): 760-6. PMID 12191059.
  31. ↑ Kulatunga-Moruzi C, Brooks LR, Norman GR (2004). "Using comprehensive feature lists to bias medical diagnosis.". J Exp Psychol Learn Mem Cogn 30 (3): 563-72. DOI:10.1037/0278-7393.30.3.563. PMID 15099125. Research Blogging.
  32. ↑ 32.0 32.1 Wolf FM et al. (1988). "Use of the competing-hypotheses heuristic to reduce 'pseudodiagnosticity'". J Med Educ 63: 548–54. PMID 3385753. [e]
  33. ↑ 33.0 33.1 Bowen JL (2006). "Educational strategies to promote clinical diagnostic reasoning". N Engl J Med 355: 2217–25. DOI:10.1056/NEJMra054782. PMID 17124019. Research Blogging.
  34. ↑ 34.0 34.1 Graber M et al. (2002). "Reducing diagnostic errors in medicine: what's the goal?". Academic Medicine 77: 981–92. PMID 12377672. [e]
  35. ↑ Windish DM et al. (2005). "Teaching medical students the important connection between communication and clinical reasoning". J Gen Intern Med 20: 1108–13. DOI:10.1111/j.1525-1497.2005.0244.x. PMID 16423099. Research Blogging.
  36. ↑ Wiese J et al. (2002). "Improving oral presentation skills with a clinical reasoning curriculum: a prospective controlled study". Am J Med 112: 212–8. PMID 11893348. [e]
  37. ↑ 37.0 37.1 Eddy DM, Clanton CH (1982). "The art of diagnosis: solving the clinicopathological exercise". N Engl J Med 306: 1263–8. PMID 7070446. [e]
  38. ↑ Croskerry P (2000). "The cognitive imperative: thinking about how we think". Academic Emergency Medicine 7: 1223–31. PMID 11073470. [e]
  39. ↑ Croskerry P (2002). "Achieving quality in clinical decision making: cognitive strategies and detection of bias". Academic Emergency Medicine 9: 1184–204. PMID 12414468. [e]
  40. ↑ Croskerry P (2003). "Cognitive forcing strategies in clinical decisionmaking.". Ann Emerg Med 41 (1): 110-20. DOI:10.1067/mem.2003.22. PMID 12514691. Research Blogging.
  41. ↑ Dubeau CE et al. (1986). "Premature conclusions in the diagnosis of iron-deficiency anemia: cause and effect". Medical Decision Making 6: 169–73. PMID 3736379. [e]
  42. ↑ Lloyd FJ, Reyna VF (2001). "A web exercise in evidence-based medicine using cognitive theory". J Gen Intern Med 16: 94–9. PMID 11251760. [e] PubMed Central
  43. ↑ 43.0 43.1 Windish DM (2000). "Teaching medical students clinical reasoning skills". Acad Med 75 (1): 90. PMID 10667884. [e]
  44. ↑ Struyf E, Beullens J, Van Damme B, Janssen P, Jaspaert H (2005). "A new methodology for teaching clinical reasoning skills: problem solving clinical seminars". Med Teach 27 (4): 364–8. DOI:10.1080/01421590500046411. PMID 16024422. Research Blogging.
  45. ↑ Cheng PW et al. (1986). "Pragmatic versus syntactic approaches to training deductive reasoning". Cognitive Psychology 18: 293–328. DOI:10.1016/0010-0285(86)90002-2. PMID 3742999. Research Blogging.
  46. ↑ Jenicek M (2006). "The hard art of soft science: Evidence-Based Medicine, Reasoned Medicine or both?". Journal of Evaluation in Clinical Practice 12: 410–9. DOI:10.1111/j.1365-2753.2006.00718.x. PMID 16907682. Research Blogging.
  47. ↑ Boshuizen HP et al. (1997). "Measuring knowledge and clinical reasoning skills in a problem-based curriculum". Medical education 31: 115–21. PMID 9231115. [e]
  48. ↑ Richardson WS, Wilson MC, Keitz SA, Wyer PC (2007). "Tips for Teachers of Evidence-based Medicine: Making Sense of Diagnostic Test Results Using Likelihood Ratios". J Gen Intern Med. DOI:10.1007/s11606-007-0330-1. PMID 18064524. Research Blogging.
  49. ↑ Puhan MA, Steurer J, Bachmann LM, ter Riet G (2005). "A randomized trial of ways to describe test accuracy: the effect on physicians' post-test probability estimates". Ann. Intern. Med. 143 (3): 184–9. PMID 16061916. [e]
  50. ↑ 50.0 50.1 Poses RM et al. (1995). "You can lead a horse to water--improving physicians' knowledge of probabilities may not affect their decisions". Medical Decision Making 15: 65–75. PMID 7898300. [e]
  51. ↑ Perneger TV, Agoritsas T (2011). "Doctors and Patients' Susceptibility to Framing Bias: A Randomized Trial.". J Gen Intern Med. DOI:10.1007/s11606-011-1810-x. PMID 21792695. Research Blogging.
  52. ↑ 52.0 52.1 52.2 Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, Whelan C, Green M (2006). "Instruments for evaluating education in evidence-based practice: a systematic review". JAMA 296: 1116–27. DOI:10.1001/jama.296.9.1116. PMID 16954491. Research Blogging.
  53. ↑ 53.0 53.1 Straus SE, Green ML, Bell DS, Badgett R, Davis D, Gerrity M, Ortiz E, Shaneyfelt TM, Whelan C, Mangrulkar R; Society of General Internal Medicine Evidence-Based Medicine Task Force (2004). "Evaluating the teaching of evidence based medicine: conceptual framework". BMJ 329: 1029–32. DOI:10.1136/bmj.329.7473.1029. PMID 15514352. Research Blogging.
  54. ↑ Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R (2002). "Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine.". BMJ 325 (7376): 1338-41. PMID 12468485. PMC PMC137813. [e]
  55. ↑ Ramos KD, Schafer S, Tracz SM (2003). "Validation of the Fresno test of competence in evidence based medicine.". BMJ 326 (7384): 319-21. PMID 12574047. PMC PMC143529. [e]
  56. ↑ Kunz R, Wegscheider K, Fritsche L, Schünemann HJ, Moyer V, Miller D et al. (2010). "Determinants of knowledge gain in evidence-based medicine short courses: an international assessment.". Open Med 4 (1): e3-e10. PMID 21686291. PMC PMC3116678. [e]
  57. ↑ West CP, Jaeger TM, McDonald FS (2011). "Extended evaluation of a longitudinal medical school evidence-based medicine curriculum.". J Gen Intern Med 26 (6): 611-5. DOI:10.1007/s11606-011-1642-8. PMID 21286836. PMC PMC3101983. Research Blogging.
  58. ↑ Coomarasamy A, Khan KS (2004). "What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review". BMJ 329: 1017. DOI:10.1136/bmj.329.7473.1017. PMID 15514348. Research Blogging.
  59. ↑ Flores-Mateo G, Argimon JM (2007). "Evidence based practice in postgraduate healthcare education: a systematic review". BMC Health Services Research 7: 119. DOI:10.1186/1472-6963-7-119. PMID 17655743. Research Blogging.
  60. ↑ Leung GM et al. (2003). "Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students". BMJ 327: 1090. DOI:10.1136/bmj.327.7423.1090. PMID 14604933. Research Blogging.
  61. ↑ Stark R, Helenius IM, Schimming LM, Takahara N, Kronish I, Korenstein D (2007). "Real-time EBM: from bed board to keyboard and back". J Gen Intern Med 22 (12): 1656–60. DOI:10.1007/s11606-007-0387-x. PMID 17922170. Research Blogging.
  62. ↑ Kim S, Willett LR, Murphy DJ, O'Rourke K, Sharma R, Shea JA (November 2008). "Impact of an evidence-based medicine curriculum on resident use of electronic resources: a randomized controlled study". J Gen Intern Med 23 (11): 1804–8. DOI:10.1007/s11606-008-0766-y. PMID 18769979. Research Blogging.
  63. ↑ Haynes RB (2006). "Of studies, syntheses, synopses, summaries, and systems: the "5S" evolution of information services for evidence-based health care decisions". ACP J Club 145: A8. PMID 17080967. [e]
  64. ↑ Haynes RB et al. (2006). "McMaster PLUS: a cluster randomized clinical trial of an intervention to accelerate clinical use of evidence-based information from digital libraries". Journal of the American Medical Informatics Association : JAMIA 13: 593–600. DOI:10.1197/jamia.M2158. PMID 16929034. Research Blogging.
  65. ↑ 65.0 65.1 Davidoff F et al. (1989). "Changing test ordering behavior. A randomized controlled trial comparing probabilistic reasoning with cost-containment education". Medical care 27: 45–58. PMID 2492066. [e]
  66. ↑ Mamede S, et al. (2008) Influence of Perceived Difficulty of Cases on Physicians' Diagnostic Reasoning. Academic Medicine. 83(12):1210-1216. DOI:10.1097/ACM.0b013e31818c71d7
  67. ↑ Wolpaw, T, MHPE; Papp, KK; Bordage, G. Using SNAPPS to Facilitate the Expression of Clinical Reasoning and Uncertainties: A Randomized Comparison Group Trial. Academic Medicine April 2009;84(4):517-524 Template:DOI
  68. ↑ Audet N, Gagnon R, Ladouceur R, Marcil M (1993). "[How effective is the teaching of critical analysis of scientific publications? Review of studies and their methodological quality]" (in French). CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne 148 (6): 945–52. PMID 8448709. [e]
  69. ↑ MacAuley D, McCrum E, Brown C (1998). "Randomised controlled trial of the READER method of critical appraisal in general practice". BMJ 316 (7138): 1134–7. PMID 9552953. [e]
  70. ↑ Harewood GC, Hendrick LM (2010). "Prospective, controlled assessment of the impact of formal evidence-based medicine teaching workshop on ability to appraise the medical literature.". Ir J Med Sci 179 (1): 91-4. DOI:10.1007/s11845-009-0411-8. PMID 19707728. Research Blogging.
  71. ↑ Sidorov J (1995). "How are internal medicine residency journal clubs organized, and what makes them successful?". Arch. Intern. Med. 155 (11): 1193–7. PMID 7763125. [e]
Retrieved from "https://citizendium.org/wiki/index.php?title=Teaching_evidence-based_medicine&oldid=991541"