Published on February 20, 2014
February 2014 Western Events
CADTH CADTH is an independent, not-for-profit producer and broker of health technology assessments. Federal, provincial, and territorial health care decision-makers rely on our evidence-based information to make informed policy and practice decision about drugs and other health technologies. www.cadth.ca
Health Technologies • Pharmaceuticals • • Medical, surgical and dental devices and procedures Diagnostics
About Us • Amanda Hodgson Manager, Information Services • Caitlyn Ford Information Specialist • Our Team Literature searching for rapid reviews, health technology assessments and CADTH Common Drug Review
Objectives PART 1 • Role of the Information Specialist in HTA • Translating your research question • Applying search filters • Peer review of search strategies BREAK PART 2 • Grey literature searching • Grey Matters and Grey Matters Light • Closing and evaluation
Finding the Evidence • Finding the evidence: literature searching tools in support of systematic reviews http://cadth.ca/en/resources/finding-evidence-is
What is a Systematic Search? “Systematic reviews of interventions require a thorough, objective and reproducible search of a range of sources to identify as many relevant studies as possible (within resource limits). This is a major factor in distinguishing systematic reviews from traditional narrative reviews and helps to minimize bias and therefore assist in achieving reliable estimates of effects.” Higgins et al. Cochrane handbook for systematic reviews of interventions. 2008. Section 22.214.171.124
What is a Systematic Search? “…the aspiration of the HTA literature search should not be comprehensiveness but rather the minimization of bias.” Booth. Int J Technol Assess Health Care. 2010 Oct;26(4):431-5.
Fundamental Challenge of Searching Balancing Recall and Precision = getting "everything" while avoiding "junk" http://www.creighton.edu/fileadmin/user/HSL/docs/ref/Searching_-_Recall_Precision.pdf
How Does This Translate Into Literature Searches? Simple search for “diabetes” in Medline Search Fields Number of Results • Subject heading • Title, abstract, all free-text fields 400,000 + • Title • 2+ times in abstract • Major subject heading 280,000 + • Title • Major subject heading 160,000+
Standards for Systematic Searchers • Finding what works in health care: standards for systematic reviews. Washington D.C.: Institute of Medicine of the National Academies; 2011. See Chapter 3: Standards for Finding and Assessing Individual Studies. Standard 3.1 – • Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions. Version 5.1.0. [Updated March 2011]. The Cochrane Collaboration; 2011. See Part 2: General methods for Cochrane Reviews. Chapter 6: searching for studies.( Authors: Carol Lefebvre, Eric Manheimer and Julie Glanville on behalf of the Cochrane Information Retrieval Methods Group)
Who are “Systematic Searchers”? • Experts in technical and methodological issues of searching • MLIS/MLS/MISt – educational requirements • Work with the research team at every stage of review WHY ARE THEY IMPORTANT? • Help to mitigate potential biases in search • Know search resources and how to use them • Can advise on size and quality of search results
Role of the Information Specialist • Help define scope of review • Perform complex database and grey literature searches • Document the search • Manage citations, ensure quality of references • Perform supplemental searches • Provide advice on copyright & obtaining permissions • Maintain alerts & media scanning
Creating YOUR Literature Search • Understanding your research question • PICO • Scoping • Setting your parameters
Understand Research Question • Work with research team to refine questions – Scoping searches, identify main concepts • Determine PICO • Define search parameters – age, gender, date or language limits, study types • Get suggestions on terminology or sources from subject experts
What is your question? P population I intervention C comparator O outcome
Understand the Research Question Patient Population Adults with type 2 diabetes who have experienced inadequate glycemic control on combination therapy with metformin and a sulfonylurea Intervention Saxagliptin (2.5 mg once daily or 5 mg once daily) in combination with metformin and a sulfonylurea Comparators Other DPP-4 inhibitors, basal insulin, or placebo in combination with metformin and a sulfonylurea Outcomes Key efficacy outcomes • mortality • diabetes-related morbidity (macrovascular, microvascular) • glycemic control (A1C, FBG) • QoL (measured by any validated scale) Harms outcomes hypoglycemia total AEs SAEs WDAEs changes in body weight Other outcomes health care resource utilization Study design Published and unpublished RCTs ≥ 12 weeks duration
When PICO Doesn’t Cut It…
Understand the Research Question • Search Protocol
Translating the Search • CADTH PubMed cheat sheet http://cadth.ca/media/is/Searchdev/PubMed_Cheat_Sheet_external.pdf
Scoping – finding terms • Look at indexed articles • Create lists of relevant terms (concept map) • Sources for drug names/terms - Chem ID - Pubmed/Medline - Embase • Use both controlled vocabulary and keywords
Major Medical Databases Medline Biomedical and health sciences Embase Biomedical and pharmaceutical database PsycINFO CINAHL Behavioural sciences and mental Nursing and health allied health Cochrane Central Controlled clinical trials Specialization Strong in clinical research; North American focus Strong in pharmaceutical information; European focus; index conference abstracts Includes fields such as psychiatry, management, business, education, social science, neuroscience, law, medicine, and social work Contains only citations of clinical trials; no need to use an RCT filter Thesaurus MeSH EMTREE Thesaurus of Psychological Index CINAHL Terms Headings MeSH Date Coverage Begins 1946 1974 1840 1981 circa 1966 (some earlier) Journals Indexed 5,516 7,500 2,450 3,000 2,400 handsearched Overlap with other databases Yes Yes Yes Yes Yes Focus Includes fields such as nursing (clinical and theory), nurse education, physiotherapy, occupational therapy
Know Database Syntax Need to know indexing and “syntax” for each database searched Example: Subject Heading Search OVID Medline Diabetes Mellitus, Type 2/ OVID Embase Non insulin dependent diabetes mellitus/ PubMed "Diabetes Mellitus, Type 2"[Mesh] CINAHL MH diabetes mellitus, type 2
Visual Example of Typical Search Strategy INDICATION TERMS INTERVENTION TERMS STUDY DESIGN FILTERS
Filters Why use filters? • • • • • Reduce the number of irrelevant results Extract the higher level evidence (i.e. RCTs) Extract a specific concept (ex. QOL, adverse events) Remove off-topic results (ex. animals, editorials) Saves time for IS and researcher
Types of Filters • Specific methodology – RCTs, Systematic reviews • Clinical queries – Diagnosis, Prognosis, Adverse events • Population specific – Humans, Pediatrics • CADTH Filters http://www.cadth.ca/en/resources/findingevidence-is/string
Example Filter: RCTs 1. Randomized Controlled Trial.pt. 2. Randomized Controlled Trials as Topic/ 3. Randomized Controlled Trial/ 4. Randomization/ 5. Double-Blind Method/ 6. Single-Blind Method/ 7. Placebos/ 8. (random* or sham or placebo*).ti,ab,hw. 9. ((singl* or doubl*) adj (blind* or dumm* or mask*)).ti,ab,hw. 10. ((tripl* or trebl*) adj (blind* or dumm* or mask*)).ti,ab,hw. 11. or/1-11
Filters: Warning • No filter is infallible • Filters are regularly being tested, tweaked and updated but they almost always capture irrelevant records and sometimes exclude relevant ones
RCT filter would pick up this article…
And this article… “Randomized” in title, but not a study:
But not this one. Study design NOT found anywhere:
CADTH Peer Review Checklist for Search Strategies http://www.cadth.ca/en/resources/finding-evidence-is/peerreview-search-strat
CADTH Peer Review Checklist for Search Strategies • History and development • Current checklist • New research on the value of peer review • How can you get your search peer reviewed?
CADTH Peer Review Checklist for Search Strategies • Developed from: Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer review of electronic search strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008. http://www.cadth.ca/media/pdf/477_PRESS-Peer-ReviewElectronic-Search-Strategies_tr_e.pdf
CADTH Peer Review Checklist for Search Strategies Key Elements Translation Appropriate search filters Operators Limits Subject headings Fields Natural language Adapted databases Spelling & syntax Additional suggestions/comments
Exercise Sample Searches
Peer Review: Does it Make a Difference? • Recent research conducted to determine whether the peer review of literature search strategies has an effect on the number and quality of articles included in CADTH rapid review reports • What were the results?
Recent Research – Initial Results PHASE ONE: • 150 rapid review (RR) reports randomly selected from 2009-10 • Screening: pre-set selection criteria PHASE TWO: • Re-running pre peer reviewed and peer reviewed searches
Phase One: Screening Results 150 rapid review reports randomly selected 47 reports: Included 103 reports: Excluded
Phase One: Screening Results – Excluded Rapid Review Reports REASONS FOR EXCLUSION 3% 31/103 (30%): OVID searches 17% 30% 50% 51/103 (50%): Could not replicate searches 17/103 (17%): Blank reports/grey literature only 4/103 (3%): Level 3 reports (systematic reviews)
Phase Two Results: Included Rapid Review Reports EFFECT OF PEER REVIEW ON RETRIEVAL 19% 9/47 (19%) searches: No unique articles retrieved 43% 38% 18/47 (38%) searches: Unique articles retrieved, none included in the final report 20/47 (43%) searches: Unique articles retrieved, one or more articles included in the final report
Results: Distribution of Publication Type NUMBER OF ARTICLES Unique articles retrieved by Peer Reviewed Searches and Included in Rapid Review Reports 60 52 50 40 30 18 20 5 3 HTA/SR/MA 10 RCT 2 1 0 NRS* CPG Review article Other PUBLICATION TYPE CPG = clinical practice guidelines; HTA/SR/MA = health technology assessment/systematic review/meta-analysis; NRS = non-randomized study; RCT = randomized controlled trial. *NRS: “Any quantitative study estimating the effectiveness of an intervention (harm or benefit) that does not use randomization to allocate units to comparison groups.”
Getting your Search Reviewed • Colleagues • Listservs • Your librarian • PRESS Forum website. A forum for librarians to request and conduct peer reviews of search strategies. http://pressforum.pbworks.com/
Best Practices for Search Strategies 1) Understand research question(s) 2) Use keywords and controlled vocabulary 3) Know database indexing and syntax 4) Use filters where appropriate 5) Have searches peer reviewed 6) Use caution: don’t overcomplicate
Reporting Results QUOROM FLOWCHART
Reporting Results PRISMA FLOW DIAGRAM
Documenting the Search Who Databases Concepts Limits Dates Nondatabase sources
Documenting the Search
Documenting the Search
Documenting the Search Full strategy Other databases Grey lit
What is Grey Literature? • Grey/gray • Unpublished • Non-commercial material • In-house, vertical files • Information on the deep web • “Hard to find” • Fugitive
Grey Literature Definition INTERNATIONAL CONFERENCE ON GREY LITERATURE “Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.” (Luxembourg, 1997 - Expanded in New York, 2004) http://www.greynet.org/greynethome.html Source: GreyNet: grey literature network service. Amsterdam: GreyNet; 2008.
Examples of Grey Literature • Government Documents • Policy Documents • Clinical Guidelines • Conference Abstracts/Proceedings/Papers • Open Access Journals (some un-indexed) • Health Technology Assessments • Personal Communication • Clinical Trials
Why is Searching Grey Literature Important? • Essential component of a comprehensive search • Overcomes bias of commercial publications • Part of evidence base • Most HTAs created for healthcare decision-makers are considered “grey literature”
Comprehensive Coverage • “Unpublished trials contribute about 20% of the weight in individual meta-analysis.” Source: Krishnan RR. Evidence-based practice: how to read what you read. Psychopharmacology Bulletin 2004;37(4):88-91.
Publication Bias • “Exclusion of grey literature from meta-analyses can lead to exaggerated estimates of intervention effectiveness” McAuley et al. Lancet 2000; 356(Oct 7): 1228-31. • “Published literature tend to be larger and show larger effects of a health care intervention than those trials found in the grey literature. There was limited evidence to show whether grey trials are of poorer methodological quality than published trials. This means that those carrying out systematic reviews need to search for trials in both the published and grey literature in order to help minimise the effects of publication bias in their review.” Hopewell et al. Cochrane Database Syst Rev 2007; 18(2): MR000010.
Grey Literature as Evidence • Agency for Healthcare Research and Quality (AHRQ): Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program. Methods Guide for Comparative Effectiveness Reviews. (Prepared by the Oregon Health and Science University and the University of Ottawa Evidence-based Practice Centers under Contract Nos. 290-2007-10057-I and 290-2007-10059-I.) AHRQ Publication No. 13(14)EHC096-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2013. http://effectivehealthcare.ahrq.gov/ehc/products/486/1751/methods-guidance-reporting-bias-131118.pdf • Institute of Medicine (IOM): Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington D.C.: National Academy of Sciences; March 2011. http://www.iom.edu/Reports/2011/Finding-What-Works-in-Health-CareStandards-for-Systematic-Reviews/
Grey Literature as Evidence Health Technology Assessments (HTAs) and/or Meta-analysis Clinical Practice Guidelines Safety advisories or adverse event reports Source: Trip Database http://www.tripdatabase.com/
Grey Matters: A Practical Deep-Web Search Tool for Evidencebased Medicine http://www.cadth.ca/en/resources/ finding-evidence-is/grey-matters
Health Technology Assessments (HTAs) KEY RESOURCES • Databases: – Centre for Reviews and Dissemination (CRD) – The Cochrane Library ($) – Trip Database • Stand-alone websites: – Health Quality Ontario Publications and OHTAC Recommendations – L’Institut national d’excellence en santé et en services sociaux (INESSS) – Canadian Agency for Drugs and Technologies in Health (CADTH)
CRD Database http://www.crd.york.ac.uk/crdweb/SearchPage.asp
The Cochrane Library http://www.thecochranelibrary.com/
Trip Database http://www.tripdatabase.com/
Health Quality Ontario http://www.hqontario.ca/evidence/publications-and-ohtac-recommendations
CADTH – Health Technology Assessment Search http://www.cadth.ca/en/products/health-technology-assessment
AHRQ Technology Assessments http://www.ahrq.gov/clinic/techix.htm
Safety/Advisory KEY RESOURCES • European Medicines Agency (EMA) Patient Safety • New Zealand Medicines and Medical Devices Safety Authority Medsafe Prescriber Update articles • Medicines and Healthcare Products Regulatory Agency (MHRA) Drug Safety Update • MedWatch: The FDA Safety Information and Adverse Event Reporting Program
Health Canada • MedEffect Canada http://www.hc-sc.gc.ca/dhpmps/medeff/advisories-avis/index-eng.php • CARN http://www.hc-sc.gc.ca/dhpmps/medeff/bulletin/carn-bcei_index-eng.php • Canada Vigilance Online Database http://webprod3.hcsc.gc.ca/arquery-rechercheei/index-eng.jsp
EMA Patient Safety http://www.ema.europa.eu/ema/index.jsp?curl=pages/medicines/landing/pha_listing.jsp&mid=WC0b01ac058001 d126
Medsafe Prescriber Update Articles http://www.medsafe.govt.nz/profs/PUarticles.asp
MHRA Drug Safety Update http://www.mhra.gov.uk/Publications/Safetyguidance/DrugSafetyUpdate/index.htm
MedWatch: The FDA Safety Information and Adverse Event Reporting Program http://www.fda.gov/SAFETY/MEDWATCH/SAFETYINFORMATION/SAFETYALERTSFORHUMANMEDICALPRODUCTS/DEFAULT.HTM
Clinical Practice Guidelines • A search for clinical practice guidelines may extend to association websites or organizations dedicated to a particular disease or patient group, depending on the requirements of the search – Guidelines – Protocols – Positions statements – Patient information sheets and resources
Clinical Practice Guidelines KEY RESOURCES • Canadian Medical Association CMA Infobase: Clinical Practice Guidelines (CPGs) • National Institute for Health and Clinical Excellence NICE Guidelines • National Guideline Clearing House (NGC)
CMA Infobase http://www.cma.ca/index.php/ci_id/54316/la_id/1.htm
NICE Guidance http://www.nice.org.uk/guidance/index.jsp?action=find
National Guideline Clearinghouse http://www.guideline.gov/
Closing and Q&A • Further questions? • Evaluations • Draw
Follow us on: @CADTH_ACMTS
Half-century evidence from western Canada shows forest dynamics are primarily driven by competition followed by ... This finding suggests that internal ...
Half-century evidence from western Canada shows forest dynamics are primarily driven by competition ... This finding suggests that internal community ...
Half-century evidence from western Canada shows forest dynamics are primarily driven by competition followed ... western Canada to ... finding suggests ...
Half-century evidence from western Canada shows ... in western Canada, ... This finding suggests that forest
... event in the western Canada ... to the finding of all normal ... the Western Canada sedimentary basin; evidence from ...
The foundation emphasizes its evidence ... more versatile workforce and finding ways ... Affordable Housing and Urban Growth in Western Canada.
Taiga Plains Ecozone + Evidence for Key Findings Summary ... First reported in western Canada in ... National key finding Canada is well endowed with a ...
The history of Canada covers the ... Archeological and Aboriginal genetic evidence indicate that ... Times were especially hard in western Canada, ...
... Amateur treasure hunter's coin find proves English hero discovered Canada ... head after finding a 16th ... evidence that English ...
Records in evidence. Evidence law in Australia; ... Evidence law in Australia. ... of relevant evidence is available to courts for fact finding ...