Finding the Evidence (Western Canada)

50 %
50 %
Information about Finding the Evidence (Western Canada)
Health & Medicine

Published on February 20, 2014




Presentation from the 2014 CADTH Information Services workshop held in Western Canada.

February 2014 Western Events

CADTH CADTH is an independent, not-for-profit producer and broker of health technology assessments. Federal, provincial, and territorial health care decision-makers rely on our evidence-based information to make informed policy and practice decision about drugs and other health technologies.

Health Technologies • Pharmaceuticals • • Medical, surgical and dental devices and procedures Diagnostics

About Us • Amanda Hodgson Manager, Information Services • Caitlyn Ford Information Specialist • Our Team Literature searching for rapid reviews, health technology assessments and CADTH Common Drug Review

Objectives PART 1 • Role of the Information Specialist in HTA • Translating your research question • Applying search filters • Peer review of search strategies BREAK PART 2 • Grey literature searching • Grey Matters and Grey Matters Light • Closing and evaluation

Finding the Evidence • Finding the evidence: literature searching tools in support of systematic reviews

What is a Systematic Search? “Systematic reviews of interventions require a thorough, objective and reproducible search of a range of sources to identify as many relevant studies as possible (within resource limits). This is a major factor in distinguishing systematic reviews from traditional narrative reviews and helps to minimize bias and therefore assist in achieving reliable estimates of effects.” Higgins et al. Cochrane handbook for systematic reviews of interventions. 2008. Section

What is a Systematic Search? “…the aspiration of the HTA literature search should not be comprehensiveness but rather the minimization of bias.” Booth. Int J Technol Assess Health Care. 2010 Oct;26(4):431-5.

Fundamental Challenge of Searching Balancing Recall and Precision = getting "everything" while avoiding "junk"

How Does This Translate Into Literature Searches? Simple search for “diabetes” in Medline Search Fields Number of Results • Subject heading • Title, abstract, all free-text fields 400,000 + • Title • 2+ times in abstract • Major subject heading 280,000 + • Title • Major subject heading 160,000+

Standards for Systematic Searchers • Finding what works in health care: standards for systematic reviews. Washington D.C.: Institute of Medicine of the National Academies; 2011. See Chapter 3: Standards for Finding and Assessing Individual Studies. Standard 3.1 – • Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions. Version 5.1.0. [Updated March 2011]. The Cochrane Collaboration; 2011. See Part 2: General methods for Cochrane Reviews. Chapter 6: searching for studies.( Authors: Carol Lefebvre, Eric Manheimer and Julie Glanville on behalf of the Cochrane Information Retrieval Methods Group)

Who are “Systematic Searchers”? • Experts in technical and methodological issues of searching • MLIS/MLS/MISt – educational requirements • Work with the research team at every stage of review WHY ARE THEY IMPORTANT? • Help to mitigate potential biases in search • Know search resources and how to use them • Can advise on size and quality of search results

Role of the Information Specialist • Help define scope of review • Perform complex database and grey literature searches • Document the search • Manage citations, ensure quality of references • Perform supplemental searches • Provide advice on copyright & obtaining permissions • Maintain alerts & media scanning

Creating YOUR Literature Search • Understanding your research question • PICO • Scoping • Setting your parameters

Understand Research Question • Work with research team to refine questions – Scoping searches, identify main concepts • Determine PICO • Define search parameters – age, gender, date or language limits, study types • Get suggestions on terminology or sources from subject experts

What is your question? P population I intervention C comparator O outcome

Understand the Research Question Patient Population Adults with type 2 diabetes who have experienced inadequate glycemic control on combination therapy with metformin and a sulfonylurea Intervention Saxagliptin (2.5 mg once daily or 5 mg once daily) in combination with metformin and a sulfonylurea Comparators Other DPP-4 inhibitors, basal insulin, or placebo in combination with metformin and a sulfonylurea Outcomes Key efficacy outcomes • mortality • diabetes-related morbidity (macrovascular, microvascular) • glycemic control (A1C, FBG) • QoL (measured by any validated scale) Harms outcomes hypoglycemia total AEs SAEs WDAEs changes in body weight Other outcomes health care resource utilization Study design Published and unpublished RCTs ≥ 12 weeks duration

When PICO Doesn’t Cut It…

Understand the Research Question • Search Protocol

Translating the Search • CADTH PubMed cheat sheet

Scoping – finding terms • Look at indexed articles • Create lists of relevant terms (concept map) • Sources for drug names/terms - Chem ID - Pubmed/Medline - Embase • Use both controlled vocabulary and keywords

Major Medical Databases Medline Biomedical and health sciences Embase Biomedical and pharmaceutical database PsycINFO CINAHL Behavioural sciences and mental Nursing and health allied health Cochrane Central Controlled clinical trials Specialization Strong in clinical research; North American focus Strong in pharmaceutical information; European focus; index conference abstracts Includes fields such as psychiatry, management, business, education, social science, neuroscience, law, medicine, and social work Contains only citations of clinical trials; no need to use an RCT filter Thesaurus MeSH EMTREE Thesaurus of Psychological Index CINAHL Terms Headings MeSH Date Coverage Begins 1946 1974 1840 1981 circa 1966 (some earlier) Journals Indexed 5,516 7,500 2,450 3,000 2,400 handsearched Overlap with other databases Yes Yes Yes Yes Yes Focus Includes fields such as nursing (clinical and theory), nurse education, physiotherapy, occupational therapy

Know Database Syntax Need to know indexing and “syntax” for each database searched Example: Subject Heading Search OVID Medline Diabetes Mellitus, Type 2/ OVID Embase Non insulin dependent diabetes mellitus/ PubMed "Diabetes Mellitus, Type 2"[Mesh] CINAHL MH diabetes mellitus, type 2


Filters Why use filters? • • • • • Reduce the number of irrelevant results Extract the higher level evidence (i.e. RCTs) Extract a specific concept (ex. QOL, adverse events) Remove off-topic results (ex. animals, editorials) Saves time for IS and researcher

Types of Filters • Specific methodology – RCTs, Systematic reviews • Clinical queries – Diagnosis, Prognosis, Adverse events • Population specific – Humans, Pediatrics • CADTH Filters

Example Filter: RCTs 1. Randomized Controlled 2. Randomized Controlled Trials as Topic/ 3. Randomized Controlled Trial/ 4. Randomization/ 5. Double-Blind Method/ 6. Single-Blind Method/ 7. Placebos/ 8. (random* or sham or placebo*).ti,ab,hw. 9. ((singl* or doubl*) adj (blind* or dumm* or mask*)).ti,ab,hw. 10. ((tripl* or trebl*) adj (blind* or dumm* or mask*)).ti,ab,hw. 11. or/1-11

Filters: Warning • No filter is infallible • Filters are regularly being tested, tweaked and updated but they almost always capture irrelevant records and sometimes exclude relevant ones

RCT filter would pick up this article…

And this article… “Randomized” in title, but not a study:

But not this one. Study design NOT found anywhere:

CADTH Peer Review Checklist for Search Strategies

CADTH Peer Review Checklist for Search Strategies • History and development • Current checklist • New research on the value of peer review • How can you get your search peer reviewed?

CADTH Peer Review Checklist for Search Strategies • Developed from: Sampson M, McGowan J, Lefebvre C, Moher D, Grimshaw J. PRESS: Peer review of electronic search strategies. Ottawa: Canadian Agency for Drugs and Technologies in Health; 2008.

CADTH Peer Review Checklist for Search Strategies Key Elements Translation Appropriate search filters Operators Limits Subject headings Fields Natural language Adapted databases Spelling & syntax Additional suggestions/comments

Exercise Sample Searches

Peer Review: Does it Make a Difference? • Recent research conducted to determine whether the peer review of literature search strategies has an effect on the number and quality of articles included in CADTH rapid review reports • What were the results?

Recent Research – Initial Results PHASE ONE: • 150 rapid review (RR) reports randomly selected from 2009-10 • Screening: pre-set selection criteria PHASE TWO: • Re-running pre peer reviewed and peer reviewed searches

Phase One: Screening Results 150 rapid review reports randomly selected 47 reports: Included 103 reports: Excluded

Phase One: Screening Results – Excluded Rapid Review Reports REASONS FOR EXCLUSION 3% 31/103 (30%): OVID searches 17% 30% 50% 51/103 (50%): Could not replicate searches 17/103 (17%): Blank reports/grey literature only 4/103 (3%): Level 3 reports (systematic reviews)

Phase Two Results: Included Rapid Review Reports EFFECT OF PEER REVIEW ON RETRIEVAL 19% 9/47 (19%) searches: No unique articles retrieved 43% 38% 18/47 (38%) searches: Unique articles retrieved, none included in the final report 20/47 (43%) searches: Unique articles retrieved, one or more articles included in the final report

Results: Distribution of Publication Type NUMBER OF ARTICLES Unique articles retrieved by Peer Reviewed Searches and Included in Rapid Review Reports 60 52 50 40 30 18 20 5 3 HTA/SR/MA 10 RCT 2 1 0 NRS* CPG Review article Other PUBLICATION TYPE CPG = clinical practice guidelines; HTA/SR/MA = health technology assessment/systematic review/meta-analysis; NRS = non-randomized study; RCT = randomized controlled trial. *NRS: “Any quantitative study estimating the effectiveness of an intervention (harm or benefit) that does not use randomization to allocate units to comparison groups.”

Getting your Search Reviewed • Colleagues • Listservs • Your librarian • PRESS Forum website. A forum for librarians to request and conduct peer reviews of search strategies.

Best Practices for Search Strategies 1) Understand research question(s) 2) Use keywords and controlled vocabulary 3) Know database indexing and syntax 4) Use filters where appropriate 5) Have searches peer reviewed 6) Use caution: don’t overcomplicate

Reporting Results QUOROM FLOWCHART


Documenting the Search Who Databases Concepts Limits Dates Nondatabase sources

Documenting the Search

Documenting the Search

Documenting the Search Full strategy Other databases Grey lit


What is Grey Literature? • Grey/gray • Unpublished • Non-commercial material • In-house, vertical files • Information on the deep web • “Hard to find” • Fugitive

Grey Literature Definition INTERNATIONAL CONFERENCE ON GREY LITERATURE “Information produced on all levels of government, academics, business and industry in electronic and print formats not controlled by commercial publishing i.e. where publishing is not the primary activity of the producing body.” (Luxembourg, 1997 - Expanded in New York, 2004) Source: GreyNet: grey literature network service. Amsterdam: GreyNet; 2008.

Examples of Grey Literature • Government Documents • Policy Documents • Clinical Guidelines • Conference Abstracts/Proceedings/Papers • Open Access Journals (some un-indexed) • Health Technology Assessments • Personal Communication • Clinical Trials

Why is Searching Grey Literature Important? • Essential component of a comprehensive search • Overcomes bias of commercial publications • Part of evidence base • Most HTAs created for healthcare decision-makers are considered “grey literature”

Comprehensive Coverage • “Unpublished trials contribute about 20% of the weight in individual meta-analysis.” Source: Krishnan RR. Evidence-based practice: how to read what you read. Psychopharmacology Bulletin 2004;37(4):88-91.

Publication Bias • “Exclusion of grey literature from meta-analyses can lead to exaggerated estimates of intervention effectiveness” McAuley et al. Lancet 2000; 356(Oct 7): 1228-31. • “Published literature tend to be larger and show larger effects of a health care intervention than those trials found in the grey literature. There was limited evidence to show whether grey trials are of poorer methodological quality than published trials. This means that those carrying out systematic reviews need to search for trials in both the published and grey literature in order to help minimise the effects of publication bias in their review.” Hopewell et al. Cochrane Database Syst Rev 2007; 18(2): MR000010.

Grey Literature as Evidence • Agency for Healthcare Research and Quality (AHRQ): Finding Grey Literature Evidence and Assessing for Outcome and Analysis Reporting Biases When Comparing Medical Interventions: AHRQ and the Effective Health Care Program. Methods Guide for Comparative Effectiveness Reviews. (Prepared by the Oregon Health and Science University and the University of Ottawa Evidence-based Practice Centers under Contract Nos. 290-2007-10057-I and 290-2007-10059-I.) AHRQ Publication No. 13(14)EHC096-EF. Rockville, MD: Agency for Healthcare Research and Quality. November 2013. • Institute of Medicine (IOM): Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington D.C.: National Academy of Sciences; March 2011.

Grey Literature as Evidence Health Technology Assessments (HTAs) and/or Meta-analysis Clinical Practice Guidelines Safety advisories or adverse event reports Source: Trip Database

Grey Matters: A Practical Deep-Web Search Tool for Evidencebased Medicine finding-evidence-is/grey-matters

Health Technology Assessments (HTAs) KEY RESOURCES • Databases: – Centre for Reviews and Dissemination (CRD) – The Cochrane Library ($) – Trip Database • Stand-alone websites: – Health Quality Ontario Publications and OHTAC Recommendations – L’Institut national d’excellence en santé et en services sociaux (INESSS) – Canadian Agency for Drugs and Technologies in Health (CADTH)

CRD Database

The Cochrane Library

Trip Database

Health Quality Ontario


CADTH – Health Technology Assessment Search

AHRQ Technology Assessments

FDA Warning

Safety/Advisory KEY RESOURCES • European Medicines Agency (EMA) Patient Safety • New Zealand Medicines and Medical Devices Safety Authority Medsafe Prescriber Update articles • Medicines and Healthcare Products Regulatory Agency (MHRA) Drug Safety Update • MedWatch: The FDA Safety Information and Adverse Event Reporting Program

Health Canada • MedEffect Canada • CARN • Canada Vigilance Online Database

EMA Patient Safety d126

Medsafe Prescriber Update Articles

MHRA Drug Safety Update

MedWatch: The FDA Safety Information and Adverse Event Reporting Program

Clinical Practice Guidelines • A search for clinical practice guidelines may extend to association websites or organizations dedicated to a particular disease or patient group, depending on the requirements of the search – Guidelines – Protocols – Positions statements – Patient information sheets and resources

Clinical Practice Guidelines KEY RESOURCES • Canadian Medical Association CMA Infobase: Clinical Practice Guidelines (CPGs) • National Institute for Health and Clinical Excellence NICE Guidelines • National Guideline Clearing House (NGC)

CMA Infobase

NICE Guidance

National Guideline Clearinghouse

Closing and Q&A • Further questions? • Evaluations • Draw

Follow us on: @CADTH_ACMTS

Add a comment

Related presentations

Related pages

Half-century evidence from western Canada shows forest ...

Half-century evidence from western Canada shows forest dynamics are primarily driven by competition followed by ... This finding suggests that internal ...
Read more

Half-century evidence from western Canada shows forest ...

Half-century evidence from western Canada shows forest dynamics are primarily driven by competition ... This finding suggests that internal community ...
Read more

Half-century evidence from western Canada shows forest ...

Half-century evidence from western Canada shows forest dynamics are primarily driven by competition followed ... western Canada to ... finding suggests ...
Read more

Half-century evidence from western Canada shows forest ...

Half-century evidence from western Canada shows ... in western Canada, ... This finding suggests that forest
Read more

Laramide orogenic fluid flow into the Western Canada ...

... event in the western Canada ... to the finding of all normal ... the Western Canada sedimentary basin; evidence from ...
Read more

Canada West Foundation - Wikipedia, the free encyclopedia

The foundation emphasizes its evidence ... more versatile workforce and finding ways ... Affordable Housing and Urban Growth in Western Canada.
Read more - Taiga Plains Ecozone+ Evidence for Key ...

Taiga Plains Ecozone + Evidence for Key Findings Summary ... First reported in western Canada in ... National key finding Canada is well endowed with a ...
Read more

History of Canada - Wikipedia, the free encyclopedia

The history of Canada covers the ... Archeological and Aboriginal genetic evidence indicate that ... Times were especially hard in western Canada, ...
Read more

Canadian history turned on head after 435-year-old British ...

... Amateur treasure hunter's coin find proves English hero discovered Canada ... head after finding a 16th ... evidence that English ...
Read more

Evidence law in Australia – National Archives of Australia

Records in evidence. Evidence law in Australia; ... Evidence law in Australia. ... of relevant evidence is available to courts for fact finding ...
Read more