Crowdsourcing As a Means to Identify SNOMED CT Subsets

25 %
75 %
Information about Crowdsourcing As a Means to Identify SNOMED CT Subsets
Health & Medicine

Published on October 9, 2009

Author: HINZ

Source: slideshare.net

Description

Dave Parry
School of Computing + Mathemtical Sciences, Auckland University of Technology
www.aut.ac.nz
(P12, 1/10/09, Works Room, 5.02)

“ Crowdsourcing” as a means to identify SNOMED CT subsets – an initial approach Dave Parry School of Computing and Mathematical sciences Auckland University of Technology Dave.parry@aut.ac.nz

Agenda Why is coding difficult ? Conceptual Issues What is crowdsourcing ? Structure and software

Why is coding difficult ?

Conceptual Issues

What is crowdsourcing ?

Structure and software

Why is coding difficult ? Experts don’t agree – even when a loose standard of agreement is required (Chiang 2006) SNOMED CT is very large and changes by 5-10% each release Data is used in ways that might be unfamiliar to the originator Reliability of SNOMED-CT Coding by Three Physicians using Two Terminology Browsers Michael F. Chiang, John C. Hwang, Alexander C. Yu, Daniel S. Casper, James J. Cimino, and Justin Starren AMIA Annu Symp Proc. 2006; 2006: 131–135.

Experts don’t agree – even when a loose standard of agreement is required (Chiang 2006)

SNOMED CT is very large and changes by 5-10% each release

Data is used in ways that might be unfamiliar to the originator

So what ? Errors propagate through systems SNOMED >ICD10 >DRG Free text present in many places in systems. Systems supporting coding may do better in avoiding “Paper trail” errors (O’Malley 2005) O'Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005). Measuring diagnoses: ICD code accuracy.(International Classification of Diseases). Health Services Research, 40 (5), 1620(1620).

Errors propagate through systems

SNOMED >ICD10 >DRG

Free text present in many places in systems.

Systems supporting coding may do better in avoiding “Paper trail” errors (O’Malley 2005)

O'Malley, K. J., Cook, K. F., Price, M. D., Wildes, K. R., Hurdle, J. F., & Ashton, C. M. (2005). Measuring diagnoses: ICD code accuracy.(International Classification of Diseases). Health Services Research, 40 (5), 1620(1620).

Existing systems Patrick et al describe means of selecting the “most likely “ term or phrase. Issues with identifying subsets and confirming correctness J. Patrick, Y. Wang, and P. Budd, "An automated system for conversion of clinical notes into SNOMED clinical terminology," in Proceedings of the fifth Australasian symposium on ACSW frontiers - Volume 68 Ballarat, Australia: Australian Computer Society, Inc., 2007.

Patrick et al describe means of selecting the “most likely “ term or phrase.

Issues with identifying subsets and confirming correctness

Conceptual basis Although SNOMED CT is hierarchical, there are many relations in addition to IS-A subsumptions. Any hierarchy is based on a particular view of the domain which may not match the reality

Although SNOMED CT is hierarchical, there are many relations in addition to IS-A subsumptions.

Any hierarchy is based on a particular view of the domain which may not match the reality

Concept Concept B Concept Concept Concept A Concept Concept Concept Concept Concept Concepts related to WHU Concepts Unrelated to WHU Concepts partially related to WHU Is-a Is-a Is-a Is-a Is-a To root concepts….

1 Membership value m 0 0 1 2 3 4 5 Value of “relatedness” response Not in subset Fully in subset Partially in subset

Women’s health ultrasound Combined radiology and O+ G dept. Diagnostic for both women and fetus Potentially very large subsets Coding important clinically and administratively

Combined radiology and O+ G dept.

Diagnostic for both women and fetus

Potentially very large subsets

Coding important clinically and administratively

WHU report “ Growth measurements lie within normal limits for this gestation. Liquor volume is normal. Fetus is active. A single left fetal kidney is identified. No definite right kidney seen in the right renal fossa. Fetal bladder appears normal. “

“ Growth measurements lie within normal limits for this gestation. Liquor volume is normal. Fetus is active. A single left fetal kidney is identified. No definite right kidney seen in the right renal fossa. Fetal bladder appears normal. “

How do we get the membership values ? Via texts – popular but limited People thinking… Lots of work for small numbers Danger of capture by one particular view Hard to get coverage

Via texts – popular but limited

People thinking…

Lots of work for small numbers

Danger of capture by one particular view

Hard to get coverage

Crowdsourcing Outsourcing to a wide group Anyone who wants to Minimal work “ All of us are smarter than some of us”

Outsourcing to a wide group

Anyone who wants to

Minimal work

“ All of us are smarter than some of us”

Examples The GUARDIAN (UK) RECAPTCHA and old texts Common sense computing project

The GUARDIAN (UK)

RECAPTCHA and old texts

Common sense computing project

System description SQL server database ASP.NET programming SNOMED CT release provided by NZHIS

SQL server database

ASP.NET programming

SNOMED CT release provided by NZHIS

Original text Potential fragments that relate to SNOMED terms Potential SNOMED Concepts Expanded SNOMED Descriptions Selected Concepts Overall scheme

Possible Concepts from fragments Original text

Rating screen

Learning memberships Start system by assigning membership from inspection of hierarchy. Modify membership using responses Present highest ranked member first

Start system by assigning membership from inspection of hierarchy.

Modify membership using responses

Present highest ranked member first

Plan Collect a number of test cases and use to test software in small unit Test usability of software Launch, with fictional cases to RANZCOG community and others Publish subset data Make system available more widely

Collect a number of test cases and use to test software in small unit

Test usability of software

Launch, with fictional cases to RANZCOG community and others

Publish subset data

Make system available more widely

Discussion There is a lot of data out there. Coding to a wide set of concepts is hard. Confirming that a coding decision is correct or not is easier than selecting a code from a wide range. Coding needs to be part of the workflow

There is a lot of data out there.

Coding to a wide set of concepts is hard.

Confirming that a coding decision is correct or not is easier than selecting a code from a wide range.

Coding needs to be part of the workflow

Acknowledgements Women’s health ultrasound department at Auckland District Health Board, especially Kathy Dryden, Chief Sonographer. Ted Cizadlo and NZHIS.

Women’s health ultrasound department at Auckland District Health Board, especially Kathy Dryden, Chief Sonographer.

Ted Cizadlo and NZHIS.

Add a comment

Related presentations

Related pages

“Crowdsourcing” as a Means to Identify SNOMED CT Su bsets ...

“Crowdsourcing” as a Means to Identify SNOMED CT Su bsets ... Selection of SNOMED CT subsets and ... Crowdsourcing [3] ...
Read more

Crowdsourcing" as a Means to Identify SNOMED CT Subsets ...

“Crowdsourcing” as a Means to Identify SNOMED CT Subsets – an Initial Approach Dave Parry School of Computing and Mathematical Sciences Auckland ...
Read more

“Crowdsourcing ” as a Means to Identify SNOMED CT Subsets ...

Abstract. Crowdsourcing is a technique that uses the internet to allow large numbers of people to contribute small amounts of time and effort to a research ...
Read more

Crowdsourcing techniques to create a fuzzy subset of ...

Crowdsourcing techniques to create a ... This paper describes an approach to identify subsets of SNOMED CT via a ... as a means of creating fuzzy subsets.
Read more

El Arte Del Crowdsourcing - es.scribd.com

El Arte Del Crowdsourcing - Free download as PDF File (.pdf ... (s.d.). ―Crowdsourcing‖ as a Means to Identify SNOMED CT Subsets–an Initial ...
Read more

Dave Parry | Auckland University of Technology | Papers ...

Dave Parry, Auckland University of Technology, ... Crowdsourcing techniques to create a ... Crowdsourcing" as a Means to Identify SNOMED CT Subsets ...
Read more

Development and evaluation of a crowdsourcing methodology ...

... and outcome data.26, 27 One preliminary report proposes the use of crowdsourcing to create SNOMED CT subsets ... identify true medication ... SNOMED CT ...
Read more

Using the wisdom of the crowds to find critical errors in ...

This work evaluated the ability of crowdsourcing ... indeed identify errors in SNOMED CT ... find critical errors in biomedical ontologies: ...
Read more

Dave Parry - Academia.edu

Dave Parry studies Computer Science, Physics, and Medical Physics. ... Crowdsourcing" as a Means to Identify SNOMED CT Subsets - an Initial Approach more.
Read more