Purpose

To challenge the international Natural Language Processing (NLP) research community to create and train computational intelligence algorithms that automate the assignment of ICD-9-CM codes to clinical free text.

Introduction

It is surprisingly hard for computers to handle free text as smoothly and effectively as humans do. So far, the results of the numerous efforts to achieve this have been mixed. Indeed, at times it has appeared that the complexities of free text are such as to render the effort futile. Not so; in fact, successive attempts to address the problem of converting free text into actionable knowledge have advanced the science of natural language processing and led to demand for software that simulates and complements what people are able to do.

We are sponsoring an international challenge task on the automated processing of clinical free text. Even with advances in structured vocabularies, many hospitals continue to electronically store some patient data as free text. This practice produces terabytes of information that, beyond the clinical visit, has limited utility because of its volume and accessibility. Natural language processing can potentially uncover implicit structure in this data, rendering it accessible to targeted search engines as well as special purpose systems dedicated to billing, quality assurance and discovery. This challenge offers participants an opportunity to test their untested algorithms or apply existing ones. Additionally, the Challenge provides full access to a carefully anonymised body of clinical data suitable for training and testing.

Competition Process

All participants will be required to register. On 1 Feb 2007, participants will be given access to a training data set, which they will use to develop their algorithms.

The test data set will be made available on 1 Mar 2007. Participants will use their algorithms to process the test data, and will submit their results in XML format, along with a brief description of their methods. For complete details about the data formats and evaluation process, download the Challenge Details document.

Competition results will be announced on 1 Apr 2007 and posted to the results page.

Travel Subsidy Awards

• First Place: US$1,000.00
• Second Place: US$500.00
• Third Place: US$250.00

Other Benefits of Participation

The competition provides an international opportunity for research groups to share the applicability of their natural language processing and artificial intelligence research in the medical domain. Also, results will be published in some way, although the publication stream is not yet finalized. It will most likely include conference proceedings, journal publications and a potential book. All publications related to the challenge will include the appropriate participant(s) as co-authors.

2007 Challenge Organizers

The Medical NLP Challenge is sponsored by the Computational Medicine Center, a collaborative medical research center between Cincinnati Children's Hospital Medical Center and the University of Cincinnati Medical Center that uses data and computational systems to make disease more preventable, illness more predictive and treatment more personalized. Cincom Systems, an Ohio-based software and services company involved with the center, is providing the travel subsidy awards. Researchers from the Department of Linguistics at The Ohio State University and from the Center for Computational Pharmacology at the University of Colorado Health Services Center also participated in organizing the challenge.

Computational Medicine Center

The Computational Medicine Center combines the best in the fields of genetics, medicine, computer science and biological science, taking medicine to a new level. With funding and support from Ohio's Third Frontier Project and the National Institutes of Health (NIH), the center continues to build its team of talented research physicians and experts in bioinformatics, genomics, genetics, epidemiology, computer science, math and statistics. To learn more about the center, visit its web site, or contact John Pestian, PhD, director.

Cincinnati Children's Hospital Medical Center

With 475 registered beds and more than 8,500 employees, Cincinnati Children's Hospital Medical Center is a leading medical research and teaching hospital consistently ranking among the top 10 pediatric hospitals in the nation. Cincinnati Children's is also the second-highest ranking recipient of research grants from the National Institutes of Health among pediatric institutions.

University of Cincinnati Medical Center

The University of Cincinnati Medical Center houses some of the university's most innovative and captivating science and research laboratories, not to mention four academic colleges and a conglomerate of patient care facilities and resources. With nearly $263 million in research dollars, more than 290,000 patients and more than 3,000 aspiring health professionals, the UC Medical Center is by far one of the most innovative academic health research complexes in the nation.

The Ohio State University Department of Linguistics

The Ohio State University linguistics program has a strongly theoretical orientation, with a research focus on the development of a general theory of human language as well as detailed accounts of the structure, development and variation of individual languages. The character of the Ohio State linguistics program is also that of a "pure science" rather than an "applied science": linguistic phenenomena are studied primarily for their own sake and as an aspect of human cognition, rather than being examined for their practical application in fields such as second language education. Experimental laboratory research in speech production, speech perception, the mental processes of sentence understanding, etc., are an important feature of many areas of study in the program. The department also has a rapidly growing focus in computational linguistics.

Center for Computational Pharmacology

The mission of the Center for Computational Pharmacology at the University of Colorado Health Sciences Center is the creation of novel algorithms and knowledge-based tools for the analysis and interpretation of high-throughput molecular biology data. The center's ultimate goal is to transform the process of drug design through the use of advanced computational techniques, particularly machine learning and knowledge-based approaches applied to high-throughput molecular biology data. Researchers at the center create novel algorithms for the analysis and interpretation of gene expression arrays, proteomics, metabolomics and combinatorial chemistry assays. They also create tools for building, maintaining and applying ontologies and knowledge-bases of molecular biology, and for knowledge-driven inference from multiple biological data types. A major focus of the center is the development and application of natural language processing techniques for information extraction from and management of the biomedical literature.

2007 Final Results

Here are final results of the 2007 Computational Medicine Center International Challenge: Classifying Clinical Free Text Using Natural Language Processing.

MICRO-AVERAGED F-1 mean = 0.7670
MICRO-AVERAGED F-1 std.dev. = 0.1325
MICRO-AVERAGED F-1 median = 0.7985

GREEN MICRO-AVERAGED F-1 is 1 standard deviation from the mean
ORANGE MICRO-AVERAGED F-1 is 2 standard deviations from the mean
RED MICRO-AVERAGED F-1 is more than 2 standard deviations from the mean