Annotation of phenotypes using ontologies: a gold standard for the training and evaluation of natural language processing systems.

Journal: Database : the journal of biological databases and curation
Published Date: Jan 1, 2018

Abstract

Natural language descriptions of organismal phenotypes, a principal object of study in biology, are abundant in the biological literature. Expressing these phenotypes as logical statements using ontologies would enable large-scale analysis on phenotypic information from diverse systems. However, considerable human effort is required to make these phenotype descriptions amenable to machine reasoning. Natural language processing tools have been developed to facilitate this task, and the training and evaluation of these tools depend on the availability of high quality, manually annotated gold standard data sets. We describe the development of an expert-curated gold standard data set of annotated phenotypes for evolutionary biology. The gold standard was developed for the curation of complex comparative phenotypes for the Phenoscape project. It was created by consensus among three curators and consists of entity-quality expressions of varying complexity. We use the gold standard to evaluate annotations created by human curators and those generated by the Semantic CharaParser tool. Using four annotation accuracy metrics that can account for any level of relationship between terms from two phenotype annotations, we found that machine-human consistency, or similarity, was significantly lower than inter-curator (human-human) consistency. Surprisingly, allowing curatorsaccess to external information did not significantly increase the similarity of their annotations to the gold standard or have a significant effect on inter-curator consistency. We found that the similarity of machine annotations to the gold standard increased after new relevant ontology terms had been added. Evaluation by the original authors of the character descriptions indicated that the gold standard annotations came closer to representing their intended meaning than did either the curator or machine annotations. These findings point toward ways to better design software to augment human curators and the use of the gold standard corpus will allow training and assessment of new tools to improve phenotype annotation accuracy at scale.

Authors

  • Wasila Dahdul
    Department of Biology, University of South Dakota, Vermillion, SD, USA, Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA and National Evolutionary Synthesis Center, Durham, NC, USA Wasila.Dahdul@usd.edu.
  • Prashanti Manda
    University of North Carolina at Greensboro, Greensboro, NC, USA.
  • Hong Cui
    School of Information Resources and Library Science, University of Arizona, Tucson, USA. hongcui@email.arizona.edu.
  • James P Balhoff
    National Evolutionary Synthesis Center, Durham, NC 27705, USA; University of North Carolina, Chapel Hill, NC 27599, USA;
  • T Alexander Dececchi
    Department of Biology, University of South Dakota, Vermillion, SD, USA, Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA and National Evolutionary Synthesis Center, Durham, NC, USA.
  • Nizar Ibrahim
    Department of Biology, University of South Dakota, Vermillion, SD, USA, Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA and National Evolutionary Synthesis Center, Durham, NC, USA.
  • Hilmar Lapp
    Department of Biology, University of South Dakota, Vermillion, SD, USA, Department of Organismal Biology and Anatomy, University of Chicago, Chicago, IL, USA and National Evolutionary Synthesis Center, Durham, NC, USA.
  • Todd Vision
    University of North Carolina at Chapel Hill, Chapel Hill, NC, USA.
  • Paula M Mabee
    Department of Biology, University of South Dakota, Vermillion, SD 57069, USA; alex.dececchi@usd.edu pmabee@usd.edu.

Keywords

External Resources

Popular Topics
Recent Journals