LoReHLT Evaluations

Low Resource Languages for Emergent Incidents (LORELEI) is a DARPA-sponsored program. The goal of the program is to dramatically advance the state of computational linguistics and human language technology to enable rapid, low-cost development of capabilities for low-resource languages.

This web page serves as a portal for all information pertaining to the Low Resource (LoRe) HLT open evaluations of component technologies relevant to LORELEI. For general questions and comments, please e-mail lorehlt_poc [at] nist.gov (lorehlt_poc[at]nist[dot]gov).  For more information about a specific evaluation cycle, including evaluation documentation, see the year-specific sections below.  For general announcements and discussing regarding the LoReHLT evaluation series, you can subscribe to the LoReHLT mailing list hosted by NIST, lorehlt [at] nist.gov (lorehlt[at]nist[dot]gov), by sending e-mail to lorehlt_list-requeset [at] nist.gov (lorehlt_list-request[at]nist[dot]gov) with "subscribe" in the subject line.

LoReHLT19

Tasks

• Machine translation (MT)
• Entity Discovery and Linking (EDL)
• Situation Frame (SF)
• Sentiment, Emotion, & Cognitive State (SEC) (Pilot)

Highlights

• Surprise language evaluation (two surprise languages)
• New SEC pilot evaluation
• Two evaluation checkpoints to gauge performance based on time and training resources given

Documentation

• LoReHLT 2019 Evaluation Plan v1.0, initial release July 19, 2019
• System description template

Tools

• SF:
  • Situation Frame submission JSON schema V1

LoReHLT18

Tasks

• Machine translation (MT)
• Entity Discovery and Linking (EDL)
• Situation Frame (SF)

Highlights

• Surprise language evaluation (two surprise languages)
• Two training conditions: constrained (required) and unconstrained
• Two evaluation checkpoints to gauge performance based on time and training resources given

Registration

• Use this form to sign up to participate: https://goo.gl/forms/Xoj6mbLFU1V5BEJ93
• Non-LORELEI participants only: Complete and return the LDC LoReHLT18 data license agreement.

Documentation

• LoReHLT 2018 Evaluation Plan v 1.0.4(updated June 7, 2018)

Tools

• Situation Frame:
  • Scoring software package (version 2.0.1): LoReHLT18_SF_Scorer_v0.8_20180515.zip

Publications and System Descriptions

• LoReHLT18 system descriptions (not peer-reviewed)

LoReHLT17

Tasks

• Machine translation (MT)
• Entity Discovery and Linking (EDL)
• Situation Frame (SF)

Highlights

• Surprise language evaluation (two surprise languages)
• Two training conditions: constrained (required) and unconstrained
• Three evaluation checkpoints to gauge performance based on time and training resources given

Documentation

• LoReHLT17 Evaluation Plan v 2.3.3 (updated July 27, 2017)
• System description template (one per team, describing all tasks participated in and submissions)

Tools

• Test data encryption package: OpenSSLTest-v2.tar.bz2 (md5)
• MT:
  • BLEU and "NIST" score scoring script: mteval-v14c.pl
• EDL:
  • Scoring script: https://github.com/wikilinks/neleval
• Situation Frame (Text):
  • Scoring software package (version 2.0.1):  LoReHLT_Frame_Scorer_v2.0.1_20170626.tgz (md5)
  • System output schema (version 1.1):  LoReHLT17_v1.1_system-output-schema.tgz (md5)
• Situation Frame (Speech) -- We would like to thank the Signal Analysis and Interpretation Laboratory (SAIL) at the University of Southern California for provided the scoring software for the Situation Frame Speech task:
  • Scoring software package (version 5): https://github.com/usc-sail/lorelei-speech-evaluation
  • System output schema (version 1.0): LoReHLT17_SF-Speech_v1.0_system-output-schema.tgz (md5)

Publications and System Descriptions

• LoReHLT17 system descriptions (not peer-reviewed)

LoReHLT16

Tasks

• Machine translation (MT)
• Situation Frame (SF)
• Named entity recognition (NER)

Highlights

• Surprise language evaluation
• Two training conditions: constrained (required) and unconstrained
• Three evaluation checkpoints to gauge performance based on time and training resources given

Documentation

• LoReHLT16 Evaluation Plan v2-8  (updated June 30, 2016)
• Description of evaluation incident language components and structure (LDC-authored; uploaded May 27, 2016)
• Description of development incident language components and structure (LDC-authored; uploaded May 27, 2016)
• System description template (one per team, describing all tasks participated in and submissions)
• For questions about the individual evaluation tasks, consult the LoReHLT16 Evaluations FAQ.
• LoReHLT16 workshop agenda with links to presentation slides, password-protected

Tools

• Test data encryption package: ftp://jaguar.ncsl.nist.gov/pub/OpenSSLTest-v2.tar.bz2
• Situation Frame scoring software package: LoReHLT_Frame_Scorer_v1.1.2_20160825.tgz (md5)
• Machine translation BLEU/NIST scoring script: mteval-v14.pl

Publications and System Descriptions

• Machine Translation, Volume 32, Issue 1-2, June 2018, Special Issue: NLP in Low-Resource Languages
• LoReHLT16 system descriptions (not peer-reviewed)

Disclaimer

Any mention of commercial products within NIST web pages is for information only; it does not imply recommendation or endorsement by NIST.