NIST logo
*
Bookmark and Share

NIST Open Machine Translation 2015 Evaluation (OpenMT15)

The NIST Open Machine Translation 2015 Evaluation (OpenMT15) will be implemented according to the OpenMT15 evaluation plan. It will take place in February 2015, followed by a workshop in May 2015.

Highlights

  • Evaluation on informal data genres (SMS/Chat, Conversational Telephone Speech) for Arabic-to-English and Chinese-to-English
  • Inclusion of audio input track
  • Explore common MT measurement techniques on these informal data genres

Documentation

  • Evaluation plan (v0.6, updated October 31, 2014 - updated submission procedures)
  • Registration form
  • LDC data license agreement (v2.2, updated October 8, 2014 - Additional training data as well as development set for the new genres (SMS/chat/CTS) are added. Those who signed the license do not have to resign the agreement but should work with the LDC to make sure they have the needed corpora. This means look at the updated license, determine which of the new corpora you would like to have, and put in a request to the LDC for those corpora.)
  • System description template 

Schedule

  • February 1, 2014: Initial evaluation plan available (will be updated periodically as more information is available)
  • February 1 - December 31, 2014: Registration period (early registration highly encouraged)
  • February - December 31, 2014: Training data available from LDC with incremental releases of new genres
  • November 3, 2014 - January 31, 2015: Dry run period
  • February 2 - 6, 2015: Main evaluation period for audio track; output due at NIST February 6, 11:59am ET
  • February 9 - 13, 2015: Main evaluation period for text track; output due at NIST February 13, 11:59am ET
  • February 13, 2015: System description due to NIST
  • February 23 - May 23, 2015: Human assessment period
  • June 2015 (TBD): Workshop in the Washington DC area
  • August 2015 (TBD): Official public release of results

Dry Run Evaluation

All registered participants must demonstrate evaluation readiness by participating in the dry run evaluation. Participants must show that their submissions pass validation. They must complete the dry run evaluation by Jan 31, 2015. The dry run submissions are used to exercise the evaluation pipeline and scores from these submissions will not be used in any official report.

To participate in the dry run evaluation, registered participants need to sign up for a user account at https://openmteval.nist.gov. You must register (fill out the registration given above) before you can sign up for a user account.

Contact

mt_poc@nist.gov

[ Main ]