NIST 2009 Open Machine Translation Evaluation (MT09) Official Release of Results

Date of release: Tue Oct 27 15:48:58 2009
Version: mt09_public_v1

The NIST 2009 Open Machine Translation Evaluation (MT09) is part of an ongoing series of evaluations of human language translation technology. NIST conducts these evaluations in order to support machine translation (MT) research and help advance the state-of-the-art in machine translation technology. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. The evaluation was administered as outlined in the official MT09 evaluation plan.

Disclaimer

These results are not to be construed, or represented as endorsements of any participant's system or commercial product, or as official findings on the part of NIST or the U.S. Government. Note that the results submitted by developers of commercial MT products were generally from research systems, not commercially available products. Since MT09 was an evaluation of research algorithms, the MT09 test design required local implementation by each participant. As such, participants were only required to submit their translation system output to NIST for uniform scoring and analysis. The systems themselves were not independently evaluated by NIST.
 

Certain commercial equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the equipment, instruments, software or materials are necessarily the best available for the purpose.
 

There is ongoing discussion within the MT research community regarding the most informative metrics for machine translation. The design and implementation of these metrics are themselves very much part of the research. At the present time, there is no single metric that has been deemed to be completely indicative of all aspects of system performance.
 

The data, protocols, and metrics employed in this evaluation were chosen to support MT research and should not be construed as indicating how well these systems would perform in applications. While changes in the data domain, or changes in the amount of data used to build a system, can greatly influence system performance, changing the task protocols could indicate different performance strengths and weaknesses for these same systems.
 

Because of the above reasons, this should not be interpreted as a product testing exercise and the results should not be used to make conclusions regarding which commercial products are best for a particular application.

History

• 2009/10/27 : First public release

Evaluation Tasks

MT09 was a test of text-to-text MT technology. The evaluation consisted of three tasks, differing only by the source language processed:

• Arabic-to-English
• Chinese-to-English
• Urdu-to-English

Evaluation Conditions

MT research and development requires language data resources. System performance is strongly affected by the type and amount of resources used. Therefore, two different resource categories were defined as conditions of evaluation. The categories differ solely by the amount of data that was available for use in the training and development of the core MT engine. These evaluation conditions were called "Constrained Training" and "Unconstrained Training". See the evaluation specification document for a complete description of allowable resources for each.

Evaluation Tracks

In recent years, performance improvements have been demonstrated through the use of system combination techniques. For MT09, two evaluation tracks were supported that were called "Single System Track" and "System Combination Track". Results are reported separately for each track. As the names of each track implies, a key feature of systems entered in the Single System Track is that the resulting translations are produced by primarily one algorithmic approach, while translations from the System Combination Track result from a combination technique where 2 or more core algorithmic approaches are used.

Evaluation Data

The following table contains the approximate source word count, for each language pair and data genre, separately for the Current Test Set and the Progress Test Set. For the Chinese-to-English language pair, we consider that a Chinese word is 1.5 characters, on average.

Performance Measurement

• BLEU-4 (mteval-v13a, the official MT09 evaluation metric)
  • Invocation line: perl mteval-v13a.pl -r REFERENCE_FILE -s SOURCE_FILE -t CANDIDATE_FILE -c -b
  • Option -c : case-sensitive scoring
  • Option -b : BLEU score only
  • Version 13a of the script is a bug-fixed version that prevents a division-by-zero error to occur when a candidate translation segment is empty.
• IBM BLEU (bleu-1.04a)
  • Invocation line: perl bleu-1.04.pl -r REFERENCE_FILE -t CANDIDATE_FILE
  • By default, scoring is case-sensitive
  • The NormalizeText method was updated to properly un-escape XML entities.
• NIST (mteval-v13a)
  • Invocation line: perl mteval-v13a.pl -r REFERENCE_FILE -s SOURCE_FILE -t CANDIDATE_FILE -c -n
  • Option -c : case-sensitive scoring
  • Option -n : NIST score only
• TER (tercom-0.7.25)
  • Invocation line: java -jar tercom.7.25.jar -r REFERENCE_FILE -h CANDIDATE_FILE -N -s
  • Option -N : enables normalization
  • Option -s : case-sensitive scoring
• METEOR (meteor-0.7)
  • Invocation line: perl meteor.pl -s SYSTEM_ID -r REFERENCE_FILE -t CANDIDATE_FILE --modules "exact porter_stem wn_stem wn_synonymy"
  • Option --modules "exact porter_stem wn_stem wn_synonymy" : uses all four METEOR matching modules, in that order

Participants

The following table lists the organizations participating in MT09 and the test sets they registered to process.

Results Section

[ Current Test Set, Arabic-to-English Results ]   [ Current Test Set, Urdu-to-English Results ]   [ Progress Test Set Results ]  [ Informal System Combination Results ]

Current Test Set, Arabic-to-English Results

Introduction

This release page is limited to the Current Test for the Arabic-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Results for submissions from GALE participants, who had previous access to the test data, will be reported separately from results submitted by those that are not part of the GALE program.

Participants

The following table lists the submissions received from the sites participating in the Arabic-to-English Current Test.

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Arabic-to-English, submissions from participants involved in GALE (Table 1)

Arabic-to-English, submissions from participants not involved in GALE (Table 2)

⁽¹⁾A late and/or debugged system was also submitted, not reported here.
 

Current Test Results - System Combination Track

Arabic-to-English, submissions from participants involved in GALE (Table 3)

Current Test Results - Single System Track and System Combination Track - All metrics

Arabic-to-English, submissions from participants involved in GALE (Table 4)

Arabic-to-English, submissions from participants not involved in GALE (Table 5)

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Set, Urdu-to-English Results

Introduction

This release page is limited to the Current Test for the Urdu-to-English track.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Participants

The following table lists the submissions received from the sites participating in the Urdu-to-English Current Test.

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Current Test Results - Single System Track

Urdu-to-English (Table 1)

⁽¹⁾A late and/or debugged system was also submitted, not reported here.
 

Current Test Results - System Combination Track

Urdu-to-English (Table 2)

Current Test Results - Single System Track and System Combination Track - All metrics

Urdu-to-English (Table 3)

⁽¹⁾A late and/or debugged system was also submitted, not reported here.

Progress Test Set Results

Introduction

This page of results is limited to those of the Progress Test data for the MT09 tests of Arabic-to-English and Chinese-to-English text-to-text translation.

The Progress test was designed as a means to demonstrate true system improvement for a particular site over time, irrespective of inherent data differences that come with each new test set. In order to keep the focus on progress over time, only results from sites that participated in both OpenMT08 and OpenMT09 are reported.

There may be issues related to processing older test sets that make across-site comparisons less meaningful.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Note that all BLEU-4 scores reported on this page were computed using version 13a of the mteval script. Last year, scores were computed using mteval-v11b, which uses a different brevity penalty. This year's script (mteval-v13a) uses the 'closest reference translation length' (same as in IBM BLEU) instead of the 'shortest reference translation length', to compute the brevity penalty.

Participants

The following table lists the submissions received from the sites participating in the Progress Test. Cells in bold indicate submissions for which scores are reported:

• Submissions were valid and on-time
• The site also participated in MT08
• The MT08 system was submitted for the same training condition as the OpenMT 2009 system.

Progress Test Results - Single System Track

Arabic-to-English (Table 1)

⁽¹⁾The MT08 system was a combined system
 

Chinese-to-English (Table 2)

⁽¹⁾The MT08 system was a combined system

Informal System Combination Results

Introduction

The NIST 2009 Open Machine Translation Evaluation (MT09) is part of an ongoing series of evaluations of human language translation technology. NIST conducts these evaluations in order to support machine translation (MT) research and help advance the state-of-the-art in machine translation technology. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. The evaluation was administered as outlined in the official MT09 evaluation plan.

Informal System Combination was an informal, diagnostic MT09 task, offered after the official evaluation period. Output from several MT09 systems on the Arabic-to-English and Urdu-to-English Current tests was anonymized and provided for system combination purposes. Participants in this category produced new output based on those provided translations.

Scores reported here are limited to primary Informal System Combination submissions.

Disclaimer

These results are not to be construed, or represented as endorsements of any participant's system or commercial product, or as official findings on the part of NIST or the U.S. Government. Note that the results submitted by developers of commercial MT products were generally from research systems, not commercially available products. Since MT09 was an evaluation of research algorithms, the MT09 test design required local implementation by each participant. As such, participants were only required to submit their translation system output to NIST for uniform scoring and analysis. The systems themselves were not independently evaluated by NIST.
 

Certain commercial equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the equipment, instruments, software or materials are necessarily the best available for the purpose.
 

There is ongoing discussion within the MT research community regarding the most informative metrics for machine translation. The design and implementation of these metrics are themselves very much part of the research. At the present time, there is no single metric that has been deemed to be completely indicative of all aspects of system performance.
 

The data, protocols, and metrics employed in this evaluation were chosen to support MT research and should not be construed as indicating how well these systems would perform in applications. While changes in the data domain, or changes in the amount of data used to build a system, can greatly influence system performance, changing the task protocols could indicate different performance strengths and weaknesses for these same systems.
 

Because of the above reasons, this should not be interpreted as a product testing exercise and the results should not be used to make conclusions regarding which commercial products are best for a particular application.

History

• 2009/10/27 : First public release

Evaluation Data

System output for the Informal System Combination track included output of the Arabic-to-English and Urdu-to-English Current tests. Approximately 30% of the test data was designated as a development set for system combination. The remainder of the system output was provided as the test set.

Informal System Combination Results

Arabic-to-English (Table 1)

Urdu-to-English (Table 2)

⁽¹⁾rescored