NIST logo
*

NIST 2012 Open Machine Translation Evaluation (OpenMT12)
Official Release of Results

Date of release: Tue Aug 28 16:22:07 2012
Version: mt12_official_v1

The NIST 2012 Open Machine Translation Evaluation (OpenMT12) is part of an ongoing series of evaluations of human language translation technology. NIST conducts these evaluations in order to support machine translation (MT) research and help advance the state-of-the-art in machine translation technology. These evaluations provide an important contribution to the direction of research efforts and the calibration of technical capabilities. The evaluation was administered as outlined in the official OpenMT12 evaluation plan.

Disclaimer

These results are not to be construed, or represented as endorsements of any participant's system or commercial product, or as official findings on the part of NIST or the U.S. Government. Note that the results submitted by developers of commercial MT products were generally from research systems, not commercially available products. Since OpenMT12 was an evaluation of research algorithms, the OpenMT12 test design required local implementation by each participant. As such, participants were only required to submit their translation system output to NIST for uniform scoring and analysis. The systems themselves were not independently evaluated by NIST.

Certain commercial equipment, instruments, software, or materials are identified in this paper in order to specify the experimental procedure adequately. Such identification is not intended to imply recommendation or endorsement by NIST, nor is it intended to imply that the equipment, instruments, software or materials are necessarily the best available for the purpose. There is ongoing discussion within the MT research community regarding the most informative metrics for machine translation. The design and implementation of these metrics are themselves very much part of the research. At the present time, there is no single metric that has been deemed to be completely indicative of all aspects of system performance.

The data, protocols, and metrics employed in this evaluation were chosen to support MT research and should not be construed as indicating how well these systems would perform in applications. While changes in the data domain, or changes in the amount of data used to build a system, can greatly influence system performance, changing the task protocols could indicate different performance strengths and weaknesses for these same systems.

Because of the above reasons, this should not be interpreted as a product testing exercise and the results should not be used to make conclusions regarding which commercial products are best for a particular application.

History
  • 2012/08/28: v1 of official release.
Evaluation Tasks

OpenMT12 was a test of text-to-text MT technology. The evaluation consisted of tests for five language pairs:

  • Arabic-to-English
  • Chinese-to-English
  • Dari-to-English
  • Farsi-to-English
  • Korean-to-English
Evaluation Conditions

MT research and development requires language data resources. System performance is strongly affected by the type and amount of resources used. Therefore, two different resource categories were defined as conditions of evaluation. The categories differ solely by the amount of data that was available for use in the training and development of the core MT engine. These evaluation conditions were called "Constrained Training" and "Unconstrained Training". See the evaluation specification document for a complete description of allowable resources for each.

Evaluation Tracks

In recent years, performance improvements have been demonstrated through the use of system combination techniques. For OpenMT12, two evaluation tracks were supported that were called "Single System Track" and "System Combination Track". Results are reported separately for each track. As the names of each track implies, a key feature of systems entered in the Single System Track is that the resulting translations are produced by primarily one algorithmic approach, while translations from the System Combination Track result from a combination technique where two or more core algorithmic approaches are used.

Evaluation Data

The evaluation data of OpenMT12 consisted of several subsets of data. All available subsets per language pair were processed by participants for that language pair. All source data and reference translations were made available for inspection and experimentation by participants after the evaluation. This means that the data sets labeled as Progress tests here will not serve as Progress tests again in the future, as they will have been exposed.

  • Original Progress test: The Progress test sets used in OpenMT08 and OpenMT09, unchanged, with data to be released after OpenMT12. Data genres: Newswire and Web.
  • Expanded Progress test: The Progress test set used in OpenMT08 and OpenMT09, expanded with additional data from the same data genres.
  • Original Progress test new source subset: A combined subset of the OpenMT08 and OpenMT09 Progress tests, with new source data generated by human translation of the English reference translation. The source data was provided in two styles:
    • English-true: A more English-oriented translation; requires that the text reads well and does not use any idiomatic expressions in the foreign language to convey meaning, unless absolutely necessary.
    • Foreign-true: A translation as close as possible to the foreign language, as if the text had originated in that language.
  • New domain: Data from a variety of genres, varying in formality and structured, with a common theme that will not be made public before evaluation.

The following table lists which data subsets were part of the evaluation set and the approximate reference word count comprising all subsets for each language pair:

Language Pair Original Progress test Expanded Progress test Original Progress test new source subset New domain Approximate reference word count
Arabic-to-English Yes   Yes   140,000
Chinese-to-English Yes Yes Yes Yes 190,000
Dari-to-English     Yes   90,000
Farsi-to-English     Yes   90,000
Korean-to-English     Yes   90,000

Performance Measurement
  • BLEU-4 (mteval-v13a, the official OpenMT12 evaluation metric)
    • Invocation line: perl mteval-v13a.pl -r REFERENCE_FILE -s SOURCE_FILE -t CANDIDATE_FILE -c -b
    • Option -c : case-sensitive scoring
    • Option -b : BLEU score only
  • IBM BLEU (bleu-1.04a)
    • Invocation line: perl bleu-1.04.pl -r REFERENCE_FILE -t CANDIDATE_FILE
    • By default, scoring is case-sensitive
  • NIST (mteval-v13a)
    • Invocation line: perl mteval-v13a.pl -r REFERENCE_FILE -s SOURCE_FILE -t CANDIDATE_FILE -c -n
    • Option -c : case-sensitive scoring
    • Option -n : NIST score only
  • TER (tercom-0.7.25)
    • Invocation line: java -jar tercom.7.25.jar -r REFERENCE_FILE -h CANDIDATE_FILE -N -s
    • Option -N : enables normalization
    • Option -s : case-sensitive scoring
  • METEOR (meteor-0.7)
    • Invocation line: perl meteor.pl -s SYSTEM_ID -r REFERENCE_FILE -t CANDIDATE_FILE --modules "exact porter_stem wn_stem wn_synonymy"
    • Option --modules "exact porter_stem wn_stem wn_synonymy" : uses all four METEOR matching modules, in that order
Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

SiteID Site Location Ara2Eng Chi2Eng Dar2Eng Far2Eng Kor2Eng
BBN Raytheon BBN Technologies USA Yes Yes      
CAM Cambridge University UK Yes Yes      
CMU CMU-Oxford USA         Yes
COLUMBIA Columbia USA Yes   Yes Yes  
DFKI DFKI Language Technology Lab Germany   Yes      
EDINBURGH University of Edinburgh UK Yes Yes      
HIT Harbin Institute of Technology China Yes Yes Yes Yes Yes
ICTCASWSU Institute of Computing Technology, Chinese Academy of Sciences and Wright State University China, USA Yes Yes      
IPPH Intellectual Property Publishing House China   Yes      
KUNLPL Korea University Natural Language Processing Lab. Korea         Yes
LIUM LIUM University of Le Mans France Yes Yes      
NEUNLP Northeastern university, Natural Language Processing Lab China   Yes      
NJU Nanjing University China   Yes      
NRC NRC Canada Canada Yes Yes      
OSU Ohio State University USA     Yes Yes  
QCRI Qatar Computing Research Institute Qatar Yes        
SAIC SAIC USA   Yes Yes Yes Yes
SUNLP Stanford University NLP Group USA   Yes      
TBKBG Tubitak-bilgem Turkey Yes        
USC-ISI University of Southern California, Information Sciences Institute USA Yes Yes Yes Yes  
UVA University of Amsterdam, Informatics Institute Netherlands Yes Yes Yes Yes Yes
XAUT Xi'an University of Technology China   Yes      

Results Section

Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





Arabic-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Arabic-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

SiteID Site Location Ara2Eng
BBN Raytheon BBN Technologies USA Yes
CAM Cambridge University UK Yes (1)
COLUMBIA Columbia USA Yes
EDINBURGH University of Edinburgh UK Yes
HIT Harbin Institute of Technology China Yes
ICTCASWSU Institute of Computing Technology, Chinese Academy of Sciences and Wright State University China, USA Yes
LIUM LIUM University of Le Mans France Yes
NRC NRC Canada Canada Yes
QCRI Qatar Computing Research Institute Qatar Yes
TBKBG Tubitak-bilgem Turkey Yes
USC-ISI University of Southern California, Information Sciences Institute USA Yes
UVA University of Amsterdam, Informatics Institute Netherlands Yes


1a. Arabic-To-English Progress Test New Source Subset - "English-True" source
(143 documents containing 1537 segments)

Arabic-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_ara2eng_primary_cn 0.4093 0.4292 0.3866 0.4091 0.4290 0.3863 10.0870 9.8895 9.3929 0.5018 0.4915 0.5129 0.6357 0.6516 0.6179
CAM CAM_ara2eng_primary_cn (1) 0.4037 0.4257 0.3789 0.4034 0.4254 0.3786 9.8997 9.7245 9.2132 0.5150 0.5041 0.5266 0.6328 0.6478 0.6161
COLUMBIA COLUMBIA_ara2eng_primary_cn 0.3782 0.3921 0.3622 0.3780 0.3920 0.3620 9.5943 9.3529 9.0146 0.5282 0.5251 0.5315 0.6172 0.6332 0.5993
EDINBURGH EDINBURGH_ara2eng_primary_cn 0.3924 0.4078 0.3748 0.3922 0.4076 0.3745 9.8058 9.5603 9.1933 0.5232 0.5175 0.5294 0.6290 0.6439 0.6124
HIT HIT_ara2eng_primary_cn 0.3225 0.3490 0.2920 0.3223 0.3489 0.2916 9.0555 8.9890 8.2512 0.5656 0.5566 0.5753 0.5609 0.5847 0.5340
ICTCASWSU ICTCASWSU_ara2eng_primary_cn 0.2765 0.2963 0.2528 0.2761 0.2958 0.2524 8.4758 8.4086 7.7959 0.6059 0.5988 0.6136 0.5426 0.5670 0.5151
LIUM LIUM_ara2eng_primary_cn 0.4000 0.4240 0.3732 0.3998 0.4240 0.3729 9.9114 9.7816 9.1749 0.5154 0.5037 0.5280 0.6238 0.6381 0.6079
NRC NRC_ara2eng_primary_cn 0.4134 0.4319 0.3926 0.4133 0.4318 0.3924 10.1264 9.9360 9.4135 0.5049 0.4937 0.5169 0.6388 0.6546 0.6212
QCRI QCRI_ara2eng_primary_cn 0.3941 0.4048 0.3821 0.3940 0.4047 0.3818 9.8545 9.5409 9.3249 0.5189 0.5151 0.5229 0.6226 0.6349 0.6089
TBKBG TBKBG_ara2eng_primary_cn 0.3934 0.4107 0.3736 0.3932 0.4106 0.3734 9.8490 9.6361 9.2003 0.5118 0.5068 0.5172 0.6234 0.6402 0.6046
USC-ISI USC-ISI_ara2eng_primary_cn 0.3838 0.4014 0.3632 0.3834 0.4010 0.3627 9.6350 9.4041 9.0368 0.5321 0.5238 0.5411 0.6230 0.6425 0.6010
UVA UVA_ara2eng_primary_cn 0.3991 0.4189 0.3764 0.3987 0.4186 0.3759 9.9296 9.7215 9.2667 0.5131 0.5056 0.5212 0.6392 0.6561 0.6203



1b. Arabic-To-English Progress Test New Source Subset - "Foreign-True" source
(143 documents containing 1537 segments)

Arabic-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_ara2eng_primary_cn 0.3927 0.4122 0.3703 0.3924 0.4121 0.3698 9.8875 9.6814 9.2225 0.5259 0.5137 0.5390 0.6194 0.6368 0.5999
CAM CAM_ara2eng_primary_cn (1) 0.3827 0.4025 0.3601 0.3822 0.4020 0.3597 9.6239 9.4485 8.9690 0.5414 0.5278 0.5559 0.6150 0.6291 0.5992
COLUMBIA COLUMBIA_ara2eng_primary_cn 0.3555 0.3743 0.3336 0.3553 0.3742 0.3332 9.3034 9.1231 8.6882 0.5553 0.5491 0.5620 0.6000 0.6195 0.5781
EDINBURGH EDINBURGH_ara2eng_primary_cn 0.3725 0.3908 0.3509 0.3722 0.3906 0.3505 9.5428 9.3180 8.9350 0.5478 0.5398 0.5564 0.6122 0.6298 0.5923
HIT HIT_ara2eng_primary_cn 0.3042 0.3349 0.2685 0.3039 0.3346 0.2681 8.7748 8.7976 7.8904 0.5834 0.5714 0.5962 0.5439 0.5692 0.5152
ICTCASWSU ICTCASWSU_ara2eng_primary_cn 0.2629 0.2890 0.2327 0.2624 0.2884 0.2321 8.2261 8.2063 7.4875 0.6197 0.6107 0.6294 0.5269 0.5519 0.4983
LIUM LIUM_ara2eng_primary_cn 0.3817 0.4065 0.3541 0.3815 0.4064 0.3537 9.6672 9.5651 8.9283 0.5361 0.5160 0.5578 0.6063 0.6235 0.5870
NRC NRC_ara2eng_primary_cn 0.3933 0.4156 0.3676 0.3930 0.4154 0.3672 9.8650 9.7097 9.1458 0.5294 0.5139 0.5460 0.6225 0.6408 0.6021
QCRI QCRI_ara2eng_primary_cn 0.3724 0.3861 0.3567 0.3721 0.3860 0.3561 9.5820 9.3174 9.0316 0.5423 0.5355 0.5495 0.6071 0.6214 0.5911
TBKBG TBKBG_ara2eng_primary_cn 0.3714 0.3909 0.3489 0.3712 0.3907 0.3486 9.5540 9.3693 8.9065 0.5379 0.5282 0.5484 0.6045 0.6217 0.5851
USC-ISI USC-ISI_ara2eng_primary_cn 0.3644 0.3874 0.3369 0.3639 0.3869 0.3365 9.3897 9.2387 8.7324 0.5556 0.5425 0.5696 0.6041 0.6263 0.5792
UVA UVA_ara2eng_primary_cn 0.3789 0.4027 0.3514 0.3785 0.4022 0.3509 9.6637 9.5303 8.9540 0.5401 0.5267 0.5543 0.6205 0.6398 0.5987



2. Arabic-To-English Original Progress Test
(135 documents containing 1378 segments)

Arabic-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_ara2eng_primary_cn 0.4951 0.5379 0.4369 0.4949 0.5378 0.4368 11.5046 11.5260 10.2290 0.4385 0.4072 0.4775 0.6657 0.6932 0.6303
CAM CAM_ara2eng_primary_cn (1) 0.4736 0.5139 0.4200 0.4742 0.5140 0.4201 11.0864 11.0591 9.9359 0.4629 0.4321 0.5013 0.6524 0.6831 0.6128
COLUMBIA COLUMBIA_ara2eng_primary_cn 0.4119 0.4531 0.3580 0.4116 0.4528 0.3577 10.2611 10.3181 9.1477 0.4950 0.4657 0.5315 0.6190 0.6519 0.5768
EDINBURGH EDINBURGH_ara2eng_primary_cn 0.4395 0.4765 0.3914 0.4391 0.4761 0.3910 10.6336 10.6674 9.5117 0.4856 0.4563 0.5222 0.6342 0.6633 0.5967
HIT HIT_ara2eng_primary_cn 0.3292 0.3733 0.2707 0.3287 0.3730 0.2701 8.8369 9.3410 7.2123 0.5459 0.5140 0.5857 0.5484 0.5884 0.4960
ICTCASWSU ICTCASWSU_ara2eng_primary_cn 0.2908 0.3266 0.2434 0.2904 0.3261 0.2431 8.4868 8.8591 7.0355 0.5830 0.5583 0.6138 0.5432 0.5827 0.4912
LIUM LIUM_ara2eng_primary_cn 0.4639 0.5089 0.4063 0.4635 0.5087 0.4056 11.0146 11.1218 9.7924 0.4625 0.4241 0.5105 0.6428 0.6740 0.6031
NRC NRC_ara2eng_primary_cn 0.4751 0.5128 0.4235 0.4749 0.5125 0.4234 11.0972 11.1119 9.9357 0.4586 0.4270 0.4979 0.6512 0.6797 0.6146
QCRI QCRI_ara2eng_primary_cn 0.4607 0.5047 0.4012 0.4605 0.5044 0.4009 11.0387 11.0278 9.8490 0.4637 0.4336 0.5011 0.6485 0.6791 0.6087
TBKBG TBKBG_ara2eng_primary_cn 0.4449 0.4890 0.3878 0.4447 0.4888 0.3875 10.6685 10.7772 9.4473 0.4752 0.4429 0.5156 0.6312 0.6627 0.5911
USC-ISI USC-ISI_ara2eng_primary_cn 0.4551 0.4976 0.3938 0.4545 0.4968 0.3934 10.8785 10.8647 9.5956 0.4744 0.4448 0.5112 0.6426 0.6800 0.5939
UVA UVA_ara2eng_primary_cn 0.4638 0.5046 0.4026 0.4633 0.5037 0.4022 10.9709 10.9427 9.8285 0.4666 0.4368 0.5037 0.6564 0.6876 0.6162


(1)A late debugged submission was also made; scores for that submission are not reported here.


Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





Chinese-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Chinese-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

SiteID Site Location Chi2Eng
BBN Raytheon BBN Technologies USA Yes (2)
CAM Cambridge University UK Yes (2)
DFKI DFKI Language Technology Lab Germany Yes
EDINBURGH University of Edinburgh UK Yes
HIT Harbin Institute of Technology China Yes
ICTCASWSU Institute of Computing Technology, Chinese Academy of Sciences and Wright State University China, USA Yes (3)
IPPH Intellectual Property Publishing House China Yes
LIUM LIUM University of Le Mans France Yes
NEUNLP Northeastern university, Natural Language Processing Lab China Yes
NJU Nanjing University China Yes
NRC NRC Canada Canada Yes
SAIC SAIC USA Yes
SUNLP Stanford University NLP Group USA Yes (4)
USC-ISI University of Southern California, Information Sciences Institute USA Yes
UVA University of Amsterdam, Informatics Institute Netherlands Yes
XAUT Xi'an University of Technology China Yes


1a. Chinese-To-English Progress Test New Source Subset - "English-True" source
(143 documents containing 1537 segments)

Chinese-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_chi2eng_primary_cn (2) 0.2996 0.3385 0.2551 0.2992 0.3382 0.2547 8.6883 8.9683 7.6803 0.6169 0.5859 0.6502 0.5495 0.5797 0.5160
CAM CAM_chi2eng_primary_cn (2) 0.2864 0.3142 0.2588 0.2861 0.3139 0.2585 8.2970 8.4806 7.4930 0.6552 0.6177 0.6955 0.5520 0.5603 0.5428
EDINBURGH EDINBURGH_chi2eng_primary_cn 0.2589 0.2809 0.2343 0.2585 0.2805 0.2338 8.0112 8.0110 7.4009 0.6748 0.6591 0.6917 0.5438 0.5617 0.5240
HIT HIT_chi2eng_primary_cn 0.2770 0.2937 0.2586 0.2768 0.2933 0.2586 8.4247 8.4135 7.7748 0.6215 0.6102 0.6336 0.5247 0.5408 0.5068
ICTCASWSU ICTCASWSU_chi2eng_primary_cn (3) 0.2680 0.2903 0.2434 0.2675 0.2900 0.2428 8.2970 8.3174 7.6170 0.6380 0.6263 0.6506 0.5389 0.5543 0.5219
LIUM LIUM_chi2eng_primary_cn 0.2557 0.2747 0.2358 0.2554 0.2744 0.2356 7.9602 7.9942 7.3390 0.6619 0.6440 0.6811 0.5224 0.5317 0.5123
NEUNLP NEUNLP_chi2eng_primary_cn 0.2617 0.2767 0.2455 0.2613 0.2764 0.2451 7.9879 7.9687 7.3962 0.6660 0.6543 0.6785 0.5441 0.5613 0.5251
NJU NJU_chi2eng_primary_cn 0.2830 0.3019 0.2621 0.2826 0.3012 0.2619 8.5211 8.4821 7.8814 0.6106 0.6012 0.6206 0.5319 0.5517 0.5098
NRC NRC_chi2eng_primary_cn 0.3155 0.3359 0.2931 0.3151 0.3355 0.2927 9.0518 9.0284 8.3325 0.5976 0.5887 0.6072 0.5639 0.5780 0.5482
USC-ISI USC-ISI_chi2eng_primary_cn 0.2799 0.3125 0.2466 0.2794 0.3119 0.2462 8.0994 8.4022 7.1660 0.6838 0.6285 0.7432 0.5635 0.5815 0.5436
UVA UVA_chi2eng_primary_cn 0.2472 0.2622 0.2302 0.2470 0.2621 0.2299 8.0463 8.0063 7.4677 0.6495 0.6440 0.6554 0.5289 0.5424 0.5140
XAUT XAUT_chi2eng_primary_cn 0.2395 0.2612 0.2170 0.2391 0.2605 0.2167 7.6360 7.7950 6.9153 0.6959 0.6654 0.7287 0.5271 0.5434 0.5091

Chinese-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
DFKI DFKI_chi2eng_primary_un 0.1518 0.1632 0.1390 0.1519 0.1633 0.1389 6.4912 6.4684 6.1044 0.7253 0.7188 0.7323 0.4634 0.4728 0.4530
IPPH IPPH_chi2eng_primary_un 0.2125 0.2151 0.2096 0.2123 0.2148 0.2094 7.3263 7.1626 6.9567 0.6450 0.6468 0.6431 0.4877 0.4949 0.4798
SAIC SAIC_chi2eng_primary_un 0.2377 0.2531 0.2183 0.2374 0.2526 0.2180 7.8377 7.9077 7.1448 0.6800 0.6493 0.7129 0.5305 0.5264 0.5349

Chinese-To-English / Original System Combination / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
ICTCASWSU ICTCASWSU_chi2eng_combo1_cn 0.2870 0.3077 0.2639 0.2866 0.3073 0.2634 8.5378 8.5495 7.8420 0.6145 0.6052 0.6245 0.5360 0.5522 0.5181



1b. Chinese-To-English Progress Test New Source Subset - "Foreign-True" source
(143 documents containing 1537 segments)

Chinese-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_chi2eng_primary_cn (2) 0.2809 0.3219 0.2282 0.2806 0.3217 0.2278 8.4228 8.7381 7.3584 0.6347 0.6032 0.6687 0.5366 0.5767 0.4918
CAM CAM_chi2eng_primary_cn (2) 0.2687 0.2976 0.2385 0.2684 0.2973 0.2382 8.0577 8.2423 7.2680 0.6746 0.6379 0.7140 0.5388 0.5522 0.5242
EDINBURGH EDINBURGH_chi2eng_primary_cn 0.2467 0.2741 0.2145 0.2464 0.2739 0.2141 7.7929 7.8829 7.0965 0.6932 0.6769 0.7107 0.5331 0.5583 0.5052
HIT HIT_chi2eng_primary_cn 0.2607 0.2864 0.2317 0.2604 0.2861 0.2312 8.1649 8.2686 7.3634 0.6350 0.6241 0.6468 0.5136 0.5379 0.4863
ICTCASWSU ICTCASWSU_chi2eng_primary_cn (3) 0.2542 0.2815 0.2229 0.2539 0.2812 0.2226 8.0134 8.1601 7.2102 0.6558 0.6366 0.6764 0.5257 0.5523 0.4961
LIUM LIUM_chi2eng_primary_cn 0.2441 0.2732 0.2125 0.2440 0.2731 0.2124 7.7764 7.9493 7.0267 0.6774 0.6538 0.7027 0.5114 0.5278 0.4934
NEUNLP NEUNLP_chi2eng_primary_cn 0.2459 0.2712 0.2168 0.2455 0.2709 0.2165 7.7552 7.8548 7.0612 0.6854 0.6677 0.7043 0.5339 0.5603 0.5045
NJU NJU_chi2eng_primary_cn 0.2697 0.2980 0.2373 0.2692 0.2976 0.2368 8.2933 8.3809 7.5012 0.6279 0.6116 0.6454 0.5230 0.5504 0.4924
NRC NRC_chi2eng_primary_cn 0.3002 0.3276 0.2675 0.2998 0.3273 0.2671 8.7757 8.8527 7.9415 0.6184 0.6031 0.6349 0.5478 0.5714 0.5214
USC-ISI USC-ISI_chi2eng_primary_cn 0.2644 0.2946 0.2307 0.2640 0.2940 0.2305 7.8500 8.0311 7.0376 0.7001 0.6660 0.7368 0.5491 0.5746 0.5205
UVA UVA_chi2eng_primary_cn 0.2397 0.2604 0.2136 0.2395 0.2603 0.2133 7.8789 7.9209 7.1926 0.6640 0.6530 0.6757 0.5193 0.5429 0.4931
XAUT XAUT_chi2eng_primary_cn 0.2248 0.2494 0.1978 0.2245 0.2490 0.1975 7.3623 7.5638 6.6089 0.7209 0.6937 0.7501 0.5198 0.5431 0.4939

Chinese-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
DFKI DFKI_chi2eng_primary_un 0.1455 0.1586 0.1296 0.1456 0.1588 0.1295 6.3588 6.4024 5.8941 0.7368 0.7340 0.7398 0.4550 0.4713 0.4367
IPPH IPPH_chi2eng_primary_un 0.1998 0.2122 0.1859 0.1996 0.2119 0.1858 7.1316 7.1508 6.5764 0.6582 0.6534 0.6634 0.4780 0.4939 0.4604
SAIC SAIC_chi2eng_primary_un 0.2256 0.2472 0.2020 0.2253 0.2469 0.2017 7.6449 7.8130 6.9092 0.6909 0.6597 0.7245 0.5202 0.5240 0.5162

Chinese-To-English / Original System Combination / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
ICTCASWSU ICTCASWSU_chi2eng_combo1_cn 0.2708 0.2988 0.2390 0.2705 0.2985 0.2387 8.2166 8.3419 7.4081 0.6328 0.6178 0.6488 0.5222 0.5476 0.4939



2. Chinese-To-English Original Progress Test
(122 documents containing 1370 segments)

Chinese-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_chi2eng_primary_cn (2) 0.3292 0.3870 0.2443 0.3288 0.3867 0.2439 9.2705 9.9628 7.0658 0.5669 0.5388 0.6037 0.5429 0.5912 0.4761
CAM CAM_chi2eng_primary_cn (2) 0.3133 0.3341 0.2848 0.3129 0.3337 0.2845 8.8874 8.9344 7.9719 0.5926 0.5806 0.6082 0.5319 0.5475 0.5108
EDINBURGH EDINBURGH_chi2eng_primary_cn 0.2863 0.3202 0.2370 0.2856 0.3193 0.2366 8.6627 8.8148 7.4162 0.6167 0.6092 0.6265 0.5309 0.5596 0.4913
HIT HIT_chi2eng_primary_cn 0.2605 0.2883 0.2217 0.2604 0.2874 0.2228 7.5397 8.0191 6.1044 0.5966 0.5900 0.6054 0.4893 0.5142 0.4551
ICTCASWSU ICTCASWSU_chi2eng_primary_cn (3) 0.2896 0.3263 0.2367 0.2890 0.3256 0.2363 8.6810 9.0084 7.2240 0.5952 0.5827 0.6115 0.5298 0.5643 0.4821
LIUM LIUM_chi2eng_primary_cn 0.2758 0.3001 0.2425 0.2752 0.2995 0.2418 8.2393 8.3613 7.3055 0.6072 0.5932 0.6255 0.5028 0.5190 0.4809
NEUNLP NEUNLP_chi2eng_primary_cn 0.2834 0.3109 0.2446 0.2829 0.3101 0.2443 8.5146 8.6358 7.3716 0.6153 0.6122 0.6194 0.5277 0.5556 0.4891
NJU NJU_chi2eng_primary_cn 0.2747 0.3100 0.2252 0.2744 0.3093 0.2254 7.8141 8.3946 6.1585 0.5815 0.5699 0.5967 0.5017 0.5348 0.4559
NRC NRC_chi2eng_primary_cn 0.3372 0.3655 0.2976 0.3368 0.3651 0.2972 9.4961 9.7301 8.0916 0.5560 0.5492 0.5650 0.5527 0.5792 0.5161
USC-ISI USC-ISI_chi2eng_primary_cn 0.3185 0.3523 0.2695 0.3180 0.3515 0.2693 9.0370 9.2154 7.6024 0.5938 0.5864 0.6036 0.5496 0.5804 0.5068
UVA UVA_chi2eng_primary_cn 0.2643 0.2871 0.2319 0.2638 0.2868 0.2311 8.3712 8.6092 7.0808 0.5995 0.5976 0.6021 0.5204 0.5467 0.4840
XAUT XAUT_chi2eng_primary_cn 0.2493 0.2749 0.2133 0.2488 0.2744 0.2128 7.9613 8.2215 6.8128 0.6364 0.6269 0.6487 0.5039 0.5290 0.4695

Chinese-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
DFKI DFKI_chi2eng_primary_un 0.1594 0.1741 0.1389 0.1593 0.1742 0.1385 6.6037 6.7714 5.7865 0.6815 0.6832 0.6792 0.4458 0.4636 0.4216
IPPH IPPH_chi2eng_primary_un 0.1856 0.1892 0.1809 0.1850 0.1885 0.1803 5.8348 5.9508 5.1426 0.6332 0.6375 0.6276 0.4427 0.4551 0.4258
SAIC SAIC_chi2eng_primary_un 0.2511 0.2666 0.2302 0.2505 0.2658 0.2297 8.0963 7.8803 7.4174 0.6284 0.6141 0.6473 0.5077 0.5118 0.5022

Chinese-To-English / Original System Combination / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
ICTCASWSU ICTCASWSU_chi2eng_combo1_cn 0.3049 0.3458 0.2469 0.3050 0.3450 0.2482 8.5189 9.0364 6.8378 0.5765 0.5616 0.5961 0.5188 0.5509 0.4745



3. Chinese-to-English OpenMT12 Current Test
(73 documents containing 820 segments)

Chinese-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_chi2eng_primary_cn (2) 0.3102 0.3917 0.2129 0.3091 0.3900 0.2126 8.9241 9.6481 7.0985 0.6008 0.5538 0.6504 0.5299 0.5880 0.4668
CAM CAM_chi2eng_primary_cn (2) 0.2868 0.3390 0.2293 0.2864 0.3384 0.2291 8.3881 8.7010 7.2723 0.6340 0.5990 0.6710 0.5151 0.5428 0.4855
EDINBURGH EDINBURGH_chi2eng_primary_cn 0.2624 0.3087 0.2070 0.2612 0.3074 0.2066 8.2502 8.5227 7.1053 0.6477 0.6229 0.6740 0.5111 0.5452 0.4745
HIT HIT_chi2eng_primary_cn 0.2515 0.3071 0.1877 0.2511 0.3062 0.1877 7.5345 8.3120 5.9133 0.6158 0.5898 0.6432 0.4799 0.5237 0.4330
ICTCASWSU ICTCASWSU_chi2eng_primary_cn (3) 0.2694 0.3276 0.2029 0.2693 0.3275 0.2027 8.3531 8.8316 7.0328 0.6198 0.5841 0.6576 0.5170 0.5591 0.4722
LIUM LIUM_chi2eng_primary_cn 0.2645 0.3162 0.2059 0.2638 0.3152 0.2057 8.1164 8.4841 6.9763 0.6377 0.6082 0.6688 0.4941 0.5259 0.4603
NEUNLP NEUNLP_chi2eng_primary_cn 0.2574 0.3011 0.2016 0.2569 0.3003 0.2013 8.0667 8.3615 6.8888 0.6584 0.6325 0.6858 0.5161 0.5610 0.4673
NJU NJU_chi2eng_primary_cn 0.2635 0.3250 0.1919 0.2626 0.3233 0.1919 7.7750 8.5421 6.0743 0.6110 0.5829 0.6408 0.4896 0.5353 0.4401
NRC NRC_chi2eng_primary_cn 0.3099 0.3696 0.2401 0.3091 0.3681 0.2401 8.9618 9.3822 7.5321 0.5946 0.5667 0.6242 0.5373 0.5732 0.4987
USC-ISI USC-ISI_chi2eng_primary_cn 0.2939 0.3502 0.2267 0.2940 0.3502 0.2268 8.5366 8.9753 7.1508 0.6393 0.6009 0.6799 0.5356 0.5819 0.4852
UVA UVA_chi2eng_primary_cn 0.2379 0.2853 0.1809 0.2380 0.2857 0.1809 8.0019 8.3992 6.6783 0.6396 0.6263 0.6538 0.5030 0.5400 0.4628
XAUT XAUT_chi2eng_primary_cn 0.2326 0.2788 0.1791 0.2326 0.2789 0.1791 7.7532 8.1948 6.5468 0.6724 0.6441 0.7024 0.4927 0.5337 0.4494

Chinese-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
DFKI DFKI_chi2eng_primary_un 0.1628 0.2003 0.1188 0.1629 0.2004 0.1189 6.6141 7.0411 5.5317 0.7010 0.6891 0.7135 0.4416 0.4724 0.4091
IPPH IPPH_chi2eng_primary_un 0.1850 0.2240 0.1410 0.1849 0.2238 0.1410 6.0915 6.6777 4.9141 0.6443 0.6239 0.6658 0.4425 0.4761 0.4069
SAIC SAIC_chi2eng_primary_un 0.2250 0.2659 0.1816 0.2235 0.2631 0.1817 7.7873 7.9375 6.7192 0.6665 0.6263 0.7089 0.4933 0.5103 0.4760

Chinese-To-English / Original System Combination / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
ICTCASWSU ICTCASWSU_chi2eng_combo1_cn 0.2836 0.3447 0.2142 0.2835 0.3446 0.2142 8.1245 8.7279 6.6801 0.5989 0.5676 0.6320 0.5033 0.5430 0.4608



4. Chinese-to-English Original Progress Test + OpenMT12 Current Test
(195 documents containing 2190 segments)

Chinese-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_chi2eng_primary_cn (2) 0.3215 0.3888 0.2308 0.3208 0.3880 0.2304 9.4828 10.2865 7.3867 0.5806 0.5446 0.6240 0.5376 0.5900 0.4720
CAM CAM_chi2eng_primary_cn (2) 0.3025 0.3360 0.2608 0.3021 0.3355 0.2604 8.9965 9.2205 7.9655 0.6094 0.5876 0.6354 0.5250 0.5457 0.4997
EDINBURGH EDINBURGH_chi2eng_primary_cn 0.2766 0.3163 0.2240 0.2757 0.3151 0.2236 8.7910 9.0606 7.5704 0.6293 0.6144 0.6471 0.5228 0.5541 0.4839
HIT HIT_chi2eng_primary_cn 0.2569 0.2955 0.2073 0.2567 0.2946 0.2079 7.8068 8.4802 6.2518 0.6044 0.5899 0.6218 0.4855 0.5178 0.4453
ICTCASWSU ICTCASWSU_chi2eng_primary_cn (3) 0.2815 0.3268 0.2222 0.2811 0.3264 0.2219 8.8556 9.3234 7.4359 0.6052 0.5833 0.6315 0.5246 0.5623 0.4777
LIUM LIUM_chi2eng_primary_cn 0.2712 0.3063 0.2268 0.2706 0.3056 0.2263 8.4735 8.7598 7.4282 0.6195 0.5989 0.6443 0.4993 0.5217 0.4719
NEUNLP NEUNLP_chi2eng_primary_cn 0.2728 0.3081 0.2260 0.2723 0.3074 0.2257 8.6412 8.8890 7.4614 0.6328 0.6199 0.6482 0.5230 0.5577 0.4795
NJU NJU_chi2eng_primary_cn 0.2703 0.3158 0.2111 0.2697 0.3147 0.2112 8.0933 8.8298 6.3725 0.5935 0.5749 0.6158 0.4968 0.5350 0.4490
NRC NRC_chi2eng_primary_cn 0.3261 0.3671 0.2730 0.3256 0.3663 0.2728 9.6360 10.0245 8.1832 0.5717 0.5559 0.5907 0.5464 0.5769 0.5084
USC-ISI USC-ISI_chi2eng_primary_cn 0.3084 0.3515 0.2510 0.3082 0.3510 0.2510 9.1664 9.5137 7.7438 0.6123 0.5919 0.6367 0.5439 0.5810 0.4973
UVA UVA_chi2eng_primary_cn 0.2536 0.2864 0.2103 0.2534 0.2864 0.2098 8.5130 8.8867 7.1814 0.6158 0.6085 0.6245 0.5133 0.5441 0.4747
XAUT XAUT_chi2eng_primary_cn 0.2425 0.2764 0.1985 0.2422 0.2761 0.1983 8.1475 8.5446 6.9532 0.6510 0.6335 0.6720 0.4994 0.5308 0.4606

Chinese-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
DFKI DFKI_chi2eng_primary_un 0.1608 0.1843 0.1303 0.1607 0.1844 0.1301 6.8006 7.1226 5.8601 0.6894 0.6855 0.6941 0.4441 0.4669 0.4161
IPPH IPPH_chi2eng_primary_un 0.1854 0.2025 0.1641 0.1850 0.2020 0.1638 6.1378 6.4791 5.2293 0.6377 0.6323 0.6442 0.4426 0.4630 0.4175
SAIC SAIC_chi2eng_primary_un 0.2406 0.2664 0.2093 0.2396 0.2648 0.2091 8.2428 8.2200 7.3757 0.6439 0.6188 0.6740 0.5018 0.5112 0.4906

Chinese-To-English / Original System Combination / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
ICTCASWSU ICTCASWSU_chi2eng_combo1_cn 0.2965 0.3454 0.2330 0.2964 0.3448 0.2337 8.6573 9.2956 7.0395 0.5856 0.5639 0.6117 0.5125 0.5479 0.4684



5. Chinese-to-English OpenMT12 Novel Domain Test
(149 documents containing 2184 segments)

Chinese-To-English / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
BBN BBN_chi2eng_primary_cn (2)     0.2140     0.2138     6.8720     0.6189     0.4495
CAM CAM_chi2eng_primary_cn (2)     0.2034     0.2033     6.3858     0.6382     0.4292
EDINBURGH EDINBURGH_chi2eng_primary_cn     0.1801     0.1800     6.5970     0.6670     0.4377
HIT HIT_chi2eng_primary_cn     0.1465     0.1457     4.3693     0.6575     0.3609
ICTCASWSU ICTCASWSU_chi2eng_primary_cn (3)     0.1659     0.1658     6.2767     0.6717     0.4239
LIUM LIUM_chi2eng_primary_cn     0.1467     0.1470     5.3378     0.6794     0.3705
NEUNLP NEUNLP_chi2eng_primary_cn     0.1664     0.1663     6.2178     0.6803     0.4156
NJU NJU_chi2eng_primary_cn     0.1633     0.1631     4.5728     0.6324     0.3808
NRC NRC_chi2eng_primary_cn     0.2121     0.2119     7.0536     0.6203     0.4523
USC-ISI USC-ISI_chi2eng_primary_cn     0.1871     0.1871     6.2319     0.6558     0.4219
UVA UVA_chi2eng_primary_cn     0.1575     0.1574     5.9266     0.6526     0.4144
XAUT XAUT_chi2eng_primary_cn     0.1592     0.1590     6.1338     0.6729     0.4168

Chinese-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
DFKI DFKI_chi2eng_primary_un     0.1101     0.1102     5.2860     0.7201     0.3765
IPPH IPPH_chi2eng_primary_un     0.1341     0.1343     4.8729     0.6609     0.3809
SAIC SAIC_chi2eng_primary_un     0.1538     0.1536     6.0445     0.7135     0.4196

Chinese-To-English / Original System Combination / Constrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
ICTCASWSU ICTCASWSU_chi2eng_combo1_cn     0.1605     0.1605     4.9380     0.6471     0.3851


(2)A late debugged submission was also made; scores for that submission are not reported here.

(3)An on-time debugged submission was also made; scores for that submission are not reported here.

(4)An on-time submission was made but withdrawn; scores for that submission are not reported here.


Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





Dari-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Dari-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

SiteID Site Location Dar2Eng
COLUMBIA Columbia USA Yes
OSU Ohio State University USA Yes
SAIC SAIC USA Yes
USC-ISI University of Southern California, Information Sciences Institute USA Yes
UVA University of Amsterdam, Informatics Institute Netherlands Yes


1a. Dari-To-English Progress Test New Source Subset - "English-True" source
(143 documents containing 1537 segments)

Dari-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
COLUMBIA COLUMBIA_dar2eng_primary_un 0.0880 0.0954 0.0798 0.0878 0.0952 0.0796 5.2300 5.3014 4.8840 0.7684 0.7625 0.7746 0.3838 0.3936 0.3728
OSU OSU_dar2eng_primary_un 0.1151 0.1114 0.1191 0.1178 0.1148 0.1201 5.3838 5.1822 5.2976 0.8133 0.8253 0.8005 0.3910 0.3899 0.3922
SAIC SAIC_dar2eng_primary_un 0.1525 0.1626 0.1412 0.1522 0.1623 0.1408 5.9771 6.0864 5.4696 0.6874 0.6869 0.6878 0.4194 0.4315 0.4059
USC-ISI USC-ISI_dar2eng_primary_un 0.1191 0.1157 0.1228 0.1190 0.1156 0.1227 4.8986 4.7237 4.8097 0.7127 0.7235 0.7012 0.4003 0.3996 0.4011
UVA UVA_dar2eng_primary_un 0.2373 0.2596 0.2120 0.2371 0.2592 0.2120 7.6978 7.7615 7.0303 0.6461 0.6373 0.6556 0.4990 0.5192 0.4763



1b. Dari-To-English Progress Test New Source Subset - "Foreign-True" source
(143 documents containing 1537 segments)

Dari-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
COLUMBIA COLUMBIA_dar2eng_primary_un 0.0802 0.0815 0.0787 0.0800 0.0815 0.0783 5.0812 5.0381 4.8519 0.7811 0.7749 0.7877 0.3767 0.3818 0.3710
OSU OSU_dar2eng_primary_un 0.1115 0.1062 0.1170 0.1145 0.1098 0.1194 5.3119 5.0917 5.2481 0.8201 0.8291 0.8105 0.3851 0.3802 0.3907
SAIC SAIC_dar2eng_primary_un 0.1423 0.1473 0.1367 0.1420 0.1471 0.1363 5.8559 5.7644 5.5950 0.6987 0.6979 0.6997 0.4109 0.4170 0.4041
USC-ISI USC-ISI_dar2eng_primary_un 0.1135 0.1064 0.1212 0.1135 0.1066 0.1209 4.8434 4.5168 4.9177 0.7219 0.7328 0.7103 0.3946 0.3889 0.4009
UVA UVA_dar2eng_primary_un 0.2256 0.2416 0.2077 0.2265 0.2416 0.2095 7.5251 7.5407 6.9642 0.6611 0.6507 0.6723 0.4884 0.4999 0.4756



Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





Farsi-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Farsi-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

SiteID Site Location Far2Eng
COLUMBIA Columbia USA Yes
OSU Ohio State University USA Yes
SAIC SAIC USA Yes
USC-ISI University of Southern California, Information Sciences Institute USA Yes
UVA University of Amsterdam, Informatics Institute Netherlands Yes


1a. Farsi-To-English Progress Test New Source Subset - "English-True" source
(143 documents containing 1537 segments)

Farsi-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
COLUMBIA COLUMBIA_far2eng_primary_un 0.1816 0.1991 0.1617 0.1816 0.1990 0.1618 6.8163 6.9113 6.2459 0.6819 0.6741 0.6903 0.4609 0.4803 0.4392
OSU OSU_far2eng_primary_un 0.1373 0.1342 0.1407 0.1387 0.1358 0.1419 5.9556 5.7274 5.8422 0.7827 0.7925 0.7721 0.4353 0.4372 0.4331
SAIC SAIC_far2eng_primary_un 0.1902 0.2057 0.1726 0.1900 0.2054 0.1724 6.7930 6.9106 6.1859 0.6521 0.6450 0.6598 0.4579 0.4760 0.4374
USC-ISI USC-ISI_far2eng_primary_un 0.1898 0.1871 0.1928 0.1896 0.1869 0.1926 6.6040 6.3006 6.5094 0.7092 0.7211 0.6963 0.4767 0.4766 0.4768
UVA UVA_far2eng_primary_un 0.2665 0.2883 0.2419 0.2662 0.2878 0.2415 8.2174 8.2059 7.5688 0.6215 0.6099 0.6339 0.5242 0.5424 0.5037



1b. Farsi-To-English Progress Test New Source Subset - "Foreign-True" source
(143 documents containing 1537 segments)

Farsi-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
COLUMBIA COLUMBIA_far2eng_primary_un 0.1799 0.1971 0.1603 0.1799 0.1970 0.1604 6.8274 6.9549 6.1952 0.6793 0.6724 0.6866 0.4599 0.4820 0.4351
OSU OSU_far2eng_primary_un 0.1358 0.1317 0.1398 0.1376 0.1336 0.1407 5.9691 5.6862 5.9064 0.7793 0.7977 0.7595 0.4306 0.4354 0.4253
SAIC SAIC_far2eng_primary_un 0.1906 0.2030 0.1767 0.1905 0.2030 0.1766 6.6773 6.7903 6.0647 0.6539 0.6510 0.6569 0.4545 0.4731 0.4336
USC-ISI USC-ISI_far2eng_primary_un 0.1955 0.1899 0.2003 0.1954 0.1898 0.2003 6.6984 6.3630 6.6021 0.7025 0.7156 0.6884 0.4747 0.4784 0.4706
UVA UVA_far2eng_primary_un 0.2664 0.2896 0.2399 0.2663 0.2895 0.2398 8.2909 8.2995 7.5721 0.6168 0.6064 0.6280 0.5258 0.5469 0.5019



Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





Korean-To-English Results

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered alphabetically by siteID.

Korean-To-English Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

SiteID Site Location Kor2Eng
CMU CMU-Oxford USA Yes
KUNLPL Korea University Natural Language Processing Lab. Korea Yes
SAIC SAIC USA Yes
UVA University of Amsterdam, Informatics Institute Netherlands Yes


1a. Korean-To-English Progress Test New Source Subset - "English-True" source
(143 documents containing 1537 segments)

Korean-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
CMU CMU_kor2eng_primary_un 0.1113 0.1172 0.1049 0.1110 0.1171 0.1044 5.4745 5.5402 5.1113 0.7856 0.7824 0.7890 0.3974 0.4122 0.3810
KUNLPL KUNLPL_kor2eng_primary_un 0.1118 0.1173 0.1052 0.1119 0.1174 0.1054 5.3482 5.1900 5.2084 0.9031 0.9126 0.8929 0.4397 0.4434 0.4356
SAIC SAIC_kor2eng_primary_un 0.0943 0.0962 0.0923 0.0942 0.0962 0.0921 5.2579 5.1684 5.0722 0.8606 0.8629 0.8582 0.4252 0.4315 0.4183
UVA UVA_kor2eng_primary_un 0.0679 0.0665 0.0693 0.0679 0.0665 0.0693 2.2916 2.3803 2.0755 0.7438 0.7520 0.7351 0.2915 0.3022 0.2796



1b. Korean-To-English Progress Test New Source Subset - "Foreign-True" source
(143 documents containing 1537 segments)

Korean-To-English / UnConstrained

BLEU-4 (mteval-v13a) IBM BLEU (bleu-1.04) NIST (mteval-v13a) TER (tercom-0.7.25) METEOR (meteor-0.7)
SiteID System Overall NW WB Overall NW WB Overall NW WB Overall NW WB Overall NW WB
CMU CMU_kor2eng_primary_un 0.1047 0.1124 0.0963 0.1044 0.1124 0.0957 5.2021 5.3566 4.7619 0.7982 0.7881 0.8089 0.3815 0.3985 0.3626
KUNLPL KUNLPL_kor2eng_primary_un 0.1014 0.1076 0.0943 0.1015 0.1077 0.0944 5.0712 5.0206 4.8328 0.9182 0.9208 0.9154 0.4189 0.4313 0.4052
SAIC SAIC_kor2eng_primary_un 0.0881 0.0905 0.0854 0.0878 0.0902 0.0851 4.9513 4.9804 4.6584 0.8780 0.8720 0.8844 0.4049 0.4157 0.3929
UVA UVA_kor2eng_primary_un 0.0608 0.0609 0.0605 0.0609 0.0610 0.0606 2.1462 2.2334 1.9402 0.7525 0.7602 0.7442 0.2775 0.2894 0.2643



Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





Progress Test History

This page of results is limited to those of the Progress Test data for the OpenMT12 tests of Arabic-to-English and Chinese-to-English text-to-text translation.

The Progress test was designed as a means to demonstrate true system improvement for a particular site over time, irrespective of inherent data differences that come with each new test set. In order to keep the focus on progress over time, only results from sites that participated in OpenMT12 and at least one prior evaluation are reported.

There may be issues related to processing older test sets that make across-site comparisons less meaningful.

Scores reported are limited to primary, on-time, non-debugged submissions.

Scores are ordered by BLEU-4 score on the Overall test set.

Note that all BLEU-4 scores reported on this page were computed using mteval-v13a. (The original OpenMT08 scores were computed using mteval-v11b.)

Participants

The following table lists the organizations participating in OpenMT12 and the test sets they registered to process.

Site MT08 and/or MT09 Site OpenMT12 SiteID OpenMT12 Location Ara2Eng Chi2Eng
BBN Technologies Raytheon BBN Technologies BBN USA MT08, MT09, MT12 MT08, MT09, MT12
  Cambridge University CAM UK MT12 MT12
  CMU-Oxford CMU USA    
  Columbia COLUMBIA USA MT12  
  DFKI Language Technology Lab DFKI Germany   MT12
University of Edinburgh University of Edinburgh EDINBURGH UK MT09, MT12 MT12
Harbin Institute of Technology, Language Technology Research Center Harbin Institute of Technology HIT China MT12 MT09, MT12
Chinese Academy of Sciences, Institute of Computing Technology Institute of Computing Technology, Chinese Academy of Sciences and Wright State University ICTCASWSU China, USA MT12 MT09, MT12
  Intellectual Property Publishing House IPPH China   MT12
  Korea University Natural Language Processing Lab. KUNLPL Korea    
LIUM University of Le Mans (MT08)
LIUM University of Le Mans / SYSTRAN (MT09) (6)
LIUM University of Le Mans LIUM France MT08, MT09 (6) , MT12 MT12
  Northeastern university, Natural Language Processing Lab NEUNLP China   MT12
  Nanjing University NJU China   MT12
National Research Council Canada NRC Canada NRC Canada MT12 MT08, MT09, MT12
  Ohio State University OSU USA    
  Qatar Computing Research Institute QCRI Qatar MT12  
  SAIC SAIC USA   MT12
  Stanford University NLP Group SUNLP USA   withdrawn
TUBITAK-UEKAE Tubitak-bilgem TBKBG Turkey MT09, MT12  
University of Southern California / Language Weaver Inc. University of Southern California, Information Sciences Institute USC-ISI USA MT08, MT09, MT12 MT08, MT09, MT12
University of Amsterdam University of Amsterdam, Informatics Institute UVA Netherlands MT09, MT12 MT12
  Xi'an University of Technology XAUT China   MT12



1. Arabic-To-English Original Progress Test History


Overall NW WB
SiteID OpenMT12 System OpenMT12 MT08 MT09 MT12 MT08 MT09 MT12 MT08 MT09 MT12
BBN BBN_ara2eng_primary_cn 0.4186 (5) 0.4379 0.4951 0.4655 (5) 0.4926 0.5379 0.3566 (5) 0.3678 0.4369
EDINBURGH EDINBURGH_ara2eng_primary_cn - 0.4056 0.4395 - 0.4420 0.4765 - 0.3494 0.3914
LIUM LIUM_ara2eng_primary_cn 0.4105 - (6) 0.4639 0.4544 - (6) 0.5089 0.3530 - (6) 0.4063
TBKBG TBKBG_ara2eng_primary_cn - 0.3816 0.4449 - 0.4149 0.4890 - 0.3301 0.3878
USC-ISI USC-ISI_ara2eng_primary_cn 0.4030 (5) 0.4296 0.4551 0.4498 (5) 0.4748 0.4976 0.3408 (5) 0.3678 0.3938
UVA UVA_ara2eng_primary_cn - 0.2824 0.4638 - 0.3084 0.5046 - 0.2466 0.4026



2. Chinese-To-English Original Progress Test History


Overall NW WB
SiteID OpenMT12 System OpenMT12 MT08 MT09 MT12 MT08 MT09 MT12 MT08 MT09 MT12
BBN BBN_chi2eng_primary_cn 0.3055 (5) 0.3153 0.3292 0.3447 (5) 0.3481 0.3870 0.2509 (5) 0.2697 0.2443
HIT HIT_chi2eng_primary_cn - 0.2437 0.2605 - 0.2694 0.2883 - 0.2079 0.2217
ICTCASWSU ICTCASWSU_chi2eng_primary_cn - 0.2696 0.2896 - 0.3064 0.3263 - 0.2178 0.2367
NRC NRC_chi2eng_primary_cn 0.2480 0.2811 0.3372 0.2679 0.3130 0.3655 0.2204 0.2357 0.2976
USC-ISI USC-ISI_chi2eng_primary_cn 0.2990 (5) 0.3225 0.3185 0.3516 (5) 0.3628 0.3523 0.2237 (5) 0.2642 0.2695


(5)The MT08 submission was a system combination.

(6)A late debugged submission was made; scores for that submission are not reported here.


Top ]   [  Arabic-To-English ]   [  Chinese-To-English ]   [  Dari-To-English ]   [  Farsi-To-English ]   [  Korean-To-English ]   [  Progress Test History ]  





NIST OpenMT12 Home  ]