NIST logo
*
Bookmark and Share

NIST Open Keyword Search 2014 Evaluation (OpenKWS14)

The OpenKWS14 is currently accepting signups for the evaluation. To join the evaluation or inquire about the evaluation, send an email to openkws-poc@nist.gov to be added to the mailing lists. Developers are expected to build systems according to the Draft KWS14 Evaluation Plan linked below.

Signup procedure

  1. Read the evaluation plan to become familiar with the evaluation.
  2. Sign and return the OpenKWS14 registration form to openkws-poc@nist.gov.
  3. Sign and return the OpenKWS14 Babel data licensing agreement V2 to openkws-poc@nist.gov.
  4. Complete a Dry Run Evaluation.  
    • The dry run is an opportunity for developers to make sure they are able to generate valid system output that can be scored with the NIST scoring tools. The actual performance of the system is not of interest during the dry run so developers may feel free to use any method to generate their system output, e.g., a random system, training on the dry run data, etc. The Evaluation Infrastructure Setup Instructions enumerates the steps to complete a dry run. 
    • NIST highly encourages new teams build a Vietnamese system to familiarize themselves with the Babel resources in advance of the surprise language evaluation as part of their dry run.

Schedule

January 31 2014 OpenKWS registration closes
March 24 2014 Surprise language build pack download begins
April 2 2014, 2:00 PM EDT NIST sends password for surprise language build pack
April 14 2014 Surprise language evaluation pack download begins

April 23 2014, 2:00 pm EDT

NIST sends password for surprise language evaluation pack and keywords
April 30 2014, 2:00pm EDT Sites submit surprise language system outputs to NIST
May 7 2014 NIST reports surprise language results
July 2014 OpenKWS14 meeting in Washington DC metro area


Documentation

Data Resources

OpenKWS participants will receive three types of resources:  build packs, evaluation packs, and IndusDB releases.  See Section 2 of the evaluation plan for a description of the resources.  Teams must complete both the registration form and data license in order receive data.  Resources will be distributed as follows:

  • Build packs: Build packs will be distributed to teams that signup for OpenKWS14.
    • The Vietnamese Build pack will be provided at signup time.  
    • The 2014 Surprise Language build pack will be distributed according to the schedule above
  • Evaluation packs: Evaluation packs will be distributed to teams that signup for OpenKWS.
    • The Vietnamese Evaluation pack  will be provided at signup time. The reference transcripts will not be released to the team until they successfully submit results on the Vietnamese eval pack.
    • The 2014 Surprise Language build pack will be distributed according to the schedule above.
  • IndusDB: IndusDB releases will be issued as appropriate for the team.

Evaluation tools

NIST maintains the Framework for Detection Evaluations (F4DE) Toolkit that supports the evaluation. The package is available on the MIG Tools page.

Contact

openkws-poc@nist.gov