U.S. flag

An official website of the United States government

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Information Technology Laboratory / Information Access Division

Multimodal Information Group

OpenKWS14 Dry Run Instructions

Share

The following is the OpenKWS Evaluation Infrastructure setup and testing instructions

  1. Download the IARPA-Babel Vietnamese Build and Eval packs with using the username and password provided by NIST.
  2. Download the latest IndusDB 20140206 (Contact NIST for link).
  3. Download and install SCTK Version 2.4.8 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) according to the INSTALL document in the distribution.
  4. Download JobRunner 1.3.0 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) 
  5. Download and check the F4DE 3.1.0 or later (from http://www.nist.gov/itl/iad/mig/tools.cfm) according to the instructions under "INSTALLATION" section in the"README" file of the distribution.
  • Note, F4DE requires many CPAN libraries and other utilities.  It is vital that a you successfully run either the full "make check" or minimally the 'make mincheck' and 'make KWSEvalcheck' before proceeding to Step 6.
  • Send NIST Information to set up your Indus Scoring Server account which uses SSH public/private keys for authentication. (Instructions summarized from the "KWSEval/BABEL/Participants/README" in the F4DE release).
    1. If not familiar with SSH, please see: http://www.wikihow.com/Use-SSH
    2. Decide on the user name of your team; it must follow the pattern "kws-<team>[_<site>]" (<team> or <site> can not contain - or _)
    3. Generate an SSH Protocol 2 Key: Either ensure you have a valid SSH Protocol 2 public/private keys or generate a pair using either '% ssh-keygen -t dsa' or '% ssh-keygen -t rsa' with a non empty passphrase of 12 or more characters.
    4. email 'openkws-poc [at] nist.gov ('openkws-poc at nist.gov)' with the following:
      1. your user, site and team names
      2. the list of users expected to use the service with contact information (email + phone)
      3. Attach your public key (not inline text) to the email
      4. Send a list of the IP subnets that will need access to the ssh Indus server
  • After completing step 6 and receiving your Indus Account, follow the Indus Scorer Setup instructions  found in "KWSEval/BABEL/Participants/README" of the  F4DE release (Note that this has been updated in F4DE 3.1.0).
  • Run the test submissions through the scoring server by:
    1. Change directory to the KWSEval/BABEL/Participants/Checks directory
    2. Execute: "./SubmissionHelper_checks.sh ../KWS14-dbDir/samples-checkfiles/Comps--107_____KWS14*"
  • Unpacked password-protected zip file (TestData.zip) to make sure you can read the delivered data. The password is "abc123". The file listing should be:

    • TestData
    TestData/dir1
    TestData/dir1/subdir1
    TestData/dir1/subdir1/Readme.txt
    TestData/dir2
    TestData/TopReadme.txt

    Data and informatics
    Created December 2, 2013, Updated June 2, 2021