The scoring script has been developed and tested in Python 3 (version 3.7+). Here, we provide an example of a requirements file for conda.
requirements.txt
name: sre19
dependencies:
- python=3.7
- numpy=1.16.4
Conda is an open source package and environment management system. The Miniconda version comes with a python distribution and is very light.
> conda env create -f requirements.txt
> source activate sre19
If you already have your own Python 3.7 environnement (using virtualenv for example), use the pip command to install the required packages (e.g., numpy). Please make sure to meet the above noted requirements.
The root directory contains the main script (i.e., sre_submission_scorer.py) and with the default configuration as per the SRE18/19 evaluation plan guidelines. It also contains a sample system output file (i.e., system_output/sre18_dev_system_output.tsv), which is formatted according to the guidelines given in the SRE18/19 evaluation plan. Currently, only the dev data has been released, therefore we only provide an example for scoring the dev set.
A good test is to first run the scoring script with the provided sample system output as input. This will first validate the system output and then computes the equalized metrics specified in the SRE18/19 evaluation plan. The sample system output is located in system_output/ folder. It meets the SRE18/19 system output format requirements.
From the root directory, execute the following command:
> python3 sre_submission_scorer.py -o system_output/sre18_dev_system_output.tsv \
-l /path/to/LDC2019E59/dev/docs/sre18_dev_trials.tsv -r /path/to/LDC2019E59/dev/docs/sre18_dev_trial_key.tsv
From the root directory, execute the following command:
> python3 sre_submission_scorer.py -o /path/to/system_output.tsv \
-l /path/to/LDC2019E59/dev/docs/sre18_dev_trials.tsv -r /path/to/LDC2019E59/dev/docs/sre18_dev_trial_key.tsv
This software was developed at the National Institute of Standards and Technology (NIST/ITL/IAD/MIG) by employees of the Federal Government in the course of their official duties. Pursuant to Title 17 Section 105 of the United States Code this software is not subject to copyright protection and is in the public domain. Permission to freely use, copy, modify, and distribute this software and its documentation without fee is hereby granted, PROVIDED that this notice and disclaimer of warranty appears in all copies
This software is offered AS IS. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guarantees and NO WARRANTIES, EXPRESSED OR IMPLIED, about its quality, reliability, fitness for any purpose, or any other characteristic.
We would appreciate acknowledgement if the software is used.