Data Science Evaluation Workshop

March 17–18, 2016

National Institute of Standards and Technology (NIST), Gaithersburg, MD

The Information Access Division (IAD)/Multimodal Information Group of the National Institute of Standards and Technology (NIST) held a workshop on the Data Science Evaluation (DSE) series within the new Data Science Research Program (DSRP).

The DSRP provides a forum for community-wide evaluations and supports research progress in the burgeoning field of data science. The DSE is a central component of the DSRP and serves as a broad forum for evaluation that will occur annually in support of data-science research.

In the fall of 2015, IAD hosted a small-scale pre-pilot evaluation in the traffic domain, meant to exercise a range of evaluation metrics and measurement methods. In 2016, IAD will follow up with an open pilot evaluation in the same domain. Proposals for tracks in the DSE will lead to a series of full-scale data science evaluations in 2017.

This workshop program included:

• A description and discussion of the upcoming public NIST IAD Pilot Evaluation
• Presentations from Track Coordinator Finalists for potential tracks to be run in the full-scale evaluation in 2017
• Discussions pertaining to Data Science Challenge Problems
• Summary and analysis of the NIST IAD Pre-Pilot evaluation results

For additional questions, please contact us at datascience [at] nist.gov (datascience[at]nist[dot]gov).

Registration 

Registration is now closed.

Program

Workshop Agenda.

Slides from the workshop are available below, with each presentation linked to the item. Slides are given in chronological order.

1. Introduction and Workshop Goals
2. Overview of NIST IAD Data Science Research Program
3. Overview of NIST IAD Data Science Evaluation Series
4. Summary of Data Science Pre-Pilot Evaluation
5. NIST Perspective on Lessons Learned from Pre-Pilot
6. UF Perspective on Lessons Learned from Pre-Pilot, Part 1
7. UF Perspective on Lessons Learned from Pre-Pilot, Part 2
8. Introduction to Track Coordinator Finalist Presentations
9. DSE Track Proposal #1: Manual vs. Automated Techniques for Estimating Security Risks
10. Introduction to Breakout Sessions
11. Infrastructure Breakout Introduction
12. Infrastructure Breakout Report
13. Evaluation Breakout Introduction
14. Evaluation Breakout Report
15. Benchmarking Breakout Introduction
16. Benchmarking Breakout Report
17. DSE Track Proposal #2: Data Science for Plant Identification with Remote Sensing
18. Overview of Pilot Evaluation
19. Challenge Problems Introduction: Overview
20. Challenge Problems Introduction: Alignment
21. Challenge Problems Introduction: Cleaning
22. Challenge Problems Introduction: Prediction
23. Concluding Remarks