IREX II IQCE

 

 

 

 

IREX II IQCE - Final Evaluation Report 

September 23, 2011: The final IQCE report: Iris Quality Calibration and Evaluation (IQCE): Performance of Iris Image Quality Assessment and Algorithms .

Update to API.

April 26, 2010: The newest version is 4.4. Table of device identifier (Table 7) is updated. Clarification on scan_type values and handling of KIND_VGA images that do not satisfy margin requirement of current draft of ISO/IEC 19794-6 (N3620) have been added to Tables 9 and 10.

Update to API.

February 24, 2010: Table of device identifier (Table 7) is updated. Newest version is 4.4.

Final Version of FAQ .

February 4, 2010: Final version FAQ here.

Release of sample data

February 1, 2010: NIST will use iris images collected by Clarkson University. Clarkson University has kindly made sample images of one subject available for public release. Sample data can be downloaded from here.

Clarkson University's Quality-Face/Iris Research Ensemble (Q-FIRE) is a multi-biometric (face and iris) dataset where images are collected at five distances from camera (baseline, 5 feet, 7 feet, 11 feet, 15 feet, and 25 feet) with varying quality (resolution, ambient (or external) illumination, out-of-focus blur, multiple faces, gaze/pose angles, motion blur, and iris occlusion). Over 175 subjects are included with at least two visits each. The dataset also includes soft biometrics such as height and weight, for subjects of different age groups, ethnicity and gender with variable number of sessions/subject. Only Q-FIRE iris images (i.e. NIR illumination) will be used in IQCE.

Detail information on Q-FIRE dataset overview here.

Detail information on Q-FIRE data collection steps here.

Q-FIRE is funded by the Department of Homeland Security (DHS) Science and Technology (S&T) Directorate in cooperation with the National Science Foundation.

NIST is grateful to Clarkson University Q-FIRE team, specially Prof. Stephanie Schuckers for making this data available to us. Special thanks to Mr. Christopher Miles at DHS for supporting Q-FIRE and IQCE.

SDK submission is open

January 27, 2010: Organizations wishing to participate in IQCE should follow steps oulined in section A.2 of the latest version (version 4.4) of IQCE API specification. FAQ is posted here.

Validation package

January 27, 2010: Validation dataset is available here. Question should be directed to irex [at] nist.gov (irex[at]nist[dot]gov).

The purpose of validation is to verify the correct operation of participants' software before and after delivery to NIST. Validation is the process of checking that our linking and execution of your software gives identical results to your execution given the same input data. Participants are asked to use the images in the validation dataset and send us, along with their encrypted SDK, output file(s) as specified below:

Class X participants: One output file
1. Quality (IQAA) SDK
Format: [imagefilename] [quality_vector] [image_enhanced]

Class Y participants: Two output files
1. Quality and template generation SDK
Format:[imagefilename] [template_size] [quality_vector] [image_enhanced]
2. Matching (comparison) SDK
Format: [imagefilename1] [imagefilename2] [comparison_score]

Class Z participants: Two output files
1. Template generation SDK
Format (without quality computation):[imagefilename] [template_size] [image_enhanced]
Format (with quality computation):[imagefilename] [template_size] [quality_vector] [image_enhanced]
2. Matching (comparison) SDK
Format: [imagefilename1] [imagefilename2] [comparison_score]Quality_vector shall have the format specified in Table 4 of the latest version of IQCE API specification. A full cross comparison of images in the validation dataset shall be performed.

IREX II IQCE - Cryptographic Protection of Vendor Data

Public Key for Encrypting SDK submissions to NIST

October 4, 2009: The NIST public key for encryption of all software send to NIST is as follows. Use of this key is required. NIST does not accept responsiblity for any unencrypted software sent to NIST.

pub 1024D/856B9B28 2008-05-28 Key fingerprint = 846E 7008 996A E912 974C F8D7 1C7A 0F22 856B 9B28 uid IREX Test Liaison (IREX Test Liaison Key) irex [at] nist.gov (irex[at]nist[dot]gov) sub 2048g/5ABA2D90 2008-05-28

IREX II IQCE - Test Plan Development

Procedural updates

January 16, 2010: Organizations wishing to participate in IREX II IQCE should email irex [at] nist.gov (irex[at]nist[dot]gov). Participation is open from January 26 till May 24, 2010. The primary requirement for participation is an ability to implement the IQCE API specification. The latest version is 4.1. A validation package will be released in few days.

Final Draft of CONOPS + API

January 12, 2010: Bug fix in section 15.2. IREX II IQCE concept, evaluation plan and API version 4.4 gcc version 4.1.2-14 for RedHat Linux gcc version 3.4.4 for Windows.

December 24, 2009: The fourth and final draft of the IREX II IQCE concept, evaluation plan and API is posted. Many thanks to those who reviewed and sent us comments. NIST proposed disposition of comments here.

CDF of iris and pupil radii

December 21, 2009: Cumulative distribution functions of (best estimate) of pupil radius and iris radius of IREX I corpra are here.

3rd Draft of CONOPS + API

November 27, 2009: The third draft of the IREX II IQCE concept, evaluation plan and API is posted. Many thanks to those who reviewed and sent us comments. NIST proposed disposition of comments here. Please send us any fixes, questions, or suggestion by December 21, 2009.

Major changes:

Table 4 -IQAAs output format has been changed. The length of quality vector is extended to 64, and new quality metrics (margin, magnification, pupil shpae and interlacehas been added.

Timeline and Clause 12 has been revised. To support research and development, IQCE will embed multiple rounds of testing. Submission period begins in January 2010 and will end in May 2010. See Clause 12.

2nd Draft of CONOPS + API

October 22, 2009: The second draft of the IREX II IQCE concept, evaluation plan and API is now available. Many thanks to those who reviewed and sent us comments. Per comments and requests, the deadline for submission of final SDKs is extended. See page 2 for the update IQCE schedule.

Comments on the 2nd draft are welcome, please send us your comments by November 15.

Important dates:

October 4, 2009: Launch of IREX II IQCE - Relaese of 1st draft of test plan

October 18, 2009: Comments on 1st draft due

October 22, 2009: Release of 2nd draft of test plan

November 15, 2009: 2nd (and final) comment period ends

December 14, 2009: Release of final evaluation plan

January 26, 2010: Submission period opens (rounds of submit -> evaluate -> report)

February 4, 2010: Deadline for submission of signed Annex A Particiaption Application to NIST

May 24, 2010: Submission period ends

Initial CONOPS + API

October 4, 2009: The first draft of the IREX II IQCE concept, evaluation plan and API is now available. Comments are welcome, please send us your comments by October 18. See page 2 for the IQCE (tentative) schedule.

Scope

October 4, 2009: The IQCE activity supports a new, formal, standard addressing iris quality. That standard, ISO/IEC 29794-6 Iris Image Quality, was initiated by the Working Group 3 of the ISO SC 37 committee in July 2009. The standard will define a vector of quality components each of which is some quantitative measure of a subject-specific or image-specific covariate. The current working draft (SC 37 N 3331) defines 21 subject or image covariates and 12 metrics for assessing the utility of an iris image. The 12 quality metrics are mostly quantification of the covariates. IQCE will evaluate these quality metrics influence on iris recognition accuracy. In addition, any measure identified as being influential on accuracy will be considered. The outcome will be a refined list of quality metrics with tolerance bounds for each. Furthermore, IQCE will establish precise methods for measuring each metric. This ensures development of a clear, tested and implementable iris image quality by excluding or avoiding any over-prescriptive and non-testable statements. The overall effect of IQCE and the ISO/IEC 29794-6 standard will be to validate and support camera imaging properties and system design.

Also, IQCE aims to evaluate the effectiveness of image quality assessment algorithms (IQAAs) that produce a scalar overall image quality in predicting the recognition accuracy of particular comparison algorithms (from the supplier of the IQAA), and of other algorithms. Furthermore, per the IREX I result that quality scores are not immediately interoperable, IQCE will establish a score calibration procedure for IQAAs.

This activity encourages participation from the main commercial providers and academic institutions, non-profit research laboratories and consultancies for which prototype implementations exist.

The NIST Iris Exchange Program was initiated at NIST in support of an expanded marketplace of iris-based applications based on standardized interoperable iris imagery. IREX I was primarily conducted in support of the ISO/IEC 19794-6 standard, now under revision. It secondarily supports the recently completed ANSI/NIST ITL 1-2007 Type 17 standard, as derived from the ISO/IEC 19794-6:2005 parent, and future revisions thereof. The Iris Quality Calibration and Evaluation (IQCE) is the second activity under the IREX iris image interoperability umbrella.

NIST intends to publish the results of the IREX II IQCE before the end of 2010.

IREX II Mailing List

October 4, 2009: To be notified of future IREX II announcements please send a request email to elham.tabassi [at] nist.gov (elham.tabassi[at]nist.gov)