To ask a question, please FpVTE [at] nist.gov (subject: FAQ) (email us). All responses will be posted to this list: no individual responses will be sent. The newest Q&As are at the beginning of each section.
Test Purpose
Is FpVTE 2003 testing slap segmentation?No. The slap fingerprints (a.k.a. 4-finger simultaneous impressions) in the datasets have already been segmented using a combination of automated and manual segmentation. Is FpVTE 2003 testing fingerprint scanners?No. FpVTE is evaluating fingerprint identification and verification systems. Is this a source selection for the U.S. VISIT program?No. FpVTE is being conducted to assess the capability of currently available fingerprint systems for a range of large-scale and small-scale real world applications. NIST's role under the USA PATRIOT Act is to certify those biometric technologies that may be used in U.S. VISIT (U.S. Visitor and Immigrant Status Indication Technology, formerly known as the US entry-exit system). Any future source selection would obviously review the results from FpVTE 2003, but FpVTE is not itself a source selection. How does FpVTE differ from FVC2002?FpVTE focuses on the operational performance of large-scale identification systems in addition to verification systems. Performance using a variety of U.S. Government operational data will be evaluated. (See http://bias.csr.unibo.it/fvc2002/)
Applying to Participate
If we take the wording in paragraph 5.4 of the "Application to Participate in FpVTE 2003" literally, we would be unable to participate because we need to share the sample datasets and software with our engineering and technical groups who reside in other locations, both within the U.S. and internationally. What is the core intent of this paragraph? Additionally, we would appreciate clarity on corporate sharing of the information provided in the "Participant's Area", especially the FAQ.The intent of paragraph 5.4 is to convey that use of the FpVTE 2003 Sample Datasets and Software (FSDS) is permitted only for the purpose of preparing for and participation in FpVTE 2003. Permission to use the FSDS expires at the conclusion of FpVTE 2003. Other uses of FSDS are specifically prohibited. This means that the FSDS may be shared with those entities within the Participant's organization that are necessary in order to prepare and perform as an FpVTE 2003 participant. The information in the website's "Participant's Area" should be treated similarly. To participate in the test, should I send you the printed and signed form? Is there any other method to register, for example via fax or email?Potential participants must fill out the Application to Participate in FpVTE 2003, print and sign the form, and send to the location designated on the form: FpVTE 2003 Liaison, National Institute of Standards and Technology, Information Access Division (894), 100 Bureau Drive, Stop 8940, Gaithersburg, MD 20899-8940 Please do not fax or email the form. Is there a fee to participate in FpVTE?There is no participation fee. Participants will be required to provide hardware and software necessary for evaluating their system, as well as on-site representation during the evaluation. Can non-US companies participate in FpVTE 2003?Yes. Can we participate anonymously?No.
Test Administration
Are the three minutes an hour for system administration limited to work hours and work days, or 24/7?Participant access is limited to three minutes of supervised and videotaped direct operator access per hour, during normal work hours Monday through Friday. Where will the test be held?FpVTE 2003 evaluations will be conducted at the NIST facilities at Gaithersburg, MD. For information, see NIST Visitor's Information. When will the test be held?FpVTE 2003 evaluations will be conducted no earlier than September 29 and no later than December 31, 2003. Start times for Participants will be staggered so that not all Participants will start on the same day. If we want to have a test only in SST, is it expected that the participant will stay in the testing place for two weeks? Or longer?The SST must be completed within a period of 14 days. The Participant must be present for setup, start of the test, completion of the test, and packing up the system. Participants will be allowed time for setup and packing up. Participants may leave while the test is being conducted. Participants are, however, responsible for insuring their systems complete all tests. FpVTE personnel are not responsible for informing Participants that their system has crashed or ceased to work for any reason. This includes (but is not limited to) power outages. If we become a participant, should somebody in our company go to NIST to have a test? Is it not possible to have a test just by sending our software in executable form? Do we provide our own hardware?Participants must provide hardware and software for the evaluation. Participants are responsible for the setup and takedown of the hardware and software at the evaluation. FpVTE 2003 personnel will oversee the setup and takedown, but are not available to physically or technically support that activity. Participants may not participate in FpVTE 2003 by simply sending software in executable form. Should the participant take the system to the testing place in NIST, including hardware, to have a test?Yes. Is it possible to configure our system with just a notebook computer and our software? Or are only systems with dedicated hardware qualified for the test?The choice of system is entirely up the Participant. Participants may use either standard or custom hardware for the test, subject to space and power restrictions. Such hardware must be sufficient to access and process the FpVTE Evaluation Datasets. The Evaluation Datasets will be provided to Participants on a CD for SST and MST, and on a Universal Serial Bus (USB) Hard Drive for LST. (Participants may request SST or MST Evaluation Datasets on USB instead of CD.) If we are allowed to do two tests (MST and LST), can we perform MST and LST sequentially within the maximum of 35 (14+21) days? We can do MST and LST on the same hardware. If we can do LST after MST by changing software, we can save hardware components.LST Participants are encouraged to take both MST and LST on a single system, one after the other. The MST and LST have separate time limits: time saved on one cannot be used for the other. Do we have to have staff present during the entire multi-week evaluation?No. The FpVTE Evaluation is designed to test systems running continuously (24 hours per day) over the test period with no substantial user/operator intervention. Participants may wish to have staff present to check on system status, or conduct limited system administration tasks, but only very limited contact with systems is permitted during the test. Will FpVTE staff sign our NDA (Non-Disclosure Agreement)?No. FpVTE will evaluate systems as "black boxes" -- we do not want to know proprietary information about the inner workings of the Participants' systems. Please do not send any proprietary information, including source code. FpVTE staff will not disclose information about Participants' system architecture, other than the system descriptions written by the Participants. The responses to the System Throughput Questionnaire will not be disclosed.
Test Data
What's the sampling resolution of the images?All images are 500 pixels per inch, 8-bit grayscale images.What are the extremes of rotation that we can expect in the images? What are the extremes in offset? (Is the core always present in the presented image?) What are the extremes of image sizes?Please see the revised Test Plan, Section 3, Evaluation Data. Do the inked card images contain extraneous text, lines, or other marks?Yes. Images scanned from paper fingerprint cards are only included in LST, not SST or MST.
Test Procedures and Methodology
Will a background database of fingerprints be distributed to participants to pre-process/characterize/load before the test itself?No. All of the fingerprints to be used in the test are Sensitive data and cannot leave the NIST facility. Given the different roles of the MST and LST, how can the MST be a subset of the LST?Participants selected to take the LST are strongly encouraged to take the MST as well; participants who choose to do so can take the two tests one after the other, and take the full time period for each test. (Explanation: The LST was originally designed to include the entire MST as a subtest, but to avoid confusion, we are keeping the tests distinct and recommend that LST participants take both tests.) Are the preliminary throughput requirements in the Test Overview indicative of the test structure?No. The preliminary throughput requirements shown in the original version of the Test Overview were intended to provide a draft, order of magnitude test size, based on then-available information. They were not intended to provide any insight into the test structure. The actual throughput requirements will be based on the Tentative Participants' responses to the System Throughput Questionnaire. Where can I find more detailed information on the test structure?The Test Plan provides additional information on the test structure. The final test structure and size for SST, MST, and LST will be determined by FpVTE 2003 personnel utilizing information in the Tentative Participants' responses to the System Throughput Questionnaire. Do you evaluate based on time to complete as well as performance accuracy?Throughput performance will not be measured. We note time to complete the test as well as failure to complete the test. Participants are encouraged to size their hardware and tune their software to make maximal use of the most accurate algorithms, using the entire time period. Rationale: (1) differences in Participants' hardware make resource comparisons impractical; (2) future operational configurations may improve throughput significantly through the use of more and specialized hardware. FpVTE2003 intends to measure the accuracy of fingerprint binning, screening or filtering systems. Generally speaking, the accuracy of "fingerprint binning, screening or filtering systems" depends on filtering rate. For example, when the filtering rate is 0 % (no substantial filtering), filtering accuracy is 100%. Therefore, it is almost meaningless to measure filtering accuracy without checking filtering rate. Does FpVTE2003 intend to measure trade-off of filtering rate and filtering accuracy on LST?Not in LST. We encourage makers of filtering systems (such as those used as stages in an AFIS) to enter them separately in the MST. In MST, accuracy of a filtering system will be measured by the tradeoff between filter reject rate and filter rate. Why don't you "level the playing field" by making all Participants perform on the same hardware configuration?FpVTE 2003 attempts to evaluate existing matcher systems as black boxes. Many systems are composed of both hardware and software. Requiring some participants to port their software or to exclude special-purpose hardware (e.g. processing boards), would neither "level the playing field" nor measure existing capabilities.
LST-Specific Questions
If a dataset contains cards that have all 10 fingers, can we assume that we can choose how many fingers to be matched and which fingers to be matched?Yes. Can we assume all finger numbers are correct, e.g. no finger sequence errors, in all LST cards?Yes. FpVTE2003 intends to measure only accuracy and not to measure the total processing time (or speed). Then, it is a reasonable strategy for participants to choose "no filtering to avoid filtering errors" and to choose "matching all available fingers (up to 10 images per card) in order to achieve best possible accuracy". However, this method (no-filtering and all-finger matching) is not recommended on real system design which seeks best cost performance. It is expected that most LST participants will bring in a lot of matching engines to do no-filtering search and all-finger matching. Does FpVTE2003 accept "accuracy-oriented" test system although it could be useless to predict actual operational accuracy?The LST is specifically designed to have such substantial throughput requirements that no-filtering searches will not complete the test. It is assumed that most Participants will trade off accuracy and throughput so that the evaluation will be completed in 80-90% of the allotted test period. LST Participants are encouraged to also take the MST as one means of demonstrating no-filtering accuracy. Are we allowed to do the conditional use of secondary matching algorithms based on the first matching results? For 1:Many search, when the candidates of the first match results do not contain any absolute mate (hit), we can automatically activate the secondary matching using more powerful matching algorithm (also more time-consuming) to find mates. This function saves total matching time and it is operationally useful. However, matching scores could fluctuate when data base (background) changes.Yes, conditional use of secondary matching algorithms is allowed. For a single subtest in LST, a Query dataset is searched against a Target dataset. Within the matrix of scores generated in a single subtest, all the scores MUST be comparable: a higher score MUST mean a higher degree of similarity. Scores do NOT have to be comparable between different subtests. For example, different subtests can use different scoring methods: the scores when comparing 2-flats against 2-flats do not have to correspond to the scores comparing 10-rolls against 10-rolls.
Results
When will the results of FpVTE 2003 be made available to the public?No later than six months after the conclusion of the evaluation; the evaluation should conclude in November or December 2003. Will the FpVTE 2003 Evaluation Datasets be made available to the public?No. The FpVTE 2003 Evaluation Datasets are "Privacy Act or FOIA Protected Information" and cannot be released. Attempting to retain the test data after the completion of the Evaluation is a violation of Federal law.FpVTE Links
Evaluation Overview
Evaluation Plan
Home
