SHARP - A Benchmark for 3D Interest Point Detection Algorithms

This benchmark aims to provide tools to evaluate 3D Interest Point Detection Algorithms with respect to human generated ground truth.
Please refer to the paper for more information about this benchmark: Helin Dutagaci, Chun Pan Cheung, Afzal Godil: “Evaluation of 3D interest point detection techniques via human-generated ground truth" <filename: article_VC_interest_points.pdf>, The Visual Computer, 2012. [Bib Tex <filename: visual_computer_IP_benchmark.bib> ]

3D Model Data Set and Interest Points Marked by Human Subjects

Using a web-based subjective experiment, human subjects marked 3D interest points on a set of 3D models. The models were organized in two datasets: Dataset A and Dataset B. Dataset A consists of 24 models which were hand-marked by 23 human subjects. Dataset B is larger with 43 models, and it contains all the models in Dataset B. The number of human subjects who marked all the models in this larger set is 16.

Some of the models are standard models that are widely used in 3D shape research; and they have been used as test objects by researchers working on the best view problem. Examples are Armadillo, David’s head, Utah teapot, Bunny, etc. We chose some of the models from The Stanford 3D Scanning Repository and some others from the Watertight Models Track of SHREC 2007 (see the license). The Stanford 3D Scanning Repository and some others from the Watertight Models Track of SHREC 2007 (see the license <filename:  AIM.txt>).

The data set can be downloaded from the link:  <filename:  MODEL_DATASET.zip>. The triangular mesh models are stored in MAT files. Please refer to README <filename:  README.pdf> for details.

The interest points marked by human subjects' can be downloaded from the link: HUMAN SUBJECT’s INTEREST POINTS <filename:  HUMAN_SUBJECTs_INTEREST_POINTS.zip>.

The text files are named with respect to the following template: 
subjectname-modelname_points.txt

An example is jck-ant_points.txt <filename:  jck-ant_point.txt>. Where jck is the ailas for the human subject and camel is the name of the 3D model. Each row in the text file gives x, y, z coordinates of an interest point marked by the subjet on the 3D model.

3D Interest Point Detection Algorithms

We have compared five 3D Interest Point Detection algorithms. The interest points detected on the 3D models of the dataset can be downloaded from the link next to the corresponding algorithm. Please refer to README for details.

• Mesh saliency [Lee et al. 2005] : Interest points by mesh saliency <filename: Mesh_saliency.zip >
• Salient points [Castellani et al. 2008] : Interest points by salient points <filename:  Salient_points.zip>
• 3D-Harris [Sipiran and Bustos, 2010] : Interest points by 3D-Harris <filename: 3D-Harris.zip >
• 3D-SIFT [Godil and Wagan, 2011] : Interest points by 3D-SIFT  <filename:  3D-SIFT.zip>  (Please note that some models in the dataset are not watertight, hence their volumetric representations could not be generated. Therefore, 3D-SIFT algorithm wasn’t able to detect interest points for those models.)
• Scale-dependent corners [Novatnack and Nishino, 2007] : Interest points by SD corners  <filname:  SD-corners.zip>
• HKS-based interest points [Sun et al. 2009] : Interest points by HKS method <filename: HKS.zip>

The interest points for all of the six algorithms can also be downloaded as a single zip file: ALGORITHMs_INTEREST_POINTS <filename:  ALGORITHMs_INTEREST_POINTS.zip>

Evaluation Codes

There are two steps for evaluating an “interest point detection algoritm” with respect to the interest points marked by human subjects:

• Building the ground truth
• Evaluating with respect to the built ground truth using three evaluation measures, namely False Positive and False Negative Errors, and Weighted Miss Error.

The MATLAB code can be downloaded from the following link: MATLAB code <filename: CODES.zip >. The details about how to use the code are present in the pdf document : README <filename:  README.pdf>.

Downloads
 

• README : Document explaining the formats of data files and how to use the evaluation codes.  <filename:  README.pdf>
• DATA SET : 51 3D models in MAT format. The lists of the models for Dataset A and Dataset B are stored in two MAT files; exp_model_list_A <filename:  exp_model_list_A.mat> and exp_model_list_B <filename:  exp_model_list_B.mat>, respectively. <filename:  MODEL_DATASET.zip>
• HUMAN SUBJECT’s INTEREST POINTS <filename: HUMAN_SUBJECTs_INTEREST_POINTS.zip>: Interest points of the 51 models marked by human subjects. The lists of the subjects who marked on Dataset A and Dataset B are stored in two MAT files; subject_list_A <filename:  subject_list_A.mat> and subject_list_B <filename:  subject_list_B.mat>, respectively. 
• ALGORITHMs_INTEREST_POINTS <filename:  ALGORITHMs_INTEREST_POINTS.zip>: Interest points detected by five algorithms.
• MATLAB code <filename:  CODES.zip>: MATLAB code for generating the ground truth and evaluating interest point detection algorithms.
• OUTPUT data <filename:  OUTPUT_DATA.zip>: MAT files generated by the codes 

The entire benchmark, including the output MAT files generated by the codes, can be downloaded as a single archive file: IP_BENCHMARK <filename: IP_BENCHMARK.zip>

Citation

Helin Dutagaci, Chun Pan Cheung, Afzal Godil, “Evaluation of 3D interest point detection techniques via human-generated ground truth” <filename: article_VC_interest_points.pdf>, The Visual Computer, 2012. [Bib Tex <filename:  visual_computer_IP_benchmark.bib>]

References

• Lee, C.H., Varshney, A., Jacobs, D.W.: Mesh saliency. In: ACM SIGGRAPH 2005, pp. 659–666 (2005)
• Castellani, U., Cristani, M., Fantoni, S., Murino, V.: Sparse points matching by combining 3D mesh saliency with statistical descriptors. Comput. Graph. Forum 27(2), 643–652 (2008)
• Sipiran, I., Bustos, B.: A robust 3D interest points detector based on Harris operator. In: Eurographics 2010 Workshop on 3D Object Retrieval (3DOR’10), pp. 7–14 (2010)
• Godil, A., Wagan, A.I.: Salient local 3D features for 3D shape retrieval. In: 3D Image Processing (3DIP) and Applications II, SPIE (2011)
• Novatnack, J., Nishino, K.: Scale-dependent 3D geometric features. In: ICCV, pp. 1–8, (2007)
• Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. In: Eurographics Symposium on Geometry Processing (SGP), pp. 1383–1392 (2009)

Contact Information

For more information and feedback, please contact Helin Dutagaci or Afzal Godil