PerfLoc: Performance Evaluation of Smartphone Indoor Localization Apps

Perfloc navigation

• PerfLoc: Performance Evaluation of Smartphone Indoor Localization Apps
• User Guide
• FAQs
• Data
• Archived Announcements
• Back to Location-Based Services portfolio page

Credit: PSCR LBS Team

Prologue

January 9, 2024: The PerfLoc Prize Competition was developed by NIST during 2015-2017 and ran during 2017-2018. The Competition concluded with a single winner on May 16, 2018. However, NIST believes the data collected for the PerfLoc Competition is still of value to the R&D community, because there is still room to develop better signal processing and data fusion algorithms that would fuse various types of smartphone data collected in this project to develop indoor localization apps with higher localization accuracy. For that reason, NIST continues to make the PerfLoc data available to the R&D community.

One thing has changed compared to when the PerfLoc Competition was running in 2017-2018. It is no longer possible for app developers to upload the location estimates generated by their apps at the PerfLoc website for performance evaluation purposes and to get statistics of localization accuracy. However, the PerfLoc data is still useful, because there is training data with ground-truth location annotation that would be useful to anyone wishing to develop indoor localization apps and getting an idea of the performance of their apps.

Introduction

With billions in everyday use throughout the world, the smartphone is a very important personal communications, data processing, and entertainment tool. It is also true that there exists a strong demand for a personal navigation capability inside buildings and in other indoor environments to complement the availability of GPS/GNSS for outdoor navigation, which is already very popular. It is anticipated that a wide variety of Location Based Services (LBS) will be enabled when effective indoor localization and tracking solutions are developed.

The main goal of the PerfLoc project is to facilitate the development of the best possible smartphone indoor localization apps. We achieve this goal by making the following available to the R&D community:

1. An extensive repository of annotated smartphone sensor and RF signal strength data to enable researchers to develop smartphone indoor localization apps.
2. A web-based performance evaluation tool based on the international standard ISO/IEC 18305 to evaluate the performance of the apps developed based on the PerfLoc data repository.

The data is "annotated" in the sense that all the measurements are time-stamped and, more importantly, the data includes timestamps at which the test subject that collected the data was at one of 900+ surveyed test points deployed in four large buildings. Therefore, if an app developer uploads the location estimate generated by their app for the time instance the test subject was at a particular test point, the location estimate can be compared with the ground truth location of the test point. This makes it possible to evaluate the performance of the app over the web without any need to physically test the app in the four buildings.

Four Android phones were used in our data collection campaign. We collected a total of ~16 hours of data with each phone over 38 Test & Evaluation (T&E) scenarios in four large buildings with a total area of 30,000 square meters based on guidance from ISO/IEC 18305. We collected data from all sensors available in each phone (accelerometer, gyroscope, magnetometer, barometric pressure, step counter, light, temperature, humidity, etc.) and the strength of RF signals received from Wi-Fi access points and cellular base stations. We also collected GPS data whenever available, but it is well known that GPS availability inside buildings is spotty at best. Please consult a paper we published at the 2016 IEEE PIMRC to get more information about the data we collected as well as some tests we ran to validate the data.

It is not easy for any organization to deploy 900+ test points in four large buildings, have the test points professionally surveyed, and spend months to collect a large amount of smartphone data. The PerfLoc data set, which includes both training and test data, facilitates development of smartphone indoor localization apps. It relieves individual organizations from having to collect their own data sets. The PerfLoc data set along with our over-the-web standardized performance evaluation methodology provides a level playing field for development and performance evaluation of smartphone indoor localization apps. It makes it possible to compare the performance of such apps as opposed to the situation prior to PerfLoc where there was no way of comparing indoor localization techniques / apps that had appeared in a large body of scholarly publications. It has also made it possible for NIST to organize a prize competition based on the PerfLoc concept. After the release of the PerfLoc data and the launch of its performance evaluation portal, NIST will allow the R&D community to develop their apps and have them evaluated through the PerfLoc web portal for six months. At the end of the six-month period, NIST will invite the developers of the top three apps to make a live demo of their apps and present their work at an appropriate scientific conference. NIST will award cash prizes to the top three apps at the conference.

This web page and the pages linked to it provide all the information app developers need to use the PerfLoc data and participate in the PerfLoc Prize Competition. This includes an FAQ page, where NIST will publish any questions about PerfLoc that might be of interest to a wider audience and answers to those questions. You may find the answer to your question in the FAQ page.
 

Any questions? Please email PerfLoc [at] nist.gov (Subject: Question) (PerfLoc[at]nist[dot]gov).