Software Performance Project

Summary

NOTE:  This work was succeeded by the Software Metrology Project 

Software Metrology:  Helping physical scientists and computer scientists understand one another a little bit better so that metrology (the science of measurement) and science generally (the system of human knowledge) will be more cohesive across disciplines

Description

The mission of the Software Performance project is to strengthen the scientific foundations of software performance measurement ("metrology for IT"). 

Its goals are:

• Application: Replace unreliable common practices with rigorously-studied methods grounded in design of experiments
• Research: Solve measurement challenges created by the evolution and increased complexity of commodity hardware
• Transfer: Feed improved methods back into the community of practice

Major Accomplishments

• qdfca (Quick-Deploy Formal Concept Analysis) is a command-line filter that implements Formal Concept Analysis (FCA). It is small, scriptable, and easy to install, with no external requirements other than the standard Ruby library.
• Test Suite for Validating Application Profiling Tools, Version 1.2
  • Spin: x86_64 Linux, GCC 
  • Spin: x86_64 Windows, Intel Toolset (Composer XE 2013, VTune Amplifier XE 2013) 
  • Spin: Android, Java, Google toolset (Debug class)
    [new version released 7/24/2013 contains patches]
• Test Driver for Android, Version 1.0 (RTD automates the repeated launching of Android test apps for experimental data collection.) 
• R package bootBCa, Version 1.0
  This package provides a function to find confidence intervals using Efron's nonparametric bias-corrected and accelerated (BCa) bootstrap method. It is an enhanced derivative of the function bcanon, forked from bootstrap_2012.04-1. Adaptive determination of the number of bootstrap replications is supported and the amount of memory required is less by a factor of nboot.
  • Package 
  • Documentation (PDF) and HTML
• The Gem for Experimental Computer Science (GECS) is for managing the data resulting from experiments on software and IT systems. It realizes a data model that disambiguates the vocabulary of experimentation and measurement for a computer science audience. It also provides convenience functions to determine confidence intervals, analyze main effects and interactions, and export data in an R-compatible format for further analysis and visualization.

Associated Product(s)

Raw Data and Table Generator for, "A Case Study of Performance Degradation Attributable to Run-Time Bounds Checks on C++ Vector Access"

Raw data and source code distribution for "Estimation of uncertainty in application profiles"

Raw data and source code distribution for "Screening for factors affecting application performance in profiling measurements"

Raw data and source code distribution for "Defensive code's impact on software performance"

Raw data and source code for 2012 kernelspace microbenchmark