Software Performance Measurement

Software Performance Measurement

Website

Overview

The mission of the Software Performance Project is to strengthen the scientific foundations of software performance measurement (ITL Objective A1, Develop the foundations of measurement science for IT).Its goals are:

• Application: Replace unreliable common practices with rigorously-studied methods grounded in design of experiments;
• Research: Solve measurement challenges created by the evolution and increased complexity of commodity hardware;
• Transfer: Feed improved methods back into the community of practice.

Industry Need Addressed

Generally, any cost vs. benefits decision on how to construct, compile, deploy, or run software requires the ability to measure the performance effects well enough.The problem is that the practices, tools, and methods commonly used to measure those effects often yield results that are plausible but wrong.The underlying issues include small samples, faults and limitations in measuring instruments and statistical software, statistical assumptions being violated by the complex behaviors of modern hardware and operating systems, and analytical models of uncertainty that do not describe the actual variability of results.

A motivating example is measuring the performance cost of defensive code.Defensive code can mitigate risks that faulty software poses to economic and national security, but it is often omitted because of its perceived performance cost.When measured carefully, these costs may prove to be negligible, as they have in most cases covered in experiments done at NIST thus far.

NIST Approach

NIST overcame the obstacles to its own experiments through a combination of

• More advanced design of experiments, exploratory data analysis, and statistical methods
• Validation of instruments, models, and statistical methods
• Correction or avoidance of faults in available instruments and statistical software

Impact

Our publications and software contributions enable others to apply the methods and tools that we used to their own software performance measurement needs, enabling more reliable measurements and fewer wrong decisions.