Background: Impact Forecasting

Go To: Advancing Competitiveness Homepage

Credit: Pixabay

A Conceptual Framework for Economic Decision Making in Advancing Manufacturing Industry Competitiveness: Impact Forecasting

For more information, see NIST AMS 100-80.

Credit: Pixabay

Typically, every investment (e.g., R&D investments or projects) or purchase involves making a prediction. When companies or governments invest in research, when an individual purchases a car or house, and even when purchasing basic items such as food, there is typically a prediction being made. When the decision maker chooses whether to invest or purchase an item, they are predicting whether the benefits of the purchase/investment will exceed the costs (both financial and non-financial). Because strategic planning requires forecasting, every investment decision ultimately involves a prediction about future costs and benefits. To maximize impact per dollar in change agent manufacturing R&D, decision makers will likely need to make formalized predictions. There are a number of approaches to making predictions, which are each associated with different levels of accuracy and different levels of effort/cost to develop. 

Some prediction methods are more accurate than others, even to the point of outperforming professional intelligence analysts with access to classified information (Tetlock and Gardner 2015). Unfortunately, high performing methods are frequently underutilized due to bias toward one’s own individual insight and other such biases. Thus, creating predictions with high levels of accuracy not only presents a challenge in developing rigorous data and methods for both decision science experts and non-specialists, but it can also involve overcoming psychological obstacles as well. Even when better methods exist, humans often default to intuition, which reduces forecast accuracy. There can be a temptation or even a tendency for decision makers to use their instinct or intuition to determine their investments or projects. To some extent this should be resisted, as humans are vulnerable to being heavily influenced by immaterial feelings and emotions (Lewis 2004; Kahneman 2011; Ariely 2008). Despite one’s best efforts, it has been shown that humans are not able to fully separate emotions from logical decision making (Kahneman 2011; Ariely 2008). 

When forecasting, it is important not to overweight particular types of information (Tetlock and Gardner 2015). Sometimes too much can be read into a piece of evidence or relevant information. For instance, a piece of evidence may only suggest something is possible rather than it being probable. In manufacturing competitiveness, artificial intelligence (AI) provides a useful example. AI is changing a great deal of the industry landscape, making it seem as though AI project impact has limitless boundaries; however, manufacturing competitiveness projects are often more narrowly bounded. For instance, the application of AI in quality control imaging operates within a constrained evaluation boundary. There are only so many detectable defects and there is only a finite range of achievable cost reductions. As a result, the potential gains are valuable but are not unlimited. This example illustrates that change-agent applied R&D is primarily a process of engineering and economic optimization within bounded operational systems, rather than a speculative portfolio-selection problem.

Another example of overweighting is the tendency to overweight the importance of a solution being the most technologically advanced. Manufacturing industry change agents can fall into an assumption that technological sophistication equates to economic impact potential. The Palm Pilot, which held 70 % of the personal digital assistant (PDA) market in 1999 (Wiggins 2004), represents a historical example of the inadequacies of this assumption. At the time, PDA manufacturers were often focused on advanced complex technology; however, the Palm Pilot succeeded not by delivering the most advanced technology, but by effectively solving common user problems. People had important personal information scattered across paper planners, sticky notes, Rolodexes, and desktop computers — but no easy way to carry, update, and synchronize it all. The Palm Pilot gave people a pocket-sized place to synchronize and backup this information in a simple and easy way. Its success demonstrates that economic impact is often driven more by usability, constraint alignment, and effective problem solving than by technological sophistication alone.

When intuition is broad and fuzzy it is more vulnerable to being based on unsound reasoning. If it is necessary to use intuition, it should apply to assessing specific factors of an investment and be grounded on informed judgement. For instance, if a particular cost of an investment is unknown (e.g., cost of energy), one might use individual insight to estimate the value of this individual cost. The unknown cost should not lead to or justify using intuition to evaluate the full merits of the project because one cost is unknown. When conducting an investment analysis of potential industry R&D projects, there are frequently many unknown values. These values need to be forecasted or predicted for an analysis to be completed. In the literature on forecasting, including that of Tetlock and Gardner (2015), some common themes arise:

• Problem Clarification: It is important to ensure that relevant questions are being asked.
• Extraneous Information and Overweighting: When forecasting, it is important not to overweight particular types of information
• Overconfidence: A common challenge that is faced when forecasting is overestimating one’s own knowledge and abilities.
• Manageable Sub-Problems: The values to be forecasted can be broken up into manageable sub-problems to obtain a more accurate estimate.
• Expert Opinion: Expert opinion often provides increased accuracy in a forecast when the subject is within the expert’s domain of expertise; thus, it is important to identify the boundaries of one’s knowledge.
• Testable Predictions: Forecast accuracy is improved in the presence of feedback—that is, when predictions can be empirically evaluated and compared against observed outcomes.
• Accuracy in Numbers: Forecast accuracy is increased by including more people’s input into the forecast; therefore, it is beneficial to have a team of people or to survey individuals in order to estimate and answer manageable sub-problems.
• Learn by Doing: Increasing prediction accuracy comes from practitioner experience.

The framework presented here, utilizes many of these themes with particular attention to creating testable predictions, utilizing manageable sub-problems, learn by doing, and problem clarification. 

Framework Components, Logic, and Implementation

Credit: Pixabay The Framework	Credit: Pixabay Framework Logic	Credit: Pixabay Implementation Infrastructure
The Framework's Process	Background: Impact Forecasting	Notes and Cautions
Data Layers and Feed Back Loops	Supporting Logic and Evidence	Standards and Platforms
Metrics and Units of Observation	Collaboration: Economies of Scale and Structured Learning	Modular Design and Incremental Benefits of Adoption

Credit: AMS 100-80

Collaboration is a key component to reducing change agent costs and enabling compound learning. If you are considering adopting this framework, consider reaching out to the author Douglas Thomas, Economist: douglas.thomas [at] nist.gov (douglas[dot]thomas[at]nist[dot]gov)