The quality and effectiveness of performance measures at all levels depend on both the quality of the measurement system and the use to which it is put. The system must be congruent with the business strategy and the organization that it measures. Even the best intentioned management cannot make good strategic decisions based on a performance measurement system that is incomplete or measuring the wrong things. Also, useful performance measurements improperly applied can also lead to incorrect decisions and actions.
Characteristics of Measurement
All effective measurement systems have a multitude of characteristics. These characteristics apply in varying degree to each measurement, but must be taken into account if a measurement system is to have maximum value. The characteristics are:
- Assumptions–All measurements contain assumptions about the measurement, its purpose and its relationship to the variable being measured. These assumptions can be explicit or implicit, but they always exist. For example, the implicit assumptions of a machine utilization measurement are: 1) all production has equal value and 2) anything produced will have greater value if produced now than if produces later. Once these assumptions become explicit, it becomes clear that they are true only under very specific circumstances, if at all.
- Precision vs. Accuracy–Calculating a measurement to five decimal places is precision. Accuracy is the difference between observed and actual value in the case of existing discrete quantities (e.g. inventory quantities) or the amount of standard deviation of a probability distribution for forecasts, standards or goals (e.g. budgets, performance standards, etc.). Precision and accuracy are not substitutable.
- Congruence—Since most measurements are surrogates for the critical success factor being measured (e.g. on time delivery as a measure of customer service), it is important that the measure varies in relative direction and magnitude with the critical success factor. Since this is difficult to determine (after all, the critical success factor cannot be measured directly), several surrogate measures can be used together for validation.
- Static vs. Vector Measures–Static measures are a measure of the position of a variable at an instant in time, while vector measures indicate the velocity and direction in which a variable is moving. Traditional performance measurement systems emphasize static measures because: 1) they are easier, both to understand and to measure, 2) they are more easily quantified and compared, 3) less information is required. Conversely, vector measures require: 1) a baseline from which to measure, 2) a goal toward which the variable should be moving and 3) a time series of measurement in order to measure direction and velocity. The minimum number of measurements is two if the variable is linear and more if the variable is non linear, probabilistic or the velocity is undergoing acceleration or deceleration.
- In a continuous change environment, vector measures take on increasing importance since the focus is no longer on where the organization is (static), but the direction in which organizational performance is heading (vector).
- Soft vs. Hard measures–Hard measurements are those that can easily be quantified. Soft measures are those that can be measured only in relative terms. Soft measures fall into two categories: 1) relative and 2) statistical. Relative measures are those that can be bracketed between two more concrete measures. For example, a business can know (e.g. by survey) that its customer service has improved over a baseline, such as last year, but that it has not yet attained the goal. Assigning a quantifiable value may be possible, but only very indirectly (e.g. number of customer complaints). However the business can be fairly certain that customer service is improving via the vector measurement.
- Statistical measures and hypothesis testing can determine whether or not a particular value or set of sample values of a variable are within the same probability distribution as an expected value.
- Relative and statistical measures can be combined in powerful ways. For example, a series of inventory cycle counts can be taken and compared to a similar series from the previous year. If it can be shown that these two samples came from different probability distributions, it can be inferred that inventory accuracy is improving (or deteriorating) relative to a previously measured baseline and a pre-established goal.
- Results measures vs. Behavior measures—Most organizations today are knowledge based and most people are hired for what they know, not what they do. To hire someone for what they know and then measure how they do it is, at best, meaningless and, at worst, destructive. The only valid and useful measures for most jobs and activities today are results measures. Knowing that people are doing things right is not nearly as important as knowing that they are doing the right thing. There continue to be, of course, important and required expected behaviors (e.g. courtesy and respect). However, success in maintaining these within the organization culture, while reflected in results measures, are not measured by them.
- Intended vs. Unintended Consequences–Peter Drucker (1973) has indicated that performance measurement in a social system can be neither objective nor neutral. A primary purpose of performance measurement is to get people and organizations to improve in the direction of a certain standard or goal. If the performance measurement is an indirect measure of the goal, as it often must be, treating the measure a hard number often makes maximizing the measure a substitute for the goal itself. This can lead to the unintended consequence of meeting the measure while ignoring the goal.
- Examples of unintended consequences include: 1) end of the month push to meet sales goals, 2) cutting activities vital in the long term to make short term profit goals or meet budgets, 3) setting goals that are easily achievable (…and maximize the bonus) and 4) focusing on hard number cost goals while neglecting the softer measures of quality and service goals.
- Unintended consequences are more likely to occur when measures are: 1) less direct, 2) static, rather than vector, 3) singular rather than combinations of measures, 4) statistical expected values treated as hard number measures and 5) used for command/control of individuals or groups rather than information which can be used to adjust the process.
Performance measurement systems that have certain characteristics run the risk of not measuring anything meaningful and leading to unintended consequences that can interfere with good performance. Most at risk are those performance measurement systems that focus on: 1) precise, hard numbers that are used to measure specific performance to specific indirect parameters, rather than: 2) using vector, relative and statistical measurements combined with management judgment.
In short, type 1) above is the very performance measurement system many companies have carried over from their hierarchical command/control days and are now trying to use to coax quality, participative, continuous improvement from their organization. It is not hard to understand why many organizations meet what seems like intentional resistance to organizational change and continuous improvement initiatives, both internally and within the supply chain.
Performance Measurement and the Business Process
As mentioned above, performance measurement is still a stepchild of financial measurement in many organizations. Metrics are added haphazardly as needed to clarify or control without regard for integration with organization goals and objectives. Many organizations still believe that if each individual and function is measured to some quantifiable standard, the sum of the results will be organizational effectiveness.
Performance measurement system characteristics were discussed above. Here are some points of integration:
- Performance measurement must be an integral part of corporate strategy.
- Each measurement should be traceably shown to support overall corporate purpose.
- Measurements and methodologies must be aligned with corporate cultural values.
- Vector measures will predominate in a continuous improvement strategy.
- There must be a clear understanding of the difference between performance measures that are deterministic (e.g. matching physical inventory counts to “book” balance) and those that are statistical in nature (e.g. standards, forecasts, budgets).
- The system must focus on measurement as information, not measurement as control. The system should measure results, not behavior.
- Measurement systems must leave room for management judgment.
- Measurement systems must be constantly re-evaluated:
- When strategy or goals change.
- When systems or processes change.
- When a measurements becomes dysfunctional (i.e. exhibits unintended consequences).
- Variables measured, measurement methodology and measurement goals must all be reviewed.
Behavior Space Measurement
How does a measurement system design fulfill all these criteria? An important attempt has been made by Kaplan and Norton (1996) in the Balanced Scorecard approach. It is a comprehensive, systematic methodology that measures groups and individuals on a variety of performance metrics, usually tied to corporate goals. However, Balanced Scorecard uses a weighted average method to reduce a variety of measures to a performance index. This leads groups and employees to “perform to maximize the index”, often with little regard for the desired performance. This occurs because the measurable factors that make up the index are not substitutable for one another.
A more effective method is to define a “Behavior Space” bounded by several measurement dimensions that define individual, department and organizational performance. Although this is somewhat more complex than the index, it is well within the ability of most systems to collect, analyze and report the information involved. The increase in usable information, both to the individual and to the organization more than offsets any additional cost and effort required.
The example shown below reflects a performance measurement framework for only two levels in the organization–the Operations Manager or facility level (see Table One) and the Supervisory or department level (see Table Two). A comprehensive system would encompass all levels of the organization, from business planning to shop floor control. The model contains six attributes to be measured:
- Quality — Products and Processes.
- Service–Both internal and external customer.
- Cost — Products and Processes.
- Velocity–Cycle time of Products and Processes.
- Knowledge–Organizational resources.
- Investment Effectiveness.
Although other organizations might have additional significant attributes to their business processes, these are probably basic to all. Also, the critical success factors, goals and metrics selected are representational and not intended to be appropriate for all situations.
In using the framework to measure performance, both static and vector characteristics of all measurement are tracked and compared to both Short Term and Long Term goals. Use of the system is primarily informational rather than control oriented or punitive. As such it is used to identify 1) problem areas such as lack of resources and/or training and 2) process improvement opportunities for continuous improvement teams. The system must be constantly re-evaluated by management to insure that it 1) still represents and supports the strategy of the organization, 2) provides correct and adequate information to manage and improve the business and 3) is not contributing, directly or indirectly, to the creation of unintended consequences or waste.
Reporting Behavior Space Performance
Rather than reducing the performance report to an index which requires judgmental weighting and makes the measurement even more indirect, a graphical representation provides a more satisfying and accurate method for the following reasons:
- Each performance measure can be clearly seen in comparison to long and short term goals.
- Areas of good and deficient performance are clearly visible.
- Graphical representation clearly shows actual behavior compared to the expected behavior space.
Below, Table Three shows actual performance against the managerial measures shown in Table One followed by a set of graphical representations of the performance against goals. The percent variance is calculated by dividing the value over or under the short term goal by the range between the short term and long term goals. For example, for the Cost measure:
(17% – 10%) / (30% – 10%) = +35% Variance from short term goal.
Figure One shows the table in graphical form. Expected results, the Expected Behavior Space, lies between the inner short term goal boundary and the outer long term goal boundary. The black area represents measures where performance exceeded short term goals and gray areas represent measures where performance fell short. The graph clearly indicates the employee’s performance profile and shows where and how employee resources need to be redirected.
The above representation has several advantages over the weighted average index that is often used in the Balanced Scorecard approach:
- It shows strengths and weaknesses in performance against metrics, which the index doesn’t do.
- It does not rely on subjectively determined “weights” which tend to be arbitrary and easily manipulated.
- It gives a clear indication of performance versus expected results for each critical success factor.
- Individual metrics (e.g. Quality) can be measured vertically from level to level in the organization, an activity that is meaningless for indices.
- It eliminates the “apples and oranges” quality in an index of trying to combine unlike critical success factors. Can you really substitute Quality for Cost?
Figure Two shows the vector component of the Quality metric. It indicates performance against a constantly increasing short term goal and the gap that exists between the current short term goal and the long term goal. The Figure Two graphic would exist for each of the six metrics that defines the Behavior Space.
This is only a sample of the information that can be provided using the Behavior Space Measurement model. Each individual and any team with goals can be measured. Additional formats and methodologies have been developed to incorporate indices, where appropriate, that accurately reflect actual performance results against goals.
Designing, developing and implementing performance measurement as a system, integrated into and changing with the business process is a critical challenge for organizations that hope to move into the next millennium successfully. Traditional measures are inadequate. While the Balanced Scorecard approach is a giant step forward, it too is flawed and must be improved. Multi dimensional Behavior Space performance measurement systems move beyond Balanced Scorecard to a more robust system tied to business strategies and critical success factors that will allow the organization to attain and maintain world class performance.
Brown, Mark Graham, (1989) Keeping Score, Quality Resources, New York.
Case, John, (1995) Open Book Management–The Coming Business Revolution, Harper Business School Press, New York.
Drucker, Peter F., (1973) Management Tasks, Responsibilities, Practices, Harper & Row, New York.
Drucker, Peter F., (1992) Managing for the Future, Truman Talley Books/Dutton, New York.
Drucker, Peter F., The Information Executives Truly Need, Harvard Business Review, January/February 1995.
Hamel, Gary and C.K. Prahalad, (1994) Competing for the Future, Harvard Business School Press, Cambridge, MA.
Kaplan, Robert and David Norton, (1996) The Balanced Scorecard, Harvard Business School Press, Boston.
Maskell, Brian, (1991) Performance Measurement for World Class Manufacturing, Productivity Press, Inc., Cambridge MA.
Maskell, Brian, (1996) Making the Numbers Count, Productivity Press, Portland OR.
Tarr, James D., “Developing Performance Measurement Systems that Support Continuous Improvement Goals”, Hospital Material Management Quarterly, November, 1995.
Tarr, James D., “Performance Measurement for a Continuous Improvement Strategy”, Hospital Material Management Quarterly, November, 1996.
Zuboff, Shoshana, (1988) In the Age of the Smart Machine, Basic Books, Inc., New York.
Categorised in: Organizational Development, Supply Chain Management