|Utility Analysis: An Overview
- Raphael L. Vitalo, Ph.D.
Utility analysis is a quantitative method that estimates the dollar value of
benefits generated by an intervention based on the improvement it produces in
worker productivity. Utility analysis provides managers information they can
use to evaluate the financial impact of an intervention, including computing
a return on their investment in implementing it.
The concept of utility was originally introduced by Brogden (1949) and Brogden
and Taylor (1950) and further developed by Cronbach & Gleser (1965). The
concept has been researched and extended by Cascio (1982); Schmidt, Hunter,
and Pearlman (1982); and Reilly and Smither (1983), among others. It was introduced
as a method for evaluating the organizational benefits of using systematic procedures
(e.g., proficiency tests) to improve the selection of personnel but extends
naturally to evaluating any intervention that attempts to improve human performance.
The first assumption of utility analysis is that human performers
generate results that have monetary value to the organizations that employ them.
This assumption is also the basis on which people claim compensation for the
work they do.
The second assumption of utility analysis is that human performers
differ in the degree to which they produce results even when they hold the same
position and operate within like circumstances. Thus, salespersons selling the
same product line at the same store on the same shift will show a variation
in success over time with a few doing extraordinarily well, a few doing unusually
poorly, and most selling around the average amount for all salespersons. This
assumption is broadly supported in common experience and in research. It is,
for example, the basis on which some performers demand and receive premium compensation.
The direct implication of these assumptions is that the level of results produced
by performers in their jobs have different monetary consequences for the organizations
that employ them. Performers are differentially productive and the productivity
of performers tends to be distributed normally (Exhibit 1).
How Utility Analysis Builds on These Assumptions
The approach of utility analysis asserts that the utility of any intervention
can be valued by determining how far up the productivity distribution the intervention
moves the performer. The distance the performer is moved is translated into
a productivity gain and the dollar value of that productivity gain is what is
termed the utility (U$) of the intervention.
What Is Needed to Complete a Utility Analysis
In completing the analysis, the performer needs to generate the following:
- A method for measuring role productivity,
- A way to assign monetary value to role productivity,
- The distribution of productivity among performers of the role,
- The dollar value of a one standard deviation difference in role productivity
- A method to measure the intervention's impact on role productivity.
With these elements of information, the analyst can compute the utility of
the intervention in dollars.
To accomplish the analysis, the analyst must be skilled in the methods of quantitative
analysis in general and utility analysis in specific. This person needs to be
aware of the variety of ways one can measure human productivity, determine its
monetary value, and gauge the affects of interventions on participant performance.
Given that there are a variety of methods for computing utility, the exact
resources needed for the task will depend on the method the analyst selects.
The least set of resources anyone will need are:
- Access to the people who will be using the results of the study to make
- The identity of the intervention whose utility you will measure;
- A subject matter expert who is knowledgeable of the intervention;
- A description of each affected role including its duties, outputs, and
- The compensation scale for each affected role; and
- A subject matter expert who is knowledgeable of the role(s) affected by
Getting Ready for the Analysis
1. Understand the people whose decision-making the study will support.
Tip: You need to meet the people who will use your study's findings
in order to understand what information they are seeking, what decisions they
will use the information to make, and any issues or concerns they may have
about the study. You should also alert them to your ongoing need for their
feedback on the methods you will propose for accomplishing the study. Assure
them that you will guarantee that the methods you propose satisfy the professional
criteria, but that their feedback is needed to ensure that the methods are
also credible in their eyes and the eyes of anyone with whom they will share
2. Learn about the intervention you will assess.
Tip: Identify the intervention whose utility you will measure and
contact the subject matter expert who is knowledgeable about it. Learn about
the intervention's purpose, target population, content, operations, cost,
and any metrics used to measure its implementation and effects. Also, uncover
what the thinking is about how the intervention affects the productivity of
the performers it targets. With these facts, you can determine what information
needed for the analysis exists and what information you will need to develop.
3. Learn about the role(s) whose productivity is affected by the intervention.
Tip: Obtain a description of each affected role. Contact the
subject matter expert who is knowledgeable of each role. Learn each role's
purpose, duties, outputs, and success criteria. You also need to understand
how the role is valued from a compensation perspective. For example, is compensation
linked to output or is it paid as a salary? You will want to understand, as
well, how the company values the output of each job. If the output is sold,
is it valued by cost or price? And you need to uncover how much responsibility
each role has for the outputs its performers produce. Finally, for each job
that is salaried, obtain its compensation scale and the average salary paid
to its incumbents. If salaries are not normally distributed, you may need
to obtain either the median or modal salary instead of the mean.
4. Determine how to measure the productivity of the performers of each role.
Tip: You will need to develop a productivity measure and a method
for determining the status of each role incumbent on the measure. You will
need to use your understanding of each affected role and the assistance of
its subject matter expert. The subject matter expert will have to approve
the method of measurement you devise, otherwise your approach to measuring
productivity will not have credibility in the workplace.
In devising the productivity measure, it is preferable to base the measure
on production of correct outputsfor example, the total amount of sales
generated less returns or the number of welds made per unit of time less the
number of welds that fail inspection. Outputs are the tangible contributions
a role makes to an enterprise and measuring the quantity, quality, and complexity
of outputs generated by performers is usually a measure of productivity that
is readily accepted.
Sometimes, however, a workplace will not accept a measure of productivity
that is tied to outputs. In these situations, you still need a way to measure
how well the role is performed. Sometimes supervisor ratings of successful
performance are used or multirater approaches that use rating of supervisors,
peers, and subordinates (when appropriate).
If the workplace will not agree that different performers achieve different
levels of success or that the level of a performer's success in performing
the role can be measured, then the utility analysis cannot be done.
Once you have devised a measure of productivity, plan how you will gather
information about the status of role incumbents on the measure. Your method
must be feasible meaning that its cost must be reasonable, its result credible,
and its burden on participants acceptable.
5. Determine how to value role productivity in dollars.
Tip: The method you choose will be determined by how you measure
productivity. If you use a method that calibrates outputs produced, then you
will assign monetary value based on the dollar value of the outputs. If the
job produces an interim output, some component of a larger final product,
then determine the component's contribution to the total product and determine
the value of the role's output by adjusting the value of the final output.
Material outputs can be valued based on cost or sales price. Service outputs
that are used in-house (e.g., a marketing plan, a processed personnel action)
can be valued using market pricingthat is, what it would cost to purchase
the service from external sources.
If you are not using a measure of productivity that is tied to output, then
you can use the typical salary paid for the job (i.e., mean, median, or mode).
Salary is acknowledged as reflecting the value a role contributes to a company.
6. Decide how to measure the affect of the intervention on role productivity.
Tip: Basically, you need to find a mathematical bridge that
relates participation in the intervention and change in role productivity.
There are very many ways to accomplish this. One way is to use a control group
comparison. Here, you identify two sets of people who are comparable in all
important ways except that one set went through the intervention and the other
did not. You compare the differences in productivity of these two sets of
people. If the intervention was effective, the people who went through it
will have higher productivity scores and the difference between the groups
will represent the intervention's impact on productivity. Another way is to
use correlational methods to associate some indicator of participation or
benefit from the intervention with scores on role productivity. Be sure that
the information with which you are working satisfies the requirements of the
statistical method you use and that your approach makes sense to the people
who will use the results of the analysis. Your solution needs to satisfy both
professional standards and credibility to provide benefit.
7. Create a plan for the utility analysis.
Tip: Be sure your plan documents how you will produce each of
the information elements needed to accomplish the utility analysis. Include
in it any decision rules you will apply in making judgments. For example,
if you are also computing a return on investment ratio, what rule will you
apply to decide if the ratio is positive? Will 1.0 be sufficient? Will the
ratio need to be 2.0 or higher? In a professionally conducted analysis, all
decision rules must be documented prior to the study.
Doing the Analysis
1. Determine the productivity of performers.
Tip. Execute your plan for measuring the productivity of current
2. Determine the dollar value of a one standard deviation difference in role
Tip. Distribute the productivity scores you gather. Confirm
the distribution is essentially normal and compute its mean and standard deviation.
If the distribution is not normal, use a transformation method (e.g., z-transformation)
to normalize it. Apply your method for valuing role productivity. Derive the
dollar value of productivity achieved by average performers and the dollar
value of a one standard deviation difference in productivity (SD$).
3. Compute the effects on performer productivity associated with the performer's
participation in the intervention being evaluated.
Tip. Apply your method for measuring the affect of the intervention
on productivity. Determine how many standard deviations of change in worker
productivity the intervention produces (SD).
4. Compute the dollar value of productivity improvements generated by the
Tip. The dollar value of productivity improvements generated
by the intervention is the intervention's utility (U$). To compute utility,
multiply the number of standard deviations of change the intervention produces
in worker productivity (SD) and the dollar value of a one standard deviation
difference in productivity (SD$) (SD x SD$ = U$).
Following Up the Analysis
1. Add context to the findings.
Tip. Statistical methods are systematic and, when properly applied,
produce reliable results. They do not, however, guarantee meaningful results.
Sometimes quantitative relationships are found for which there is no reasonable
explanation. One reason this occurs is that you rarely can control all the
possible factors that may influence whether a found relationship is valid.
Sometimes what your data says is causing the effect is not, rather some other
intervening factor which you have not identified or controlled may create
the appearance of a relationship that does not actually exist. One way to
eliminate this possibility is through the use of controls during the process
of relating participation in the intervention and changes in productivity.
Another way is to explore whether the content of the intervention or the experiences
of its participants suggest a meaningful mechanism for the effects your analysis
detects. Study the intervention to see if any aspects of it suggest such a
mechanism. Gather or review existing accounts of the actual experience of
people participating in the intervention. Their experiences will help provide
qualitative information that may either suggest a lack of reasonableness to
the quantitative findings or provide a bases for making the findings understandable.
In analyzing the intervention itself, we look for content or activities that
previous research supports as reliably affecting human performance. With respect
to gathering participant experiences, our typical approach is to use focus
groups in which we gather the participants' observations of what happened,
whether in their viewpoint it affected their performance, and how it affected
2. Report the results of the analysis.
Tip. Be sure to describe the methods you used to generate your
findings and the rationale for each. After you draft your report, obtain feedback
on it from the subject matter experts of the role(s) affected by the intervention
and the intervention itself. This feedback may identify issues you need to
address and assist you in improving the communication of your results.
Learning More About Utility
Read the example of a utility analysis we completed to evaluate the monetary
return of a training intervention. Use this example to clarify how the steps
are performed. Also, study each of the references listed at the end of this
article. They provide a sound introduction to the various methods of utility
analysis one may employ.
Example of a Utility Analysis
We were asked to evaluate a contracts management course offered on a fee-for-service
basis by the human resource department of a government agency. The course trained
contract officer's technical representatives (COTRs) in how to specify requirements,
build a request for quote or proposals, evaluate bidders, select and contract
with the best supplier, manage contract performance, and ensure the delivery
of the needed products or services on time, at cost, and to specifications.
One of the questions being asked was whether the course returned a monetary
value greater than its cost. We proposed a utility analysis as the means to
assess the monetary benefits produced by the course and a return on investment
analysis to determine the ratio of benefits received to the cost expended. Prior
to these evaluations, we determined that the content offered by the course was
relevant to the COTR role and that the course participants did demonstrate increased
proficiency in their performance as a result of completing the course.
Measuring Productivity and Determining Its
With the role identified, we studied the job it accomplished by reviewing its
tasks, outputs, and performance expectations. No measure of productivity existedyet
the means for deriving a measure appeared evident. First, the COTR role had
a defined output and criterion for judging success. COTRs were responsible for
successfully satisfying a product or service need within their agency through
contracting. Successful satisfaction of the need meant the timely delivery of
products and services that met technical specifications and the accomplishment
of these ends at the cost specified. Second, there was a monetary value associated
with the output. The dollar value of every contract a COTR managed was systematically
determined. Third, there was a logical way to relate the monetary value of the
role's output and its success criterion. A COTR realized the value of a contract
to the degree that the contract was concluded on time, at cost, and to specifications.
Conversely, to the degree it was not concluded on time, at cost, and to specifications,
monetary value was lost.
While the basic logic was sound, conversations with incumbents and supervisors
quickly revealed that while the COTR was responsible for the contract,
sometimes he or she was not free to exercise complete control over its contents
or the decision-making associated with it. Therefore, some amount of the value
of the contract was outside the control of the COTR and its realization or loss
should not be credited to the performer. We also learned that contracts sometimes
yielded benefits greater than their face value and that this could be the result
of the COTR's forward thinking, selection of the means for accomplishing the
contract, speed of execution, and other factors.
To measure role productivity, we developed and tested the COTR Productivity
Rating Form. This form measured the degree to which each COTR brings in his
or her assigned contracts at cost, on time, and to specification. It also calibrated
the importance of each of these factors for the contracts managed and the typical
degree of control over each contract the COTR has. The different component measures
were converted into an overall productivity score. This score was a percentage
that represented the degree to which each COTR realized the value of the contracts
he or she manages on a yearly basis.
The COTR Productivity Rating Form was sent to the 266 supervisors of COTRs
randomly selected so that the ratings would reflect the status of COTRs in the
general population. One hundred and thirty (130) responses were received (48.9%
response rate). The response level provided estimates of productivity that were
accurate to +/- 5% at a 95% level of confidence.
Establishing the Dollar Value of Productivity
We completed three steps to determine the dollar value of improved productivity.
First, we distributed the productivity scores achieved by COTRs to determine
their average level of productivity and the productivity levels of performers
at the 15th and 85th percentiles. Exhibit 2 depicts the distribution of COTR
productivity. The average performer is 81.65% successful in extracting the controllable
value from the contracts he or she manages. In contrast, the exemplary COTR
realized 94.04% of the value from the contracts he or she manages and the poor
performing COTR extracted only 69.26%.
To calibrate the value of productivity in dollars, the study used the median
face value of contracts managed by COTRs during one year as modified by the
control the COTR has over the outcome of the contracts. The degree to which
a COTR brings in his or her assigned acquisitions at cost, on time, and to specifications
determines how much of the controllable dollar value of those contracts is realized.
The median face value of contracts fulfilled per year by COTRs was $500,000.
Corrected for the degree of control COTRs have over outcomes, as perceived by
their supervisors, the median potential single year benefit a 100% productive
COTR produces is $397,525. By multiplying the average actual productivity of
COTRs (81.65%) against the controllable dollar value of the contracts a COTR
manages on a yearly basis ($397,525), the study estimated the dollar benefits
generated by the average performing COTR at $324,583.13. Poor performing COTRs
that is, performers achieving at or below the 15th percentile of all COTRsgenerated
only $274,344.10 of value each year. Exemplary performing COTRs, defined as
incumbents whose productivity was at or above the 85th percentile of all COTRs,
generated $373,895.44 of value.
Establishing the Dollar Value of Productivity
To determine the monetary value of improvement in productivity, the study computed
the dollar value of one standard deviation in change (SD$) in role productivity.
The SD$ for the current distribution of performers is $49,239.04. This means
that if some intervention advanced the productivity of a COTR by one standard
deviation, that COTR would generate $49,239.04 in additional benefits to the
agency each year.
Measuring the Course's Affect on COTR Productivity
The correlation between COTR's job proficiency and productivity ratings served
as the mathematical bridge for estimating the course's impact on performer productivity.
The elements required to use this bridge were the amount of proficiency change
produced by the course, the regression coefficient (beta) relating job
proficiency scores to productivity ratings, and the standard deviation of productivity
scores. Applying these elements, the course advances COTRs upward in productivity
by .1547 standard deviations (Exhibit 3).
3. Measuring the Course's Impact on COTR Productivity
in Proficiency Produced by the Course
in Productivity Produced by Increased Proficiency
Deviation (SD) Difference in Productivity Scores
in Productivity Expressed in Standard Deviation Units
Determining the Course's Utility
As stated above, utility is the dollar value of the increased productivity
of a single COTR that is generated by the course. To determine the utility of
the course, the study translated the distance the course advanced COTRs along
the productivity continuum into dollars. As reported, the course advanced COTRs
.1547 standard deviations up the productivity continuum. We previously determined
that one standard deviation change in productivity has a monetary value of $49,239.04.
Multiplying this amount by the .1547 provides us the course's utility (.1547
x $49,239.04 = $7,617.27). This figure ($7,616.18) is the dollar value of the
improvement in productivity evidenced by each COTR as a result of training (Exhibit
4. Computing the Utility (U$) of Contracts Management Training
in Productivity Expressed in Standard Deviation Units
Value of a 1 Standard Deviation Change (SD$) in Productivity
Value of Productivity Improvement Produced by COTR Course
Assessing Return on Investment
The return on investment was computed using the method of dividing the dollar
value of the productivity benefits generated by the course by the cost of participating
in the course (benefit to cost ratio). In this study a desirable ROI was defined
as any value greater than 1. The study determined the per student cost for
completing the COTR course. It added the fee charged to departments for each
COTR taking the course with the cost of lost opportunity associated with the
COTRs not performing their regular job during the 10-day period of the instruction.
This fee ($700) included all expenses associated with the course. The cost of
lost opportunity was computed by dividing the salary of the typical COTR who
participated in the course (GS-14, Step 1) by the number of hours that define
full time employment in the Government (2,087). This per hour cost is then multiplied
by the 80 hours that the COTR is off the job. The opportunity cost per student
was $2,385.63. The total cost for participating in the course was computed as
$3,005.63 per COTR.
The ROI for the course was 2.53 ($7,616,18/$3,005.63) for one year of COTR
performance following completion of the course. This means that for every dollar
invested in completing the course, the sponsoring department receives $2.53
in benefits the first year. Any reasonable assessment of return should
recognize that the benefits of the course extended forward. Given the general
stability of the content the course teaches, a three year period for return
on investment was considered conservative. Within 3 years, the total productivity
improvement benefit is $22,848.54 and the ROI is 7.60 meaning, for every
dollar spent, $7.60 in agency benefits is generated (Exhibit 4).
How Productivity Was Improved
The completion of the two focus group discussions with COTRs who completed
the contracts management course provided insight into the course's mechanism
of impact. Participants uniformly confirmed their experience of benefit from
the course. They listed 17 ways their performance was improved by what they
learned. One major element they emphasized was that the course provided a cognitive
map of the contracting process that allowed them to see ahead, to plan and prepare,
and feel more confident in the conduct of their role. As well, the course equipped
them to produce the products required by the role and to know how to judge the
adequacy of each product. Also stressed was the learning about the various players
in the contracting process, their responsibilities, the importance of communicating
with them, and the importance of creating a teamed effort. Equally important,
the course participants felt they grasped the principles that ensured the integrity
of the contracting process and that they were able to see how these principles
apply in different contracting situations. Finally, participants also reported
the training coursebook provided with the course served as a continuing learning
resource that they turned to as they encountered new contracting experiences.
References and Suggested Readings
Bernstein, Allen L. (1966) A handbook of statistical solutions for the behavioral
sciences. New York: Holt, Rinehart and Winston.
Brogden, H.E. & Taylor, E.K. (1950) The dollar criterion: applying cost
accounting concepts to criterion selection. Personnel Psychology, 3,
Burke, Michael J. & Frederick, James T. (1986) A comparison of economic
utility estimates for alternatives SDy estimation procedures. Journal
of Applied Psychology, 71, 334-339.
Byron, James & Vitalo, Raphael L. (1991) Quality improvement through exemplar-based
productivity analysis. American Productivity & Quality Center, Brief #82,
Cascio,W. (1982) Applied Psychology in personnel management. Reston,
VA: Reston Publishing Company (see Chapter 7, Utility: the concept and its measurement).
Cronbach, L.J. & Gleser, G.C. Psychological tests and personnel decisions.
(2nd ed.), Urbana: University of Illinois, 1965.
McDaniel, Michael; Schmidt, Frank L.; & Hunter, John E. (1987) Job experience
as a determinant of job performance. Paper presented at the 95th Annual Convention
of the American Psychological Association, August 1987.
Myers, Jerome L. (1966) Fundamentals of experimental design. Boston:
Allyn and Bacon, Inc.
Reilly, Richard R. & Smither, James W. (1983) An examination of two alternative
techniques to estimate the standard deviation of job performance in dollars.
Journal of Applied Psychology, 70, 651-661.
Rossi, Peter H.; Freeman, Howard E.; & Wright, Sonia R. (1979) Evaluation:
a systematic approach. Beverly Hills: Sage Publications.
Schmidt, F.L.; Hunter, J.E.; & Pearlman, K. (1982) Assessing the economic
impact of personnel programs on workforce productivity. Personnel Psychology,
U.S. Department of Health Education and Welfare (1975) A practical guide
to measuring project impact on student achievement. Washington, DC: U.S.
Government Printing Office (Stock Number 017-080--1400-2).
Published April 2004
Help Us Provide
You Better Content.
Tell us your thoughts about this
Be sure to name the article in your