The Federal Chief Information Officers Council has run a
pilot program for a commercial performance assessment method,
“applied information economics” (AIE), designed to measure
seemingly subjective outcomes such as “improved decision
making” in order to see if it could effectively measure
IT contributions in federal agencies, said a CIO report.
It said the Department of Veterans Affairs volunteered for
the pilot with its information security program, that it
priced out at $114 million before providing six senior IT
security specialists and a team leader. Through the AIE
pilot VA figured it could prevent $20 million in costs due
to security-related fraud and productivity losses.
AIE “quantifies costs, benefits, and risks in terms of
dollars and probabilities of occurrence. AIE determines
which costs and benefits to measure based upon whether
additional information from measurements will improve a
specific investment decision, estimates the costs of
measurement and the degree of accuracy needed for each
measure, and defines measurement methods,” said the
report. With AIE, performance measures are used to both
guide investment and measure effectiveness.
VA’s information security program was designed to cut
back on viruses, unauthorized intrusions and environmental
events leading to four types of losses — fraud,
productivity, mission setbacks and legal liability. So,
in a performance measurement model, the outputs of the
ISP were a reduction in frequency and severity of IT
security-related incidents and related losses, according
to the report, which was authored by Patrick Plunkett,
co-chairman of the Community of Practice for IT
Performance Management.
The AIE contractor decided the goal was to figure out
which investments resulted in the greatest reduction in
losses.
That could be measured, according to the contractor, by
reducing uncertainty through observation. The idea is
that you don’t have to exactly measure quantities to use
them as a basis for decision-making. Often, determining
whether a quantity is greater or lesser than a certain
cut-off point is enough to decide if an investment
improves decision-making, for example. AIE quantifies the
variables that go into such measurements, so that the
numbers are there to back up decisions and make them.
The contractor’s AIE rationale was as follows: if something
is better, then it is different, and that difference is
measurable. AIE is also based on the assumption that if
something is defined well enough, it is measurable, said
the report.
It said AIE is strong as an investment analysis that
quantifies costs, benefits, risks and measurement methods,
and selects them according to cost and accuracy. However,
CIO cautioned that such a performance measurement tends
to be complicated and requires an analytical background.
