Management Advisory Report: Comprehensive Measures for Interim Business Systems Modernization
Status Reporting Are Needed
July 2002
Reference Number: 2002-20-128
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
July
10, 2002
MEMORANDUM FOR
DEPUTY COMMISSIONER FOR MODERNIZATION
&
CHIEF INFORMATION OFFICER
FROM: (for) Pamela J. Gardiner /s/ Daniel R.
Devlin
Deputy Inspector General for
Audit
SUBJECT: Final Management Advisory Report - Comprehensive Measures for
Interim Business Systems Modernization Status Reporting Are Needed (Audit # 200120041)
This
report presents the results of our review and analysis of measures for the
Business Systems Modernization (BSM) program.
The overall objective of this review was to assess the Internal Revenue Service’s (IRS) plans for
monitoring and reporting to the Congress the BSM’s program-level status for
cost, schedule, and quality.
The IRS has recently begun
requesting BSM funding for an entire year.
This is an increase in the funding period from previous requests. Since the time between funding requests has
increased, the IRS has identified the need to produce periodic status reports
on the modernization program for the Congress and other interested
stakeholders.
In summary, the Business
Systems Modernization Office (BSMO) is currently generating monthly internal
reports on the status of modernization projects, and plans to design a
comprehensive periodic report for external decision-makers. BSMO management has acknowledged that a
different set of reports may be needed for external reporting.
To
assist the BSMO in developing an external reporting process, we reviewed “best
practices” literature, and conducted interviews with individuals from private
industry, the IRS, and other government agencies. This report provides details
on our research and suggestions to assist the BSMO in developing a
comprehensive set of interim status reports.
Management’s Response: BSMO management requested an extension to respond to our
draft report from June 28, 2002, to July 5, 2002. As of July 8, 2002,
management had not responded to the draft report.
Copies of this report are also being sent to the IRS
managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you
have questions or Scott E. Wilson, Assistant Inspector General for Audit
(Information Systems Programs) at (202) 622-8510.
Comprehensive Measures for Interim Business Systems Modernization Status Reports Should Be Developed
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix V – Comprehensive Reporting Model for Government Agencies
Appendix VI – Examples of Recommended Status Report Measures
The Internal Revenue Service (IRS) is currently modernizing its computer systems and its business processes and practices. This multi-billion dollar effort, known as Business Systems Modernization (BSM), is projected to last up to 15 years.
The Congress controls funding for the BSM effort by allocating funds from an investment account. The IRS provides the Congress justification for a release of funds by submitting BSM expenditure plans. Until recently, the IRS submitted expenditure plans sporadically that resulted in a lengthy approval process and, in some cases, funding gaps.
In a previous audit, we recommended that the IRS consider alternative funding strategies that would benefit the IRS and the Congress. The IRS now submits annual expenditure plans for the upcoming fiscal year, which should stabilize the BSM funding process. Since the time between BSM expenditure plans has increased, the IRS has identified the need to produce periodic status reports on the BSM program for the Congress and other interested stakeholders.
The objective of this audit was to assess the IRS’ plans for monitoring and reporting to the Congress the BSM’s program-level status for cost, schedule, and quality. The IRS requested an audit of how effectively the Business Systems Modernization Office (BSMO) was overseeing the PRIME contractor. This audit is one of several audits designed to provide input on this topic.
To accomplish our objective, we obtained and reviewed
documentation and conducted interviews with individuals from private industry,
the IRS, and other government agencies.
The audit was conducted at the IRS’ National Headquarters Office and the
BSMO facilities in New Carrollton, Maryland, between October 2001 and February
2002 in accordance with Government Auditing Standards.
Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
Since the time between BSM expenditure plans has increased, the IRS has identified the need to produce periodic status reports on the BSM program for the Congress and other interested stakeholders. Without comprehensive status reports to help keep the Congress informed on the progress of the BSM program between expenditure plans, the BSMO could experience lengthy funding approvals and funding gaps.
While the BSMO has not started issuing interim external reports to the Congress, it has been actively working on performance reporting. In 2001, the BSMO was developing its Performance Management Program based on a best practices framework. This program identifies specific project performance measures. The program also includes data collection and verification, analysis, and reporting processes. Per BSMO management, BSM risks and issues are used to determine what measures should be collected. The BSMO has also created a reporting tool called a “dashboard”, which displays project measurements. We reviewed the dashboard and agreed that it would be a good tool to help monitor the modernization contractors’ performance.
The dashboard, however, is not the best vehicle for reporting overall BSM program progress to external stakeholders, such as the Congress. The dashboard is fairly technical and presents information at a point in time, rather than as a trend.
Based on our initial review of the dashboard, we believe a separate reporting vehicle should be developed that focuses on the following topics:
· Quality measures – Measurements could be made to determine the percentage of user requirements that were delivered and the number of deficiencies found in a system once released. See example reports in Appendix VI – Pages 24 and 26.
· Expenditure plan comparisons versus task order comparisons – Measurements could be made against goals released to external stakeholders in the expenditure plans versus measurements made against internal contracts. See example reports in Appendix VI – Pages 23 and 28.
· Trending results versus static results – Measurements could be presented as a trend to give a clearer picture of program progress over time, rather than providing only one point in time. See example reports in Appendix VI – Pages 22 and 23.
Management Actions: During our review, the BSMO was tracking cost and schedule measures. Also, the BSMO was developing non-cost and non-schedule measures. BSMO management stated that three additional measures (product quality, business value, and functional stability) would be tracked by June 2002. All measures will be displayed on the BSMO dashboard, which is currently used by BSMO executives.
The BSMO had not yet initiated interim status reporting to the Congress because it had been developing the foundation for comprehensive performance measures. Since external reports had not been designed, we decided to determine what information would be appropriate for external reporting. To accomplish this, we discussed the type of information that would be helpful with a Congressional staff member, reviewed relevant legal requirements, and obtained information on “best practices” from both private industry and other government agencies.
Congressional Needs
We obtained Congressional input through discussion with a key staff member for the Senate Finance Committee. The major categories of interest for monitoring BSM progress were:
· Cost trends – This would provide perspective on the accuracy of IRS cost estimates over time, and be a factor in considering the appropriate level of future funding for the BSM program.
· Progress of the BSM program – This would provide perspective on how well the IRS is meeting its commitments to deliver benefits to taxpayers. Trends in the IRS’ ability to deliver on its commitments would be a gauge as to the level of congressional oversight necessary for the BSM program.
Relevant
Legal Requirements
We reviewed the Government Performance and Results Act (GPRA) to determine any requirements for performance reporting. The GPRA requires comparing actual program results with established performance goals, and suggests providing multiple iterations of data for trend purposes. The GPRA also suggested that program activities be logically grouped. For example, the BSM program conducts several different types of activities (program level activities, systems development projects, etc.).
Best
Practices
We studied the Software Acquisition Capability Maturity Model (SA-CMM) and interviewed a software acquisition measurements expert. We also reviewed Software Engineering Institute documentation to obtain potential performance measurements. The BSMO has a goal of being rated at Level II on the CMM scale by the end of 2002. Part of the requirements to be rated at a CMM Level II is that organizations have processes for overseeing contracts and documented procedures for internal operations. In addition, an organization should maintain a corrective action system for issues discovered as a result of contract oversight activities. Periodic status reports could help the IRS meet these CMM requirements.
We also reviewed various articles on measuring performance in the Government and in the private sector. One reporting method is known as the Balanced Scorecard, which we believe could be used to ensure that all facets of performance measures are included in the scope of external reports.
Finally, we conducted interviews with nine government agencies that are or have been involved in a major systems modernization effort. We requested information on the type of measures collected and the type of reports produced. See the table in Appendix IV for our results. See Appendix V for the most comprehensive reporting model we obtained.
Based on our analysis and knowledge of the BSM program, we captured potential measures and reports for each Balanced Scorecard component. We have included examples of these measures and reports in Appendix VI. While the information presented in Appendix VI is quite voluminous, the purpose for including each chart/graph is merely to provide examples of measurements and measurement formats. We favor the model outlined in Appendix V, but believe that selected aspects of the other examples can be combined with this model to create a concise, quality report.
In conclusion, the IRS is in the early phases of a potential 15-year effort to modernize its systems. Initial performance analysis efforts focused mainly on cost and schedule measures. The National Partnership for Reinventing Government stated that initial efforts to measure performance usually focus “…on what is done, that is primarily output, activity, or work related measures. The next stage moves toward more outcome-oriented measures, recognizing the need to measure impact or results. Most Federal agencies are at this stage now as they struggle to implement the Government Performance and Results Act and achieve their mission or business results. Beyond this stage, comes the recognition of the importance of the organization’s learning and growth and customer perspectives. High performing results-based management organizations in the private sector have been focusing not just on financial, but also on internal business, customer, and employee perspectives.”
To assist the BSMO in progressing to the next stage of measurement for the BSM program, we suggest that BSMO management create comprehensive, yet concise, interim status reports that provide value for internal and external stakeholders by taking the following items into consideration. It may be practical to begin reporting using only those measures that are currently available, and improving reporting as time progresses.
· Project data needed for status reporting should be collected from all modernization contractors.
· All components or dimensions of the Balanced Scorecard should be included.
· Data should be presented in a trend format, when possible.
· Financial measurements should be compared to expenditure plan data.
· Projects should be grouped for analysis, e.g., program management efforts, infrastructure projects, non-infrastructure projects.
· A corrective action system should be in place that is linked to the reporting process. This could be accomplished as part of the dashboard effort, versus the external reporting vehicle.
· The reporting and monitoring aspects of this effort should be documented as a process improvement for software acquisition maturity rating purposes.
· The usefulness and success of performance metrics should be reviewed periodically for improvement.
Management’s Response: BSMO management requested an extension to respond to our draft report from June 28, 2002, to July 5, 2002. As of July 8, 2002, management had not responded to the draft report.
Appendix I
Our overall audit objective was to assess the Internal Revenue Service’s (IRS) plans for monitoring and reporting to the Congress the Business Systems Modernization (BSM) program-level status for cost, schedule, and quality. To accomplish this objective, we
I. Determined if the IRS’ processes for gathering and maintaining project data were sufficient to support program-level analysis and reporting.
A. Determined how other government agencies that were involved in systems modernization were monitoring projects and tracking program status.
B. Researched project management literature (Software Engineering Institute, Center for Project Management, etc.) to identify best practice cost, schedule, and quality metrics.
C. Interviewed Business Systems Modernization Office (BSMO) and MITRE Corporation staff to determine the availability and source of project data.
D. Interviewed two BSMO contracting specialists and conducted interviews with IRS procurement personnel to determine what quality measurements were possible.
E. Determined the processes for ensuring that project data being gathered for monitoring purposes was accurate.
F. Interviewed IRS management and Management Information Center/MITRE staff to determine if the IRS had adequately defined cost/schedule baselines for modernization projects.
G. Determined if the process of gathering and maintaining project data was a documented, and thus, repeatable process.
H. Determined if all modernization projects were subject to the same monitoring requirements.
II. Evaluated the IRS’ plans to publish a program-level status report for the BSM.
A. Determined how other government agencies that were involved in systems modernization were reporting status at the program level.
B. Interviewed BSMO/MITRE Corporation staff to determine what program-level measurements were being gathered or could be gathered using existing project data.
C. Interviewed the IRS Director of Financial Policy, Planning, and Programs and BSMO/MITRE Corporation staff to identify suggestions and processes for program-level reporting of BSM results.
Appendix II
Major Contributors to This Report
Jimmie Johnson,
Senior Auditor
Perrin T. Gleaton, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Associate Commissioner, Business Systems Modernization M:B
Deputy Associate Commissioner, Program Management M:B:PM
Deputy Associate Commissioner, Systems Integration M:B:SI
Director, Budget Policy, Planning and Programs M
Chief Counsel CC
National Taxpayer Advocate TA
Director,
Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaison:
Associate Commissioner, Business Systems Modernization M:B
Appendix IV
Explanation:
We interviewed nine
government agencies that are or have been involved in a major software
development project. Based on our
interviews and documentation obtained, we created this summary chart to show
nine issues and whether the issues were covered in each agencies reporting
process. If the issue was covered, the
block below the issue is blackened. The
most comprehensive report was from agency five. This report is depicted in Appendix V.
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Appendix V
The reporting
model was removed due to its size. To
see the reporting model, please go to the Adobe PDF version of the report on
the TIGTA Public Web Page.
Comprehensive Reporting
Model for Government Agencies
The reporting model
was removed due to its size. To see the
report model, please go to the Adobe PDF version of the report on the TIGTA
Public Web Page.
Comprehensive Reporting
Model for Government Agencies
The reporting model
was removed due to its size. To see the
report model, please go to the Adobe PDF version of the report on the TIGTA
Public Web Page.
The reporting model
was removed due to its size. To see the
report model, please go to the Adobe PDF version of the report on the TIGTA
Public Web Page.
Appendix VI
Examples of Recommended Status Report Measures
Explanation: The Balanced Scorecard approach
is becoming a recognized industry best practice as a tool for improving
strategic planning. The Balanced
Scorecard supplies a framework to translate a strategy into operational terms. The Scorecard includes four dimensions, or
components. These components are as
follows.
· Internal Business Process – “To satisfy our shareholders and customers, what business processes must we excel at?”
·
Financial – “To succeed financially, how should we appear to our stakeholders?”
Examples of Recommended Status Report
Measures
Internal Business Measures
Explanation: The number of defects (deficiencies found in
software products) could be used to measure the quality of software development
and testing. This chart reflects a
32-week trend in defect reports (DR), or deficiency reports.
The chart was removed due to its size. To see the chart, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
Innovation and Learning Measures
Explanation: Attainment and verification of Capability Maturity Model levels could be measured to show the Business Systems Modernization (BSM) program’s maturity.
The chart was removed due to its size.
To see the chart, please go to the Adobe PDF version of the report on the
TIGTA Public Web Page.
The chart was removed due to its size. To see the chart, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
Innovation and Learning Measures
Explanation: This bar chart of audits of internal processes could be used to show how well processes are being followed. The reason codes listed on the horizontal axis represent process problems identified during audits. When these charts are produced periodically, a trend can be observed. As a process becomes more mature, the total number of findings may decrease and the reasons may change. For instance, if reason code one is “ lack of documentation” and reason code five is “process quality is not enforced”, you would expect to see reason code one problems disappear and reason code five problems to decrease as an organization matures.
Process Audit Results
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Customer Measures
Explanation: This is a chart reflecting program cost/schedule measures against past expenditure (spend) plans. This chart could be used to indicate a trend in meeting (or not meeting) past cost/schedule goals. In this example chart, the trend shows that schedule and cost variances within each expenditure plan have steadily decreased.
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Customer
Measures
Explanation: This is a chart reflecting program cost/schedule measures against current expenditure (spend) plans. This chart could be used to indicate a trend in meeting (or not meeting) current cost/schedule goals. Once developed, quality measures could also be included. In this example chart, the program is under cost and ahead of schedule (percentages are below 0) for January through approximately July/August. However, beginning in August/September the program begins to exceed expenditure plan cost and schedule goals (percentages begin to creep above 0).
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Customer
Measures
Explanation: This is a chart reflecting program cost/schedule measures against the current expenditure (spend) plan. The numbers shown indicate the month. This chart could be used to indicate a trend in meeting (or not meeting) current cost/schedule goals. In this example chart, the program is getting more and more behind schedule and over cost for the first four months. The program then begins correcting itself and ends up ahead of schedule and under cost by the twelfth month.
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Customer
Measures
Explanation: This chart represents the percentage of requirements that projects are adhering to (all requirements, most requirements, few requirements).
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Customer
Measures
Explanation:
These charts could be used to measure the percentage and type of
requirements that have been changed.
These measurements would indicate the stability of user requirements.
Compliance
with Requirements
The charts were
removed due to their size. To see the
charts, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
Customer
Measures
Explanation: These charts are a
representation of quality measures. The
number of deficiencies found during testing could be measured to indicate the
quality of software development activities.
The charts were
removed due to their size. To see the
charts, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
Financial
Measures
Explanation: These charts, as well as the charts on the next two pages, are examples of status of funds measurement. A chart similar to one of these could be adapted to reflect the amount of funds remaining that were allocated to the IRS.
The charts were
removed due to their size. To see the
charts, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
Financial
Measures
Explanation: This chart is another example of status of funds measurement. A chart similar to this could be adapted to reflect the amount of funds that were allocated to the IRS.
Funding ($M)
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.
Financial
Measures
Explanation: This chart is another example of status of funds measurement. A chart similar to this could be adapted to reflect the amount of funds remaining that were allocated to the IRS.
The chart was
removed due to its size. To see the
chart, please go to the Adobe PDF version of the report on the TIGTA Public Web
Page.