The Cost and Schedule Estimation Process for the Business
Systems Modernization Program Has Been Improved, but Additional Actions Should
Be Taken
September 2003
Reference
Number: 2003-20-219
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
September
29, 2003
MEMORANDUM FOR
CHIEF INFORMATION OFFICER
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Assistant
Inspector General for Audit (Small Business and
Corporate
Programs)
SUBJECT: Final Audit Report - The Cost and Schedule Estimation Process for
the Business Systems Modernization Program Has Been Improved, but Additional
Actions Should Be Taken (Audit # 200320024)
This
report presents the results of our review of the cost and schedule estimation
process for the Business Systems Modernization (BSM) program. The overall objective of this review was to determine if the Business Systems Modernization
Office (BSMO) and the PRIME contractor have developed a cost and schedule
estimation system that will produce reliable estimates and follow applicable
regulations and best practices.
In summary, the BSM program
has been plagued with cost and schedule overruns and has been criticized for
its ineffective cost and schedule estimation capabilities. Until recently, the PRIME contractor’s cost
and schedule estimation process was decentralized at the project level, and the
BSMO had not independently reviewed cost and schedule estimates or the PRIME
contractor’s cost and schedule estimation system. In response to criticism
and the need to improve processes, the BSMO has taken the lead by tasking the
PRIME contractor to build a program-wide cost and
schedule estimation system that is compliant with best practices in the
Information Technology industry. In addition, the BSMO is nearing completion
of the initial validation of the PRIME contractor’s cost and schedule estimation
system and has created a process for independently reviewing cost and schedule
estimates.
The BSMO and the
PRIME contractor have made progress in establishing a program-wide cost and
schedule estimation system by developing estimation procedures. In addition, the BSMO and PRIME contractor began
other estimation activities, such as establishing a historical database,
reviewing and adjusting cost and schedule estimation models, and developing a
risk-adjusted model to include an
analysis of uncertainty when preparing estimates.
These procedures and
activities incorporate industry best practices. However, the cost and schedule estimation system had not been
validated, and procedures had not been fully implemented during our audit time
period. Because significant work
remains, we could not determine if the cost and schedule estimation system
would produce reliable estimates and follow applicable regulations and best
practices.
To provide recommendations
for improvement as the cost and schedule estimation capability was being
enhanced, we reviewed the current BSMO and PRIME contractor activities and
process documentation. We determined
that the Chief Information Officer should ensure the BSMO takes additional
steps to further enhance the cost and schedule estimation system. Accordingly, we recommended that the Chief
Information Officer ensure that all modernization contractors follow cost and
schedule estimation guidance, the cost and schedule model calibration process
is documented, cost and schedule estimation guidance is revised, and an
independent assessment of the cost and schedule system is made once
improvements are made. We also
recommended improving system validation and estimate review processes.
Management’s
Response: BSMO management requested an extension to respond to our draft report
from September 19, 2003, to September 26, 2003. As of September 26,
2003, management had not responded to the draft report.
Copies of this report are
also being sent to the Internal Revenue Service managers affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector
General for Audit (Information Systems Programs), at (202) 622-8510.
Progress
Has Been Made in Establishing a Cost and Schedule Estimation System
Continued Management Focus Is Needed to Further Improve Cost and Schedule Estimation Capabilities
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
The Internal Revenue Service (IRS) is currently modernizing
its computer systems and business processes and practices. This effort is known as the Business Systems
Modernization (BSM). Since the inception of the BSM, the General
Accounting Office (GAO) has designated the program as high risk, in part
because of its size, complexity, and immense importance to improving IRS
performance and accountability.
In December 1998, the IRS hired the Computer Sciences Corporation as the PRIME contractor to design and develop the BSM program and projects. The expectation was that the PRIME contractor would strengthen the IRS’ ability to manage and control modernization initiatives. In addition, the IRS created the Business Systems Modernization Office (BSMO) to coordinate and oversee the work of the PRIME contractor.
While the partnership between the PRIME contractor and the BSMO was meant to strengthen systems acquisition and development capabilities, the BSM program has been plagued with cost and schedule overruns and criticized for its ineffective cost and schedule estimation capabilities.
In February 2002, the GAO reported that the BSM cost and
schedule estimates being provided to the Congress were contractor-provided
“rough order of magnitude” estimates that were not subjected to meaningful,
reliable validation by the IRS. The GAO
recommended that the IRS adopt effective cost and schedule estimation
practices.
In addition, we have reported that the BSMO and PRIME
contractor have experienced difficulties in meeting cost and schedule
estimates. For example, we reported in
October 2002 that the BSM program experienced a $75
million, or 24 percent, cost increase for the 20 projects that had been
initiated since 1999. We also testified this year before the
Congress that one of the more critical BSM projects, the Customer Account
Database Engine, would miss its planned delivery date by at least 20 months.
In response to criticism and the need to
improve processes, the BSMO has taken the lead by tasking the PRIME contractor
to
build a program-wide cost and schedule estimation system that is compliant with
best practices in the Information Technology industry. The BSMO is currently validating the PRIME
contractor’s cost and schedule estimation system, as well as establishing a
capability for validating estimates received from the PRIME contractor.
We conducted our audit work at the IRS National Headquarters and the BSM offices in New Carrollton, Maryland, between April and July 2003 in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The Federal Acquisition Regulation (FAR) provides that using
an acceptable estimation system benefits both the Federal Government and a
contractor by increasing the accuracy and reliability of individual proposals. Until recently, the PRIME contractor’s cost
and schedule estimation process was decentralized at the project level, and the
BSMO had not independently validated cost and schedule estimates or the PRIME
contractor’s cost and schedule estimation system. For example, program-level guidance on creating and validating
cost and schedule estimates was not available, and historical data on cost and
schedule performance within the BSM program were not easily accessible.
The BSMO and the PRIME contractor have made progress in
establishing a program-wide cost and schedule estimation system by developing
estimation procedures and beginning other estimation activities. These procedures and activities incorporate
industry best practices.
· The PRIME contractor developed a guidebook to provide assistance with developing cost and schedule estimates. This guidebook is being updated quarterly to provide additional guidance and respond to recommended improvements.
· The BSMO developed a draft procedure to be followed by the BSMO validation team in performing the validation of the PRIME contractor’s cost and schedule estimation system capabilities. The procedure includes a checklist based on the Software Engineering Institute (SEI) guidance to be used in conducting the validation. At the end of our audit work, the BSMO was nearing completion of the initial validation of the cost and schedule estimation system.
· The BSMO developed a draft procedure to provide guidance for an independent review of contractor-provided cost and schedule estimates. The procedure includes a checklist based on the SEI guidance to be used in conducting the reviews. In addition, the BSMO conducted a pilot of three project estimates to verify the adequacy of the procedure.
· The PRIME contractor began establishing a database containing historical cost and schedule data from PRIME projects to be used in preparing future estimates.
· The PRIME contractor began calibrating its estimation models. Calibration involves reviewing and adjusting estimation models to ensure that they represent the PRIME contractor’s specific cost and product history.
· The BSMO and the PRIME contractor recently began developing a risk-adjusted model to include an analysis of uncertainty when preparing estimates. This model is being developed to meet a recent Office of Management and Budget requirement for budget justifications.
To provide recommendations for improvement as the cost and schedule estimation capability is being enhanced, we reviewed the current BSMO and PRIME contractor activities and process documentation. We determined that the BSMO could take the following additional steps to further improve the cost and schedule estimation system:
· Complete a schedule for developing the historical cost and schedule database.
· Require all modernization contractors to follow cost and schedule estimation system policies and procedures.
· Document the process for calibrating estimation models and complete a schedule for the calibration.
· Improve the cost and schedule estimation guidebook.
· Conduct a future independent assessment to ensure that the cost and schedule estimation system has the ability to produce reliable estimates.
· Improve BSMO procedures for validating the PRIME contractor’s cost and schedule estimation system and reviewing contractor-provided estimates.
The PRIME contractor is in the process of developing a database to include historical cost and schedule data from PRIME projects. The database is needed to provide a solid historical measurement basis for more accurate estimates. Project managers will use the database to develop estimates for their projects.
Due to competing priorities, not all project data have been provided to the PRIME contractor’s Estimation and Measurement staff for inclusion in the database. To further complicate the completion of this task, the project teams have not been collecting consistent data. Also, the process is time-consuming, and there is no detailed schedule for completing the database.
Without a detailed schedule, the BSMO and PRIME contractor
do not know when the database will be completed. Until the database is completed, program-wide historical data
will not be available to assist project managers in preparing estimates.
Management Action: During the audit, we communicated to the BSMO that the PRIME contractor should prepare a detailed schedule for completing the historical cost and schedule measurement database. During discussions of a preliminary version of this report, the BSMO and the PRIME contractor provided a schedule detailing when the highest priority projects would become part of the historical cost and schedule database.
Require all modernization contractors to follow cost
and schedule estimation system policies and procedures
The PRIME contractor’s estimation
policies and procedures are applicable to the BSM program. However, there is no requirement for
non-PRIME contractors involved in the modernization effort to follow the
policies and procedures. In addition,
there are currently no plans to include non-PRIME contractor cost and schedule
data in the historical database.
For example, the
Custodial Accounting Project is a modernization project that is being managed
and developed by a non-PRIME contractor.
Non-PRIME contractors are not required to
follow the policies and procedures in the PRIME contractor’s cost and schedule
estimation guidebook and have not been requested to provide any measures on
their cost and schedule estimates to the PRIME contractor for inclusion in the
historical database for modernization.
Without specific requirements for all modernization contractors to follow established policies and procedures, there will not be consistency in the estimation process. In addition, if measures are not collected from all modernization projects, the database will not include all modernization data. The more historical data available in the database for preparing estimates, the more reliable future estimates will be to improve the overall estimation process.
Document the process for calibrating estimation models and complete a schedule for the calibration
The PRIME contractor is in the process of calibrating the
estimation models that are currently being used to prepare cost and schedule
estimates. The FAR provides that the
Federal Government may use various cost analysis techniques and procedures to
ensure a fair and reasonable price, given the circumstances of the acquisition. These techniques and procedures include the
reasonableness of estimates generated by appropriately calibrated and validated
models.
The PRIME contractor does not have a documented process for the calibration of the estimation models, as required by IRS draft procedures. In addition, there is no detailed schedule to prioritize and complete the calibration of the estimation models. Since there is no process or schedule for the calibrations, the PRIME contractor has not identified the time required to complete the calibrations. Until all models are calibrated, the estimates are being prepared using estimation models that have not been calibrated and, therefore, do not meet the FAR requirement or IRS draft procedures.
Management Action: During the audit, we communicated to the BSMO that the PRIME contractor should document the process for cost and schedule estimation model calibrations and prepare a detailed schedule for completing the calibration of the estimation models. During discussions of a preliminary version of this report, the BSMO and PRIME contractor provided a status update. The PRIME contractor has created a draft, high-level process for calibrating estimation models. In addition, the PRIME contractor has also created a schedule for completing the calibration of an established industry estimation model, as well as several other estimation models used by the PRIME contractor.
Improve the cost and schedule estimation guidebook
The PRIME contractor is responsible for the establishment of estimation processes to more accurately estimate costs and schedules associated with task order proposals, impact assessments, and change requests. The PRIME contractor developed a guidebook to provide assistance with developing estimates. The guidebook is in its early stages of development and does not include guidance on:
· What documentation is required to support estimates. SEI guidance provides that the estimation process should be documented. During one of the pilot estimate reviews, engineering judgment was used extensively for the estimates. Engineering judgment is acceptable; however, insufficient documentation was provided for how the engineers applied past experience to estimate the current job.
In addition, the PRIME contractor did not provide
a schedule for a change request. The
PRIME contractor stated that a schedule was not required for the change
request, but the BSMO insisted that a schedule was needed. The guidebook and the Enterprise Life Cycle
(ELC) Change Management Process procedure do not include the specific
information that should be included with change requests. If all documentation is not provided to the
BSMO estimate review team, the team may not be able to determine if future
estimates are reasonable.
· When estimators are required to use a second method in developing estimates. The BSMO estimate review team noted on all three pilot project estimates that there was no second method used when developing estimates. The purpose of using a second method is to validate the estimate and increase its reliability. In addition, using a second method is an SEI best practice and is recommended in the PRIME contractor’s estimation guidebook.
· How and where estimation lessons learned should be documented. The ELC requires that lessons learned be documented, including recommendations for improvement for future projects. Without proper guidance, significant lessons learned on cost and schedule estimation may not be documented, shared with other project personnel, and used to improve the estimation process.
· How to prepare estimates for the operations and maintenance phase of projects. The estimation guidebook includes guidance for all ELC phases, except the operations and maintenance phase. The IRS plans to task modernization contractors with the operation and maintenance of systems in the future; therefore, sound management practices dictate that guidance needs to include procedures on how to prepare estimates for the operations and maintenance phase.
Detailed procedures are needed so that project teams know how to provide support for cost and schedule estimates. Without detailed guidance, the staff involved in the estimation process will not know what is specifically required by the policies and procedures, and there will be inconsistencies in how the guidance is followed.
Conduct a future independent assessment to ensure that the
cost and schedule estimation system has the ability to produce reliable
estimates
Validation is the process of demonstrating a
model’s ability to function as a credible estimation tool. Validation ensures that estimation
system policies and procedures are established and enforced, and that key
personnel have proper experience and are adequately trained. In addition, validation ensures that proper
information system controls are established to monitor system development and
maintenance activities, and that the model is a good predictor of costs.
The BSMO established a
validation team to conduct the validation of the PRIME contractor’s cost and
schedule estimation system. During the
validation process, the team assisted the PRIME contractor in areas that needed
to be completed so that they could continue with their validation. This involvement has increased the amount of
time needed to complete the validation.
Since
the BSMO has been closely involved in both assisting the PRIME contractor with
establishing the system while simultaneously validating the system, a
third-party opinion may be needed to ensure that the cost and schedule
estimation system follows best practices and will produce reliable
estimates.
In our opinion, the
validation of the cost and schedule estimation system should be independently
conducted. Without an independent
assessment, the BSMO cannot be confident that the cost and schedule estimation
system has the ability to produce reliable estimates and follows applicable
best practices. Because the BSM program
has been estimated to last as long as 15 years and cost up to $7 billion, any
investment in ensuring that cost and schedule estimates are as reliable as
possible is prudent.
Improve BSMO procedures for validating the PRIME contractor’s cost and schedule estimation system and reviewing contractor-provided estimates
The BSMO developed two draft procedures to provide guidance for its estimation processes. One procedure is for the validation of the PRIME contractor’s cost and schedule estimation system, and the other provides guidance on how to conduct an independent review of contractor-provided cost and schedule estimates. Although the procedures are comprehensive, they are focused on the process, not on how to use the results to improve the process.
The BSMO focused initially on establishing processes. The next step will be to add detail to the processes and begin tracking results. We identified the following areas that need additional guidance in the draft BSMO cost and schedule estimation system validation procedure and the draft procedure for independent review of contractor-provided cost and schedule estimates:
· Tracking identified issues – The BSM Risk Management Plan provides that all modernization organizations use the Item Tracking Reporting and Control (ITRAC) system to record and update the status of issues. The BSMO system validation and estimate review procedures provide for limited follow-up of issues identified; however, there is no guidance on including identified issues in the ITRAC system or on tracking the issues through to resolution. Any issues identified in the validation of the cost and schedule estimation system and during the estimate reviews should be tracked in the ITRAC system to ensure that they are resolved. If the issues are not tracked in the ITRAC system, all parties may not be aware of the issues, and the issues may not be resolved timely or at all.
· Providing results to the IRS Procurement function – BSMO procedures do not require that system validation and estimate review reports be provided to the IRS Procurement function. The Procurement function conducts reviews of contractor pricing and cost data to ensure rates charged are appropriate. Understanding the cost and schedule estimation system is important because it is the basis for the cost estimates. The results of the BSMO reviews may affect the degree upon which the Procurement function relies on the contractor’s cost data. Without the sharing of information, the Procurement function’s reviews will not take into consideration issues identified by the BSMO.
· Trending of review results – According to SEI guidance, the estimating capability of an organization should be quantified, tracked, and evaluated. Evidence of maturity in this area includes management tracking and reviewing the effectiveness of its estimation processes. To track and review effectiveness, the BSMO needs to include guidance on trending review results so that areas needing improvement can be identified.
· Providing issues to the system validation team – The cost and schedule estimation system validation team needs to be aware of all issues identified during the estimate reviews so that it can take the issues into account during its periodic validation of the cost and schedule estimation system. Therefore, sound management practices dictate that the estimate review procedure needs to include a requirement that the reports be provided to the cost and schedule estimation system validation team.
If procedures do not include guidance on how to use results to improve the process, the same issues may continue to occur, and the BSM program will not have evidence that cost and schedule estimation capabilities are improving.
To further improve cost and schedule estimation processes, the Chief Information Officer (CIO) should ensure that:
1. All contractors working on BSM projects
follow the PRIME contractor’s policies and procedures for preparing cost and
schedule estimates and provide data for inclusion in the historical database.
Management’s Response: BSMO management requested an extension to respond to our draft report from September 19, 2003, to September 26, 2003. As of September 26, 2003, management had not responded to the draft report.
2. The PRIME contractor
documents the process for cost and schedule estimation model calibrations.
3. The PRIME contractor revises the cost and schedule estimation guidebook and applicable ELC references to provide details of what specific documentation is required to support estimates. The CIO should also require the BSMO to ensure that guidance is clarified regarding when a second method is required for preparing estimates.
4. The SEI is requested to conduct an
independent review of the cost and schedule estimation system once the initial
validation is complete and policies and procedures are fully implemented.
5.
The BSMO updates draft procedures to include guidance on:
a. Including all issues identified during the cost and schedule estimation system validation and estimate reviews in the ITRAC system.
b. Providing cost and schedule estimation system validation and estimate review reports to the IRS Procurement function.
c. Trending estimate review results.
d.
Providing all
estimate review findings to the cost and schedule estimation system validation
team.
Appendix I
Detailed Objective, Scope,
and Methodology
The overall objective of this review was to determine if the Business Systems Modernization Office (BSMO) and the PRIME contractor have developed a cost and schedule estimation system that will produce reliable estimates and follow applicable regulations and best practices. To accomplish our objective, we:
I. Determined if the BSMO validation of the PRIME contractor’s cost and schedule estimation system would be sufficient to ensure the reliability of cost and schedule estimates.
A. Determined the status of the BSMO validation of the PRIME contractor’s cost and schedule estimation system and planned corrective action(s).
B. Interviewed Internal Revenue Service (IRS) Procurement function personnel to determine the function’s involvement in the validation of the PRIME contractor’s cost and schedule estimation system.
II.
Determined if the
BSMO estimate review process was adequate to provide an independent review of
contractor-provided cost and schedule estimates.
A. Obtained and reviewed the IRS Cost and Schedule Basis of Estimate Review Checklist for the piloted estimate reviews to determine any identified issues.
B. Obtained and reviewed the Cost and Schedule Estimate Report for one of the piloted reviews to determine the completeness of the review.
C. Interviewed staff responsible for the estimate review to determine the process used for conducting the reviews.
III. Determined the process and status of the PRIME contractor’s activities for the cost and schedule estimation system.
A. Determined if the PRIME contractor had an adequate process in place to establish the centralized, historical database.
B. Determined the process being used for calibrating the cost and schedule estimation system models.
IV. Determined if the cost and schedule estimation system policies and procedures follow best practice guidance and if improvements could be made.
Appendix II
Major Contributors to This Report
Margaret E. Begg, Assistant Inspector General
for Audit (Information Systems Programs)
Scott A. Macfarlane, Director
Troy D. Paterson, Audit Manager
Michelle Griffin, Senior Auditor
Bruce
Polidori, Senior Auditor
Beverly K. Tamanaha, Senior Auditor
Appendix III
Commissioner C
Deputy Commissioner Operations
Support OS
Associate Commissioner, Business
Systems Modernization OS:CIO:B
Deputy Associate Commissioner,
Program Management OS:CIO:B:PM
Acting Director, Portfolio
Management Division OS:CIO:R:PM
Chief Counsel
CC
National Taxpayer Advocate TA
Director, Office of Legislative
Affairs CL:LA
Director, Office of Program Evaluation and
Risk Analysis RAS:O
Office
of Management Controls OS:CFO:AR:M
Audit Liaison: Associate Commissioner, Business Systems Modernization OS:CIO:B