The Business Systems Modernization Office Needs to
Strengthen Its Processes for Overseeing the Work of the PRIME Contractor
March 2002
Reference
Number: 2002-20-059
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
March
1, 2002
MEMORANDUM FOR
DEPUTY COMMISSIONER FOR MODERNIZATION &
CHIEF INFORMATION OFFICER
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – The Business Systems Modernization
Office Needs to Strengthen Its Processes for Overseeing the Work of the PRIME
Contractor (Audit # 200120028)
This report presents the results of our
review of the Internal Revenue Service’s (IRS) oversight of the contractor
hired to integrate its systems modernization efforts. The overall objective of this review was to determine whether the Business Systems
Modernization Office (BSMO) had established effective processes to ensure that
the contractor (PRIME contractor) was delivering high-quality goods and
services in a timely and cost-effective manner. To address this objective, we analyzed the results from
four audits we previously conducted of individual systems modernization
projects for issues and trends that may affect the systems modernization
program as a whole.
In
summary, the BSMO developed various management processes intended to ensure the
PRIME contractor delivers quality goods and services within expected time
frames. As of the completion of our
audit, however, many of these processes were still maturing and some had not
yet been effectively implemented. This
contributed to delays and cost increases in the four projects we reviewed. To date, the IRS and the PRIME contractor
have been overly optimistic about their timetable for delivering modernized
systems given the immaturity of their management processes. As the BSMO continues its ongoing work to
strengthen its processes for overseeing the PRIME contractor, improvements in
the areas noted below will be needed if the IRS’ systems modernization effort
is to avoid major delays and cost increases in the future. Further delays could erode confidence in the
IRS’ ability to deliver modernized systems that are needed to dramatically
improve both internal operations and service to taxpayers.
·
While the BSMO has made improvements in its contracting
practices, processes had not been implemented to require defined and negotiated
requirements and costs prior to the contractor personnel beginning work. Instead, the BSMO issued “level-of-effort”
task orders to the PRIME contractor, which paid them for hours worked rather
than deliverables produced until negotiations were completed. It is important to start work with
negotiated contracts so that team members are clear on the specific
deliverables required and the associated quality standards they need to
address. In one of the projects we
reviewed, approximately $3.9 million was paid to the PRIME contractor over a
5-month period for hours worked on a task order that was never finalized and
was eventually cancelled.
·
Performance-based contracting processes had not been fully
implemented. Although quality review
processes had been established to monitor contractor deliverables, most of the
task orders we evaluated did not contain incentives or disincentives to further
encourage contractor timeliness and quality.
While the BSMO can withhold payment to the contractor until quality
products are delivered, this option does not provide any compensation for
delays in delivering promised improvements to taxpayers. In addition, when quality problems were
identified, the BSMO did not always require that the contractor stop and
address the problems before progressing to the next project phase.
·
Project monitoring processes did not adequately capture
project cost information. Currently,
actual costs are measured against estimated costs for the current phase
(milestone) of the project rather than the project’s full budget. In addition, accurate measurements of
internal IRS costs are not included in project monitoring data, and the
measurement system has not been validated.
Without an accurate accounting of all project costs, it is difficult for
the IRS and the Congress to determine if a project is worth the investment.
·
Project managers did not establish accountability for
upcoming tasks or include reserve time for unplanned events in project
schedules. Currently, the PRIME
contractor assigns tasks to a group rather than to the individuals in that
group. Therefore, the BSMO has no
assurance that the individual employees with the skills necessary to do the job
will be assigned to critical project tasks.
In addition, reserve time for unplanned events was not built into
project schedules even though these events have continually occurred on all
projects. Allowing for unscheduled
events, such as the time needed to resolve problems identified during the
testing phase of projects, will provide more realistic estimates of project
delivery dates.
In order to address the above
conditions, task orders with clearly defined requirements and expectations
should be completed for the next project development phase prior to exiting the
current phase. The task orders should
be performance-based, including incentives and disincentives, when contracting
for key phases in the project life cycle.
To enable better monitoring of project development, progress should be
measured against the entire project budget rather than just the current
development phase, the IRS should include accurate internal cost data in these
measurements, and the BSMO should validate the accuracy of the measurement
system. Lastly, key individuals should
be included on assignment schedules for upcoming project tasks, and the project
teams should review and implement “lessons learned” from previous projects in
developing project schedule and cost estimates. Addressing these conditions should improve the development of ongoing
and future modernization projects.
Management’s Response: Management’s response was due on February
25, 2002. As of February 26, 2002,
management had not responded to the draft report.
Copies
of this report are also being sent to the IRS managers who are affected by the
report recommendations. Please contact
me at (202) 622-6510 if you have questions or Scott E. Wilson, Assistant
Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
Task Orders Did Not Contain Contractor Incentives and Disincentives
Project Monitoring Information Was Not Complete or Accurate and Had Not Been Validated
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Related Treasury Inspector General for Tax
Administration Audit Reports
The Internal Revenue Service (IRS) is in the process of modernizing its information technology systems. This effort is known as Business Systems Modernization (BSM). The IRS formed the BSM Office (BSMO) to manage this effort and to oversee the PRIME Alliance, a group of contractors headed by the Computer Sciences Corporation (CSC). The CSC, also known as the PRIME contractor, was hired to help design and integrate the various modernization projects.
To obtain funding for BSM from the Congress, the BSMO prepares a BSM Expenditure Plan that lists each project and the estimated costs and delivery dates. After the funding is received from the Congress, the BSMO issues task orders to the CSC that identify specific work and costs for defined project deliverables.
The BSMO and the CSC are required to follow a structured systems development process called the Enterprise Life Cycle (ELC). The ELC requires that certain project deliverables are completed and ready for review at key points (milestones) in the development process. The BSMO is responsible for establishing processes and procedures to ensure that the deliverables produced by the CSC meet defined requirements and are produced in a timely and cost-effective manner.
The BSMO established a quality review process to determine whether products delivered by the CSC meet required criteria at certain points. The CSC and the BSMO use project tracking techniques to monitor progress.
The overall objective of this review was to determine whether the BSMO had established effective processes to ensure that the CSC was delivering high-quality goods and services in a timely and cost-effective manner. To accomplish this, we analyzed the results of four audits conducted in Fiscal Year 2001 on the following projects to identify issues and trends that affect the BSM program as a whole (see Appendix IV for details on these four project audits).
Customer Communications - 2001 Release (CC01) - upgrades the IRS’ telephone communications system to more efficiently and effectively handle taxpayer calls.
· Telecommunications Enterprise Strategic Program (TESP) - provides the telecommunications needs for modernization projects and develops a long-term IRS telecommunications strategy.
· e-Services - provides the means for tax practitioners and other authorized parties to conduct business electronically with the IRS.
· Customer Relationship Management - Examination (CRM-Exam) - provides the ability for revenue agents to more efficiently and accurately compute complex corporate taxes.
This audit was conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to this report are listed in Appendix II.
At the time of our audits, the BSMO did not have processes in place to require task orders to be defined and negotiations finalized prior to allowing work to begin on the projects. Instead, the BSMO issued “level-of-effort” task orders, for which the CSC was paid for hours worked rather than deliverables produced. Project cost and schedule estimates were exceeded in part because the BSMO did not have clearly defined requirements and expectations prior to initiating work on projects.
In some cases contract negotiations were not finalized for several months after work began on the task orders. It is important to start work with negotiated contracts so that team members are clear on the specific deliverables required and the associated quality standards they need to address. Until negotiations are completed and requirements are final, the project team works towards what they believe will be the finalized requirements, and the BSMO may have to pay for hours spent on deliverables that eventually are not needed or used.
For example, approximately $3.9 million was paid to the CSC over a 5-month period for hours worked on a TESP task order that was never finalized and was eventually cancelled. On the CRM-Exam project, approximately $700,000 was paid to the CSC over a 60-day period for hours worked on a task order before requirements and costs were finalized.
One reason the BSMO allowed the CSC to begin work prior to completing negotiations on the task orders and finalizing requirements was to enable the CSC to provide input to the project requirements. Another reason was to avoid gaps in funding the projects that could result in the loss of valuable contractor expertise.
While help defining the requirements was needed, delays in completing the task order negotiations contributed to significant delays in completing project work and cost increases from estimates originally presented to the Congress. The project milestones we reviewed were delayed from 4 to 9 months, while cost increases ranged from nearly $700,000 to over $13 million from original estimates presented to the Congress.
For example, the CRM-Exam project team originally estimated they would complete their project planning phases in 10 months at a cost of $2.2 million. However, the planning phases took 7 months longer to complete than estimated, and the costs had increased over $2.5 million to nearly $4.8 million. According to BSMO management, these cost increases and delays were due in part to the fact that estimates developed for the Congress are produced prior to the development of proposals and completion of negotiations for the individual task orders. As a result, once requirements were finally defined, costs and time to address these requirements were substantially more than originally estimated.
Another project that incurred delays and cost increases was the e-Services project. The e-Services project team originally estimated they would complete the design phase of the project by September 2000 at a cost of $3.8 million. When we completed our audit work on the e-Services project, the completion date had slipped 9 months to June 2001, and the estimated cost had increased over $13 million to $17.2 million. According to BSMO management, the delays and cost increases were due in part to a significant change in requirements subsequent to the initial estimates, and the project was ahead of other projects it was dependent upon for direction. However, had the requirements been better defined at the beginning of the project, the delays and cost increases could have been significantly reduced.
According to Federal Acquisition Regulations, performance-based contracting methods require statements of work that define requirements in clear, concise language identifying specific work to be accomplished. Recent General Accounting Office testimony indicates that there is a very high risk that systems will not be developed timely and within budget, or not meet the needs of the business processes they are intended to support, when requirements are not well defined.
Management Actions
At the time of our audit, the BSMO was developing procedures to require defined task orders prior to allowing project teams to progress into the next phase. However, the procedures documenting this new effort were not available for our review prior to the completion of our audit work.
To ensure the CSC delivers high-quality goods and services in a cost-effective and timely manner, the BSMO should:
1. Finalize and implement procedures that require the completion of fully defined and negotiated task orders for the next project phase prior to exiting the current development phase or milestone. The BSMO’s milestone exit review should ensure that task orders for the next phase or milestone contain measurable performance standards, specific deliverables, costs, and due dates.
Management’s Response: Management’s response was due on February 25, 2002. As of February 26, 2002, management had not responded to the draft report.
Although the BSMO indicated that
it was following performance-based contracting processes, the task orders we
evaluated did not uniformly contain incentives or disincentives to encourage
quality and timeliness. BSMO officials indicated they did not uniformly include
incentives and disincentives in task orders because they wanted to avoid
developing an adversarial relationship with the CSC. Officials in BSMO also stated that disincentives were not needed
since the BSMO had the option of withholding payments from the CSC until
satisfactory performance was achieved.
While withholding payments to the CSC may address quality concerns, it does not provide any compensation to the IRS for project delays that could have a significant impact on other projects or on promised taxpayer benefits.
Government policy states that
agencies should negotiate a contract type and price that will result in
reasonable contractor risk and provide the contractor with the greatest
incentive for efficient and economical performance.
Although incentives and
disincentives were not included in the task orders to address potential
problems with the quality of contractor deliverables, quality review processes
have been developed to ensure that all CSC deliverables are reviewed prior to
allowing the CSC to move on to a new phase in the project development. Using these processes, the BSMO identified
some quality problems with products provided by the CSC in the projects we
reviewed.
For example, the BSMO quality
review process identified various concerns with significant CC01 project work
products, such as the Quality Management Plan, the System Life Cycle
Management Plan, the System Design Report, and the key Security Documents, as the project moved from the design phase into development,
and additional work was required to address the issues identified. The delays in completing these work products
impacted the project direction and expectations and resulted in further delays
in development and deployment.
The BSMO also identified several
concerns with deliverables from the TESP and e-Services project teams. The IRS executive team required the TESP
project team to address its issues prior to approving the project to move
forward into its next phase. Although
the e-Services project was allowed to proceed into the next phase, we believe
several issues that the quality review team identified should have been
addressed prior to allowing the project team to proceed.
Even though the BSMO worked with the CSC to rectify these issues, in some cases it allowed the CSC to proceed into the next phase in project development prior to these issues being addressed, and in others quality problems caused delays in approval of a milestone. Because incentives and disincentives were not included in the task orders, the BSMO was not able to obtain compensation for these delays and quality problems.
To ensure the IRS’ interests are protected, the BSMO should require that:
2. Task orders for system design, development, and implementation be performance-based whenever possible. These task orders should include incentive provisions to reward contractors for good performance and quality assurance deduction schedules to discourage unsatisfactory performance. The incentive and disincentive provisions should be based on measurement against predetermined performance standards and review plans.
The BSMO and CSC were not adequately monitoring and measuring the costs of the projects as they were being developed. Monitoring of project progress is an important process to enable successful project completion within required time frames and budgets.
To accomplish project monitoring, the CSC is using a “best practices” process called earned value management. This process measures actual cost and work accomplished against the budgeted cost and planned work scheduled. Variances between these actual and planned factors are analyzed and provided to management for decision-making.
While a project monitoring process has been implemented, improvements are needed in the following areas to enable more accurate monitoring for all projects:
· Costs are currently being measured against individual milestone estimates rather than complete project budgets.
· Accurate measurements for the IRS’ internal costs are not currently included in project monitoring data.
· The system used by the CSC to produce project monitoring data has not been validated by the BSMO.
Costs are measured
against individual milestone estimates rather than complete project budgets
ELC guidelines state that earned value techniques should measure the cost of the project over its life cycle. However, the current earned value measurement process only measures costs of the project against estimates for the current milestone.
According to project personnel, the current task order process makes monitoring against complete project budgets difficult because costs are negotiated by project phase rather than by complete project. As a result, it is difficult to use the current earned value data to determine whether each project is progressing according to the course established when it was initiated.
The Clinger-Cohen Act of 1996 requires the agency Chief Information Officer to monitor the performance of information technology programs of the agency. This includes evaluating the performance of those programs on the basis of applicable performance measures and advising the agency head regarding whether to continue, modify, or terminate the programs or projects. Current earned value data is of little value in assisting program management in determining whether the project team is on track to meet original requirements, time periods, and costs and whether investment in the project should be continued, because it is limited by project phase.
Accurate
measurements for the IRS’ internal costs are not currently included in project
monitoring data
The BSMO has not yet established an effective means to accurately track and report IRS labor, hardware, and software maintenance costs associated with projects. Therefore, these costs are not always included as part of the earned value analysis.
For example, we compared the data provided by the IRS for expenses related to its employees working on the e-Services project to a figure we calculated based on 17 employees working on the project for 16 months. We determined that the reported expenses for IRS employees working on the e-Services project were understated by approximately $1.6 million during the period October 1999 through January 2001.
Project personnel from e-Services indicated that it is difficult to ensure project employees, especially those who do not work full-time on the project team, use the correct codes for charging their time. However, the ELC indicates that an appropriate measure of total project cost should include any indirect costs, and without an accurate accounting of these costs, actual return on investment cannot be accurately calculated.
The system used by
the CSC to produce project monitoring data has not been validated
The CSC is required to provide accurate and complete earned value data. In response to an earlier audit report, Significant Risks Need to Be Addressed to Ensure Adequate Oversight of the Systems Modernization Effort (Reference Number 2000-20-099, dated June 2000), the BSMO stated that it would review the system the CSC uses to produce earned value measures and other project monitoring tools.
However, at the time of our four project reviews, the BSMO had not validated the system because the CSC had not provided the information necessary for this review. It is critical to ensure that the system is producing accurate data because the BSMO relies on this data to make crucial business decisions.
Management Actions
For Fiscal Year 2002, the IRS plans to begin an earned value monitoring process, based on a recurring evaluation of the CSC’s internal management control practices and samples of internal and external data. The procedures documenting this new effort were not available for our review prior to the completion of our audit.
To enable effective monitoring of project progress, the BSMO should take the following actions:
3. Require the CSC to begin evaluating project
progress against the entire project budget rather than just the current
milestone figures.
4. Provide accurate IRS internal cost data to the CSC for inclusion in project monitoring data.
5. Follow through with plans to validate the system used to produce earned value project measures.
Processes to ensure individual accountability for tasks were
not followed by the CSC. Project managers assigned tasks to a group rather than to
the individuals in that group. As a
result, the BSMO had no assurance that the CSC assigned employees with the
necessary skills to complete critical project tasks and that those employees
would be available when needed. The ELC
states that, during the execution stage, the project manager must develop
individual Task Assignment Schedules based on the project work packages.
When we discussed this issue with
the BSMO and the CSC, the CSC indicated it does not believe individual task
assignment information is useful in managing the projects.
In addition, project managers did
not follow best practices to ensure adequate time was allocated for tasks. Reserve time was not built into the project
schedules to allow for project delays or unplanned events, even though these
types of events have continually occurred on all projects. Reserve time is especially critical when the
project schedule is aggressive, as it was in several of the projects we
evaluated. Allowing for
unscheduled events, such as the time needed to resolve issues identified during
the testing phase of projects or quality concerns with other deliverables,
would provide more realistic estimates of project delivery dates.
When we discussed this issue with
the BSMO and the CSC, the CSC indicated that its practice is not to separately
allocate reserve or recovery time.
The BSMO and CSC will continue to be overly optimistic about their timetable for delivering modernized systems if more effective techniques are not developed for estimating reserve time and ensuring work is assigned to available individuals with the required skills. Further delays in projects could erode confidence in the IRS’ ability to deliver modernized systems that are needed to dramatically improve both internal operations and service to taxpayers.
Management Actions
The BSMO is working closely with the CSC to improve project schedule estimates. Using experience gained with the initial projects, the BSMO and CSC plan to incorporate additional time into project schedules to provide for thorough reviews by IRS stakeholders.
To increase the likelihood of timely delivery of projects, the BSMO should ensure that:
6.
Individual Task Assignment Schedules are used for critical
project tasks.
7.
Project managers use “lessons learned” from previous BSM
projects in developing time estimates for critical tasks. Until those lessons are effectively
implemented, project schedules should include reserve time to compensate for
delays or unplanned events.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine whether the Business Systems Modernization Office (BSMO) had established effective processes to ensure that the Computer Sciences Corporation (CSC), also known as the PRIME contractor, was delivering high-quality goods and services in a timely and cost-effective manner. To accomplish this, we analyzed the results of the following project audits we conducted in Fiscal Year 2001 for issues and trends that may affect the Business Systems Modernization (BSM) program as a whole:
· Customer Communications - 2001 Release (CC01) - upgrades the Internal Revenue Service’s (IRS) telephone communications system to more efficiently and effectively handle taxpayer calls.
· Telecommunications Enterprise Strategic Program (TESP) - provides the telecommunications needs for modernization projects and develops a long-term IRS telecommunications strategy.
· e-Services - provides the means for tax practitioners and other authorized parties to conduct business electronically with the IRS.
· Customer Relationship Management - Examination (CRM-Exam) - provides the ability for revenue agents to more efficiently and accurately compute complex corporate taxes.
Audit teams performed fieldwork for the four project audits and for this trend analysis audit in the IRS National Headquarters and the BSMO facilities in New Carrollton, Maryland.
These four audits, as well as this trend analysis review, were performed between February 2000 and October 2001.
To complete our work on this review, we conducted the following tests:
I. Obtained sufficient background on the subject area by reviewing the following documents:
-
Capability
Maturity Model guidelines issued by the Software Engineering Institute.
-
Enterprise
Life Cycle (ELC) guide.
-
Relevant
BSMO documents.
-
Core
Business Systems Executive Steering Committee meeting minutes and pre-meeting
reading material.
-
Audit plans
and reports from the four project audits.
-
The BSM
Quality Assurance Assessment of the Acquisition Management Process.
-
Task orders
for all four project audits.
- Relevant General Accounting Office and Inspector General reports.
II. Identified common successes and failures among the four projects reviewed.
A. Obtained and reviewed reports, audit
plans and workpapers from the four subject audits.
1.
Reviewed
audit workpaper files.
2.
Interviewed
audit team members and managers to identify extent of project scope and to
discuss issues identified.
B. Identified trend results in the four
audits (both positive and negative).
1.
Reviewed
briefing documents, memoranda, and reports.
2.
Created a
matrix with results from each audit.
3.
Analyzed
the matrix for trends (positive and negative).
C. Interviewed audit team members as
needed to clarify issues.
D. Reviewed
documentation of previously conducted interviews of BSMO program officials to
evaluate project integration and dependencies and the effect of issues on the
BSM program.
III. Determined causes of issues affecting multiple projects.
A. Reviewed audit workpapers to identify potential causes for issues identified as affecting more than one project.
B. Reviewed documentation of previously conducted interviews of project managers, Contracting Officer Technical Representatives, Government Task Managers, and BSMO Personnel to determine causes, significance of issues, and potential recommendations.
C.
Interviewed
audit teams as needed to clarify results and comments from project managers.
IV. Determined the impact of issues on the BSM program and quantify outcome of proposed corrective actions.
A. Reviewed workpapers and interviewed auditors to identify tests in which outcomes have been quantified and determined whether corresponding issues can be projected against other BSM projects.
B. Reviewed documentation of interviews of project managers and BSMO personnel to evaluate significance of issues and potential outcomes of planned corrective actions.
V. Developed recommendations to address issues and improve project quality and timeliness throughout the BSM program.
A. Reviewed workpapers to identify project level recommendations for issues affecting multiple projects.
B. Interviewed audit teams to clarify issues.
C. Determined whether project level recommendations can be rolled into program level corrective actions.
D. Reviewed documentation of previously conducted interviews of project managers and BSMO personnel to evaluate proposed recommendations.
Appendix II
Major
Contributors to This Report
Scott E. Wilson, Assistant Inspector General for Audit (Information Systems Programs)
Scott Macfarlane, Director
Tammy L. Whitcomb, Audit Manager
Eulala Davis, Senior Auditor
Charles Winn, Senior Auditor
George Franklin, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Associate Commissioner, Business Systems Modernization M:B
Advisor to Associate Commissioner, Business Systems Modernization M:B
Deputy Associate Commissioner, Program Management M:B
Deputy Associate Commissioner, Systems Integration M:B
Director, Tax Administration Modernization M:B:TAM
Director, Infrastructure Modernization M:B:IF
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaison:
Associate Commissioner, Business Systems
Modernization M:B
Appendix IV
Related Treasury Inspector General for Tax
Administration Audit Reports
Progress in
Developing the Customer Communications Project Has Been Made, But Risks to
Timely Deployment in 2001 Still Exist (Reference
Number 2001-20-055, dated March 2001).
The Customer
Relationship Management Examination Project Experienced Delays and Increased
Costs, But Lessons Learned Should Improve Future Modernization Projects (Reference
Number 2001-20-140, dated August 2001).
The Telecommunications
Modernization Project Provided Some Benefits, But Process Improvements Are
Needed for Future Projects (Reference Number 2001-20-143,
dated
August 2001).
Improvements Are Needed in the Management of the e-Services Project to Enable Timely Progress Towards Future Goals (Reference Number 2001-20-144, dated September 2001).