The Custodial Accounting Project Team Is Making Progress; However, Further Actions Should Be Taken to Increase the Likelihood of a Successful Implementation

 

March 2004

 

Reference Number:  2004-20-061

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

March 3, 2004

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:     Gordon C. Milbourn III /s/ Gordon C. Milbourn III

                 Acting Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - The Custodial Accounting Project Team Is Making Progress; However, Further Actions Should Be Taken to Increase the Likelihood of a Successful Implementation (Audit # 200320025)

 

This report presents the results of our review of the Custodial Accounting Project (CAP) Release 1.0.  The overall objective of this review was to assess the status of the Internal Revenue Service’s (IRS) and the CAP contractor’s readiness to deliver the CAP Release 1.0.  To accomplish our objective, we reviewed test planning and execution, transition planning, and business continuity planning activities.  In addition, we reviewed any deviations from cost and schedule estimates.

In summary, financial accountability has proven to be problematic Government-wide.  The Office of Management and Budget considers 85 percent of major Federal Government agencies, including the Department of the Treasury, to have serious problems with financial accountability.  Part of the IRS’ solution for correcting its financial management weaknesses includes developing and implementing the CAP.

The IRS and the CAP contractor have made progress toward implementing the CAP Release 1.0.  The CAP team has planned a series of tests to determine if the system will operate within expectations, completed significant parts of the planned testing, and made some important transition decisions.  While the team has made progress, it faces challenges in the following areas:  testing, transition planning, and business continuity planning.

To assist the IRS in facing these challenges, we recommended that the Chief Information Officer (CIO) follow through on plans to increase the Business Systems Modernization Office’s involvement in testing, ensure the CAP contractor revises test scripts for future tests, and enhance transition planning by reviewing newly published guidelines.  We also recommended that the CIO build out and test the CAP disaster recovery environment as soon as possible, and reconsider the CAP disaster recovery classification.

In addition to challenges in these three areas, the CAP contractor has experienced significant cost increases and schedule delays.  To understand the reasons for this, we categorized the causes of each cost increase and schedule delay.  Since the IRS is in the process of implementing previous recommendations concerning cost and schedule increases and the use of firm fixed price contracts, we made no additional recommendations in these areas. 

Because the CAP team is in the midst of critical testing and transition activities, we communicated the results of our analyses intermittently during the audit.  Therefore, this report is historical in nature and may not reflect the most current activities or processes.  We plan to conduct a follow-up audit to provide additional analyses and recommendations as the CAP team works toward implementation.

Management’s Response:  Management fully agreed with four of our recommendations and disagreed with one recommendation.  The IRS responded that it has already established an embedded testing team with representatives from the CAP and the MITRE Corporation to further enhance the CAP testing process.  In addition, IRS management plans to ensure revised test procedures are included in an upcoming contract, update the Transition to Support Plan to incorporate transition acceptance criteria and critical success factors, share the program-level disaster recovery strategy with the CAP contractor, and ensure the contractor develops a project-level disaster recovery plan.  Management’s complete response to the draft report is included as Appendix V.

Office of Audit Comment:  The CIO disagreed with our recommendation to reconsider the CAP disaster recovery classification.  The CIO stated that it was the Chief Financial Officer’s (CFO) decision as owner of the system to determine the criticality of the system, and the CFO had chosen the noncritical classification.  We understand the responsibility for determining the CAP classification for disaster recovery purposes may rest within another part of the IRS; however, we still believe the IRS should reconsider the noncritical disaster recovery classification. 

The noncritical classification means that, if the CAP goes down after a disaster, the system could be down until restoration of the disaster location or until a new system is purchased.  The CAP Release 1.0 will support one of the IRS’ mission critical processes.  In addition, the General Accounting Office has stated that one of the biggest obstacles facing IRS management is the lack of financial management systems that can produce information needed for day-to-day decisions.  In the event of a disaster, the CAP will not be restored timely to support a mission critical process and to provide financial data for day-to-day decision making.  While we still believe our recommendation is worthwhile, the IRS could change the classification of the system in the future.  Therefore, we do not intend to elevate our disagreement concerning this matter to the Department of the Treasury for resolution.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

Table of Contents

Background

The Custodial Accounting Project Team Is Making Progress

Additional Focus Is Needed to Monitor Testing Results

Recommendation 1:

Recommendation 2:

Transition Planning Can Be Improved

Recommendation 3:

Business Continuity Risks Need to Be Addressed

Recommendations 4 and 5:

Cost Increases and Schedule Delays Persist

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Breakout of Cost Increases for the Custodial Accounting Project Since the Previous Audit

Appendix V – Management’s Response to the Draft Report

 

Background

The Custodial Accounting Project (CAP) will be a single, integrated data repository of taxpayer account information, integrated with the general ledger and accessible for management analysis and reporting.  The first release of the CAP will extract taxpayer account data from the Individual Master File (IMF) for the Taxpayer Account Subledger.  The second release of the CAP will extract data from the Business Master File (BMF) and other sources for the Taxpayer Account Subledger.

Financial accountability has proven to be problematic Government-wide.  The Office of Management and Budget considers 85 percent of major Federal Government agencies, including the Department of the Treasury, to have serious problems with financial accountability.  Weaknesses in the Internal Revenue Service’s (IRS) financial management systems include:

·        Deficiencies in controls to properly manage unpaid assessments, which result in both taxpayer burden and lost revenue to the Federal Government.

·        Deficiencies in controls over tax refunds, which permit the disbursement of improper refunds.

·        Inadequacies in the financial reporting process, which limit the timeliness and reliability of information for decision making. 

These weaknesses have caused the IRS to expend tremendous resources to prepare reliable financial statements.  Additionally, these weaknesses may adversely affect any decision by IRS management and/or the Congress that is based upon information obtained from IRS systems.

Part of the IRS’ solution for correcting these financial management weaknesses includes developing and implementing the CAP.  During our audit, the CAP team was making progress in preparing for implementation by planning and executing significant testing and making significant transition decisions.  While the CAP team has made progress, it faces challenges to improve testing, transition planning, and business continuity planning.  In addition, progress has been more costly and slower than planned.

The Business Systems Modernization Office (BSMO) and the CAP contractor were making changes to CAP testing activities during our review period.  Changes that have occurred since we concluded our analysis in early November 2003 are not reflected in this report.  As a result, this report may not reflect the most current activities or processes.  We plan to conduct a follow-up audit to assess the progress being made toward implementing the CAP Release 1.0. 

This review was performed at the BSMO facilities in IRS National Headquarters and New Carrollton, Maryland, and the CAP contractor offices in Merrifield, Virginia.  To provide timely feedback during critical testing activities, we communicated the results of our analyses intermittently during the audit.  These communications are discussed throughout the report.

The audit was conducted between January and December 2003 in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

The Custodial Accounting Project Team Is Making Progress

In August 2000, the CAP team was approved to move forward with the development, testing, and implementation of the CAP.  During our audit, the IRS and the CAP contractor were testing the system and preparing for implementation.

The IRS and the CAP contractor have made progress testing and preparing CAP Release 1.0 for implementation.  The team has planned a series of tests to determine if the system will operate within expectations, completed significant parts of the planned testing, and made some important transition decisions.

Testing plans were prepared

The CAP team has prepared a complementary set of test plans to help ensure that the developed system meets expectations.  We reviewed plans for the following test phases and found that the plans generally followed the standards set forth in the Enterprise Life Cycle (ELC).

·        System Integration Test (SIT) – Ensures that system components are properly integrated.  More specifically, this test ensures the components of the system can properly exchange information with each other and with other systems.

·        System Acceptance Test (SAT) – Confirms that the overall system functions correctly from the end user’s point of view.

·        Release System Integration and Test (RSIT) – Verifies that the CAP can properly integrate with other designated modernization projects.

Each test plan included entrance and exit criteria, specific test procedures to be followed, and the expected results for each test procedure. 

Significant testing has been completed or is in progress

During our audit work, the CAP team completed the SIT for the CAP Release 1.0.  In addition, the CAP team started the SAT and RSIT.

Important transition decisions were made

A Transition To Support (T2S) Plan includes the information necessary to successfully migrate a system to the IRS environment.  The CAP T2S Plan was initially due in June 2002; however, it could not be completed at that time because a decision had not been made as to who would maintain and operate the CAP once Release 1.0 was deployed. 

In Spring 2003, the IRS decided that the CAP contractor would maintain the system, while the IRS Enterprise Operations Office would operate the system.  In addition, funds from the Information Systems appropriation would be used to pay for operations and maintenance costs.  Once the IRS had communicated these decisions, the CAP contractor delivered a draft version of the CAP T2S Plan in July 2003 for review.

Additional Focus Is Needed to Monitor Testing Results

As mentioned previously, significant testing has been completed or is underway.  The most formal testing that has been completed is the SIT.  Therefore, the majority of our efforts involved analyzing the SIT.

We reviewed the test results from the 6 completed SIT scripts (consisting of 425 test procedures) to determine whether all test results were accurately reported and all test procedures were actually performed.  We found the following:

·        Testing officials reported inaccurate test information during critical meetings.  The ELC requires that meetings be held to determine if a project team is ready to proceed from one test phase to another.  The CAP team correctly held these meetings.  However, testing officials reported inaccurate information during these meetings. 

First, testing officials reported that seven of eight test scripts had passed the SIT.  In actuality, only six of the eight test scripts had passed the SIT.  The remaining two test scripts were deferred until the SAT.

In addition, testing officials reported that only two problem reports were outstanding upon exiting the SIT.  Using a file of problem reports dated September 25, 2003, we identified an additional eight problem reports that were outstanding after the SIT had concluded.  We reported this to the BSMO on November 9, 2003.  The CAP contractor responded that these problem reports were not discussed during critical meetings because short-term solutions had been applied to fix the problems, documentation was incorrect and the problems were not outstanding, or the intention was to fix the problem as part of the SAT.

·        The CAP contractor did not always complete the test procedures as written.  Therefore, the actual testing results did not always match the predetermined expected results.  We identified 24 test procedures that could not be completed as written.  The CAP contractor informed us that this occurred for several reasons:  lack of sufficient data in the SIT environment, lack of systems software in the SIT environment, and lack of data to generate logs and reports. 

We also identified 23 planned test procedures that were either not applicable or for which alternative test procedures were conducted.  Testers determined certain test procedures were not applicable, could be accomplished more efficiently, or needed additional instructions.  The ELC states that testers should correct test procedures as problems are discovered.  However, the test procedures were not corrected.  We communicated these concerns to the BSMO on September 24, 2003.

·        Some test procedures were not updated with actual test results.  Some test procedures failed initially.  Once retested, the test procedures passed.  However, the actual results were not updated in the testing documentation for seven test procedures.  We communicated this concern to the BSMO on September 24, 2003.

We determined these inaccuracies had little effect on the project because the IRS and the CAP contractor can ensure shortfalls from the SIT are handled as part of the SAT.  However, it will be important to ensure that all testing results are known prior to completing the SAT.  The IRS CAP Project Manager informed us that she was not initially aware of detailed test result inaccuracies due to staffing shortages.

Management Actions:  Once we conveyed our SIT concerns to the CAP contractor, the contractor stated it would 1) ensure all closed SAT test records are reviewed before problem reports are closed, 2) correct test procedures that would be reused in a later set of tests known as the Deployment Site Readiness Test (DSRT), 3) update testing documentation with actual testing results, and 4) update testing results to make the test results clearer.  The CAP contractor also prepared documentation to defer the testing of 49 test procedures from the SIT phase to the SAT phase.

The IRS CAP Project Manager informed us that the BSMO was concerned about the inaccuracies in reporting test results, but she was comfortable with the corrections being made by the CAP contractor.  With additional staffing now available, the IRS CAP Project Manager also informed us that she has assigned BSMO and MITRE representatives to enhance oversight and reduce risk in this area.

Recommendations

To improve insight into the testing process and results, the Chief Information Officer (CIO) should:

1.      Continue increased involvement by BSMO and/or MITRE officials in the CAP testing process.

Management’s Response:  The IRS responded that it has already established an embedded testing team with representatives from the CAP and MITRE to further enhance the CAP testing process.  In addition, CAP and MITRE support will continue through Milestone 5.

2.      Ensure the CAP contractor follows through on plans to revise test procedures for the DSRT.

Management’s Response:  IRS management plans to ensure revised DSRT procedures are included in an upcoming contract.

Transition Planning Can Be Improved

The CAP contractor recently delivered a draft version of the CAP T2S Plan to document how the system will migrate from the CAP contractor to the IRS.  We reviewed a draft version of the CAP T2S Plan and determined it complied with ELC requirements.  However, it could be improved by applying transition best practices.

Recently, the PRIME contractor enhanced the ELC to require a Transition Management Plan, in place of a T2S Plan.  The Transition Management Plan guidance is a result of the accumulation of best practices by the PRIME contractor.  Since the CAP team started development prior to creation of the new requirement, it is not required to deliver a Transition Management Plan. 

The CAP team could improve transition planning by reviewing the revised ELC guidance and determining if adhering to any additional requirements would improve the CAP T2S Plan.  In particular, the requirements for transition acceptance criteria and critical success factors would be helpful in planning the CAP transition.  Since the final CAP T2S Plan was due to the IRS for review in early December 2003, we communicated this improvement opportunity to the BSMO on October 16, 2003.

Recommendation

To enhance transition planning, the CIO should:

3.      Determine if any sections of the new ELC guidelines could be applied to the draft CAP T2S Plan to make the Plan more comprehensive.  For example, the project team should consider incorporating transition acceptance criteria and critical success factors.

Management’s Response:  The IRS responded that it had met with the CAP contractor to convey the need to review the new ELC guidelines and determine whether best practice recommendations would produce a more comprehensive CAP T2S Plan.  In addition, the CAP contractor will update the T2S Plan to incorporate transition acceptance criteria and critical success factors.

Business Continuity Risks Need to Be Addressed

The IRS’ business continuity program includes disaster recovery, business resumption, and other related efforts to ensure IRS operations continue in the event of an unexpected outage.  For the CAP, we determined several risks need to be addressed to ensure CAP operations can continue in such an event.  We communicated our concerns to the BSMO on August 1, 2003.

Disaster recovery capabilities will not be built out or fully tested prior to implementation

In case of a disaster after system implementation, certain components needed to fully restore CAP functionality are not currently available.  For example, a mid-level computer system that will be used to log into the CAP application has not been replicated at a disaster site.

According to BSMO officials, funds have been earmarked for the next 2 fiscal years to improve disaster recovery capabilities for all modernization projects.  Until that time, disaster recovery capabilities will be less than optimal.  Until all components needed for a full restoration of CAP capabilities are put in place, a full test of disaster recovery capabilities cannot be conducted.  According to project officials, not having full disaster recovery capabilities was caused by past budget cuts. 

Without full disaster recovery capabilities and testing, the IRS runs the risk that the CAP will not be able to fully recover in the event of a disaster.  According to IRS officials, this risk is minimized because the majority of the CAP application runs in a mainframe environment, which can be restored in the event of a disaster.

Business contingency tests were not initially conducted

The SIT Plan defines two tests known as the Application Recovery Test and the Technical Recovery Test that were to be conducted at the Martinsburg Computing Center.  These tests would provide indicators of the CAP’s ability to recover adequately from various simulated system failures.  We determined that these tests were not conducted because the CAP contractor did not inform the Martinsburg Computing Center these tests needed to be conducted.

Management Action:  Upon being informed these tests had not been conducted, the CAP contractor rescheduled the tests for the SAT.

CAP Release 1.0 is currently classified as a noncritical system

The draft CAP Technical Contingency Planning Document describes the contingency planning framework and identifies strategies for recovery in the event of a disaster.  Our review of the draft CAP Technical Contingency Planning Document revealed that the CAP Release 1.0, a system that will cost taxpayers nearly $100 million, was classified as a noncritical system.  This classification means that if the CAP goes down after a disaster, the system could be down until restoration of the disaster location or until a new system is purchased. 

We asked Chief Financial Officer (CFO) and BSMO personnel why the CAP was classified as noncritical.  Based upon information from the personnel we interviewed, we could not determine the exact reason for the classification.  However, a CAP team member informed us that the CAP would not be relied upon to prepare financial statements during Fiscal Year 2004 and that the classification could be changed at any time in the future.  This, along with the fact that the IRS has been able to sustain a clean opinion on its financial statements without the CAP, may be the reason the system is currently considered noncritical.

Notwithstanding these points, we determined that the CAP Release 1.0 will support one of the IRS’ mission critical processes – preparing audited figures for unpaid assessments, tax receipts, and refunds.  Therefore, we believe the IRS should consider upgrading the CAP’s classification as the CAP moves toward implementation.

Recommendations

To ensure IRS officials are able to restore the CAP after a disaster with the least disruption to the IRS mission, the CIO should:

4.      Completely build out and test the CAP disaster recovery environment as soon as possible.

Management’s Response:  IRS management plans to share the program-level disaster recovery strategy with the CAP contractor and ensure the contractor develops a project-level disaster recovery plan. 

5.      Reconsider the CAP disaster recovery classification in the draft Technical Contingency Planning Document.

Management’s Response:  The CIO disagreed with our recommendation to reconsider the CAP disaster recovery classification.  The CIO stated it was the CFO’s decision as owner of the system to determine the criticality of the system, and the CFO had chosen the noncritical classification. 

Office of Audit Comment:  While we understand the responsibility for determining the CAP classification for disaster recovery purposes may rest within another part of the IRS, we still believe the IRS should reconsider the noncritical disaster recovery classification.  The noncritical classification means that, if the CAP goes down after a disaster, the system could be down until restoration of the disaster location or until a new system is purchased.  The CAP Release 1.0 will support one of the IRS’ mission critical processes.  In addition, the General Accounting Office has stated that one of the biggest obstacles facing IRS management is the lack of financial management systems that can produce information needed for day-to-day decisions.  In the event of a disaster, the CAP will not be restored timely to support a mission critical process and provide financial data for day-to-day decision making.  While the current disaster recovery classification is noncritical, the IRS could change the classification of the system in the future.

Cost Increases and Schedule Delays Persist

In a previous audit, we reported the estimated costs to complete the first release of the CAP had increased from a planned $47 million to over $61 million.  In addition, the schedule for implementation had slipped from May 2002 until May 2003.  Cost increases and schedule delays have continued since that audit.

Cost increases

The majority of the cost increases noted during our previous audit were due to a work stoppage beginning in late 2000.  In September 2000, the Congress directed the BSMO to limit spending on the CAP until deficiencies in the CAP business case were corrected and the management of the CAP was integrated with the Business Systems Modernization (BSM) program.

During this audit, we determined that costs have continued to increase.  The latest published cost to complete the CAP Release 1.0 is now nearly $98 million.  The vast majority of this increase continues to be due to integrating with the BSM program.  Based upon our review of documentation and discussions with project officials, we categorized the many different cost increases since our previous review into the following general categories.  See Appendix IV for a delineation of all cost increases and how each cost increase was categorized.

Breakout of Cost Increases Since Our Previous Audit

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Schedule increases

At the end of our previous audit, the CAP Release 1.0 was due to exit Milestone 5 in May 2003.  At the time we completed our fieldwork, the Milestone 5 exit date had been re-scheduled for February 2004.  This is a 9-month slippage in completing the CAP Release 1.0 since our previous review.  The causes of the schedule slippage are similar to the causes of the cost increases detailed previously.

Future cost and schedule increases

At the end of our review, the CAP contractor was replanning the amount of time that would be needed to complete testing for the CAP Release 1.0.  Due to the contract type, any additional time needed to complete Release 1.0 will require additional funds from the IRS.  Since this information was not available to us at the end of the audit, it was not included in the previous analysis. 

One of the reasons cost increases continue to mount for the Federal Government is that the CAP contractor is not operating under a firm fixed price contract.  In the past, we have recommended firm fixed price contracts be used whenever possible, especially for projects in the development and deployment stages such as the CAP. 

Management Actions:  On September 29, 2003, the CIO issued a policy that states contracts for BSM projects be fixed price, with certain exceptions.  As part of this policy statement, the CIO requested that the Associate Commissioner, BSM, and the Director, Procurement, provide plans for implementing the new policy. 

Since the IRS is in the process of implementing previous recommendations concerning cost and schedule increases and the use of firm fixed price contracts, we are making no additional recommendations in these areas.  We will consider following up on previous recommendations in the future.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to assess the status of the Internal Revenue Service’s (IRS) and the Custodial Accounting Project (CAP) contractor’s readiness to deliver the CAP Release 1.0.  To accomplish our objective, we:

I.                    Determined whether project officials developed the project test plans in accordance with Enterprise Life Cycle (ELC) requirements.

II.                 Determined whether the CAP contractor accurately reported test results and resolved documented problems.

A.     Compared the test results to the test plans to determine whether all tests were performed.

B.     Traced summary test results to the detailed test results to determine whether the summary test results were reported accurately. 

C.     Determined whether components that were scheduled for testing for the CAP Release 1.0 were adequately addressed during the System Integration Test (SIT). 

D.     Reviewed a download of test failures for the SIT and determined whether failures were adequately addressed.  In addition, we documented whether the IRS ensured the test failures were effectively addressed.

III.               Determined how the IRS plans to operate and maintain the CAP Release 1.0 after deployment.

A.     Documented how the project team plans to resolve issues related to supporting the project after deployment.

B.     Reviewed the draft Transition To Support Plan.

IV.              Determined whether project officials were adequately planning for contingencies.

V.                 Determined the extent of changes to the project’s estimated cost, schedule, and/or benefits and whether the changes had been adequately explained and supported.

NOTE:  Additional work was scheduled concerning the CAP System Acceptance Test (SAT) and project dependencies.  Due to delays in CAP testing, we were unable to complete sufficient work to report on the CAP SAT.  Due to delays in other modernization projects, we are also not reporting on project dependencies due to their decreased significance for the CAP Release 1.0.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary V. Hinkle, Director

Scott A. Macfarlane, Director

Troy D. Paterson, Audit Manager

James A. Douglas, Senior Auditor

Wallace C. Sims, Senior Auditor

Tina Wong, Senior Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Commissioner, Business Systems Modernization  OS:CIO:B

Deputy Associate Commissioner, Program Management  OS:CIO:B:PM

Acting Director, Portfolio Management  OS:CIO:R:PM

Director, Internal Management Modernization  OS:CIO:B:PM:IMM

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaison:  Associate Commissioner, Business Systems Modernization  OS:CIO:B

 

Appendix IV

 

Breakout of Cost Increases for the Custodial Accounting Project Since the Previous Audit

 

Source:  Analysis based upon information provided by the Business Systems Modernization Office.

 

Appendix V

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.