The primary goal of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) End-to-End (E2E) Systems Integration Test was to demonstrate that tax processing systems would perform correctly on or after January 1, 2000. The Y2K E2E Systems Integration Test was controlled by the IRS’ Information Systems Product Assurance function, with significant assistance from contractors. The IRS planned to complete its E2E Systems Integration Test of all mission critical systems by late December 1999.

The E2E Systems Integration Test consisted of three segments. The overall objective of this review was to assess the effectiveness of the preparation for and execution of the last segment, Test 3, of the IRS’ Y2K E2E Systems Integration Test. Execution of Test 3 was divided into two separate tests (Test 3A and Test 3B), whose combined purpose was to perform an assessment of the effects on tax processing when system dates were advanced to and beyond January 1, 2000. This report presents our initial assessment of Test 3.

In planning and executing the E2E Systems Integration Test, the Century Date Change (CDC) Testing Program Office was responsible for creating the first E2E systems integration test of the IRS’ mission critical tax processing systems. The systems integration test was designed to assure that taxpayer information will be processed correctly through all the paths in the IRS’ production computer environment. The CDC Testing Program Office built an extensive test environment, separate from the IRS’ production systems. The test environment substantially replicated the IRS’ tax processing environment and included the hardware and telecommunications infrastructure used to support major IRS systems such as those processing tax returns, providing on-line access to taxpayer accounts, and allowing electronic filing. This test environment allowed the IRS to test the effects of advancing the systems clocks to the year 2000 on a large variety of hardware and telecommunications equipment. The CDC Testing Program Office also performed a wide-ranging analysis of the IRS’ computer programs and systems to identify the specific tests needed for the integration test.

The Product Assurance function managed the E2E Systems Integration Test by holding daily and weekly meetings to report on and discuss the issues and test status. Through these meetings, the CDC Testing Program Manager resolved many of the day-to-day problems that affected the running of the test. In addition, the Product Assurance function used a contractor to conduct an independent review of the E2E Systems Integration Test materials to determine whether the test data and test cases were complete and accurate. However, due to continuing delays in completing planned work and other factors, necessary testing was at risk of not being completed by December 31, 1999, as scheduled.

Although the final phase of the E2E Systems Integration Test, Test 3B, began on October 4, 1999, the IRS had still not finished identifying all the specific E2E Systems Integration Tests at the beginning of November 1999. Without finalized tests and related test cases, the IRS would not have had an objective method for measuring the completeness and success of Test 3B.

As of November 1, 1999, there were a considerable number of tasks that the IRS still had to perform before the E2E Systems Integration Test data could be considered complete. These tasks included:

• Developing test cases for a large number of specific tests and for recently identified interfaces for processing data between computer systems and applications.
• Assuring that the quality review of all test cases had been completed.
• Verifying that test data were not inadvertently deleted from the E2E Systems Integration Test.

There was a risk that the IRS would not successfully complete a full E2E Systems Integration Test of its mission critical systems by December 31, 1999. Failure to complete the E2E Systems Integration Test could have resulted in the IRS experiencing delays and errors in its computer processing on and after January 1, 2000. There were several issues that contributed to this risk.

• According to a contractor’s report, dated October 29, 1999, 29 percent of the tests had only 1 test case. From a statistical standpoint, a single test case provides a zero percent confidence level that an item has been successfully tested. The IRS may not have a valid basis for ascertaining whether the systems represented by these specific tests have been adequately tested in a year 2000 environment before they are placed into production.
• There was no documentation showing that the IRS used a consistent methodology to develop the specific tests (the IRS uses the term ‘threads’ to refer to specific tests) or test cases.
• As of November 10, 1999, there were 52 issues with the test data that still needed to be resolved before the beginning of testing that began on November 13, 1999.
• An IRS contractor delivered a report on its review of the adequacy of the test cases on December 3, 1999. The issues identified in that report had little time for correction.
• All Systems Acceptability Testing (SAT) should have been completed before the beginning of Test 3B, the final phase of Test 3. At the beginning of the last phase of the E2E Systems Integration Test, SAT had not been completed for six mission critical IRS systems. As of November 5, 1999, four of these systems had still not completed SAT.

The E2E Systems Integration Test also encountered other problems, including frequent instances of misplaced tapes with data files needed for testing, incorrect versions of application programs being sent to the test sites, incorrect tape labeling, and the need to repeat the running of computer programs. In addition, during the first pass of the E2E Systems Integration Test, not all programs were tested as planned.

Although the CDC Project Office had identified the completion of the E2E Systems Integration Test as a major risk, we were concerned that the cumulative effects of these issues had not received appropriate attention.

Due to the time sensitivity of the E2E Systems Integration Test, we discussed these issues with management during our review. When we presented our concerns to Product Assurance management, the CDC Testing Program Manager stated that the E2E System Integration Test would be completed on December 17, 1999.

After we provided a briefing on our concerns to Product Assurance management, they presented us with the analyses and actions they had undertaken to address our issues. We did not evaluate the actions the IRS had taken or to what extent the IRS’ actions reduced the risks identified in this report, due to the limited number of days remaining before the year 2000 at the time we surfaced our concerns.

The Chief Information Officer (CIO) needs to assess whether all risks were adequately addressed. In addition, the CIO should assure that the strategy to deal with Y2K issues after January 1, 2000, includes any E2E Systems Integration Test issues that could not be adequately addressed before December 31, 1999.

Management’s Response: A draft of this report dated February 28, 2000, was sent to IRS management for comment. As of the date of this final report, the IRS was in the process of developing a response outlining the actions it has taken, or will take, to address the issues in the report.