This report presents the results of our review of the effectiveness of the preparation for and execution of the last segment of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) End-to-End (E2E) Systems Integration Test. In summary, we determined that the IRS needed to identify all specific tests required for the E2E Systems Integration Test and address the risk that a full E2E Systems Integration Test might not be successfully completed by December 1999. Due to the time sensitivity of the E2E Systems Integration Test, we discussed these issues with management as they were identified. After we provided our concerns to IRS management, they presented us with the analyses and actions they had undertaken to address our issues. We did not evaluate the actions the IRS had taken or to what extent the IRS’ actions reduced the risks identified in this report, due to the limited number of days remaining before the year 2000, at the time we surfaced our concerns.

We recommended that the Chief Information Officer (CIO) assess whether all risks were adequately addressed. In addition, the CIO should assure that the strategy to deal with Y2K issues after January 1, 2000, includes any E2E Systems Integration Test issues that could not be adequately addressed before December 31, 1999.

Executive Summary

Objective and Scope

Background

Results

Conclusion

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Executive Summary

The primary goal of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) End-to-End (E2E) Systems Integration Test was to demonstrate that tax processing systems would perform correctly on or after January 1, 2000. The Y2K E2E Systems Integration Test was controlled by the IRS’ Information Systems Product Assurance function, with significant assistance from contractors. The IRS planned to complete its E2E Systems Integration Test of all mission critical systems by late December 1999.

The E2E Systems Integration Test consisted of three segments. The overall objective of this review was to assess the effectiveness of the preparation for and execution of the last segment, Test 3, of the IRS’ Y2K E2E Systems Integration Test. Execution of Test 3 was divided into two separate tests (Test 3A and Test 3B), whose combined purpose was to perform an assessment of the effects on tax processing when system dates were advanced to and beyond January 1, 2000. This report presents our initial assessment of Test 3.

In planning and executing the E2E Systems Integration Test, the Century Date Change (CDC) Testing Program Office was responsible for creating the first E2E systems integration test of the IRS’ mission critical tax processing systems. The systems integration test was designed to assure that taxpayer information will be processed correctly through all the paths in the IRS’ production computer environment. The CDC Testing Program Office built an extensive test environment, separate from the IRS’ production systems. The test environment substantially replicated the IRS’ tax processing environment and included the hardware and telecommunications infrastructure used to support major IRS systems such as those processing tax returns, providing on-line access to taxpayer accounts, and allowing electronic filing. This test environment allowed the IRS to test the effects of advancing the systems clocks to the year 2000 on a large variety of hardware and telecommunications equipment. The CDC Testing Program Office also performed a wide-ranging analysis of the IRS’ computer programs and systems to identify the specific tests needed for the integration test.

The Product Assurance function managed the E2E Systems Integration Test by holding daily and weekly meetings to report on and discuss the issues and test status. Through these meetings, the CDC Testing Program Manager resolved many of the day-to-day problems that affected the running of the test. In addition, the Product Assurance function used a contractor to conduct an independent review of the E2E Systems Integration Test materials to determine whether the test data and test cases were complete and accurate. However, due to continuing delays in completing planned work and other factors, necessary testing was at risk of not being completed by December 31, 1999, as scheduled.

Although the final phase of the E2E Systems Integration Test, Test 3B, began on October 4, 1999, the IRS had still not finished identifying all the specific E2E Systems Integration Tests at the beginning of November 1999. Without finalized tests and related test cases, the IRS would not have had an objective method for measuring the completeness and success of Test 3B.

As of November 1, 1999, there were a considerable number of tasks that the IRS still had to perform before the E2E Systems Integration Test data could be considered complete. These tasks included:

• Developing test cases for a large number of specific tests and for recently identified interfaces for processing data between computer systems and applications.
• Assuring that the quality review of all test cases had been completed.
• Verifying that test data were not inadvertently deleted from the E2E Systems Integration Test.

There was a risk that the IRS would not successfully complete a full E2E Systems Integration Test of its mission critical systems by December 31, 1999. Failure to complete the E2E Systems Integration Test could have resulted in the IRS experiencing delays and errors in its computer processing on and after January 1, 2000. There were several issues that contributed to this risk.

• According to a contractor’s report, dated October 29, 1999, 29 percent of the tests had only 1 test case. From a statistical standpoint, a single test case provides a zero percent confidence level that an item has been successfully tested. The IRS may not have a valid basis for ascertaining whether the systems represented by these specific tests have been adequately tested in a year 2000 environment before they are placed into production.
• There was no documentation showing that the IRS used a consistent methodology to develop the specific tests (the IRS uses the term ‘threads’ to refer to specific tests) or test cases.
• As of November 10, 1999, there were 52 issues with the test data that still needed to be resolved before the beginning of testing that began on November 13, 1999.
• An IRS contractor delivered a report on its review of the adequacy of the test cases on December 3, 1999. The issues identified in that report had little time for correction.
• All Systems Acceptability Testing (SAT) should have been completed before the beginning of Test 3B, the final phase of Test 3. At the beginning of the last phase of the E2E Systems Integration Test, SAT had not been completed for six mission critical IRS systems. As of November 5, 1999, four of these systems had still not completed SAT.

The E2E Systems Integration Test also encountered other problems, including frequent instances of misplaced tapes with data files needed for testing, incorrect versions of application programs being sent to the test sites, incorrect tape labeling, and the need to repeat the running of computer programs. In addition, during the first pass of the E2E Systems Integration Test, not all programs were tested as planned.

Although the CDC Project Office had identified the completion of the E2E Systems Integration Test as a major risk, we were concerned that the cumulative effects of these issues had not received appropriate attention.

Due to the time sensitivity of the E2E Systems Integration Test, we discussed these issues with management during our review. When we presented our concerns to Product Assurance management, the CDC Testing Program Manager stated that the E2E System Integration Test would be completed on December 17, 1999.

After we provided a briefing on our concerns to Product Assurance management, they presented us with the analyses and actions they had undertaken to address our issues. We did not evaluate the actions the IRS had taken or to what extent the IRS’ actions reduced the risks identified in this report, due to the limited number of days remaining before the year 2000 at the time we surfaced our concerns.

The Chief Information Officer (CIO) needs to assess whether all risks were adequately addressed. In addition, the CIO should assure that the strategy to deal with Y2K issues after January 1, 2000, includes any E2E Systems Integration Test issues that could not be adequately addressed before December 31, 1999.

Management’s Response: A draft of this report dated February 28, 2000, was sent to IRS management for comment. As of the date of this final report, the IRS was in the process of developing a response outlining the actions it has taken, or will take, to address the issues in the report.

Objective and Scope

This report presents the results of our on-line review of the Internal Revenue Service’s (IRS) preparation for the continuing operations of its computer systems in the year 2000 (Y2K). The overall objective of this review was to assess the effectiveness of the preparation for and execution of Test 3 of the IRS’ Y2K End-to-End (E2E) Systems Integration Test.

We evaluated the methodology of the testing approach and the adequacy of the tests conducted to assure the accurate operation of the IRS’ computer systems and programs. Also, as part of this review, we monitored the advancing of system and software internal clocks to January 1, 2000, and beyond. The final phase of the E2E Systems Integration Test, which was designed to test the IRS’ mission critical systems using year 2000 dates and applications, began on October 4, 1999, and was scheduled for completion no later than December 18, 1999.

We conducted this review within the Office of the Chief Information Officer (CIO), including work at the Tennessee Computing Center, the Detroit Computing Center, and the Integration, Test and Control Center in New Carrollton, Maryland. We conducted the audit work from September through December 1999. The audit was performed in accordance with Government Auditing Standards.

Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

A memorandum detailing the findings listed in this report was sent to IRS management December 20, 1999, to allow management the opportunity to address the issues prior January 1, 2000. A draft report was sent to management for comment on February 28, 2000, with a response requested in 30 calendar days. As of the date of this final report, the IRS was in the process of developing a response outlining the actions it has taken, or will take, to address the issues in the report.

Background

The IRS’ preparation for year 2000 tax processing was one of its most critical undertakings. As the final step in its Y2K readiness efforts, the IRS planned to verify the accurate operation of all programs and equipment that would be used on and after January 1, 2000, to process approximately 213 million tax returns, issue 94 million refunds, answer 120 million calls for taxpayer assistance, and collect nearly $1.8 trillion. Failure to identify, re-code, and test each date-based system could have corrupted taxpayer data and resulted in the generation of millions of erroneous notices, refunds, bills, interest calculations, and taxpayer account adjustments.

To confirm system and tax processing readiness for the year 2000, the Information Systems Product Assurance function, along with significant contractor support, developed a series of E2E systems integration tests. The primary goal of the E2E Systems Integration Test was to demonstrate that tax processing systems would perform correctly on or after January 1, 2000. The E2E Systems Integration Test consisted of three segments. Execution of the last segment, Test 3, consisted of two parts: Test 3A and Test 3B. Test 3A began on April 12, 1999, and ended July 23, 1999, while Test 3B began on October 4, 1999, and was scheduled to be completed in late December 1999.

Test 3A was conducted to perform an initial assessment of the effects on tax processing systems and applications when internal clocks are advanced to the year 2000 and beyond. Test 3B, the final step in the E2E Systems Integration Test, was performed to prove that:

• All mission critical tax processing programs performed correctly when system dates are on or after January 1, 2000.
• The telecommunications infrastructure supported advanced date processing.
• Application systems and platforms, including systems software, hardware, operating systems, and commercial-off-the-shelf products, interfaced correctly into and beyond the year 2000.

One of the major E2E Systems Integration Test challenges was defining tests to assure that taxpayer information would be processed correctly through all the paths in the IRS’ production computer environment. To meet this challenge, the IRS developed a model that breaks down high-level test threads into scenarios and scenarios into test cases. The data input to meet the criteria for specific scenarios constitutes a test case. Over 2,000 scenarios were defined. After the scenarios were completed, research was conducted to identify one or more test cases to satisfy each scenario. In total, 6,961 test cases were identified to provide full testing coverage of the IRS’ mission critical systems. By using sets of test data with pre-determined test results, the IRS sought to ensure correct performance of tax processing hardware and systems applications for the year 2000 and beyond.

Results

In planning and executing the E2E Systems Integration Test, the Century Date Change (CDC) Testing Program Office was responsible for creating the first system integration test of the IRS’ mission critical tax processing systems. The CDC Testing Program Office built an extensive test environment, separate from the IRS’ production systems. The environment substantially replicated the IRS’ tax processing environment and included the hardware and telecommunications infrastructure supporting the major IRS systems for processing tax returns, providing on-line access to taxpayer accounts, and allowing electronic filing. This test environment allowed the IRS to test the effects of advancing the systems clocks on a large variety of hardware and telecommunications equipment.

The CDC Testing Program Office also performed a wide-ranging analysis of the IRS’ computer programs and systems to identify the specific tests needed for the integration test. The first phase of Test 3 was successful in demonstrating that the internal clocks on all mission critical systems’ hardware could be set forward and that the hardware environments were able to operate with dates set to the year 2000.

The Information Systems Product Assurance function managed the E2E Systems Integration Test by holding daily and weekly meetings to report on and discuss issues and test status. Through these meetings, the CDC Testing Program Manager resolved many of the day-to-day problems that affected the running of the test. In addition, the Product Assurance function used a contractor to conduct an independent review of the E2E Systems Integration Test materials to determine whether the test data and test cases were complete and accurate.

Although the IRS’ primary administrative system, the Automated Financial System (AFS), is a mission critical system, the IRS decided to exclude it from the IRS-wide E2E Systems Integration Test. We determined that this decision was reasonable and that the IRS provided adequate oversight of the AFS contractor’s E2E Systems Integration Test. Furthermore, the exclusion of the AFS from the IRS-wide E2E Systems Integration Test did not pose a significant risk to the IRS’ tax processing systems.

Our audit work did, however, identify areas that needed management’s attention. The Chief Information Officer (CIO) needed to:

• Identify all specific tests required for the E2E Systems Integration Test.
• Address the risk that a full E2E Systems Integration Test may not have been completed by December 1999.

Due to the time sensitivity of the E2E Systems Integration Test, we discussed these issues with management during our review. After we provided our concerns to Product Assurance’s management, they presented us with the analyses and actions they had undertaken to address our issues. We did not evaluate these efforts and did not determine to what extent the IRS’ actions reduced the risks identified in this report, due to the limited number of days remaining before the year 2000 at the time we surfaced our concerns.

The Internal Revenue Service Needed to Finish Identifying All the Specific Tests Required for the End-to-End Systems Integration Test

Although the E2E Systems Integration Test 3B began on October 4, 1999, the IRS had not finished identifying all the specific tests required for the E2E Systems Integration Test. A key element of any project is the establishment of a baseline. According to the IRS’ documentation, a baseline is a set of data that has been formally reviewed, agreed to, and documented, and thereafter, serves as a basis for managing changes to the base set of data. As of November 1, 1999, the IRS had not established a baseline for all the specific tests for the E2E Systems Integration Test and did not have a process for controlling changes to the threads and test cases that constitute the test elements. Without finalized threads and test cases and a controlled baseline, the IRS would not have had an objective baseline for measuring the completeness and success of the E2E Systems Integration Test 3B.

As of November 1, 1999, there were a considerable number of tasks that the IRS needed to perform before the E2E Systems Integration Test data could be considered complete:

• In a report dated August 13, 1999, a contractor hired by the CDC Testing Program Office to conduct an Independent Validation & Verification (IV&V) of Test 3A identified 336 threads without test cases. In a subsequent report by the same contractor, dated October 29, 1999, the number of threads without test cases had increased to 362. The latter report includes 255 (70 percent) of the threads from the prior report, thus indicating that little action was taken to verify which threads were still needed and how many test cases still needed to be developed after the issuance of the first report.
• In addition to the 362 threads without test cases, the IV&V report identified 64 system interfaces without threads. The IRS also needed to develop tests and test cases for these interfaces to assure that its E2E Systems Integration Test would be adequately executed.
• The contractor had a significant number of test cases to review. The October 29, 1999, IV&V report indicates the IRS’ contractor was able to review only 73 percent of the test cases because the IRS had indicated that the other test cases (27 percent) were not ready for review.
• The IRS maintains databases for all the E2E threads and test cases. We determined that there might be threads that were inadvertently deleted and, therefore, would not be tested. For example, we attempted to determine the status of 100 consecutive numbers in the Test 3A and Test 3B master thread databases and found that 32 numbers were missing from the Test 3A master thread database. Of these, 30 of the same numbers were missing from the Test 3B database. In addition, eight other numbers were missing from the Test 3B database that were present in Test 3A.

Since there was no documentation showing why the numbers are missing, we were concerned that missing threads and related test case numbers may have been accidentally deleted. For example, we originally determined that there was no thread for number 128 in either the Test 3A database or Test 3B database and questioned the contractors assigned to develop this thread about this situation. The contractors advised us that the thread dealt with an on-line application (called AMREQ) that allows an employee to request that the information in the IRS’ on-line database be properly synchronized with the IRS’ Masterfile. They were unable to explain what happened to this thread and surmised that it had been deleted. Subsequently, this thread was reinstated.

The CDC Testing Program Office used a contractor to provide an independent assessment of the threads and test cases. The CDC Testing Program Office decided not to take action on at least two of the recommendations made in the August 1999 IV&V report. The CDC Testing Program Manager informed us that, at the time of the IV&V report, the Program Office did not take action on the recommendations because it was still working on changing test data for year 2000 dates. The Program Manager stated that, as part of this conversion effort, the work was prioritized so that test cases were developed according to the sequence of their use in the E2E Systems Integration Test. In addition, one of the contractors responsible for developing threads provided us with a possible reason for the large number of threads without test cases on the test case database. The contractor informed us that, because of database problems, it was not possible in October 1999 to identify threads as either active or inactive on the thread database. Since it was not possible to list a thread’s status, the only options for dealing with an inactive thread would have been to delete it completely from the database or leave it in the database.

The same contractor also advised the CDC Testing Program Office in August 1999 that a baseline was needed to deal with fluctuations in the number of test cases, and a consistent and accurate method of generating test case totals should be adopted. The CDC Testing Program Manager felt that fluctuations were normal because of the dynamic nature of the E2E Systems Integration Test and that the purpose of their baselining was to keep track of the transmittals of programs to the test environment. As a result, no schedule was established for verifying that the threads without test cases were, in fact, inactive threads, nor was any action undertaken to establish a baseline of test cases.

After the subsequent IV&V report in late October 1999 continued to identify the problem of threads without test cases, the CDC Testing Program Office directed the contractors to review the threads without test cases and determine whether these threads were inactive or were still needed for the E2E Systems Integration Test. For those threads that were deemed necessary for the E2E Systems Integration Test, the IRS would have to develop test cases. The contractors were still performing this work in November 1999, which could have caused delays in testing. As of November 5, 1999, it was uncertain to us how many test threads needed to be completed, how many test cases needed to be developed, and whether there was a baseline of test cases.

Due to the time sensitivity of the E2E Systems Integration Test, we discussed these issues with management during our review. We recommended that Product Assurance management take the following actions:

• Set up a schedule for completing work on the threads and test cases.
• Verify that the actions to deactivate or delete the threads were appropriate.
• Assure that test cases are completed for all the active threads.

After we provided a briefing on our concerns to Product Assurance management, they presented us with the analyses and actions they had undertaken to address our issues, including:

• A list of inactive threads and the reasons they were inactive.
• A list of deleted threads and the reason they were deleted.
• A list of open issues as of November 23, 1999, and when they anticipated a resolution.
• A list of threads with a single test case and explanations for threads with only one test case.

We did not evaluate these efforts and did not determine to what extent the IRS’ actions reduced the risks identified in this report, due to the limited number of days remaining before the year 2000 at the time we surfaced our concerns.

There Was a Risk That a Full End-to-End Systems Integration Test Would Not Be Successfully Completed by December 31, 1999

There was a risk that the IRS would not successfully complete Test 3B, a full E2E Systems Integration Test of its mission critical systems, by December 31, 1999. Failure to complete the E2E Systems Integration Test could have resulted in the IRS experiencing delays and errors in its computer processing on and after January 1, 2000. There were several issues that contributed to this risk:

• The IRS had initially planned for a second running of the E2E Systems Integration Test, called a regression test, for those systems that were not ready to be tested in the initial pass of Test 3B, or had problems in the initial running of the test. The time allotted to the regression phase was increased after the original planning for the E2E Systems Integration Test to accommodate systems that were not ready at the beginning of Test 3B. When the regression testing began on November 13, 1999, the CDC Testing Program Office decided to suspend the initial running of Test 3B for the systems that had not yet been tested or had not completed the initial pass of Test 3B. The testing for these systems was then added to the regression test. These changes to the IRS’ original plan for the E2E Systems Integration Test made the time frames for completing the E2E Systems Integration Test very tight.
• According to a contractor’s IV&V report, dated October 29, 1999, 29 percent of the threads had only 1 test case. From a statistical standpoint, a single test case provides a zero percent confidence level that an item has been successfully tested. The successful processing of only one case could be the result of something other than the correct running of the test case. Therefore, the IRS may not have a valid basis for ascertaining whether these systems were adequately tested in a year 2000 environment before they were placed into production.
• There was no documentation showing that the IRS used a consistent methodology to develop the threads or test cases. Of the three parties tasked with thread development, the Product Assurance function provided us with one contractor’s high-level overview, but this documentation did not include specific details on the thread development. Since there was no consistent, documented process for developing threads, it would have been difficult for the IRS to determine whether problems encountered during Test 3B were the result of thread development errors or software development errors. This deficiency could also have added further delays to the difficult task of diagnosing and correcting problems during the final running of Test 3B.
• As of November 10, 1999, there were 52 issues with the test data that still needed to be resolved. These issues, which were identified by an IRS contractor, should have been fully resolved before the beginning of the regression test. Since the regression test began on November 13, 1999, it was unclear to us at the time how the regression testing could be successful unless these issues were resolved.
• An IRS contractor delivered a report on its review of the adequacy of the test cases on December 3, 1999. The issues identified in that report had little time for correction.
• All Systems Acceptability Testing (SAT) should have been completed before the beginning of Test 3B, the last phase of the E2E Systems Integration Test. At the beginning of Test 3B, there were six mission critical IRS systems whose SAT was not complete, including the Notice Review Processing System - Online Review Notice Retype, Automated Underreporter (AUR), Integrated Submission and Remittance Processing, Lockbox Checks/Balances, Tax Return Database, and Telephone Routing Interactive System - Spanish version. As of November 5, 1999, only two of these systems, AUR and Lockbox Checks/Balances, had completed SAT.
• Tapes with test files were misplaced on several occasions, resulting in delays in testing. Problems occurred, in part, because packages of tapes are received as part of a mass shipment. Initially, the individual packages are not verified to the delivery listing when the transport agent makes its delivery to large IRS sites such as the Tennessee Computing Center. This same problem may exist at other IRS locations, such as the New Carrollton Federal Building, where they have experienced similar difficulties with package receipts.
• There were frequent testing delays because incorrect versions of programs were sent to the E2E Systems Integration Test environment (testbed). The CDC Testing Program Office identified 31 instances in which incorrect versions of programs were found on the testbed after October 6, 1999. The IRS had an open management response item from a previous Treasury Inspector General for Tax Administration report, The Internal Revenue Service Can Improve Management Controls Over the Year 2000 End-to-End Systems Integration Test (Reference Number 200020008, dated November 1999) to assure that the software and hardware residing on the E2E Systems Integration Test environment are the same hardware and software that will be implemented into production. The IRS’ management response to our earlier report did not foresee the issue of version control on the E2E Systems Integration Testbed. Inadequate version controls could have resulted in untested versions being released into production.
• Recurring issues added to the time needed to conduct testing. For instance, continued delays were experienced because of incorrect tape labeling, including errors with both the external, physical label of the tape and the internal label that the system uses at the beginning of a computer program to ensure that the correct data are about to be processed. There was also the need to repeat the running of computer programs because of errors in specifying the correct type of computer tape for input or output. The cumulative effect of these delays added to the risk that the E2E Systems Integration Test would not have been completed by December 31, 1999.

The CDC Project Office established a risk process to identify, categorize, prioritize, and monitor risks. These risks were rated in priority. Only level one priority risks were reported to the Combined Management Program for Century Date Change and Filing Season Executive Steering Committee, chaired by the IRS Commissioner. The CDC Project Office had identified the completion of the E2E Systems Integration Test as a major risk area, but the issues outlined above were not included in the CDC Project Office’s risk statement.

We were concerned that IRS management had not considered the cumulative effect of the factors impeding the completion of the E2E Systems Integration Test. Delays in the above areas could have adversely affected the completion of the regression test and could have resulted in the E2E Systems Integration Test not being fully completed by December 31, 1999.

Due to the time sensitivity of the E2E Systems Integration Test, we discussed these issues with management during our review. After we presented our concerns to Product Assurance management, the CDC Testing Program Manager stated that E2E regression testing, which was the final phase of the E2E Systems Integration Test, would be completed on December 17, 1999.

Product Assurance management informed us verbally that they had taken several actions to reduce the risk presented by the above issues, including:

• Obtaining a commitment from the IRS system developers to exercise additional care when transmitting programs to the testbed.
• Directing the contractors responsible for developing test cases to respond in writing to the IV&V reports.
• Preparing a standard operating procedure for contractors and IRS personnel to use when developing test cases.
• Assessing the SAT status of projects with incomplete SAT testing and determining which would be available for the E2E Systems Integration Test.

As a part of the audit work supporting this report, we did not evaluate the actions the IRS took or to what extent the IRS’ actions reduced the risks identified in this report. The IRS still needs to assess whether these actions have been successful in reducing the risk that some mission critical systems may not have been sufficiently tested during the E2E Systems Integration Test.

1. The CIO needs to assess whether all the identified risk factors have been adequately addressed.
2. The CIO should assure that the strategy for dealing with Y2K issues after January 1, 2000, includes any E2E Systems Integration Test issues that could not be adequately addressed before December 31, 1999.

Conclusion

The E2E Systems Integration Test effort has been a major undertaking in the overall endeavor to prepare the IRS for continued operations in the next century. The IRS needs to ensure that all critical systems are tested and functioning properly before entering into production.

Detailed Objective, Scope, and Methodology

The overall objective of this review was to assess the effectiveness of the preparation for and execution of Test 3 of the Internal Revenue Service’s (IRS) End-to-End (E2E) Systems Integration Test for year 2000 tax processing. To complete this objective, we:

1. Assessed the adequacy of the thread coverage, through the review of the thread development process for Test 3.
1. Reviewed the methodology/criteria for thread identification and prepared a model for describing the thread development process.
2. Determined whether a sufficient number of threads were identified and developed to adequately test system interfaces.
3. Analyzed a judgmental sample of 77 of the developed threads.
4. Reviewed the thread selection process to determine if any key IRS operational functions were omitted and assessed the risk of not including these in the E2E Systems Integration Test.
2. Evaluated whether the E2E Systems Integration Test cases adequately addressed the IRS’ ability to process transactions on and after January 1, 2000.
1. Attended or monitored the E2E Systems Integration Test status or progress meetings to obtain testing information, identify significant problems/concerns, and determine if these concerns were being timely resolved.
2. Determined if an adequate number of test cases were developed to ensure that each thread tested the IRS’ transaction processing.
3. Determined the sufficiency of threads and test cases.
4. Reviewed test activity and documentation on each system selected for the E2E Systems Integration Test.
5. Assessed the impact on testing resources (e.g., test platform capacity, staffing for testing, etc.) in adding systems to the E2E Systems Integration Test.
3. Assessed the impact on the overall testing effort when intermediate test-related tasks and tests were not completed by projected deadlines.
1. Determined whether any slippage from the IRS’ efforts to make its systems, hardware, and/or software Y2K compliant adversely affected the completion of Test 3.
2. Determined the worst case scenario in the event of missed deadlines.
3. Determined whether deadlines for completing test cases were being met and evaluated the effects of any slippage.
4. Evaluated the test planning process by reviewing tax processing systems, the testing approach for scenario completeness and appropriateness, and the test plan documentation for completeness, supervisory review, acceptance, and approvals.
5. Assessed the readiness for the test execution processes by visiting test sites, interviewing personnel, and monitoring the teleconferences used for testing.

4. Evaluated the effects of not including the IRS’ Automated Financial System (AFS) in the E2E Systems Integration Test.

1. Interviewed Product Assurance and Information Systems management to determine why the IRS did not include the AFS in the E2E Systems Integration Test.
2. Determined the impact/risk on the IRS’ year 2000 tax processing.

Major Contributors to This Report

Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)

Gary Hinkle, Director

Vincent Dell’Orto, Audit Manager

Barbara Sailhamer, Acting Senior Auditor

Mark Carder, Auditor

Steve Gibson, Auditor

Linda Screws, Auditor

Report Distribution List

Deputy Commissioner Modernization C:DM

Deputy Commissioner Operations C:DO

Chief Information Officer IS

Deputy Chief Information Officer (Operations) IS

Deputy Chief Information Officer (Systems) IS

Director, Century Date Change Project Office IS:CD

Director, Information Resources Management IS:IR

Director, Office of Program Evaluation and Risk Analysis M:O

Director, Product Assurance IS:PA

National Director for Legislative Affairs CL:LA

Office of the Chief Counsel CC

Office of Management Controls M:CFO:A:M

Audit Liaison, Office of Program Oversight IS:IR:Q