Objective and Scope

This report presents the results of our on-line review of the Internal Revenue Service’s (IRS) Year 2000

End-to-End (E2E) System Integration Test. The objective of this review was to assess the IRS’ planning efforts for the E2E System Integration Test II and Test III, and the execution of Test II.

Planning for this review started in November 1998 and fieldwork was completed in May 1999. As part of this review, we observed the execution of Test II, which was a limited testing of approximately 69 tax processing systems in an integrated environment. Test II was conducted in two segments, the first of which started October 19, 1998, and ended December 18, 1998. The second segment started on January 18, 1999, and ended on April 27, 1999. We also reviewed planning efforts for Test III during which 94 systems will be tested to ensure that data processed through the IRS’ tax processing systems perform correctly on and after January 1, 2000. Test III will be conducted from February 1, 1999, through December 17, 1999, with the testing of the year 2000 production programs commencing in October 1999.

We conducted this review within the office of the Chief Information Officer. Audit tests and observations were conducted at the Martinsburg Computing Center (MCC), the Tennessee Computing Center (TCC) and the Integration, Test and Control Center (ITCC) in New Carrollton, Maryland. The audit was performed in accordance with Government Auditing Standards.

Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

Background

The IRS’ Year 2000 E2E Systems Integration Test is being controlled by the Information Systems Product Assurance function, with significant levels of contractor support. The primary goal of the Year 2000 E2E Systems Integration Test is to create a testbed that replicates the IRS’ tax processing environment, including the processing of tax returns, providing on-line access to taxpayer accounts, electronic tax filing, and the supporting hardware and telecommunications infrastructure. Planning for the E2E Systems Integration Test began in May 1998, and testing will run to December 1999. The test is ultimately intended to prove that tax processing applications will perform correctly on or after January 1, 2000.

Results

During our monitoring of the Test II execution and

Test III planning, we found that the Product Assurance test team, the contractors supporting the IRS’ E2E testing efforts, and the Century Date Change (CDC) Project Office were working diligently to reduce problems with the testbed and to coordinate large numbers of resources in various sites to deliver the testing effort. However, we identified the following issues that needed to be addressed:

We initially reported these findings and recommendations in a memorandum dated March 8, 1999. The original memorandum is included as Appendix IV. The IRS provided a written response on April 12, 1999. The response is included as Appendix V.

Product Assurance Should Perform Quality Checks of the Contractors’ Deliverables to Assure They Meet the Internal Revenue Service’s Intended Objectives

The monitoring and maintenance of the E2E Systems Integration Test testbed are integral to the successful completion of the E2E test and subsequent transition of the testbed components to production late in 1999. The hardware and software tested in the E2E testbed will establish the baseline configuration that will be used in production for the 2000 filing season.

The IRS is heavily dependent on contractor support for successful completion of its E2E testing efforts. Contractor developed inventory deliverables for the E2E Systems Integration Test testbed were not consistent. In requesting documentation of the E2E testbed inventory of equipment and software, we received two sets of information from different contractor sources. Our review of these sources showed they did not match.

The contractor was not aware of the differences. After we identified the discrepancies, the contractor initiated an effort to resolve and/or justify differences and to develop one finalized inventory for use by Product Assurance.

In addition, the contractor had oversight responsibility for certifying the baseline of hardware and software that constitute the testbed for E2E Test II. After the completion of the first segment of Test II, IRS officials were sent packages containing inventory items for which they were responsible. The responsible IRS officials were required to certify these packages and then return them to the contractor. This effort was not completed prior to the start of Test II, since the IRS has a relatively low number of employees assigned to the E2E effort in relation to the number of contractors, making it difficult to review the large number of contractor deliverables. In addition, the fast-paced schedule of the E2E Systems Integration Test has contributed to the difficulty in timely reviewing and analyzing all contractor products.

In addition, regular quality reviews help ensure that the IRS is obtaining useful products from its contractors. Without the quality reviews, IRS management cannot be assured that the information provided by the contractors is useful and pertinent and advances the goals of the project.

1. Product Assurance management should provide for a quality review of all contractor deliverables and assure the work completed by those contractors is consistent with the objectives of the E2E Systems Integration Test.

Management’s Response: Product Assurance management has taken action by appointing a team of IRS employees to monitor the activities of the contractors. Tasks performed by the contractors are monitored via multiple weekly and biweekly meetings. In addition, the E2E Test Program Manager personally reviews each deliverable before acceptance.

Configuration Management Controls Over the Year 2000 End-to-End Systems Integration Test Platform Should Be Improved

Configuration management is an important component of any testing, development, or production system. The objective of configuration management is to identify the components (baselines) of a development or maintenance effort and to control changes to those baselines to ensure traceability throughout the usable life of the system. The first step in configuration management activities is identification and documentation. This step establishes a baseline from which changes to the configuration are managed or controlled.

A strong configuration management process, as part of the E2E Systems Integration effort, will make identification of problems with the vast hardware and software platforms easier. Also, a baselined, documented and controlled testbed will be needed for the transition from the testbed to production in December 1999.

One objective of configuration management is to account for all information technology components and provide a basis for sound change management. No single organization in the IRS has control over the configuration management of the E2E Systems Integration Test testbed. Portions of the E2E testbed have not been inventoried and only fragmented efforts for documenting the testbed were identified.

Requests for inventory information on the E2E testbed revealed two of the four sites used for the E2E Systems Integration Test were not inventoried. The testing facilities in Oxon Hill, Maryland, and the ITCC in the New Carrollton Federal Building were not inventoried at the time of our review. Contractor personnel explained that these sites had lower priority and, at the time, their inventory efforts were focused on the larger portions of the testbed in the MCC and the TCC.

Various factors have contributed to the absence of centralized configuration management of the E2E testbed. Initially, TRW, Inc. (the Integration Support Contractor) was tasked with configuration management of the E2E testing effort. This responsibility was eliminated from TRW’s responsibility early in the E2E testing effort. No single organization within the IRS has assumed responsibility for this activity. Our efforts to identify a configuration management control point yielded multiple organizations assuming various levels of responsibility, and using multiple configuration management tools.

Without a complete and accurate inventory of the hardware and software to be included in the E2E test, there can be no assurance that all systems will perform adequately after January 1, 2000.

Recommendations

2. IRS management should assign responsibility for configuration management of the E2E testbed to one controlling organization. This organization should adhere to prescribed configuration management criteria and document and control all hardware and software used as part of the E2E testing effort.

Management’s Response: IRS management agreed to have the CDC Project Office assess the ability of the IRS to ensure that the hardware and software residing on the E2E testbed are, in fact, the same hardware and software that will be implemented in the production environment. At the completion of the assessment, if the CDC Project Office has been able to determine a means by which it can ensure the likeness of the testbeds and the production environment, then the first iteration of the test will occur. Otherwise, they will recommend and implement an alternative strategy to minimize the risk.

3. IRS management should ensure a complete inventory of the entire E2E testbed is completed prior to the completion of additional tests.

Management’s Response: IRS management agreed to ensure a complete inventory of the entire E2E testbed is completed prior to any additional tests.

Conclusion

During our monitoring of the Test II execution and Test III planning, we found that the Product Assurance test team and its contractor support were working diligently to reduce problems with the testbed and to coordinate large numbers of resources in various sites to deliver the testing effort. However, contractor developed inventory deliverables for the E2E Systems Integration Test testbed were not consistent. In addition, no single organization has control over the configuration management of the E2E Systems Integration Test testbed.

We originally reported these issues in a memorandum dated March 8, 1999. Management provided a response on April 12, 1999. Information Systems informed us verbally and through electronic mail that the IRS has met its dates as indicated in the management response except for one item, Action #2B, which addresses the issue of implementing procedures for ensuring the likeness of the E2E testbeds and the production environment. We have not evaluated the implementation of the corrective actions provided in the management response as part of this audit.

Detailed Objective, Scope, and Methodology

The overall objective of this review was to assess the Internal Revenue Service’s (IRS) planning efforts for the End-to-End (E2E) System Integration Test II and Test III, and the execution of Test II.

1. Through interviews of necessary officials (i.e., Product Assurance, Year 2000 Telecommunications Project Office, Office of Technical Support, TRW, Inc., etc.) and review of relevant documentation, we determined if the E2E System Integration Test is being sufficiently planned and coordinated. Specifically, we:

1. Obtaining an update on the Product Assurance Integrated Test and Control Center’s (ITCC) efforts to establish the Tier II configuration management process.
2. Determining the status of the Unisys configuration management tool (SQA) in relation to the E2E System Integration Test (i.e., has the tool been procured, implemented, tested, or utilized for the E2E System Integration Test?).
3. Reviewing the planned use of Endevor for the IBM mainframe environment to determine if the product will be used to control both systems software and applications.

1. Monitoring status meetings conducted by TRW.
2. Monitoring status meetings conducted by Product Assurance.
3. Monitoring status meetings conducted by the CDC Project Office.

1. We evaluated the establishment of the test platforms to determine if the parallel processing environment can effectively run tests. Specifically, we:

A. Assessed the progress made in establishing and testing the telecommunication links between the test sites.

B. Determined how delays in establishing the IRS’ Year 2000 compliant minicomputer (Tier II) platforms will affect the establishment of the Tier II testing platforms at the ITCC.

C. Assessed the impact on testing resources (i.e., test platform capacity, staffing for testing, etc.) in adding systems to the E2E System Integration Test.

D. Assessed the status of the Unisys 4800 testbed for the Tennessee Computing Center (TCC). This machine was initially scheduled for delivery at the TCC in the fall of 1998, but was delayed until the winter of 1999.

2. We observed Test II activities at the Martinsburg Computing Center, the TCC, and the ITCC and reviewed results to determine if the test provides the assurance that its system software, applications, telecommunications, and Commercial-off-the-Shelf products operate correctly in the year 2000. Specifically, we:

1. Determined if the E2E System Integration Test team is executing all the planned test runs.

Major Contributors to This Report

Scott Wilson, Associate Inspector General for Audit (Information Systems Programs)

Michael Phillips, Acting Director

Kent Sagara, Acting Deputy Director

Vincent Dell’Orto, Audit Manager

Anthony Knox, Senior Auditor

Kim Hyatt, Auditor

Melvin Lindsey, Auditor

Report Distribution List

Deputy Commissioner Operations C:DO

Deputy Commissioner Modernization C:DM

Chief Information Officer IS

Deputy Chief Information Officer (Systems) IS

Deputy Chief Information Officer (Operations) IS

Assistant Commissioner (Product Assurance) IS:PA

Assistant Commissioner (Program Evaluation and Risk Analysis) M:OP

Century Date Change Program Director IS:CD

Office of the Chief Counsel CC

Office of Information Resources Management IS:IR

Office of Management Controls M:CFO:A:M

National Director for Legislative Affairs CL:LA

Audit Liaison, Office of Program Oversight IS:IR:O

Memorandum #1 – Review of the Internal Revenue Service’s Year 2000 End-to-End Systems Integration Testing Effort – Test II Planning/Execution and Test III Planning

Management’s Response to Memorandum #1