Review of the Service’s Efforts to Prepare
Its Tier II Infrastructure for the Year 2000
Reference No. 091206 Date: November 20, 1998
Table of contents
Objectives and Scope
An implementation plan for the Tier II effort is needed
Planning and coordination of the testing effort need to be enhanced
coordination and accountability with the field and customer organizations need to be improved
certain Y2K and Tier II program office INOMS requirements are not reflected in the INOMS handbook
The accuracy and completeness of the Tier II hardware and cOTS software inventories need to be improved
contingency plans for delays in vendor schedules and infrastructure upgrades are needed
Detailed Objectives and Scope of Review
One of the most critical issues the Service faces this year and next is the need to make its computer systems Year 2000 (Y2K) compliant. The Service’s ability to successfully meet this enormous challenge will largely be determined by the quality of their program management and executive leadership. The Service is a $1.7 trillion financial services organization dependent on its automated systems to process tax returns, issue refunds, deposit payments, and provide employee access to timely and accurate taxpayer account data. Failure to identify, renovate, and test each of these system calculations could result in catastrophic disruption to taxpayers and the government.
The objective of this review was to evaluate the overall efforts of the Service in identifying and converting its Tier II infrastructure. Although the majority of the Service’s tax processing occurs at the Tier I level, there is a significant amount of processing at the Tier II level, as well. As a result, some Tier II systems feed data to the Tier I systems. Examples of processing systems in Tier II include the Electronic Management System (EMS) and Telefile.
The Service must better focus, manage, and control the century Date change (cDc) effort to assure business continuity. Our review identified the following areas where critical improvements are needed:
The century Date change Project Office and the Tier II Program Office have made progress in the way the Y2K effort is being tracked and managed. However, the conversion of the Tier II infrastructure by January 1999, is questionable. In May 1998, we reported to management the need to establish an implementation plan for the Tier II effort, enhance the planning and coordination of the Tier II testing effort, and improve coordination and accountability with the field and customer organizations.
Information Systems management agreed with the issues we reported in May 1998, and established a Tier II Program Office to address these issues. A work breakdown structure to schedule testing and implementation for mission critical Tier II systems was developed, and an overall implementation schedule was to be developed by July 30, 1998. coordination improvements were implemented to address Tier II issues. However, the corrective actions planned in response to the coordination issues in our memorandum addressed Tier II only. The conditions we observed were not confined to Tier II, but included coordination issues among all Tiers.
As our review progressed, we reported to management the need to ensure consistency between Y2K needs and Integrated Network and Operations Management System (INOMS) instructions, and to improve the accuracy and completeness of the inventory. We are now reporting the need to establish contingency plans for delays in vendor schedules and infrastructure upgrades.
Accuracy of INOMS is critical because it is the primary tool the cDc Project Office and the Tier II Program office are using to track the Y2K conversion process. We identified that the platform inventory is complete but nearly half of the 837 platforms reviewed were recorded inaccurately. The commercial-Off-The-Shelf (cOTS) software inventory was incomplete and inaccurate. Approximately 28% of the 411 products were not recorded on INOMS, and 22% were recorded with the wrong version. In a separate test for accuracy, 77 of 168 products were recorded with the incorrect version.
Each of the findings identified above is addressed as steps in the awareness and assessment phases outlined in GAO’s Assessment Guide for Year 2000. According to the guide, these phases should have been completed by mid-1997 to ensure conversion is completed for Y2K. However, the Service is still in the process of finalizing their Tier II inventory and establishing detailed management plans for the Tier II initiative. Based on our assessment, conversion of the Tier II processing systems by January 1999, is in jeopardy.
We recommend the following:
Auditor's Note: Management responded to each of the above recommendations when they were issued in a memorandum. However, this recommendation was designed to address broad cDc coordination issues, not just those related to Tier II. A broader corrective action is needed to address this concern.
Management Response: Management’s earlier response (see preceding page) has been summarized in the report. However, their response to this complete report was not available for inclusion at the time it was issued. We were informed that management is developing actions to address our concerns and will provide us with a written description of their proposed corrective actions at a later date.
Objectives and Scope
This report represents the results of our review of the Service’s efforts to prepare its Tier II infrastructure for the century date change. We initiated this review as part of Internal Audit’s Year 2000 (Y2K) strategy. Our primary objective was to assess the Information Systems (IS) and non-Information Systems (non-IS) Y2K efforts to determine if the Service has assurance that its Tier II infrastructure will operate in the next century. The audit was performed in accordance with generally accepted government auditing standards from March to August 1998.
We conducted testing in the century Date change (cDc) Project Office and the Tier II Program Office. Testing was also conducted in certain Northeast, Midstates, and Southeast Region District Offices; the Austin, Kansas city, Memphis, Brookhaven, Andover, Atlanta, Philadelphia and cincinnati Service centers; and the Detroit and Martinsburg computing centers.
Our tests were designed to provide an overall assessment of the management of the Tier II effort and the accuracy and completeness of the Integrated Network Operations Management System (INOMS) Tier II inventory. We also performed some testing to determine whether the Service can rely on vendors' efforts to make commercial Off-The-Shelf (cOTS) products Y2K compliant.
For purposes of this report, references to "field office" includes regional and district offices and service centers. The detailed audit objectives and scope of review is included in Attachment I.
Management’s response to a memorandum presenting our first three findings and related recommendations has been summarized in this report. However, their response to the complete report was not available for inclusion at the time it was issued. We were informed that management is developing actions to address our concerns and will provide us with a written description of their proposed corrective actions at a later date.
The Y2K date change is one of the most critical problems facing most organizations today. On February 11, 1997, the Deputy commissioner issued a memorandum on Inventory of computer Systems to Facilitate century Date compliance. The memorandum provided reporting requirements as well as compliance and enforcement actions for executives to ensure the cDc project is completed timely. The Deputy commissioner urged all executives to do "whatever it takes" to support the timely and successful completion of this effort.
To ensure Y2K compliance, the Service must evaluate all computer systems and applications. Although the majority of the Service’s tax processing occurs at the Tier I level, there is a significant amount of processing at the Tier II level, as well. Tier II processing systems include the, Electronic Management System (EMS), and Telefile.
The cDc Project Office was established to ensure all current and future IRS systems are Y2K compliant prior to January 1, 2000. Applications are to be converted, tested, and implemented and cOTS products are to be converted by January 31, 1999. Since we began our review, IS has established a new Tier II Program Office to provide Tier II program management and facilitate the timely integration of Y2K compliant applications and cOTS products on compliant platforms.
Non-IS efforts include appointing a Y2K Executive council to identify and assess all non-IS Tier I and II applications and system software for Y2K impact. The Executive council included officials from the service centers, regional and district offices, and headquarters non-IS areas. In addition to the council, the Service has established a service center and a regional Y2K executive. These executives are responsible for coordinating with the IS cDc Project Office and the field offices to ensure non-IS Y2K efforts are accomplished.
The cDc Project Office and the Tier II Program Office have taken steps to ensure the Service’s Tier II systems are ready for the century date change. The Tier II Program Office has identified Y2K compliant versions of operating systems and Database Management Systems (DBMS) for Tier II platforms and the field offices are obtaining these targeted products. In addition, the field offices have complied with the Tier II Program Office request to clean-up questionable Tier II platforms on INOMS. The field offices have also taken steps to protect the applications currently running on the Tier II platforms that will not be upgraded for the Y2K. Our review of a sample of inactive platforms and products showed that nearly all were classified correctly and the majority were being excessed or retired.
However, the Service’s overall management of the Y2K effort and the Tier II infrastructure conversion can be improved. Our review identified several areas that need to be addressed for the service-wide Y2K and Tier II Y2K initiative.
We issued a memorandum to management on May 8, 1998, which detailed the following issues:
In response to the above memorandum and concerns of the commissioner, the Tier II Program Office was established to coordinate the conversion effort.
This report initially details the issues presented in the memorandum along with management's responses and any related auditor comments. In addition to the above issues, we have identified the following additional issues that need to be addressed:
An implementation plan for the Tier II effort is needed.
The cDc Project Office has developed a Project Management Plan (PMP) for the implementation of the cDc Project. The PMP describes the conversion strategy and the 14-step conversion process, and establishes interim target dates for the conversion of system applications.
The GAO Assessment Guide for Year 2000 states that a Y2K program plan should be developed that includes schedules for all tasks and phases of the Y2K program. The Service’s PMP does not, however, establish interim target dates for the conversion or replacement of Tier II system platforms or their related cOTS products. The Project Office has established January 31, 1999, as the final target date for the conversion of these products.
The lack of a Tier II implementation plan greatly reduces the Project Office’s ability to monitor the effect of decisions and delays on the Service’s efforts to complete the Tier II initiative by the January 1999, target date. For example, the Project Office, in conjunction with the Tier II Program Owner, cannot provide the field with necessary milestones that must be met for Tier II to meet the overall target date. Without these interim milestones to measure progress, the Service has an increased risk of not meeting the overall Tier II conversion date of January 1999.
In our closing meeting on September 15, we were informed that a draft Integrated Management Schedule (IMS) had been presented to the commissioner at the end of August. This covered a portion of the Tier II systems being monitored by the program office. A full management schedule showing all systems had not yet been developed.
Management’s Response:In response to Internal Audit's memorandum issued May 8, 1998, the Tier II Program Office began developing a Project Management Plan for Tier II. The plan will address the testing and implementation of compliant Tier II systems as well as the retirement of non-compliant cOTS products and platforms.
The Tier II Program Office planned to have this document completed by July 30, 1998. However, as of our closing meeting on September 15, 1998, a full implementation plan had not yet been completed.
Planning and coordination of the testing effort need to be enhanced.
The GAO Assessment Guide for Year 2000 states that one of the activities to be completed during the assessment phase of the Y2K conversion is the establishment of a testing plan. In addition, agencies should define the requirements for Y2K testing facilities including the need to acquire test facilities. Despite these guidelines, at the time of our initial review the Service did not have a detailed coordinated testing plan for Tier II.
In an April 7, 1998, memorandum from the cDc Project Office, the field and customer organizations were asked to accept ownership and begin conversion of the Tier II platforms and cOTS products. The Project Office indicated that the Distributed Systems Management Branch would independently test the Y2K compatibility and interoperability of the Tier II system components as they were made available from the vendors.
At the time of our initial memorandum (May 8, 1998), the field and customer organizations were told not to wait for the completion of this testing before they begin application testing on the platforms. A number of the Tier II system components had not yet been made available from the vendor. As vendors made Tier II system components available, the compatibility and interoperability testing and the application testing in the field were to be conducted simultaneously.
Because some of the field and customer sites do not have separate testing environments, system upgrades could potentially be made in a production environment. This dramatically increases the risk that problems identified with the compatibility or operability of the Tier II system components could interrupt current Service processing. In addition, simultaneous testing of applications and system components increases the difficulty of identifying the origin of problems encountered during testing. This results in an increased risk of delays in the overall conversion.
We reported this issue to management in a memorandum dated May 8, 1998, with the recommendations that follow. Since that time, a significant effort has taken place to identify responsible executives for each of the mission-critical systems and to obtain concurrence from those executives on required testing and conversion milestones. We will follow-up on the conversion effort for mission-critical Tier II systems in the second phase of our review.
Management’s Response:In response to Internal Audit's memorandum issued May 8, 1998, the Tier II Program Office developed a detailed testing schedule for all mission critical Tier II systems. The schedule established interim milestones for completion of testing and the transmittal of the system into production. In addition, progress on each system will be tracked weekly.
coordination and accountability with the field and customer organizations need to be improved.
The cDc Project Office, tier owners, and representatives from field offices are working together to identify, standardize, and convert the Service's computer systems. The Service has established a network of executives and coordinators to address Y2K issues. A national executive has been designated for the service centers and the regional and district offices. In addition, a regional Y2K executive has been established in each region and the service has also established a Y2K and an INOMS coordinator in each regional office and service center.
These efforts require extreme coordination and cooperation from all parties involved to ensure the success of the Y2K conversion. According to the personnel involved, needed accountability requires a centralized control point in each major field and customer organization to improve the following areas:
Our analysis of the Y2K executive and Y2K coordinator structure identified problems in the use of these positions to effectively manage and carry out the Y2K effort. The regional executives are not being fully utilized and they are not informed of all cDc Project Office requests. The field Y2K executive positions do not appear to be considered critical positions. For example, the Midstates Regional executive was detailed to headquarters early in our review and was not replaced for several months. Also, the national executive position for the field and customer organizations was vacant for five months.
The Y2K program owner responsible for each of the Y2K initiatives communicates with the Y2K coordinators. The coordinators may be asked to provide the cDc Project Office with additional information, to identify inventory or to validate information provided by the Project Office. Each of these requests may carry an independent response date. Discussions with the field Y2K and INOMS coordinators indicate the coordinators are frustrated with the number of requests and the lack of coordination when establishing target response dates.
We analyzed requests forwarded by the cDc Project Office to the Y2K coordinators between January 1997 and March 1998. Our analysis showed a number of the requests asked for similar information for one or more Y2K initiatives. In addition, the response dates established for the receipt of this information conflicted among the various Y2K initiatives. Also, requests were not always forwarded to all Y2K executives and other Y2K designated officials.
GAO’s Year 2000 computing crisis: An Assessment Guide states that a committee needs to be established to continually coordinate with the programmatic and functional area managers. The Guide also indicates that it is important for the technical and management staff of the core business areas to work closely with the Y2K project teams in the assessment and testing process.
The lack of coordination of field and customer organization requests by the cDc Project Office reduces the level of cooperation being provided by these organizations. coordination of requests for action or information can reduce the duplication of effort currently required to satisfy these requests and will ensure response dates are feasible and obtainable. In addition, a central contact point in the field and customer organizations for the Y2K effort will improve the cDc Project Office’s ability to readily ascertain the status of the Y2K effort in these organizations.
We initially reported this issue in a memorandum dated May 8, 1998, with the recommendations that follow. We received a response that was focused solely from a Tier II perspective (see auditor comment below). We believe additional corrective action is necessary to improve coordination of the entire cDc effort.
We recommend that the cDc Project Office:
Management’s Response:In response to Internal Audit's memorandum issued on May 8, 1998, the Tier II Program Office established Tier II executive contacts for the regions, districts and service centers. coordination of Tier II efforts is being accomplished through these executives.
This action addresses the coordination issue related to the Tier II initiative, but does not address the overall coordination issues relative to Y2K as a whole. The conditions we observed were not confined to Tier II, but included coordination issues among all Tiers. Therefore, additional corrective action is needed to address the broader century Date change issue.
certain Y2K and Tier II program office requirements are not reflected in the INOMS handbook.
During our review, we identified two specific instances where the Y2K needs conflict with INOMS instructions and documentation. This condition is causing confusion in the field and customer organizations and can jeopardize the success of the Y2K effort. Following are situations where the needs of the cDc Project Office and the Tier II program office have not been reflected in the INOMS handbook.
conflicting direction between the cDc Project Office/Tier II program office and the INOMS Procedural Guide creates confusion for the field and customer organizations and directly affects the completeness and accuracy of the INOMS inventory. Discussions with field personnel in various locations indicated that they rely on the INOMS documentation for guidance and there was confusion and frustration with the inconsistencies between INOMS and cDc direction.
The accuracy and completeness of the Tier II hardware and cOTS software inventories need to be improved.
INOMS is the primary tool the cDc Project Office and the Tier II Program Office are using to track the Y2K conversion progress. The Tier II Program Office is using INOMS to identify platforms that need to have Y2K operating systems and cOTS software products. In addition, INOMS is being used to identify each cOTS software product currently in use so Y2K compliant versions can be identified.
The cDc Project Office and the Tier II Program Owner have made repeated requests to the field and customer organizations to clean up their ADP inventory on INOMS. Requests have also come from the Deputy commissioner, Associate commissioner for Modernization, and the chief Information Officer. However, our audit tests show that the INOMS inventory for Tier II contains a number of inaccuracies. As part of our testing in the 26 sites, we:
Our testing identified that the Tier II platform inventory is relatively complete, but the items have some inaccuracies. However, the cOTS software inventory for Tier II is not complete, and also has significant inaccuracies.
Tier II platform inventory is complete but inaccurate
Our audit testing in the 26 sites visited identified 837 platforms. Of those, only 14 were not recorded on INOMS. However, 386 (46%) of the platforms had errors in at least one of the following areas:
Of the 386 platforms with errors, 260 (67%) were NcR/AT&T 3430 models. Many of these were incorrectly classified as microcomputers (see misclassification errors above), and some were also recorded as 3230 models (see wrong product name, manufacturer, or model above). We have discussed the field and customer organizations' continued confusion about how to record the NcR 3430s with the Tier II Project Owner and cDc Project Office staff. During our audit testing, the Y2K Executive for Service centers sent e-mail to all service centers clarifying how an NcR 3430 is to be recorded on INOMS.
Tier II cOTS software inventory is incomplete and inaccurate
We conducted two tests to verify the cOTS software inventory. Initially we selected a statistical sample of cOTS software products nationwide to verify the accuracy of these items on INOMS. Secondly, we selected two or three platforms in each site to verify that the cOTS software products running on those platforms were accurately recorded on INOMS.
Our statistical sample included 168 products that were located in the 26 sites visited. Of the 168 products, 77 (46%) had an incorrect version on INOMS. This was the most significant error identified in this accuracy test.
Our second test identified 411 cOTS software products running on 55 platforms. Of these:
According to GAO’s Year 2000 computing crisis: An Assessment Guide, agencies should conduct an enterprise-wide inventory as part of the assessment phase of the Y2K conversion process. The guide states that a thorough inventory ensures that all systems are identified and linked to a specific business area or process. In addition, the inventory should be used to develop a comprehensive system portfolio including platforms, database management systems and operating system software and utilities.
The INOMS inventory continues to be incomplete and inaccurate despite each district and service center director’s annual certification of its accuracy. The inaccuracies of the INOMS inventory affect the Tier II Program Owner’s ability to ensure all Tier II products have been identified and are being considered as part of the Y2K conversion effort. Inventory inaccuracies can also affect the Service’s overall assurance that all Tier II systems will be able to operate through the beginning of the new century.
Maintenance contract information becomes significant in the budgetary planning for the conversion of the Tier II infrastructure. The Tier II Program Office is assuming vendors will provide Y2K upgrades under existing maintenance contracts. Based on this assumption, estimated budget needs for the infrastructure conversion may be significantly understated.
The completeness and accuracy of the INOMS inventory significantly impacts the Project and Program Offices' ability to effectively manage the Y2K conversion of the Tier II infrastructure. Our testing shows that no reliance can currently be placed on the completeness or accuracy of INOMS as it relates to the cOTS products running on the Tier II systems in the field. In addition, reliance cannot be placed on the accuracy of platform information on INOMS for Tier II.
Items that are not recorded in the inventory run the risk of not being considered for conversion. In addition, inaccuracies in the status and maintenance information on INOMS could negatively impact the estimated budget needs for the Tier II conversion effort.
The systems associated with and potentially impacted by these INOMS inaccuracies include: Automated Insolvency System (AIS), Automated Lien System (ALS), Automated Underreporter (AUR), Electronic Fraud Detection System (EFDS), Electronic Management System (EMS), Integrated case Processing (IcP), and Telephone Routing Interactive System (TRIS).
To address the remaining systems, we recommend that the cIO:
In addition, we recommend that:
contingency plans for delays in vendor schedules and infrastructure upgrades are needed.
The conversion of Tier II systems depends in large part on the ability of outside vendors to deliver Y2K compliant products within the Service’s established timeframes. In addition, the conversion of the infrastructure is dependent on the timeliness of the field office upgrades of the Tier II infrastructure components. Our review shows that the Tier II infrastructure could be in jeopardy if contingency plans are not put in place to adapt for vendor and upgrade delays.
We evaluated vendor’s progress in delivering the Y2K compliant versions by retesting a statistical sample of 220 cOTS software products. Our review identified 133 unique cOTS software products. We were unable to determine the status of the compliant versions for 27 of the 133 products because the product and manufacturer information was not available on INOMS. In addition, 23 products have been moved to either
Tier I or Tier III; 32 products are components of an operating system and 5 products have incorrect product names on INOMS.
Of the remaining 46 software products, we found that 9 (20%) did not have a Y2K compliant version currently available from the vendor.
Our analysis of the field offices’ progress in upgrading the Tier II infrastructure components showed platforms are not being upgraded. We reviewed a judgmental sample of 55 platforms to determine if the operating system and Database Management System (DBMS) had been upgraded to the IRS target versions. Of the 55 platforms reviewed, we found that one platform was running the target operating system and one platform was running the target DBMS.
Vendor delivery schedules and the efforts of the field offices are critical to the success of the Service’s efforts to upgrade the Tier II infrastructure. The inability of vendors to supply the Service with Y2K compliant products in a timely manner jeopardizes our ability to validate those products and place them into production prior to the January 1999 target date.
In addition, a process should be established to monitor the progress of items covered by the contingency plan to ensure the action plans accurately address each condition and are carried out timely.
The cDc Project Office and the Tier II Program Office have made progress in the way the Y2K effort is being tracked and managed. However, weaknesses still exist in the management methods being used to ensure the Service will be in a position to operate in the next century. The inventory system, which is the primary tool being used to track the progress of the Y2K initiative, is flawed. Guidance for input to the system conflicts with cDc Project Office and Tier II Program Office needs. In addition, the inventories captured on INOMS are inaccurate and incomplete. These basic problems significantly increase the risk that items will not be identified for consideration in the Service’s conversion efforts.
Each of the activities discussed in this report is included as steps in the awareness and assessment phases outlined in GAO’s Assessment Guide for Year 2000. According to the guide, these phases should have been completed by mid-1997 to ensure conversion is completed by the Y2K. However, the Service is still in the process of finalizing their Tier II inventory and establishing detailed management and test plans for the Tier II initiative. Based on our assessment, conversion of the Tier II processing systems by January 1999 is in jeopardy.
Tammy L. Whitcomb
Detailed Objectives and Scope of Review
The overall objective of this review was to assess the IS and non-IS Tier II Y2K efforts to determine if the Service has assurance that its Tier II infrastructure will operate in the next century.
Because we found consistent INOMS inaccuracies in the offices visited in the Midstates, Southeast, and Northeast Region geographic areas (certain districts, service centers, computing and development centers) we decided not to continue testing in Western Region and National Office. We discussed this decision with the cDc Project Office, and they agreed they would prefer an expedited report, even though we could no longer statistically project the results to the entire Tier II population.
To accomplish our overall objective, we conducted the following tests: