The Internal Revenue Service Needs to Strengthen the Controls Over the Year 2000 Conversion of Its Telecommunications Equipment
February 2000
Reference Number: 2000-20-030
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
February 9, 2000
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – The Internal Revenue Service Needs to Strengthen the Controls Over the Year 2000 Conversion of Its Telecommunications Equipment
This report presents the results of our review of the effectiveness of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) telecommunications implementation processes. When reading this report, please keep in mind that the draft report was issued on December 17, 1999, and is reflective of our assessment of the IRS’ telecommunications conversion efforts through the end of November 1999. Since that time, January 1, 2000, has come and gone without any significant glitches in the telecommunications infrastructure. However, it is still prudent to address the issues in this report since the 2000 filing season for tax returns has not yet reached its peak.
In summary, we found that, although improvements had been made since our previous report, the IRS still had significant problems with the completeness of its telecommunications equipment inventory. An accurate inventory is critical to ensure that all telecommunications equipment is identified and assessed for Y2K compliance. We also found that the IRS did not have an effective method of assuring that only Y2K compliant telecommunications equipment is in production.
We recommended that the Chief Information Officer (CIO) ensure the final Y2K planning efforts include strategies to mitigate the risk of telecommunications failures at local IRS sites with unvalidated inventories. The CIO should also assure that the Y2K assessment is complete for the items whose compliance status we could not determine and that the assessment process is completed for all other telecommunications equipment. We also recommended that the CIO develop and implement an interim configuration management process for assuring the Y2K readiness of the IRS telecommunications environment. If this is not possible, the CIO should ensure the final Y2K planning efforts mitigate the risk that non-Y2K compliant equipment is in production.
A draft copy of this report was sent to IRS management for comment on December 17, 1999. As of the date of this final report, IRS was in the process of developing a response outlining the actions it has taken or will take to address the issues in the report.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or your staff may call Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix V – Management's Response to Memorandum #1
Appendix VI – List of Audit Sites
An accurate inventory of telecommunications equipment is critical to ensure the Internal Revenue Service (IRS) identifies and updates components affected by the Year 2000 (Y2K) date change and thus assures that its networks and phone systems will operate correctly in a Y2K environment. The IRS relies on its inventory system, the Integrated Network and Operations Management System (INOMS), as the primary tool to manage and report on the progress of its Y2K conversion efforts. Previous audit work has identified serious deficiencies on the INOMS database.
Since it is projected that the IRS will receive over 120 million telephone calls from taxpayers in Fiscal Year (FY) 2000, its telecommunications equipment and networks must be Y2K ready to assure there is no significant disruption in the telephone systems and access to taxpayer data. If there is slow response time on the telecommunications network, customer service employees will have to wait for information to come up on their computer screens, thus answering fewer taxpayer telephone calls. If the network goes down, thousands of employees may not have on-line access to taxpayer data.
The objective of this review was to assess the effectiveness of the IRS’ Y2K telecommunications products implementation processes, particularly management’s progress in providing oversight and ensuring goals and objectives are being met. We obtained our results by conducting audits on-site at 21 IRS field locations. The majority of the audit work was conducted under contract by PricewaterhouseCoopers.
Results
While the inventory statistics showed improvement since the completion of audit work conducted for our previous audit report, The Internal Revenue Service Needs to Significantly Improve the Inventory Used to Monitor Its Year 2000 Conversion Efforts (Reference Number 2000-20-021, dated December 22, 1999), there were still significant problems with the completeness of the telecommunications inventory as of November 1, 1999. IRS management used our interim results to begin taking corrective actions to improve the inventory, including a wall-to-wall inventory at major IRS sites as well as other efforts directed at verifying the inventory. These efforts do not, however, include many other IRS sites and will leave these IRS sites with unvalidated inventory information.
Our review identified two other areas of concern:
Because of the time sensitivity of the Y2K issue, we shared the issues and recommendations in this report with IRS management before issuing the draft report so they could begin addressing these issues.
The Internal Revenue Service Continues to Experience Inaccuracies in Its Inventory of Telecommunications Equipment
The Century Date Change Project Office has established the INOMS database as the tool used to manage the Y2K conversion of the telecommunications inventory. The IRS uses the Treasury Communications System (TCS) contract for a significant part of its data communications needs, including telecommunications equipment and services. Although the TCS contractor maintains its own inventory of TCS equipment, the IRS decided to include all of the TCS equipment on the INOMS, as a way to provide overall control over its Y2K conversion efforts. The IRS does not, however, consistently use the INOMS to manage its information technology inventory. Local telecommunications specialists are using various databases and spreadsheets to monitor the inventory, but they are not reconciling the information on these listings to the INOMS. As a result, information that may be in the databases and spreadsheets is not entered onto the INOMS.
In the 21 IRS field sites visited, we performed a verification of a physical sample of equipment to the INOMS inventory and found that only 65 percent of voice and data telecommunications equipment and 87 percent of the local area network servers were recorded on the INOMS. We also found that only 57 percent of TCS equipment was captured on the INOMS database. Telecommunications equipment is heavily integrated in computer application systems (data) and telephone and message routing systems (voice). Failure to identify and assure Y2K readiness of all telecommunications equipment may lead to system failures that could prevent critical customer service functions from operating, such as responding to taxpayer questions and accessing taxpayer data while the taxpayer is on the telephone.
The Internal Revenue Service Does Not Have an Effective Method of Assuring That Only Year 2000 Compliant Telecommunications Equipment Is in Production
Configuration management is defined as the continuous control of changes made to a system throughout the development and operational life of the system. The IRS does not have an effective method for maintaining configuration management over its telecommunications equipment. Although in July 1999, the IRS reported that over 97 percent of its telecommunications inventory was Y2K compliant, our results indicate this figure is significantly overstated. For the 21 IRS sites visited, we sought IRS documentation on the Y2K compliance of 618 pieces of voice and data equipment and were unable to determine if the IRS had assessed the compliance status of 120 (19 percent) of those pieces. Our sample also identified that 17 of the remaining 498 items did not meet the IRS standard for Y2K compliance. The disparity between the figures reported and the actual status of the inventory indicates the IRS lacks a workable configuration management process for assuring the Y2K readiness of telecommunications equipment.
The IRS faces the risk that many of its telecommunications products will be non-compliant on January 1, 2000. This could result in lengthy efforts to fix or replace this equipment during the 2000 filing season. Since it is projected that the IRS will receive over 120 million telephone calls from taxpayers in FY 2000, its telecommunications equipment and networks must be Y2K ready to assure there is no significant disruption in the telephone systems and access to taxpayer data.
The Internal Revenue Service Needs a Process to Forecast the Effects of Growth on Future Network Performance and Capacity
The IRS does not currently have a process to measure and monitor growth in its telecommunications needs and to forecast the effects of such growth on the performance of its networks. Such a process requires the systematic collecting of network performance and other data over time and the use of modeling to analyze such historical data. The goal of establishing and using such a model is to identify future network capacity and performance problems and deal proactively with them. Several current and future initiatives will add to the demands on the network. Although factors affecting network performance are not exclusively Y2K issues, the IRS faces the risk of inadequate network performance during the 2000 filing season. Network failures can result in thousands of IRS employees being unable to perform their jobs due to IRS reliance on data and telecommunications networks. For example, there was a network failure in the area served by the Cincinnati Service Center in July 1999 and employees could not use the Integrated Data Retrieval System for two to three days.
Summary of Recommendations
The Chief Information Officer (CIO) should ensure the final Y2K planning efforts include strategies to mitigate the risk of telecommunications failures at local IRS sites with unvalidated inventories. The CIO should assure that the Y2K assessment is complete for the 120 items whose compliance status we could not determine and should also assure that the assessment process is completed for all other telecommunications equipment. We also recommended that the CIO develop and implement an interim configuration management process for assuring the Y2K readiness of its telecommunications environment. If this is not possible, the CIO should ensure the final Y2K planning efforts mitigate the risk that non-Y2K compliant equipment is in production on January 1, 2000.
The CIO should also develop a process to monitor and manage the traffic over the IRS data networks. As an initial step, the CIO needs to prepare a documented plan for this development effort. The development and implementation of this plan should be tracked and monitored through the Combined Management Program for Century Date Change and Filing Season Executive Steering Committee process since changes or upgrades to the network could adversely affect IRS network performance next year.
Management’s Response: A draft copy of this report was sent to IRS management for comment on December 17, 1999, with the response due within 30 calendar days. As of the date of this final report, IRS was in the process of developing a response outlining the actions it has taken or will take to address the issues in this report.
The overall objective of this review was to assess the effectiveness of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) telecommunications products implementation processes, particularly management’s progress in providing oversight and in ensuring goals and objectives are being met. The review focused on control areas within the Century Date Change (CDC) Project Office, which consist of risk and compliance assessment, testing processes, implementing Y2K compliant telecommunications, and progress remediating non-compliant products.
Audit work was conducted in accordance with Government Auditing Standards. We conducted audit work from April to November 1999 in the National Office and on-site at 21 sampled IRS field offices, including service centers, a computing center, and district offices. These sites were selected because they represent the various aspects of IRS operations. The majority of the audit work was conducted under contract by PricewaterhouseCoopers. A complete list of these sites is included in Appendix VI. We also conducted a limited assessment of the wall-to-wall inventory efforts recently undertaken by the IRS.
Appendix I contains the detailed objective, scope, and methodology for our review. A listing of major contributors to this report is shown in Appendix II.
A draft copy of this report was sent to IRS management for comment on December 17, 1999, with the response due within 30 calendar days. We did not receive a management response prior to the due date. Management has indicated they will provide us with a final document describing their proposed corrective actions at a later date.
The IRS is currently involved in an organization-wide initiative to address the potential issues that will arise as a result of the upcoming century rollover. As the year changes from 1999 to 2000, many computers and information systems will experience problems due to their interpretation of 2-digit years. The Y2K problem is a serious threat to the normal business operations of the IRS since the majority of tax processing functions use dates.
The General Accounting Office describes several steps necessary to complete the Y2K assessment. An initial step in addressing the Y2K problem is conducting a detailed inventory of all date-sensitive equipment throughout an organization. In its Y2K efforts, the IRS is using the Integrated Network and Operations Management System (INOMS) database as the central source of inventory data. This database is driving the Y2K assessment process for information systems throughout the IRS and is being used by the CDC Project Office to assess the IRS’ progress in making its systems Y2K compliant. The assessment phase includes determining the Y2K compliance of a product and whether the product should be converted, replaced, or retired. This assessment phase is followed by a remediation phase which includes the actual conversion, replacement, or retirement of the product.
The technology infrastructure of the IRS is heavily reliant on the consistent operation of telecommunications equipment. This equipment includes, but is not limited to, phone switches, routers, hubs, etc. The majority of voice and data traffic, both internal and external to the IRS, is dependent on telecommunications equipment. Since it is projected that the IRS will receive over 120 million telephone calls from taxpayers in Fiscal Year (FY) 2000, its telecommunications equipment and networks must be Y2K ready to assure there is no significant disruption in the telephone systems and access to taxpayer data. In addition, the IRS’ operations are dependent on the electronic transfer of data in its processing of tax returns and refunds.
The IRS has established processes that, if consistently followed, will provide adequate verification and testing of the Y2K compliance of its telecommunications components. The IRS also has engaged a contractor to perform an independent review of the Y2K compliance of it commercial off-the-shelf (COTS) and telecommunications products.
The IRS’ inventory statistics showed improvement since our previous audit report, The IRS Needs to Significantly Improve the Inventory Used to Monitor Its Year 2000 Conversion Efforts (Reference Number 2000-20-021, dated December 22, 1999), in which we reported that only 45 percent of telecommunications equipment sampled was found on the INOMS. Nevertheless, our review continued to identify significant problems with the completeness of the telecommunications inventory.
At the request of the Department of the Treasury Assistant Secretary for Management and Chief Financial Officer, we performed a limited review of the IRS’ wall-to-wall inventory efforts in October and November 1999. The overall inventory accuracy at the site visited, after the wall-to-wall inventory, improved to 95 percent. However, a majority of the equipment that we could not find on the INOMS is classified as telecommunications equipment.
Due to the importance of the inventory in the overall IRS Y2K project and the limited amount of time remaining, the IRS should take immediate action to validate and verify the telecommunications inventory and assure that remediation has occurred. Our previous inventory report provided a series of recommendations that address ways to improve the IRS’ inventory for the long term.
In addition to the problems in the Y2K inventory, our review identified two other issues requiring the IRS’ attention:
Because of the time sensitivity of the Y2K issue, we shared the issues and recommendations in this report with IRS management before issuing the draft report so they could begin addressing them.
The Internal Revenue Service Continues to Experience Inaccuracies in Its Inventory of Telecommunications Equipment
We verified the telecommunications inventory at 21 IRS field sites (1 computing center, 5 service centers, 10 district offices, 3 regional offices, and 2 posts-of-duty). At these sites we sampled 618 pieces of IRS-owned telecommunications equipment and 115 local area network servers. We also sampled 122 pieces of equipment acquired through a contract managed by the Department of the Treasury. Our results showed that the IRS continues to have significant weaknesses in its Y2K telecommunications inventory. As of September 1, 1999, we found:
The CDC Project Office has established the INOMS database as the tool used to manage the Y2K conversion of the telecommunications inventory. However, the IRS does not consistently use the INOMS to manage its information technology inventory. In some instances, local telecommunications specialists are using various databases and spreadsheets to monitor the inventory, but they are not reconciling the information on these listings to the INOMS. As a result, information that may be in the databases and spreadsheets is not entered onto the INOMS.
In our previous audit report, we detailed several other factors that have led to the deficiencies in the telecommunications inventory on the INOMS, including the nature of telecommunications equipment, the lack of consistent policies and procedures pertaining to the telecommunications inventory process, the separation and weak communications between the procurement and inventory management processes/functions, and the reluctance by the owners of telecommunications program areas to use the INOMS in managing Y2K progress.
The lack of consistent use of the INOMS has caused a disparity between what is being reported on the status of telecommunications Y2K remediation and the actual status of the inventory. The reliance on an inaccurate inventory management system has created the risk that the IRS will not complete the conversion of its telecommunications inventory by January 1, 2000.
After receiving our interim data and confirming it through their own validation reviews, IRS management determined the need to perform a wall-to-wall inventory at "mission critical" sites, including computing centers, service centers, and selected district offices. The computing centers and service centers were originally scheduled to be completed by September 30, 1999. However, a report from the CDC Project Office dated October 18, 1999, shows that the schedule for completing the wall-to-wall inventory had been extended into November 1999. Additionally, the wall-to-wall inventories at 11 of the 12 computing and service centers are shown to have missed October milestones. E ven if the wall-to-wall inventory efforts are completed, conversion may not be completed before January 1, 2000.
At the request of Department of the Treasury management, we performed limited audit work at the Kansas City Service Center to evaluate the progress of the wall-to-wall inventory efforts. The inventory accuracy had improved to 95 percent. In our sample of 825 items, a total of 41 items (5 percent) were not found. However, of the 41 missing items, 31 were telecommunications equipment.
We also attempted to locate equipment that was not found on the INOMS during our previous site visits. We searched for these items again on the INOMS, after the wall-to-wall inventory had been completed. The results of this assessment are shown in Table 1.
|
Service Center |
Number of Items in Sample |
Number of Items Not Found on the INOMS |
Percentage of Items Not Found on the INOMS |
|
Philadelphia |
39 |
18 |
46% |
|
Andover |
44 |
18 |
41% |
|
Cincinnati* |
34 |
22 |
65% |
|
Austin* |
62 |
15 |
24% |
|
Atlanta* |
61 |
19 |
31% |
|
TOTAL *Inventory certification not completed (11/5/99). |
240 |
92 |
38% |
Table 1: Items Missing from the INOMS
These results indicate that the IRS may continue to have a part of its telecommunications equipment not inventoried and converted before January 1, 2000.
Recommendation
The Internal Revenue Service Does Not Have an Effective Method of Assuring That Only Year 2000 Compliant Telecommunications Equipment Is in Production
Configuration management is the continuous control of changes made to systems throughout the development and operational life of the system. The IRS lacks an effective method of maintaining configuration management over its telecommunications equipment. As a result, the information provided by the IRS to external parties is inaccurate. In its July 25, 1999, report to the Office of Management and Budget, the IRS reported that over 97 percent of its telecommunications inventory had been converted. Our results indicate this figure is significantly overstated.
When we sought to determine whether a sample of 618 pieces of equipment we surveyed at 21 sites was Y2K compliant, our results showed:
We also reviewed a sample of 59 critical telecommunications products to assess adherence to documentation requirements. Program owners are responsible for completing formal documentation that includes a Compliance Approval Sheet that describes the product and summarizes the compliance findings and a Compliance Assessment Checksheet that gives the criteria by which the product’s compliance is judged. We found that compliance assessment procedures were not properly completed for 5 out of 59 critical telecommunications products.
These results indicate the IRS lacks a workable process for assuring that only Y2K ready equipment is in production on January 1, 2000. The IRS has not maintained effective control over its telecommunications equipment, as evidenced by the continued problems with its inventory. In addition, there does not appear to be any formal communication between the field offices and the Telecommunications Project Office. As a result, compliance standards for several products have not been communicated to the field offices by the Telecommunications Project Office.
In February 1999, the Commissioner testified before the Congress that there are over 100,000 pieces of equipment in the telecommunications network. Our results indicate there is a significant risk that a large number of these pieces of equipment will be non-compliant on January 1, 2000, and the IRS will not be aware of this circumstance. Because of the interdependencies existing among the components of a data network, the failure of one piece could jeopardize the functionality of the whole network. This could result in lengthy efforts to fix or replace this equipment during the 2000 filing season. Since it is projected that the IRS will receive over 120 million telephone calls from taxpayers in FY 2000, its telecommunications equipment and networks must be Y2K ready to assure there is no significant disruption in the telephone systems and access to taxpayer data.
Recommendations
The Internal Revenue Service Needs a Process to Forecast the Effects of Growth on Future Network Performance and Capacity
The IRS does not currently have a process to measure and monitor growth in its telecommunications needs and to forecast the effects of such growth on the performance of its networks. Such a process requires the systematic collecting of network performance and other data over time and the use of modeling to analyze historical data. The goal of establishing such a model is to use it to identify future network capacity and performance problems and deal proactively with them. The implementation of several current and future IRS initiatives, such as a new electronic mail system, a collection system (the Integrated Collection System), and a research system (the Service Electronic Research Project) will add to the demands on the network. Although factors affecting network performance are not exclusively Y2K issues, the IRS faces the risk of inadequate network performance during the 2000 filing season.
Should networks fail, the IRS would face serious interruptions in many of its basic functions. For example, if there is slow response time on the network, customer service employees will have to wait for tax information to come up on their computer screens, thus answering fewer taxpayer telephone calls. Network failures can also result in thousands of IRS employees being unable to perform their jobs due to the IRS’ reliance on data and telecommunications networks. For example, there was a network failure in the area served by the Cincinnati Service Center in July 1999 and employees could not use the Integrated Data Retrieval System for two to three days.
After concerns about network performance were raised to us by Information Systems field staffs during site visits in June 1999, we requested a network capacity study or plan that provides guidance in the management of network traffic. We were informed that all such documentation was in the draft phase and data for modeling network performance was being developed. We were also informed that planning for the development of this process was in its initial stages and that there was no documented plan. Without such a plan, it will be difficult to secure the resource commitments needed to implement a performance and capacity forecasting process.
During site visits to IRS field offices from June to August 1999, we found there is a lack of communication from the National Office to field Information Systems staffs regarding performance of IRS networks. Field management is concerned with the absence of a network capacity study or plan that provides guidance in the management of network traffic. Specifically, at the Ohio, Rocky Mountain, Southwest, Northern California, and Los Angeles Districts, questions concerning the ability of the networks to handle the increased workload caused by system upgrades were raised during interviews with district Information Systems management. Information Systems staffs told us that they had requested guidance for network performance management but no documentation was available.
There are several indications that the IRS may experience significant additional demands on its network in the year 2000. At the sites visited, we were informed that early in the conversion from one electronic mail system to another, dramatic "slow downs" in network traffic were experienced. In the Integrated Collection System and Service Electronic Research Project rollouts, new workstations are being placed on the network, some requiring Internet access capability. As new systems and workstations are placed on the network, the network performance diminishes. This has been attributed, in part, to the use of relatively slow 56 kilobits per second (Kbit/s) connections between some IRS offices.
The IRS has scheduled the 56 Kbit/s connections to be upgraded to T-1 connections capable of transmitting data at 1.5 megabits. It appears, however, that efforts to upgrade the network are being undertaken without a model for monitoring network performance.
These issues have not been tracked at the meetings of the Combined Management Program for Century Date Change and Filing Season Executive Steering Committee or treated as presenting a risk to IRS processing in the year 2000 (regardless of whether the risk is attributed to Y2K compliance or the ability to handle additional traffic). In addition, TRW, Inc. (the contractor for the TCS and one of the major providers of telecommunications systems services) indicated that the IRS has not performed a capacity study on its network.
Recommendations
While the IRS’ inventory statistics showed improvement since our previous inventory report, there were still significant problems with the completeness of the telecommunications inventory. There was also a lack of configuration control over telecommunications equipment. These conditions negatively affected the IRS’ ability to accurately access the progress of the Y2K conversion of its telecommunications equipment. Due to the importance of telecommunications in the overall IRS Y2K project, the IRS needs to take actions to validate its inventory and assure that only Y2K compliant components are in production on January 1, 2000.
Appendix I
Detailed Objective, Scope, and MethodologyThe overall objective of this review was to assess the effectiveness of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) telecommunications products implementation processes, particularly management’s progress in providing oversight and in ensuring goals and objectives are being met. To accomplish this, we:
risk-ordered criticality and identifying risks that may have been overlooked.
Appendix II
Major Contributors to This ReportScott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)
Gary Hinkle, Director
Vincent J. Dell’Orto, Audit Manager
Myron Gulley, Auditor
Chinita Coates, Auditor
Appendix III
Report Distribution ListDeputy Commissioner Modernization C:DM
Deputy Commissioner Operations C:DO
Chief Information Officer IS
Deputy Chief Information Officer (Operations) IS
Deputy Chief Information Officer (Systems) IS
Director, Century Date Change Project Office IS:CD
Director, Enterprise Operations IS:EO
Director, Information Resources Management IS:IR
Director, Information Systems Field Operations IS:F
Director, Office of Program Evaluation and Risk Analysis M:O
Director, Service Center Operations IS:SC
Director, Telecommunications IS:TL
National Director for Legislative Affairs CL:LA
Office of the Chief Counsel CC
Office of Management Controls M:CFO:A:M
Audit Liaison, Office of Program Oversight IS:IR:O
Appendix IV
Memorandum #1: A Significant Amount of Telecommunications Equipment Could Not Be Identified on the Integrated Network and Operations Management System
Memorandum has been removed due to its size. To see the complete Memorandum, please go to the Adobe PDF version of this report.
Appendix V
Management's Response to Memorandum #1
Response has been removed due to its size. To see the complete Response, please go to the Adobe PDF version of this report.
Appendix VI
List of Audit SitesDetroit Computing Center
Andover Service Center
Atlanta Service Center
Austin Service Center
Cincinnati Service Center
Philadelphia Service Center
Midstates Regional Office
Southeast Regional Office
Western Regional Office
Georgia District Office
Los Angeles District Office
Michigan District Office
Northern California District Office
North Texas District Office
Ohio District Office
Pennsylvania District Office
Rocky Mountain District Office
Southwest District Office
Virginia-West Virginia District Office
(2) Rocky Mountain Posts-of-Duty