The Internal Revenue Service Needs to Significantly Improve
the Inventory Used to Monitor Its Year 2000 Conversion Efforts
December 1999
Reference Number: 2000-20-021
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
December 22, 1999
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner
/s/ Pamela J. GardinerDeputy Inspector General for Audit
SUBJECT: Final Audit Report - The Internal Revenue Service Needs to Significantly Improve the Inventory Used to Monitor Its Year 2000 Conversion Efforts
This report presents the results of our assessment of whether the Internal Revenue Service’s (IRS) telecommunications equipment was inventoried completely and accurately and whether the preliminary risk assessment of this equipment was appropriately factored into the IRS’ overall Year 2000 (Y2K) conversion plans. We also assessed whether the Y2K initiatives for inventorying tax processing commercial off-the-shelf (COTS) products are progressing as planned and goals are being completed effectively and timely. Network operating systems were also included in our telecommunications work because Information Systems management has classified network operating systems (NOS) as telecommunications items.
In summary, we found the IRS
still has a significant amount of work remaining to sufficiently address the Y2K issues related to the inventories of telecommunications equipment, mid-range computer commercial software products, and NOS. Due to the time sensitivity of Y2K, these issues were reported to IRS officials throughout the review.To minimize the Y2K-related risks associated with inaccurate or incomplete inventories, we recommended that the Chief Information Officer (CIO) develop and implement specific inventory processes and procedures to be followed by system managers and administrators at the field offices to ensure a comprehensive inventory. Also, the CIO should conduct independent assessments of the inventory at critical IRS installations to ensure inventory accuracy and completeness for Y2K efforts. Over the longer term, we also recommended that the IRS evaluate and implement automated inventory and network management tools.
A draft copy of this report was sent to IRS management for comment on November 5, 1999. We received a draft copy of a management response from the Century Date Change Project Office. The text of this response is included in the body of this report. Management has indicated they will provide us with a final document describing their proposed corrective actions at a later date.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or your staff may call Scott Wilson, Associate Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
The Inventory of Network Operating Systems Is Substantially Understated
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – List of Audit Sites
An accurate inventory of all hardware and software is critical to ensure the Internal Revenue Service (IRS) identifies and updates components affected by the Year 2000 (Y2K) date change. The IRS relies on its inventory system, the Integrated Network and Operations Management System (INOMS), as the primary tool to manage and report on the progress of its Y2K conversion efforts.
The objective of this review was to determine whether the IRS has completely and accurately inventoried its telecommunications equipment, network operating systems (NOS), and commercial-off-the-shelf products (COTS) as part of its Y2K conversion efforts. We obtained our results by conducting audits on-site at the IRS’ 3 computing centers, 5 of its centers for processing tax returns, and 16 other locations. The majority of the audit work was conducted under contract by PricewaterhouseCoopers.
Results
The IRS still has a significant amount of work remaining to sufficiently address the Y2K issues related to the telecommunications equipment inventory, mid-range computer commercial software products inventory, and NOS inventory. Due to the importance of the inventory in the overall Y2K conversion effort, the IRS should take immediate action to correct the incompleteness and inaccuracies in its inventory database.
The Inventory Significantly Understates the Amount of Telecommunications Equipment in Use Throughout the Internal Revenue Service
When we compared samples of telecommunications equipment located at 24 IRS sites to the inventory information maintained on the INOMS database, only 45 percent of the items at the sites were included in the inventory. There was a substantial variation in the completeness of the inventory, depending on the type of IRS facility. The IRS’ computing centers had the most complete inventories at approximately 74 percent, while local offices had the lowest completeness rate at 20 percent.
The significant understatement in the amount of telecommunications equipment used in IRS offices and operations poses a significant risk that some IRS offices may not be able to operate correctly on and after January 1, 2000. Since it is projected that the IRS will receive over 120 million telephone calls from taxpayers in Fiscal Year 2000, its telecommunications equipment and networks must be Y2K ready to assure there is no significant disruption in the telephone systems and access to taxpayer data.
A variety of factors have contributed to the incompleteness of the IRS’ telecommunications inventory:
The Inventory of Mid-Range Computer Commercial Off-the-Shelf Software Products Has Significant Completeness and Accuracy Deficiencies
Our review of the mid-range computer (Tier II) COTS software products (such as Unix operation systems and Oracle database products) inventory identified significant disparities between the COTS installed on computers and the corresponding inventory records. At each site, we requested high-level documentation, independent of the INOMS, regarding each system’s characteristics for comparison to the INOMS. Only 51 percent of Tier II COTS products listed in the system documentation were entered on the INOMS database. Also, only 33 percent of Tier II COTS products on the INOMS database were found on the system documentation provided by the site personnel.
Without a complete inventory of Tier II COTS on the INOMS, the IRS may not identify and correct potentially serious Y2K problems. The inaccuracies may also lead to misguided renovation efforts, neglect of major issues, and inefficient use of resources.
The Inventory of Network Operating Systems Is Substantially Understated
Our review also identified a significant discrepancy on the number of NOS installed at IRS sites and the number of NOS listed on the INOMS inventory. A NOS, such as Microsoft’s Windows NT, is used to control the access to stored data, the flow of input and output information, and the scheduling and execution of multiple concurrent tasks on a local area network (LAN). Of the 244 LAN servers we sampled, only 60 had an associated NOS on the INOMS database. There should be a one-to-one relationship between NOS and servers. A review of the entire INOMS database shows that there are only 742 NOS records on the INOMS, while the IRS uses an estimated 2,500 to 3,000 LAN servers to conduct operations.
The complete and accurate inventory of all NOS is a critical aspect of the IRS’ Y2K efforts. Since the IRS is dependent on its local area networks to perform a significant part of its tax administration work, the Century Date Change (CDC) Project Office and NOS Program Office must identify the existence and location of each NOS and conduct the necessary research and testing to assure that any NOS, unable to process past the year 2000 is updated or replaced before January 1, 2000. A NOS that does not process the century date correctly can cause disruption for all the workstations residing on a given LAN. If a large number of NOS remain non-compliant, there is a potential that workstations in several locations will be disrupted.
Summary of Recommendations
To minimize the Y2K-related risks associated with inaccurate or incomplete inventories, we recommended that the Chief Information Officer (CIO) develop and implement specific inventory processes and procedures to be followed by system managers and administrators at the field offices to ensure a comprehensive inventory. Also, the CIO should conduct independent assessments of the inventory at critical IRS installations to ensure INOMS accuracy and completeness for Y2K efforts. Over the longer term, we also recommend that the IRS evaluate and implement automated inventory and network management tools.
Management’s Draft Response: As of the date of this report, a final management response had not been received due to the extensive efforts underway to prepare for Y2K. However, we summarized below a draft response from the CDC Project Office and incorporated a more detailed description of the response in the body of the report.
The CDC Project Office initiated a wall-to-wall inventory reconciliation process to be completed at critical IRS sites by December 31, 1999. The wall-to-wall inventory started August 2, 1999. The scope of the wall-to-wall inventory encompasses telecommunication hardware and COTS software. The inventory teams physically reconcile the INOMS database with inventory found or not found on the floor and update the INOMS to reflect the results of the physical inspection and software interrogation.
Information Systems (IS) is revamping the entire inventory process to sustain the wall-to-wall effort after Y2K through the Single Point Inventory Function (SPIF). As planned, the new SPIF will provide guidelines and procedures that establish accountability in receipt, distribution, excessing, and disposal of ADP hardware, software, and telecommunications inventory throughout the various IRS locations. IS will update the guidelines as required to incorporate the new procedures and update the existing inventory and property management documents to reflect the newly developed organizational structure and policies.
IS also established the Asset Management Modernization Project Office to modernize the IRS Asset Management Program. The prime vendor responsible for the IRS modernization initiative is addressing the concerns raised in our report and the inventory management processes in its deliverables. IS expects incremental enhancement of the INOMS including new scanner technology by September 2000. The delivery date for the prototype of the modernized system, Information Technology Asset Management System, is scheduled for March 2001.
This report presents the results of our work to assess whether the Internal Revenue Service’s (IRS) telecommunications equipment has been inventoried completely and accurately, that the preliminary risk assessment of this equipment has been appropriately factored into the IRS’ overall Year 2000 (Y2K) conversion plans, and that the Y2K initiatives for inventorying tax processing commercial off-the-shelf (COTS) and network operating systems (NOS) products are progressing as planned and are being completed effectively and timely.
Our work was conducted in accordance with Government Auditing Standards. We conducted our review from November 1998 to August 1999 in the National Office and on-site at sampled service centers and computing centers, as well as at other IRS offices. A complete list of these sites is included in Appendix IV. We provided our interim results to IRS management throughout the audit period due to the time sensitive nature of the Y2K issue. IRS management used our interim results to begin taking corrective action.
Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.
As of the date of this report, a final management response had not been received due to the extensive efforts underway to prepare for Y2K. However, we incorporated the draft management response from the CDC Project Office in the body of the report. IRS management has indicated they will provide us with a final document describing their proposed corrective actions at a later date.
The IRS is currently involved in an organization-wide initiative to address the potential issues that will arise as a result of the upcoming century rollover. As the year changes from 1999 to 2000, many computers and information systems will experience problems if they are not fixed due to their interpretation of two-digit years. The "Y2K problem" is a serious threat to the normal business operations of the IRS because the majority of tax processing functions use dates.
An initial step in addressing the Y2K problem is conducting a detailed inventory of all date-sensitive equipment throughout an organization. A comprehensive inventory is essential to understanding the scope and extent of effort required in a Y2K project. In its Y2K efforts, the IRS is using the Integrated Network and Operations Management System (INOMS) database as the central source of inventory data. This database is driving the Y2K assessment process for information systems throughout the IRS.
The technology infrastructure of the IRS is heavily reliant on the consistent operation of telecommunications equipment and NOS. This equipment includes, but is not limited to, automated call distributors, voice response units, telephone switches, routers, hubs, etc. The majority of voice and data traffic, both internal and external to the IRS, is dependent on telecommunications and NOS equipment. Many mission critical IRS systems rely on mid-range computer COTS products.
The IRS still has a significant amount of work remaining to sufficiently address the Y2K issues related to the following areas:
The CDC Project Office is using the INOMS database as the official inventory and the primary source of Y2K-readiness status. The deficiencies of the inventory database present a significant problem because the CDC Project Office uses the inventory to determine, manage and report on the progress of Y2K efforts. Due to the importance of the inventory in the overall IRS Y2K project, and the limited amount of time remaining, the IRS should take immediate action to correct the incompleteness and inaccuracies in its inventory database.
Our review did identify that the IRS has used an automated tool, the Bellcore Risk Assessment Tool, to assist in its efforts to perform a risk assessment of its telecommunications inventory. The results from this tool were used to identify the telecommunications equipment with the greatest risk for century date change problems.
To reduce the potential risk presented by non-compliant COTS and NOS products, the IRS hired a contractor to conduct a validation of all tax processing COTS and NOS products that are listed on the INOMS.
The Inventory Significantly Understates the Amount of Telecommunications Equipment in Use Throughout the Internal Revenue Service
When telecommunications equipment located at the sampled 24 IRS sites was compared to the inventory information maintained on the INOMS database, we identified completeness rates ranging from 20 to 74 percent, with an average completeness rate of 45 percent (see Table 1). Our sample was selected to represent various aspects of IRS operations, including Computing Centers, Service Centers, and District Offices. For each site, we judgmentally selected the items with the greatest potential for century date problems and excluded items with no known century date impact.
This significant understatement in the amount of telecommunications equipment used in IRS offices and operations poses a significant risk that all IRS offices may not be able to operate without significant interruption on and after January 1, 2000.
There was a substantial variation in the completeness of the inventory, depending on the type of IRS facility. Computing Center inventories were approximately 74 percent complete. The rate of completeness of the inventory for service centers was 49 percent while district offices had a completeness rate of 24 percent. Posts-of-duty had the lowest completeness rate of 20 percent (see Table 1). In general, the continued operation of the computing centers and service centers is required to assure timely tax processing. The district offices and posts-of-duty are essential to assure effective customer service and tax law enforcement.
|
Items In Sample |
Items Found on the INOMS |
Percent of Sample Found on the INOMS |
|
|
Computing Centers |
444 |
330 |
74% |
|
Service Centers |
608 |
300 |
49% |
|
District Offices |
659 |
156 |
24% |
|
Posts-Of-Duty |
60 |
12 |
20% |
|
TOTALS |
1771 |
798 |
45% |
Table 1: Telecommunications Equipment Found on the INOMS
A significant amount of equipment also lacked an inventory barcode, although attaching a barcode is supposed to be one of the first steps in inventorying equipment. During 1998, the IRS hired a contractor to conduct a 100 percent physical inventory of telecommunications equipment throughout IRS offices. Despite the recent 100 percent inventory efforts, approximately 18 percent of our audit sample did not appear to have a barcode label attached when we performed our visual inspection of the equipment in November 1998 through January 1999.
A variety of factors have caused the incompleteness of the telecommunications inventory:
The following pages contain a more detailed description of these factors:
Telecommunications inventory is often difficult to track due to its size and portability. Additionally, management at each IRS facility has the ability to purchase their own networking hardware and to move telecommunications equipment between locations. Further complicating inventory efforts, this equipment is usually installed or stored in an out-of-the-way location such as a telephone closet. Due to a combination of these factors, telecommunications equipment is often excluded or improperly recorded on the INOMS inventory.
Telecommunications policies and procedures prior to Y2K activities were basically non-existent or inconsistent. During our field visits, we found that most field personnel were unable to provide copies of written inventory procedures to the audit teams. Also, the property disposal procedures in place are not always followed. In our review of the INOMS database, we found a significant amount of "deadwood" equipment (i.e., existing INOMS records for equipment that had been disposed of).
Some of the deficiencies in the inventory database can be attributed to the separation and lack of integration between the procurement and inventory management processes. IRS policy dictates that all recently purchased automated data processing (ADP) equipment should be tagged with a barcode and entered onto the INOMS database within ten days. However, these steps are not being performed on a consistent basis in the field offices. Also, purchase order information is often absent when the equipment arrives at the receiving dock.
Many of the telecommunications program offices appear to be managing their Y2K efforts independent of the INOMS. During the review process, several telecommunications program office individuals expressed that they do not trust or use the INOMS in their Y2K activities. Instead, they track their progress on independent spreadsheets. These telecommunications program offices often ignore the INOMS and are not concerned about the completeness or accuracy of telecommunications inventory data on the INOMS.
The problems with the inventory expose the IRS to considerable risks. The senior management of the CDC Project Office and the IRS are dependent on the completeness and accuracy of the INOMS inventory database in the execution and reporting of their Y2K activities. This review indicated significant inventory deficiencies in the database that could affect the ability of the IRS to identify and complete all Y2K remediation tasks related to telecommunications equipment. Further, telecommunications equipment is heavily integrated in computer application systems (data) and telephone and message routing systems (voice). Failure to identify and correct all telecommunications equipment may lead to system failures that could prevent critical customer service functions from operating, such as responding to taxpayer questions and accessing taxpayer data while the taxpayer is on the telephone. Since it is projected that the IRS will receive over 120 million telephone calls from taxpayers in Fiscal Year 2000, its telecommunications equipment and networks must be Y2K ready to assure there is no significant disruption in the telephone systems and access to taxpayer data.
We held preliminary discussions with IRS Information Systems management concerning the inventory issues. Management took corrective actions to improve the completeness of the inventory database, including uploading the results of the 100 percent inventory of telecommunications equipment that had not been entered onto the INOMS database. An inventory assessment (performed by the IRS on March 17, 1999) showed an overall completeness rate of approximately 63 percent at the sampled sites. This assessment confirmed that there are significant problems despite some improvement after we performed our inventory work. Management has also included telecommunications as part of its Tier II Independent Audit and Readiness Verification process. Part of this process includes site visits to validate the telecommunications inventory.
Recommendations
Management’s Draft Response: In addressing recommendations 1 through 4, the CDC Project Office indicated that the following actions were taken to improve IRS’ Y2K inventory:
Long-term Recommendations
Management’s Draft Response: In addressing recommendations 5 through 9, the CDC Project Office indicated that the following actions will be implemented to improve the IRS’ inventory controls over the long term:
The Inventory of Mid-Range Computer Commercial Off-the-Shelf Software Products Has Significant Completeness and Accuracy Deficiencies
Our review of the Tier II COTS software products inventory identified significant disparities between the COTS software installed on computers and the corresponding INOMS inventory records. At each site, we requested systems documentation, independent of the INOMS, regarding each system’s characteristics (i.e., hardware manufacturer and model, operating system, COTS products used on the system, applications, and system administrators) to verify the accuracy of the INOMS data.
Where this type of information was available, we reconciled the Tier II COTS systems documentation to the Tier II COTS products found on the INOMS database. We found that only 51 percent of Tier II COTS products identified in system documentation were found on the INOMS database (see Table 2).
|
Tier II COTS Products Identified By Site Documentation |
Tier II COTS Products Identified by Documentation and Found on the INOMS |
Percentage Found |
|
|
Computing Centers |
125 |
58 |
46% |
|
Service Centers |
592 |
307 |
52% |
|
District Offices |
8 |
7 |
88% |
|
TOTALS |
725 |
372 |
51% |
Table 2: Tier II COTS Products Found on the INOMS
We also reconciled the INOMS database to the system documentation and found only 33 percent of Tier II COTS products on the INOMS database were identified in the system documentation provided (see Table 3).
|
Tier II COTS Products on the INOMS |
Tier II COTS Products on the INOMS and Found on Documentation |
Percentage Found |
|
|
Computing Centers |
334 |
79 |
24% |
|
Service Centers |
804 |
309 |
38% |
|
District Offices |
68 |
7 |
10% |
|
TOTALS |
1206 |
395 |
33% |
Table 3: Tier II COTS Products Found in Documentation
There were also significant discrepancies between the COTS products listings provided by the systems administrators and the "Model/Version" field on the INOMS database. Of the Tier II COTS products we found on the INOMS database and the system’s documentation, 59 percent had discrepancies between the "Model/Version" field on the INOMS and the documentation (see Table 4).
|
|
Tier II COTS Products Identified by Documentation and Found on the INOMS |
Tier II COTS Products Identified by Documentation and Found on the INOMS with Model/Version Discrepancies |
Percentage with Discrepancies |
|
Computing Centers |
58 |
34 |
59% |
|
Service Centers |
307 |
181 |
59% |
|
District Offices |
7 |
6 |
86% |
|
TOTALS |
372 |
221 |
59% |
Table 4: Tier II COTS Products with Model/Version Discrepancies
There are several factors contributing to the incompleteness of the Tier II COTS products inventory, including:
Due to the IRS’ reliance on the inventory information provided by the INOMS, these discrepancies present critical risks to the IRS’ efforts to address Y2K issues. Without a complete inventory of Tier II COTS products on the INOMS, the IRS may not identify and remediate potentially serious Y2K problems. Since, in many instances, application programs receive date information from COTS products, a date error might occur if the COTS product does not correctly interpret the century date. The inaccuracies may also lead to misguided renovation efforts and inefficient use of resources.
Recommendations
The following recommendations are specific to the Tier II COTS products inventory:
Management’s Draft Response: In addressing recommendations 10 through 12, the CDC Project Office indicated that the following actions will be taken:
The Inventory of Network Operating Systems Is Substantially Understated
Our review found a significant discrepancy in the number of sampled NOS and the INOMS database information. A NOS, such as Microsoft’s Windows NT, is used to control the access to stored data, the flow of input and output information, and the scheduling and execution of multiple concurrent tasks on a local area network (LAN). Of the 244 LAN servers we sampled, only 60 had an associated NOS on the INOMS database. There should be a one-to-one relationship between NOS and servers on the inventory. A review of the entire INOMS database shows that there are only 742 entries for NOS records on the INOMS, while the IRS uses an estimated 2,500 to 3,000 LAN servers to conduct operations.
Many of the factors contributing to the incompleteness of the NOS inventory are the same as those discussed previously, including a lack of shared ownership for the INOMS data and a lack of training for inventory specialists. In addition, until Y2K efforts began, the capability for the consistent tracking of server hardware and the associated software was not viewed as being needed.
The complete and accurate inventory of all NOS is a critical aspect of the IRS’ Y2K efforts. Since the IRS is dependent on its LANs to perform a significant part of its tax administration work, the CDC Project Office and NOS Program Office must know the existence and location of each NOS in order to conduct the necessary research, testing, and potential remediation efforts. Remediation efforts for NOS may include replacing or installing software patches. However, the IRS will be unable to complete implementation and testing activities without fully addressing every NOS on every server. A NOS that does not process the century date correctly can cause disruption for all the workstations residing on a given LAN. If a large number of NOS remain non-compliant, there is a potential that workstations in several locations will be disrupted.
Many of the previous recommendations, if implemented, would also improve the NOS inventory. In addition, we have included a specific recommendation to deal with NOS issues.
Recommendation
Management’s Draft Response: The CDC Project Office states that Information Systems will continue analysis of the INOMS to determine the ratio of NOS records to corresponding servers. Information Systems will then determine if this issue needs to be included in the Y2K End-Game plan by December 15, 1999.
The IRS still has a significant amount of work remaining to sufficiently address the Y2K issues related to the telecommunications equipment inventory, mid-range computer commercial software products inventory, and NOS inventory. The deficiencies of the inventory database present a significant problem because the CDC Project Office uses the inventory to determine, manage and report on the progress of Y2K efforts to internal and external entities. Due to the importance of the inventory in the overall IRS Y2K project and the limited amount of time remaining, the IRS should take immediate action to correct the incompleteness and inaccuracies on its inventory database.
Appendix I
Detailed Objective, Scope, and MethodologyThe overall objective of this review was to assess whether the Internal Revenue Service’s (IRS) telecommunications equipment was inventoried completely and accurately, that the preliminary risk assessment of this equipment was appropriately factored into the IRS’ overall plans; and that the Year 2000 (Y2K) initiatives for inventorying tax processing commercial off-the-shelf (COTS) products are progressing as planned and goals are being completed effectively and timely. Network operating systems (NOS) were included in our telecommunications work because Information Systems management has classified NOS as telecommunications items.
The focus of this review was to evaluate the accuracy and completeness of the telecommunications and COTS inventories. The review also assessed how effectively the preliminary risk assessments of telecommunications equipment and COTS and NOS products have been factored into the IRS’ overall plans. To accomplish this, we:
Appendix II
Major Contributors to This ReportScott Wilson, Associate Inspector General for Audit (Information Systems Programs)
Gary Hinkle, Director
Michael Phillips, Director
Vincent J. Dell'Orto, Audit Manager
Kent Sagara, Audit Manager
Regina Dougherty, Senior Auditor
Myron Gulley, Auditor
Appendix III
Report Distribution ListDeputy Commissioner Modernization C:DM
Deputy Commissioner Operations C:DO
Chief Information Officer IS
Deputy Chief Information Officer (Systems) IS
Deputy Chief Information Officer (Operations) IS
Assistant Commissioner, National Operations IS:O
Assistant Commissioner, IS Field Operations IS:FO
Assistant Commissioner, Service Center Operations IS:SC
Assistant Commissioner, Program Evaluation and Risk Analysis M:OP
Century Date Change Program Director IS:CD
Office of the Chief Counsel CC
Office of Management Controls M:CFO:A:M
Office of Information Resources Management IS:IR
National Director for Legislative Affairs CL:LA
Audit Liaison, Office of Program Oversight IS:IR:O
Appendix IV
List of Audit SitesDetroit Computing Center
Martinsburg Computing Center
Tennessee Computing Center
Atlanta Service Center
Brookhaven Service Center
Memphis Service Center
Ogden Service Center
Philadelphia Service Center
Brooklyn District Office
Central California District Office
Georgia District Office
Illinois District Office
Indiana District Office
Michigan District Office
North Florida District Office
Northern California District Office
Pacific Northwest District Office
Philadelphia District Office
Bellevue, Washington Post-of-Duty
Jacksonville, Florida Call Center Post-of-Duty
Jacksonville, Florida Post-of-Duty
King of Prussia, Pennsylvania Post-of-Duty
Merriville, Indiana Post-of-Duty
San Francisco, California Post-of-Duty