Management Advisory Report: Lessons the Internal Revenue Service Can Apply From Its Year 2000 Efforts to Improve the Management of Its Systems
August 2000
Reference Number: 2000-20-117
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
August 11, 2000
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Management Advisory Report: Lessons the Internal Revenue Service Can Apply From Its Year 2000 Efforts to Improve the Management of Its Systems
This report presents the Treasury Inspector General for Tax Administration's observations on the lessons that the Internal Revenue Service (IRS) can apply from its Year 2000 (Y2K) efforts in managing its existing systems and in the development of modernized systems. Since the IRS has already recognized and is applying many of the lessons it learned during the Y2K effort, we are not making formal recommendations in this report but rather are including several additional suggestions for the IRS to consider.
In summary, we found that effective project tracking, a standardized control structure, and use of steering committees were significant critical success factors that should continue to be deployed. We suggested that the IRS also continue to implement program management techniques across all its information technology initiatives, continue its efforts to implement a comprehensive program to manage its information technology assets, develop more efficient methods for deploying information technology components to its field locations, and enhance future systems integration testing processes.
IRS management agreed with the report and provided highlights of its efforts to use various project management techniques, deploy information technology components, and conduct systems integration tests. Management’s comments have been included as Appendix IV to this report.
Copies of this report are also being sent to the IRS managers who are affected by the issues raised in the report. Please contact me at (202) 622-6510 if you have questions, or your staff may call Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
Improvements Can Be Made to Future Systems Integration Testing
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Management's Response to the Draft Report
The Internal Revenue Service’s (IRS) Year 2000 (Y2K) remediation efforts demanded extraordinary efforts from all areas of the organization. Although the Y2K conversion has been successfully completed, the IRS can still realize long-term benefits from this vast undertaking that cost approximately $1 billion. The IRS can preserve the resources, methodologies, and knowledge gained through the Y2K remediation efforts and deploy them in the future.
We contracted with PricewaterhouseCoopers (PWC) to provide technical assistance to our audits of the IRS’ Y2K conversion initiatives. At the completion of the Y2K audit work, we requested that PWC assist in identifying long-term organizational benefits that the IRS might derive from its experience in managing the Y2K conversion activities. This management advisory report presents suggestions to the IRS based on lessons learned through reviews of the IRS’ Y2K conversion activities. It also includes suggestions based on our assessment of the last phase of the IRS’ End-to-End Systems Integration Test.
Results
There were a number of factors that contributed to the Y2K success, including effective project tracking, a standardized control structure, and the use of steering committees. The continued use of these management techniques can contribute significantly to the success of future IRS systems enhancement activities. We suggest that the Chief Information Officer use similar project management processes across all major systems initiatives, incorporate industry best practices into its asset management program, develop a standardized method for deploying changes and upgrades across the entire spectrum of the IRS’ systems and networks, and establish baselines and guidelines to enhance the efficiency and effectiveness of future systems integration tests.
Management's Response: The Commissioner agreed with the report and provided highlights of the IRS’ efforts to use various project management techniques, deploy information technology components, and conduct systems integration tests.
Management’s complete response is included as Appendix IV to this report.
We contracted with PricewaterhouseCoopers (PWC) to provide technical assistance to our audits of the Internal Revenue Service’s (IRS) Year 2000 (Y2K) conversion initiatives. At the completion of the Y2K audit work, we requested that PWC assist in identifying long-term organizational benefits that the IRS might derive from its experience in managing the Y2K conversion activities. We also reviewed the last phase of the IRS’ End-to-End (E2E) Systems Integration Test to assess its effectiveness and to identify potential improvements for future E2E systems integration efforts.
This management advisory report presents suggestions to the IRS based on lessons learned through reviews of the IRS’ Y2K conversion activities and E2E Systems Integration Test. The suggestions contained in this report are based on observations made throughout PWC’s engagement, its experience with a variety of government and private entities, and its knowledge of industry best practices.
PWC provided assistance from March 1998 through February 2000. PWC’s work was performed at numerous IRS sites nationwide. Our reviews of the last phase of the IRS’ E2E Systems Integration Test were conducted from November 1999 through February 2000 within the Office of the Chief Information Officer (CIO), including work at the Tennessee Computing Center, the Detroit Computing Center, and the Integration, Test, and Control Center in New Carrollton, Maryland.
Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.
The IRS’ preparation for Y2K tax processing was one of its most critical undertakings. The IRS established the Century Date Change (CDC) Project Office in 1996 to provide oversight and to direct the conversion of all IRS information technology (IT) systems and non-IT elements that would be affected by the Y2K date change. The primary goal of the CDC Project Office was to ensure that all IRS systems were compliant prior to January 1, 2000. In helping to achieve this goal, the CDC Project Office was assisted in the conversion by support organizations, work groups, teams, steering committees, and technical contractor support. The Office received active support and visibility from the Commissioner and the CIO and was provided the necessary budget support.
As the final step in its Y2K readiness efforts, the IRS planned to verify the accurate operation of all programs and equipment that would be used on and after January 1, 2000, to process approximately 213 million tax returns. To confirm system and tax processing readiness for the Year 2000, the Information Systems Product Assurance function, with significant contractor support, developed a series of E2E systems integration tests.
The IRS’ Y2K remediation efforts demanded extraordinary efforts from all areas of the organization. Although the Y2K conversion has now been successfully completed, the IRS can still realize long-term benefits from this vast undertaking that cost approximately $1 billion. The IRS can preserve the resources, methodologies, and knowledge gained through the Y2K remediation and deploy them in the future. We have identified five major areas in which the IRS can apply the lessons learned from its Y2K remediation efforts:
Management's Response: The Commissioner agreed with the report and provided highlights, in Appendix IV, of the IRS’ efforts to use various project management techniques, deploy information technology components, and conduct systems integration tests.
Effective Project Tracking, a Standardized Control Structure, and Use of Steering Committees Were Significant Critical Success Factors That Should Continue to Be Deployed
There were several significant critical success factors that provided an effective framework for managing the IRS’ Y2K efforts. These critical success factors included:
Implementation of a Century Date Change Project Office
The CDC Project Office proved that centralizing certain aspects of project tracking could promote consistency and efficiencies across the IRS. By successfully implementing many of the features expected of a well-run project, the CDC Project Office was able to harness the resources and segment tasks so that the Y2K conversion could be performed in a systematic and orderly fashion. The CDC Project Office established the overall project policies and procedures for the IRS to follow in the conversion effort and created extensive conversion schedules and milestones. The Integrated Network and Operations Management System (INOMS), an inventory which contains information for all IRS applications and commercial off-the-shelf (COTS) products, was modified to meet the CDC Project Office’s need for an automated tracking mechanism and served as the primary control for tracking the overall conversion effort.
The CDC Project Office established a standardized process for reporting on Y2K conversion progress. This process clearly highlighted tasks that were falling significantly behind schedule so that executive management attention could be channeled to areas where it was most needed. The CDC Project Director also set up a risk identification system that was used to identify specific risk areas so that a schedule could be established for addressing the risks and progress in reducing each risk could be reported to the Project Office.
Establishment of a project-specific standardized methodology
The IRS’ success in adhering to a standardized methodology demonstrates the importance of structuring an organized approach to solve a widespread problem. The CDC Project Office established a 14-step process that provided an effective control structure. This process consisted of verifiable milestones and was used to measure both the Y2K conversion status of each individual item and to aggregate agency-wide conversion status at the IRS.
The primary activities within the 14-step process involved identifying the items for conversion, performing an analysis to assess the scope of conversion requirements, making the necessary changes to each item in the inventory, performing various levels of testing, and certifying the items ready to operate after December 31, 1999. In the CDC Project Director’s view, this structure provided an "auditable" framework that could be used by independent parties to verify the status of conversion efforts. As a result, oversight groups such as the Treasury Inspector General for Tax Administration (TIGTA) and the General Accounting Office were able to verify progress in the IRS’ Y2K efforts.
Establishment of steering committees to oversee the progress of Y2K initiatives
During the Y2K initiative, several steering committees were established to address enterprise-wide issues that posed threats to Y2K conversion progress and success. The Commissioner established a steering committee of senior IRS executives to oversee the overall Y2K effort. This committee was very effective in bringing immediate agency-wide attention to problems that could have seriously detracted from the IRS’ Y2K conversion success.
Steering committees were also used at lower management levels with good results. For example, an Inventory Steering Committee comprised of representatives from across the IRS was established to address issues related to the INOMS database. One of the prime benefits of this group was that it improved communications between field and headquarters personnel.
We suggest the CIO review major legacy, maintenance, and systems modernization projects to determine whether their governance can be improved by applying standardized tracking mechanisms similar to those used for the CDC Project.
Program/Project Management Processes, Procedures, and Controls Need to Be Implemented and Enforced Across the Internal Revenue Service
The implementation of a centralized program management function over major initiatives provides several major benefits:
Although the CDC Project Office established and used effective project management controls for the overall Y2K conversion effort, several major subtasks were executed with little or no project management discipline. This situation was most evident in those instances where the IRS had to inventory and replace equipment and software at its many field locations. As a result, the CDC Project Office found itself revising the status of Y2K completion schedules to show slippage and increasing the cost estimates for the IRS’ Y2K conversion efforts.
Several subtasks, such as the efforts to assure the Y2K compliance of telecommunications equipment and of IRS workstations, showed deficiencies that are typical of inadequate project management. These shortcomings included:
Many of these shortcomings were encountered by the IRS in working with the Treasury Communications Systems (TCS) contractor to establish a baseline telecommunications inventory. At an early stage in the Y2K conversion effort, the IRS recognized that it needed to ensure that all equipment purchased and maintained through the TCS contract was properly accounted for. To accomplish this, the IRS decided that the TCS inventory for Y2K purposes should be maintained in a common database with telecommunications components that the IRS purchased from other sources. The contractor was also tasked with performing a physical inventory at 415 IRS sites. These efforts required both an initial download from the TCS contractor’s database and a monthly update from the contractor’s database to the IRS database.
From the very outset of the attempts to update inventory data, the major deficiency and underlying cause of problems was poor communication. Both the IRS and the contractor had different business practices, different data requirements, and different definitions of what constituted a telecommunications component. There were frequent delays in completing the update of the IRS database with TCS data, and the process for managing this updating activity was not clearly established. The task involving the inventorying of 415 IRS sites was not subjected to appropriate quality control and, ultimately, the IRS determined that the inventory results were inaccurate and unusable. The responsibility for validating the results of the contractor’s inventory and making appropriate updates to the central IRS inventory was never firmly set. The problems in using the contractor’s inventory information were not effectively communicated. As a result, it was not until July 1999 that it became clear that the inventory information had, for the most part, not been used to update the IRS’ inventory.
The symptoms of poor project management are often accompanied by significant risks:
These risks manifested themselves during the IRS Y2K conversion efforts. The poor quality of a contractor’s inventory efforts has already been discussed as one example of inadequate project management.
The IRS’ efforts to assess the impact of the Year 2000 on its workstations presents another example where the better application of project management disciplines would have been useful. In June 1999, the IRS realized that it needed to replace almost 20,000 workstations because the current workstations would not be able to run the Y2K compliant software. This situation had not been included in the IRS’ budget and resulted in the need for the IRS to request emergency funding to replace these workstations.
The IRS also encountered problems with its status reporting of its inventory. In July 1999, the IRS reported that over 97 percent of its telecommunications inventory was Y2K compliant, but a review by TIGTA of 21 sites indicated that the compliance status of 19 percent of the equipment could not be determined.
We suggest the CIO review current legacy and maintenance systems and telecommunications efforts to identify those that would benefit from a centralized project management function and establish guidelines for identifying such projects in the future.
The Internal Revenue Service Should Continue Its Initiative to Implement a More Comprehensive Program to Manage Its Information Technology Assets
The IRS encountered numerous and continual problems identifying and updating its information technology inventory throughout the Y2K conversion effort. The IRS needs to maintain the inventory that was assembled initially for its Y2K efforts as the basis of a more comprehensive program to manage and protect its extensive information technology investments. As a response to its Y2K experiences, the IRS has initiated significant steps to better manage its inventory and to implement a broad management program for its information technology assets. These steps include the establishment of an Asset Management Modernization Project Office.
For an asset management program to be successful, it must be comprehensive and include controls, centralized tracking and reporting, and links among all phases of the asset life cycle from acquisition through retirement. The problems the IRS encountered in assembling its Y2K inventory resulted, in part, because these management structures were either missing or inadequate. In addition, there were other factors that contributed to the continual inventory problems encountered by the IRS, including:
Without an effective IT asset management process, the IRS’ ability to make future changes to its systems in a cost effective and timely manner will be impaired. The IRS’ Y2K efforts included situations where an inordinate amount of time and effort was devoted to determining what system components actually needed to be changed and how many pieces or copies of each component existed. For example, at one point, the IRS inventory showed that it had over 200,000 workstations and laptops, although the IRS only had approximately 100,000 employees. It was known that this number overstated the actual number of workstations and laptops, but the IRS had to expend a great deal of effort to identify the workstations and the software configurations associated with those workstations that were actually in use. The IRS encountered a related problem in determining how many copies of various widely used COTS software products existed and how many had to be replaced.
At the time we performed our inventory reviews, the IRS lacked many of the features that have been identified as industry best practices in asset management. These best practices include:
We suggest the CIO assure that the industry best practices listed above are incorporated into the IRS’ future IT Asset Management Program.
The Internal Revenue Service’s Methods for Deploying Computer Systems and Telecommunications Components Can Be More Efficient
During the Y2K conversion efforts, we encountered many situations in which it was difficult to determine whether equipment or software had actually been updated to be Y2K compliant. In part, this occurred because of the inventory problems discussed earlier in this report. Another factor contributing to this problem was the IRS’ lack of a standardized process for deploying changes in its hardware, software, and telecommunications environment to its many field locations.
If the IRS had used a standardized deployment process during its Y2K activities, the task of verifying the status of Y2K conversion efforts would have been greatly simplified and costs of confirming Y2K progress would have been reduced. A significant part of the IRS’ Y2K efforts were devoted to establishing and verifying the software and hardware configuration of its systems and networks.
Although the IRS has clearly defined transmittal (deployment) processes for its major mainframe systems, the IRS generally lacks a comprehensive configuration management program across all its systems and does not have a formal transmittal process for many of its non-mainframe systems and telecommunications projects.
Where deployment processes do exist, procedures are often inconsistent across the various computer and telecommunications environments, and the methods for confirming the implementation of new or updated products are ineffective.
The lack of a feedback element to verify that system changes have occurred was not limited to the telecommunications environment. Four of nine minicomputer systems we queried in August 1999 lacked feedback mechanisms to ensure that upgrades to COTS software had, in fact, been installed. In addition, the problems that the IRS had in identifying its actual workstation needs (mentioned earlier in this report) occurred, in part, because the IRS did not have effective configuration information for some of its systems.
The absence of a structured deployment process can lead to other problems, such as inaccurate status reporting to senior management and the lack of management control over the reporting process. During the Y2K conversion efforts, such inaccurate status reporting did occur and a great deal of effort was expended to obtain correct information about what products had been correctly upgraded to be Y2K compliant.
In the future, the IRS will need to implement more cost effective ways to make changes to its systems and to verify that systems are operating with approved equipment and software. The IRS has begun to address the issue of establishing a configuration management process. As the IRS strives to improve its methods for implementing control over its IT environment, there are a number of industry best practices that we suggest the IRS consider. These practices for managing the deployment of systems, systems components, and telecommunications resources include:
Improvements Can Be Made to Future Systems Integration Testing
We identified the following steps the IRS could take to improve future systems integration testing efforts:
The Product Assurance function has placed the entire E2E methodology (including lessons learned, test plans and reports, and templates for the creation of each plan and report) on its website for future on-line reference. This should facilitate future testing efforts.
Baselines for future E2E Systems Integration Tests need to be properly established beforehand and controlled throughout the testing process
A key management mechanism for any project is the establishment of a baseline against which actual performance can be assessed. Once a baseline is established, changes to the baseline should be evaluated and approved and the reasons for changes should be documented. We determined that a baseline was not established against which changes and performance could be assessed.
The Product Assurance function had to rely on the Independent Verification and Validation contractor to perform a great deal of reconciliation work between the testers’ documentation and the Test Case Database (where the Systems Integration Tests were documented) before there was an objective baseline for measuring the completeness and success of the E2E Systems Integration Test.
Since this was the IRS’ first effort to conduct an E2E systems integration test, it is understandable that the IRS could not identify all the required test threads in advance of the actual test. In the future, however, the IRS will not have the resources to perform extensive verification of its test cases during the test. A more cost-effective method would be to establish a baseline of test cases and threads before the beginning of the test and control them through a disciplined change management process. The IRS invested heavily in identifying test threads for its Y2K E2E Systems Integration Test. The extensive work in identifying the interrelationships among its computer systems as captured in the Test Case Database can serve as an excellent starting point for preparing this baseline.
For future annual testing efforts, we suggest the CIO establish baselines of IRS data against which changes can be reviewed, evaluated, and documented and continue to update and use the test thread database as the basis for future systems integration tests.
Elements of the E2E Systems Integration Test administration could be improved for future testing efforts
The E2E Systems Integration Test experienced delays because of the need for frequent rerunning of programs and because of delays in identifying test problems and resolving them. In the future, the IRS will likely continue to conduct systems integration tests after Systems Acceptability Testing (SAT) but with less time available. More advanced planning is needed to assure that the test schedule is not compromised by numerous minor technical problems.
Our experience during the test identified several areas where administrative practices within the E2E Systems Integration Test could have been improved and time and resources could have been saved in identifying and solving test problems. These areas include: assuring operational issues are addressed prior to the execution of test runs; providing validity checks for key fields used to identify testing problems; and implementing procedures to ensure correct versions of programs are transmitted for testing.
A significant number of operations problems required tests to be rerun or delayed. An analysis of the E2E Systems Integration Test Help Desk Database showed that a significant number of operational problems required tests to be rerun or delayed. Many of these problems could have been avoided through better planning and coordination among the various staffs involved in the test. We found delays caused by job control language errors, test sequencing and scheduling conflicts, and problems entering input and sending output because of misidentified equipment. Each of the errors represents a test that ended without the expected results and required a scheduled rerun. Without the reruns, the Product Assurance staff could have used its time more effectively.
Specific problem identifiers were not included in search fields of the IRS’ problem tracking database that is part of the INOMS. The Y2K E2E Help Desk reported that several testing problems were not included in problem reports for Y2K E2E Testing due to incorrect information in the "System Name" field of the INOMS database. Incorrectly entering information in the INOMS database increases the likelihood of programs with unresolved problems being placed into production. In addition, implementation of corrective actions for potential redundant problems was hindered when recurring problem research could not be effectively conducted and specific types of problems were not accurately tracked and/or monitored.
The Y2K E2E Systems Integration Test on numerous occasions received incorrect versions of production software. Incorrect versions of processing programs resulted in unexpected reruns of E2E tests and delays in the overall test schedule.
Version control should be achieved through established procedures using a tool specific to the particular system being tested. The Information Systems Internal Revenue Manual states that the transmittal system is established to provide a means of effectively monitoring and controlling releases of application software, documentation, and run control information.
For future annual testing efforts, we suggest that the CIO analyze the lessons learned in the End of Test Status Report and establish testing guidelines that, at a minimum, include:
The overall project management structure used by the IRS in addressing its Y2K conversion efforts should serve as a model that can be applied to all major upgrade and improvement projects. The IRS needs to implement and enforce program/project management processes, procedures, and controls across all its major systems initiatives, including major changes and upgrades to its legacy systems.
In initiating the major undertaking of implementing a broad asset management program for information technology, the CIO needs to assure all facets of industry best practices in the management of information technology assets are considered and implemented. The IRS also needs to standardize and make more efficient its methods for deploying computer systems and telecommunications components to its field locations.
Through our observation of the IRS’ Y2K E2E Systems Integration Test, we found many successful efforts. However, future systems integration testing efforts could be improved by establishing testing baselines against which changes in the test plan can be evaluated and addressing operational issues that hindered completion of the entire testing schedule.
Appendix I
Detailed Objective, Scope, and MethodologyIn this audit, we reviewed Treasury Inspector General for Tax Administration (TIGTA) and PricewaterhouseCoopers (PWC) findings from Year 2000 (Y2K) reviews and analyzed the data to develop lessons learned to apply to future Internal Revenue Service (IRS) initiatives. We also assessed the effectiveness of the preparation for and execution of Test 3 of the IRS End-to-End (E2E) Systems Integration Test for Y2K tax processing to identify potential improvements in future E2E systems integration efforts. Specifically, we:
Appendix II
Major Contributors to This ReportScott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)
Gary V. Hinkle, Director
Vincent J. Dell’Orto, Audit Manager
Anthony Knox, Senior Auditor
Van Warmke, Senior Auditor
Mark Carder, Auditor
Steve Gibson, Auditor
Myron Gulley, Auditor
Olivia Jasper, Auditor
Barbara Sailhamer, Auditor
Appendix III
Report Distribution ListDeputy Commissioner Modernization C:DM
Deputy Commissioner Operations C:DO
Chief Information Officer IS
Deputy Chief Information Officer, Operations IS
Deputy Chief Information Officer, Systems IS
Director, Century Date Change Project Office IS:CD
Director, Enterprise Operations IS:E
Director, Information Systems Field Operations IS: F
Director, Office of Security and Privacy Oversight IS:SPO
Director, Product Assurance IS:PA
Director, Service Center Operations IS:SC
Director, Telecommunications IS:T
Office of Information Resources Management IS:IR
Director for Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis M:O
Office of Management Controls CFO:A:M
Office of Chief Counsel CC
National Taxpayer Advocate C:TA
Audit Liaison, Program Oversight & Management Controls Office IS:IR:OMA
Appendix IV
Management's Response to the Draft ReportThe response has been removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.