The Internal Revenue Service Can Improve Its Capacity and Performance Program to Better Manage the Availability of Its Computer Systems
February 2000
Reference Number: 2001-20-046
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
February 13, 2001
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: (for) Pamela J. Gardiner /s/ Scott E. Wilson
Deputy Inspector General for Audit
SUBJECT: Final Audit Report - The Internal Revenue Service Can Improve Its Capacity and Performance Program to Better Manage the Availability of Its Computer Systems
This report presents the results of our review of the Internal Revenue Service’s (IRS) capacity and performance management program. In summary, we found that the IRS adequately manages the capacity and performance of its mainframe computer systems, but a more comprehensive program is needed for managing its mid-range systems. Additionally, best practices and other procedures need to be documented to ensure there is consistency in the IRS
’ capacity planning and performance management program.We recommended that the Chief Information Officer ensure the program for managing the capacity and performance of mid-range systems is comprehensive, that essential technical information needed for the program is developed, and that currently available information is adequately publicized. Management agreed with our recommendations and developed appropriate corrective actions. Management’s response has been incorporated into the report where appropriate, and the full text of the response is included as an appendix.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or your staff may call Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs), at (202) 622-5896.
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Management’s Response to the Draft Report
Management of the capacity and performance of the Internal Revenue Service’s (IRS) computer systems is critically important to the IRS’ success in providing top quality customer service to taxpayers. To achieve this goal, the IRS is increasingly relying on the ability of its computer systems to provide quick and convenient access to taxpayer information. Therefore, the IRS’ capacity and performance management effort must be comprehensive in ensuring that its computer systems have sufficient capacity to process taxpayer information and meet computer systems users’ performance needs.
The objective of this review was to assess the efficiency, economy, and effectiveness of the IRS’ capacity and performance management program for mainframe and mid-range computer systems. We conducted fieldwork at the IRS National Headquarters; the Martinsburg, Detroit, and Tennessee Computing Centers; the Memphis Customer Service Site; and the former Baltimore District Office.
Results
The IRS has developed a program to adequately manage the capacity and performance of some of its most critical computer systems. However, two areas were identified where the IRS could improve its efforts to better meet the increasing demand being placed on its computer systems.
The Internal Revenue Service Adequately Manages the Capacity and Performance of Its Mainframe Computer Systems
The IRS’ program for managing the capacity and performance of its mainframe computer systems includes all of the elements specified for such a program by industry best practices. Collectively, the technical staff of the program has a significant amount of experience with the IRS’ complex computer environment. As a result, the IRS is adequately managing the current capacity needs for these computer systems and planning for future capacity needs. The IRS is also actively monitoring the performance of these critical tax processing systems.
A More Comprehensive Effort Is Needed to Effectively Manage the Capacity and Performance of Mid-Range Computer Systems
The IRS does not have a comprehensive program for managing the capacity and performance of its mid-range systems, totaling 628 computers. These systems run many of the IRS’ taxpayer programs, including those for electronic filing and identifying taxpayers who are underreporting their taxes. The IRS’ program for managing its mid-range systems is primarily focused on analyzing the current utilization rates of a relatively small number of its mid-range systems. Little long-term trending or monitoring occurs, and limited proactive analyses are performed for these computer systems. As a result, the IRS cannot ensure that its mid-range computer systems have adequate capacity. Since the IRS will have spent over $33.7 million by the end of Calendar Year 2000 and plans to spend at least an additional $31.8 million over the next 2 years to upgrade and consolidate its mid-range computer systems, increased capacity and performance analysis of the IRS’ mid-range computer systems is needed to ensure that adequate information is available for making acquisition decisions, determining whether current capacity is being used effectively, and identifying whether excess capacity exists.
Comprehensive Guidelines and Best Practices Would Improve Management of Both Mainframe and Mid-Range Computer Systems
The IRS does not have comprehensive documentation of the procedures, methods, and techniques needed to manage the capacity and performance of its mainframe and mid-range computer systems. Such documented guidance is needed for two primary reasons: 1) to capture the technical expertise of the IRS’ most experienced mainframe Computer Performance Evaluators, a significant number of whom are or will be eligible to retire within the next 5 years, upon which the success of the IRS’ capacity and performance program has in part been based, and (2) to communicate essential technical information to both new and inexperienced mainframe and mid-range computer technical staff. Without this technical documentation of the methods and techniques to be used by the technical staff, the IRS will find it difficult to maintain or expand its capacity and performance management program and to assure that in the future its technical staff will possess the expertise needed to assume system performance activities.
Summary of Recommendations
The Chief Information Officer (CIO) needs to ensure the program for managing the capacity and performance of its mid-range systems is comprehensive and includes a strategy for evaluating all of the IRS’ consolidated mid-range systems. Additionally, since the IRS may lose a significant portion of its experienced capacity and performance management staff to retirement over the next 5 years, the CIO should ensure that essential technical information currently being used or needed to monitor the capacity and performance of its mainframe and mid-range computer systems is defined, documented, and communicated to the necessary technical staff. In addition, the CIO should ensure that the Capacity Management Branch (CMB) web site and the information available at the site are adequately publicized throughout the organization.
Management’s Response: The CIO has developed a strategy to manage the capacity and performance issues for the IRS’ mid-range computer systems, which includes installing monitoring software on mid-range computer systems to automate data collection and reduce resources needed to manage these systems. In addition, the CIO has identified several field technical staff who can be assigned to support the CMB. However, if reassigning the field technical staff is not enough, the CMB will hire contractors to increase support in critical areas.
The CMB will develop written guidance on procedures, methods, techniques and other information unique to each CMB report, which will be made available on the CMB web site. The CMB will publicize its web site in a variety of ways, including e-mail, IRS publications, placement on IRS web sites, and an awareness campaign.
This report presents the results of our review of the Internal Revenue Service’s (IRS) capacity planning and performance management program. This review was directed at assisting the IRS in achieving its goal of providing "top quality service to each taxpayer" by evaluating whether the IRS’ capacity and performance program is capable of ensuring that its computer resources are keeping pace with the organization’s changing functional and performance demands and will continue to do so in the future.
The overall objective of this review was to assess the efficiency, economy, and effectiveness of the IRS’ capacity and performance management program for mainframe and mid-range computer systems. Specifically, we:
Audit work was conducted at the IRS National Headquarters; the Martinsburg, Detroit, and Tennessee Computing Centers, the Memphis Customer Service Site; and the former Baltimore District Office from January to August 2000. This review was performed in accordance with Government Auditing Standards.
Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.
To meet the goals of providing top quality customer service, the IRS is increasingly relying on the ability of its computer systems to provide quick and convenient access to taxpayer information. To provide more convenient access, the IRS has been expanding the number of hours per week that its systems are available for accessing information. For example, in August and September 2000, the Martinsburg Computing Center expanded the access time for two of the IRS’ key tax processing systems - the Integrated Data Retrieval System (IDRS) and the Automated Collection System.
To accommodate the expanding need for longer periods of quick, on-line access to information by IRS business users, the IRS’ computer systems will need to have sufficient capacity and to be continually monitored to assure that they will meet the demands for increased on-line access while still completing other essential IRS work in a timely manner. For example, the IRS schedules its processing so that many transactions updating taxpayer accounts are processed in batches at times when on-line access (realtime) is not available, typically during nights and weekends. Consequently, as the hours of on-line availability are expanded, there is less time to process batch transactions, making the IRS’ ability to process a greater number of transactions increasingly difficult without sufficient capacity. The IRS is currently in the process of upgrading one of its mission-critical mainframe computer systems to ensure that the system continues to have the processing capacity needed to process its weekend batch workload in the available time frames. To assure that its computer systems are able to provide quick and convenient access, IRS management must have an effective program for managing capacity and performance.
Our review of industry best practices found that effective capacity and performance management programs include three essential processes. One is trending, which involves analyzing historical system capacity and performance data to identify future workload trends. A second is proactive analysis, which includes analyzing short-term workload patterns to address performance problems, possibly before they occur and escalate into significant degradations in performance. The third is monitoring, which is similar to proactive analysis, but involves continuously examining computer systems to address performance problems as timely as possible.
Performance problems are generally experienced by end-users. The problems are usually related to one of the following: the inability to access computer systems or long delays in retrieving and/or updating information.
For the IRS, capacity and performance management is generally the responsibility of the Capacity Management Branch (CMB), located in the Information Systems, System Support Division. Supporting the CMB’s efforts are the computer performance evaluators (CPEs) and systems administrators (SAs) located at IRS facilities, including service and computing centers. The CPEs are responsible for the performance management of the IRS’ mainframe computer systems, and SAs are responsible for the overall administration of the IRS’ mid-range computer systems.
Complicating the IRS’ capacity and performance management effort are the size and complexity of the IRS’ computer system environment. As of August 2000, this environment included 41 mainframe computers from 3 manufacturers and 628 minicomputers from 22 manufacturers. As a result, managing the performance and capacity of these computer systems requires a comprehensive program that considers both the number and size of these computer systems and their unique characteristics.
The IRS has developed a program to adequately manage the capacity and performance of some of its most critical computer systems. The IRS’ program, to varying extents:
Also, the IRS is making progress in defining and documenting overall end-user performance goals and expectations for the computer systems located at the IRS’ computing centers. For example, the Detroit Computing Center had initiated a program to develop service level agreements for critical and newly developed mainframe and mid-range applications in order to define end-user expectations and monitor the achievement of users’ goals.
While the IRS’ efforts in managing the capacity and performance of its mainframe computer systems are adequate, there is a need for improvement in managing its mid-range computer systems. For example, the program for mid-range computer systems does not always include consistent monitoring and proactive analysis of these computer systems, due in part to the large number of such computer systems used by the IRS.
In addition, there is a need for the IRS to develop comprehensive documentation of the procedures, methods, and techniques needed for managing the capacity and performance efforts for both its mainframe and mid-range computer systems.
The Internal Revenue Service Adequately Manages the Capacity and Performance of Its Mainframe Computer Systems
The IRS’ program for managing capacity and performance of its mainframe computer systems includes all of the elements of such a program as specified by industry best practices. Collectively, the technical staff of the program has a significant amount of experience with the IRS’ complex computer environment. Specifically, the IRS’ mainframe program includes the following processes:
As a result, this program includes managing both the planning for future capacity needs and the adequate monitoring of performance.
A More Comprehensive Effort Is Needed to Effectively Manage the Capacity and Performance of Mid-Range Computer Systems
The IRS needs a more comprehensive program for managing the capacity and performance of its mid-range computer systems, to make it comparable to its program for mainframe systems. Without an effective program to manage its mid-range systems, the IRS may not have sufficient information to make investment decisions for its mid-range systems. Additionally, without a more comprehensive program, the IRS may not be able to assure its end-users that the computer systems will be available when needed.
Many of the IRS’ taxpayer programs are run on mid-range computer systems, including those for electronic filing and identifying taxpayers who are underreporting their taxes. The current state of the IRS’ capacity and performance management efforts for these mid-range systems can be summarized as follows:
The IRS’ efforts do not address all of the mid-range computer systems due in part to the large number of such computer systems used by the IRS. As of August 2000, the IRS used 628 mid-range computer systems from 22 different manufacturers to support its tax processing applications. According to the CMB, the Branch does not have sufficient resources to analyze all of the IRS’ mid-range computer systems or to assess capacity and performance trends for them. Consequently, the CMB focuses on assessing the current utilization of a selected number of computer systems.
Unlike the mainframe program, the IRS’ mid-range capacity and performance management program does not have personnel solely committed to performing the tasks and analyses to consistently monitor or proactively analyze the mid-range computer systems. For the mainframe program, CPEs are committed to performing the capacity and performance management tasks. The CPE-type activities for performance management of mid-range systems are performed by the SAs, who are also responsible for the overall administration of these systems. As a result, performance management duties often receive a lesser priority.
Without a comprehensive program for its mid-range computer systems, the IRS cannot ensure that adequate information will be available to plan for the acquisition of additional capacity, assess whether current capacity is being used effectively, or know whether excess capacity is available. Since the IRS will have spent over $33.7 million by the end of Calendar Year 2000 and plans to spend at least an additional $31.8 million over the next 2 years to consolidate and upgrade its mid-range computer systems, the need for this type of capacity and performance analysis will become increasingly important.
In its 1999 capacity reports, the CMB had already identified 26 mid-range systems that had exceeded acceptable thresholds for processing, memory, and/or storage capacity. However, since not all mid-range systems have been studied, and those studies that had been conducted covered only current utilization, the IRS does not have all of the information available to be able to strategically plan for current and future capacity needs. In addition, since budget requests are input nearly 2 years in advance, the IRS may not have adequate information to ensure sufficient funds to add capacity will be available when needed. Furthermore, without personnel to consistently monitor and proactively analyze the IRS’ mid-range computer systems, those computer systems nearing capacity may experience slower response times, or worse, may not be available for use, thus affecting end-users’ abilities to do their work in a timely fashion.
The IRS’ consolidation of its mid-range computer systems provides an example of how capacity planning information is needed. The IRS is currently in the process of consolidating its mid-range computer systems in its computing and customer service centers. In planning this effort, information on the current and future capacity needs of the computer systems to be consolidated would have been useful in estimating the total cost of the project. In addition, though some capacity information is now being used to amend the schedule of computer systems to be consolidated, such information would have been useful in developing the migration schedule. The information would have allowed the IRS to prioritize the schedule during planning, based in part on when computer systems would be at or nearing acceptable capacity thresholds.
The IRS’ mid-range consolidation project will provide the IRS with an opportunity to improve the program for managing the capacity and performance of its mid-range computer systems. The consolidation, while occurring over the next several years, should have the following effects:
Thus, the IRS should consider organizationally reallocating and realigning the available SAs to carry out the capacity and performance management tasks for the newly consolidated systems. With fewer mid-range computer systems to manage and a number of SAs specifically tasked with system capacity and performance activities, the CMB should have sufficient resources to study the current and future capacity trends for all consolidated mid-range computer systems.
Recommendations
Management’s Response: The CIO has developed a strategy to manage the capacity and performance issues of the IRS’ mid-range computer systems. This strategy includes analyzing the capacity of all systems scheduled for consolidation. In addition, monitoring software will be installed on mid-range computer systems to automate data collection and reduce the resources needed to manage these systems.
Management’s Response: The CIO has identified several CPEs and SAs who can be assigned to support capacity management or other appropriate organizations. In addition, the CMB will need to fill key technical positions to replace the loss of experienced staff. However, if reassigning the field technical staff is not sufficient, the CMB will hire contractors to increase support in critical areas.
Comprehensive Guidelines and Best Practices Would Improve Management of Both Mainframe and Mid-Range Computer Systems
The IRS does not have comprehensive documentation of the procedures, methods, and techniques needed to manage the capacity and performance of its mainframe and mid-range computer systems. Such documented guidance is needed for two primary reasons:
Without the technical documentation of the methods and techniques to be used by the CPEs and SAs, the IRS will find it difficult to expand the capacity and performance management program of its mid-range computer systems and to maintain the current level of its program by assuring that its future technical staff will possess the expertise needed to assume system performance activities.
The overall duties and responsibilities of the CPEs and SAs are defined in the Internal Revenue Manual (IRM). However, the IRM does not specify the detailed procedures, methods, and techniques needed to execute these responsibilities. The CMB is tasked, through its mission statement, with providing functional and technical guidance to business, Information Systems, and field organizations, which include the field CPEs and SAs. The guidance for managing capacity and performance for both the mainframe and mid-range computer systems should be comprehensive, to ensure that the best practices for managing these systems are consistently used by all CPEs and SAs.
In the absence of specific guidance, many of the CPEs and SAs either develop their own informal procedures or use vendor documentation. In some cases, these procedures do not provide all of the information needed by the CPEs and SAs. For example, one group of SAs wanted to more proactively tune their computer systems; however, they did not have any guidance on how to do this.
The CMB’s primary source of guidance to the CPEs and SAs is the capacity reports prepared by the CMB. These reports contain the methodology used to analyze a system’s capacity and performance as well as the results, conclusions, and recommendations based on the analysis. The reports are posted to the CMB’s web site, which also includes performance graphs for most of the IRS’ mainframe computer systems and two mid-range systems as well as administrative information on the CMB and suggested third-party training providers. The CMB plans to continue improving the site by adding additional reports and graphical information.
Although the CMB has made progress in placing information on its web site, the site does not have the detailed information needed by the CPEs and SAs. Our discussions with CPEs and SAs identified that some did not know about the web site and others did not find the information useful. The CMB has recognized the need to aggressively market its web site by taking the opportunity to present an overview of its products and services during visits to IRS field sites but has not yet undertaken an extensive effort to make its web site known within the IRS.
In developing comprehensive procedures, techniques, and methods, the IRS would also have the opportunity to capture a significant amount of experience that may be lost due to the retirement of CMB analysts. Over the next 5 years, the CMB is facing a potential turnover due to retirements of a third of its staff, with approximately 20 percent of the staff currently eligible. The CMB has collectively lost 102 years of experience due to retirements since June 2000, with the potential to lose an additional 221 years of service and experience due to retirements over the next 5 years. Since the CMB’s staff not only conducts capacity analyses but also assists in diagnosing and resolving emerging system problems, this loss could limit the CMB’s ability to manage the capacity and performance of the IRS’ computer systems. Both CMB and CPE personnel have indicated that it takes several years of experience with the IRS’ computer systems for CPE analysts to become fully proficient in dealing with capacity and performance issues within the IRS’ complex computer environment.
Recommendations
Management’s Response: The CMB will develop written guidance that describes procedures, methodologies, data gathering techniques, frequency of reports, and other information unique to each CMB report. This guidance will be made available on the CMB web site.
Management’s Response: The CMB will publicize its web site in several ways, including e-mail, announcements in IRS publications, and an awareness campaign. In addition, other organizations will be asked to help publicize the CMB web site. The CMB will also register the web site on the IRS’ Intranet search engine.
While the IRS has developed a program to adequately manage the capacity and performance of its most critical computer systems, there are two areas where improvements can be made. The IRS needs to comprehensively manage the capacity and performance of its mid-range systems as well as define, document, and communicate essential technical information for managing the capacity and performance of its mainframe and mid-range computer systems. Such improvements will enable the IRS to better meet the increasing demand being placed on its computer systems as well as protect its current and future investment in its computer systems.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to assess the efficiency, economy, and effectiveness of the Internal Revenue Service’s (IRS) capacity and performance management program. This review assessed the IRS’ capacity and performance management program for mainframe (Tier I) and mid-range (Tier II) systems only. Specifically, we:
II. Determined whether the IRS had an effective program for managing the system capacity and performance needs of its Tier I and II systems.
III. Determined whether the IRS had an effective process for monitoring and addressing system capacity and performance problems on an as-needed basis for Tier I and II systems.
Appendix II
Major Contributors to This Report
Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)
Gary Hinkle, Director
Vincent J. Dell’Orto, Audit Manager
Barbara Bartuska, Senior Auditor
Michael Howard, Senior Auditor
Arthur Granger, Auditor
Olivia Jasper, Auditor
Nikki Thomas, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Deputy Chief Information Officer, Operations IS
Deputy Chief Information Officer, Systems IS
Director, Enterprise Operations IS:E
Director, Information Resources Management IS:IR
Director, Information Systems Service Center Operations IS:SC
Director, Strategic Planning and Client Services IS:SP
Director, Systems Development IS:SD
Director, Systems Support Division IS:SS
Director, Detroit Computing Center IS:E:DC
Director, Martinsburg Computing Center IS:E:MC
Director, Tennessee Computing Center IS:E:TC
Chief, Program Oversight and Coordination IS:SP
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Director, Legislative Affairs CL:LA
Chief Counsel CC
National Taxpayer Advocate TA
Office of Management Controls N:CFO:F:M
Audit Liaison:
Chief Information Officer IS
Appendix IV
Management’s Response to the Draft Report
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.