Mainframe Computer Performance Is Being Actively Monitored, but Defined-Service Agreements and Software Licensing Can Be Improved
September 23, 2011
Reference Number: 2011-20-074
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
Phone
Number | 202-622-6500
Email Address | TIGTACommunications@tigta.treas.gov
Web Site |
http://www.tigta.gov
HIGHLIGHTS
MAINFRAME COMPUTER PERFORMANCE IS
BEING ACTIVELY MONITORED, BUT DEFINED-SERVICE AGREEMENTS AND SOFTWARE LICENSING
CAN BE IMPROVED
Highlights
Final Report issued on September
23, 2011
Highlights of Reference
Number: 2011-20-074 to the Internal
Revenue Service Chief Technology Officer.
IMPACT ON TAXPAYERS
The Internal Revenue Service (IRS) mainframe computing environment provides
the processing for mission-critical tax processing systems. The IRS can improve management of the capacity and
performance of its mainframe computers by formalizing performance measures in
its defined-service agreements. There is
also an opportunity for the IRS to realize cost savings in mainframe software contracts
that are dependent upon the capacity of the International Business Machines Corporation
(IBM) mainframe computers. If actions
are not taken to renegotiate capacity‑dependent software contracts, the
IRS could incur unnecessary software costs.
WHY TIGTA DID THE
AUDIT
This
review is included in our Fiscal Year 2011 Annual Audit Plan and addresses the
major management challenge of Modernization.
The overall objective of this review was to evaluate the efficiency and
effectiveness of the capacity and performance management of the IRS mainframe
environment.
WHAT TIGTA FOUND
The IRS has
incorporated Information Technology Infrastructure Library best practice
principles into its mainframe capacity management policies and procedures. However, only four of 20 defined-service
agreements included measurable performance metrics. Without a structure in place to measure and
report actual performance relative to performance metric requirements, the IRS
will be unable to verify and ensure that the quality of service provided by its
mission-critical tax processing systems will meet the expectations of its
customers and stakeholders.
TIGTA also
determined that the IRS has an opportunity to realize cost savings in its software
license costs that are dependent upon the capacity of its mainframe computers. If the IRS had changed its basis for
measuring the capacity of its IBM mainframes prior to a hardware upgrade in October
2010, it potentially could have saved more than $580,000 in software license costs.
WHAT TIGTA
RECOMMENDED
TIGTA
recommended that the Associate Chief Information Officer, Enterprise
Operations, include specific and measurable qualitative and quantitative metric
measurements in the defined-service agreements and establish a method of
reporting actual performance, relative to agreed-upon performance metric
requirements, to business unit application owners. TIGTA also recommended that, to realize cost
savings in mainframe software contracts, the Chief Technology Officer should
change the method by which mainframe capacity is measured and attempt to
renegotiate capacity-dependent contracts to achieve more favorable terms for
the IRS.
In its response to the report, the IRS agreed with TIGTA’s
recommendations. The IRS plans to 1)
develop and incorporate performance measures into the defined services design,
2) establish a method of reporting actual performance achievements relative to
agreed-upon performance metric requirements, 3) change the basis for measuring
capacity of its IBM mainframe computers in future software upgrades, and 4)
attempt to revise or restructure mainframe computer capacity-dependent software
agreements for maximum efficiency. The
IRS disagreed with TIGTA’s $580,358 outcome measure. TIGTA maintains the appropriateness of this
measure.
September 23, 2011
MEMORANDUM FOR CHIEF TECHNOLOGY OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Mainframe Computer Performance Is Being Actively Monitored, but Defined-Service Agreements and Software Licensing Can Be Improved (Audit #201120015)
This report presents the results of our review of the efficiency and effectiveness of the capacity and performance management of the Internal Revenue Service’s (IRS) mainframe environment. This audit is included in our Fiscal Year 2011 Annual Audit Plan and addresses the major management challenge of Modernization.
Management’s complete response to the draft report is included as Appendix VII.
Copies of this report are also
being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if
you have questions or Alan R. Duncan, Assistant Inspector General for Audit
(Security and Information Technology Services) at (202) 622-5894.
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
V – Internal Revenue Service Mainframe Computer Performance
Appendix
VI – Glossary of Terms
Appendix
VII – Management’s Response to the Draft Report
Abbreviations
|
CADE |
Customer Account Data Engine |
|
COBIT® |
Control Objectives for Information Technology |
|
DSA |
Defined-Service Agreements |
|
EOps |
Enterprise Operations |
|
IBM |
International Business Machines Corporation |
|
IRS |
Internal Revenue Service |
|
IT |
Information Technology |
|
ITIL® |
Information Technology Infrastructure Library |
|
MIPS |
Millions of Instructions Per Second |
|
MITS |
Modernization and Information Technology Services |
|
MSU |
Millions of Service Units |
|
SLA |
Service-Level Agreement |
The Internal Revenue Service (IRS) mainframe[1] computing environment provides the processing for mission-critical tax processing systems, including the Business Master File, the Customer Account Data Engine (CADE), the Integrated Data Retrieval System, and the Individual Master File. The Enterprise Storage and Capacity Planning Branch in the Large Systems and Storage Infrastructure Division of the Enterprise Operations (EOps) organization primarily manages capacity and performance of the IRS mainframe computer environment. The Enterprise Storage and Capacity Planning Branch has two mainframe sections, the International Business Machines Corporation (IBM) platform section and the Unisys Corporation platform section.
The IBM mainframes reside at the Enterprise Computing Centers in Martinsburg, West Virginia (Martinsburg Computing Center), and Memphis, Tennessee (Memphis Computing Center). The mainframe platform at the Martinsburg Computing Center includes two IBM z/196 machines that are organized into logical partitions that are paired and connected so they can cooperate as one unit to provide workload balancing. The IBM environment at the Memphis Computing Center consists of one IBM z/9 mainframe. The Memphis Computing Center z/9 mainframe, scheduled to be upgraded to an IBM z/196 machine in Fiscal Year 2011, will provide 100 percent disaster recovery capability for the Martinsburg Computing Center mainframes, as needed.
IBM mainframe processing costs are billed to the IRS based on the amount of processing capacity, measured in Millions of Instructions per Second (MIPS), allocated to the mainframes. To reduce software costs, the IRS has implemented IBM specialty processors, called zIIPs. Work typically performed by the mainframe general processors is off-loaded to the specialty processors, thus reducing the number of MIPS allocated to processing capacity and lowering costs.
The IRS Unisys mainframe environment contains two Dorado 280 mainframe computers, with one located at the Martinsburg Computing Center and one at the Memphis Computing Center. Because vendor hardware support will be discontinued as of December 31, 2011, the IRS has decided to upgrade the Dorado 280s to Dorado 780s during Fiscal Year 2011. While the new machines have been purchased, an implementation date for the production workload has not been announced.
IBM platform: workload, capacity allocation, and capacity
utilization
The first Martinsburg Computing Center IBM mainframe processes the production workloads for the Automated Collection System, the CADE, the Individual Master File, and the Business Master File. It also provides development, test, and disaster recovery environments for other applications. The second Martinsburg Computing Center IBM mainframe processes the production workloads for the Business Master File, the CADE, the Individual Master File, and the Web Currency and Banking Retrieval System. The Memphis Computing Center IBM mainframe processes the production workload for the Integrated Collection System.
The capacity of the IRS mainframes is measured in MIPS. While the IRS mainframes run at nearly 100 percent capacity during periods of peak processing, as the charts in Appendix V demonstrate, this is not an area of concern. Best practices indicate that modern mainframe systems are capable of running at high levels of utilization and that it is the performance of higher priority workloads that should be managed, not utilization.
Figure 1 shows how the IBM mainframes performed from January 2009 to June 2011. For each IBM mainframe (Martinsburg Computing Center #1, Martinsburg Computing Center #2, and Memphis Computing Center) the number of MIPS allocated at the end of the year and the average machine utilization for the year are shown.
Figure 1: IBM Mainframe
Computer Performance
January 2009 to June 2011
|
|
2009
MIPS Allocated |
2009
Average Utilization |
2010 MIPS Allocated |
2010
Average Utilization |
2011
MIPS Allocated |
2011
Average Utilization |
|
Martinsburg
Computing Center #1 |
6,400 |
62% |
7,500 |
57% |
8,800 |
63% |
|
Martinsburg
Computing Center #2 |
6,100 |
60% |
7,500 |
51% |
8,000 |
61% |
|
Memphis
Computing Center |
900 |
54% |
900 |
43% |
900 |
49% |
Source: Obtained from IRS IBM Weekly Utilization
Reports for Calendar Years 2009 through 2011.
A more detailed presentation of the IBM mainframe performance over the last 3 years is located in Appendix V.
Unisys platform: workload, capacity allocation, and capacity utilization
The Unisys mainframe at the Martinsburg Computing Center processes all of the centralized Individual Taxpayer Information File workload for the 10 IRS campuses. The Unisys mainframe at the Memphis Computing Center processes all of the centralized Business Taxpayer Information File workload for the 10 campuses.
The Unisys mainframe at the Martinsburg Computing Center is configured to run with 950 MIPS for normal weekday processing. For weekend processing, the Unisys mainframe at the Martinsburg Computing Center borrows MIPS from the development and test environments to increase capacity to 1,200 MIPS for managing the increased workload. The borrowed MIPS are returned to their respective systems on Monday mornings to support weekday processing. The Unisys mainframe at the Memphis Computing Center is configured to run with 675 MIPS for normal weekday processing.
The Unisys mainframe environment is mature and stable although the transaction processing workload has steadily increased. No significant changes have been made in processing capacity for several years. Each of the Annual Capacity Reports for the Unisys mainframe environment for Calendar Years 2008 through 2010 states there is sufficient capacity to manage the processing workloads for the foreseeable future.
The IRS currently executes two primary production systems that process individual taxpayer account data, respectively known as the Individual Master File and the CADE. The IRS is currently undergoing a major development effort to develop and deploy CADE 2 in January 2012 to replace the existing Individual Master File and CADE applications. The CADE 2 is designed to provide state-of-the-art individual taxpayer account processing and technologies to improve service to taxpayers and enhance IRS tax administration. Once completed, the new modernization environment should allow the IRS to more effectively and efficiently update taxpayer accounts, support account settlement and maintenance, and process refunds on a daily basis, which will contribute to improved service to taxpayers. With the transition from the CADE to the CADE 2, it is expected that 48 percent of the MIPS (approximately 8,000) will be available to be repurposed to other applications.
This review was performed at the Modernization and Information Technology Services (MITS) organization offices in New Carrollton, Maryland, and the Enterprise Computing Center in Martinsburg, West Virginia, during the period January through June 2011. We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
Information Technology Infrastructure Library Best Practice Principles Have Been Incorporated Into the Capacity Management Policies and Procedures
The Information Technology Infrastructure Library (ITIL®) is a set of concepts and practices for information technology (IT) service management. The ITIL focuses on the key service management principles pertaining to service strategy, service design, service transition, service operation, and continual service improvement. Within the service design principle, capacity management is a key process.
The purpose of capacity management is to
provide a point of focus and management for all capacity and
performance-related issues, relating to both services and resources, and to
match the capacity of the IT to the agreed-upon requirements of the
business. A key success factor in capacity
management is ensuring that it is considered during an application’s design.
In September 2010, the Chief Technology Officer outlined a goal to have the MITS organization implement ITIL best practices over the next several years. The MITS Process Re-Engineering Executive Steering Committee governs the implementation of the ITIL. Responsibility for implementing key ITIL concepts has been assigned to EOps executives, with an implementation plan due in September 2011. We reviewed the capacity management policies and procedures and determined that the EOps organization has incorporated ITIL best practice principles.
Performance Measurement Requirements in Defined-Service Agreements Are Not Formally Established to Facilitate the Management and Reporting of Mainframe Performance
The ITIL states that the primary purpose of the capacity management process is to identify and understand IT services and their use of resources, working patterns, and peaks and troughs and to ensure that the services meet their Service-Level Agreement (SLA) targets, i.e., to ensure that IT services are performed as required. The focus is on managing service performance, as determined by the performance requirements contained in the SLAs that were agreed upon with their IRS business unit customers.
The Information Systems Audit and Control Association’s Control Objectives for Information Technology (COBIT®) represents an authoritative, internationally accepted IT governance control framework for use by business managers, IT professionals, and IT audit professionals. The COBIT states that an organization should define and agree to SLAs for all critical IT services based on customer requirements and IT capabilities. The SLAs should cover customer commitments, service support requirements, quantitative and qualitative metrics for measuring the service signed off on by the stakeholders, funding and commercial arrangements (if applicable), and roles and responsibilities, including oversight of the SLA. Metrics that should be considered include system availability, reliability, performance, capacity for growth, levels of support, continuity planning, security, and demand constraints. The COBIT also recommends that organizations continuously monitor specified service-level performance criteria and provide reports on achievement of service levels in a format that is meaningful to the customers. The monitoring statistics should be analyzed and acted upon to identify negative and positive trends for individual services, as well as for services overall.
The Customer Relationships and Integration Division within the EOps organization is responsible for managing and administering all EOps organization SLAs. Its stated mission is to serve as a focal point for establishing relationships between the IRS business units (customers) and the appropriate EOps division (service providers). The EOps organization eliminated the need for individual applications to have SLAs. The SLAs for individual applications that were previously in use were converted into 20 Defined-Service Agreements (DSA) or were merged into a larger document called the EOps Service Standards Document. The EOps organization uses the term DSA to refer to the industry generic term, SLA. The DSA describes the services received from the service provider, and the EOps Service Standards Document describes how the EOps organization (the service provider) will provide the services. The conversion of SLAs to DSAs and a Service Standards Document was completed and will undergo an annual review.
During our review of
the 20 DSAs, we found that the EOps organization is not consistently including
measurable performance metrics such as availability, reliability, performance,
and capacity in these agreements. Only 4
(20 percent) of the 20 DSAs contained any measurable performance metrics. The remaining 16 (80 percent) documents
contained no measurable metrics. More
specifically, one of the four DSAs that had measurable metrics was the Federal Information
Security Management Act Reportable Application Support Service DSA. It contained a list of 160 additional
applications, and only 51 (32 percent) of the applications contained the measurable
performance metrics for response times. EOps
management explained that the EOps organization has not included qualitative
and quantitative metrics in its DSAs because IRS business unit customers have
not asked it to do so. Further, EOps
management indicated that their Customer Relationships and Integration Division
does not have the technical expertise needed to provide
performance metric requirements.
Our review of
mainframe performance monitoring determined the EOps personnel responsible for
capacity management of the IBM and Unisys mainframe environments are actively
monitoring mainframe performance against their own informal measures. The IBM capacity managers create an annual
capacity report, as well as various day-to-day application-specific
reports. The Unisys capacity managers create
periodic reports on daily, weekly, and weekend transaction processing.
In addition, the EOps Service
Standards Document states that the EOps Customer Relationships and Integration Division
will provide measurable performance metrics for customer verification of its quality
of service. However, the Customer Relationships
and Integration Division is not reporting actual performance
achievements against specific DSA performance requirements to the EOps
organization’s business unit customers.
If specific and measurable performance metric requirements are not documented and agreed upon in the DSAs, then these documents cannot be effectively used to hold service providers to the minimum level of service required by critical applications. Without a structure in place to report actual performance achievements relative to performance metric requirements to application owners, IRS management will be unable to verify and ensure that the quality of service provided to its mission-critical tax processing systems will meet the expectations of its customers and stakeholders.
Recommendations
Recommendation 1: The Associate Chief Information
Officer, Enterprise Operations, should include specific and measurable
qualitative and quantitative metric requirements in DSAs that can be used to
define the quality of service required by EOps organization customers. In this effort, quantitative metrics such as
availability, reliability, performance, capacity for growth, levels of support,
continuity planning, security, and demand constraints should be considered.
Management’s Response: The IRS agreed with this recommendation. The IRS is working to develop measures as they relate to the Mainframe Computing Defined Services. In accordance with the recommendation, the IRS will consider the categories mentioned in the defined services design.
Recommendation 2: The Associate Chief Information Officer, Enterprise Operations, should establish a method of reporting actual performance achievements, relative to agreed-upon performance metric requirements, to business unit application owners in a format that is useful for customer verification of the EOps organization’s quality of service.
Management’s Response: The IRS agreed
with this recommendation. Enterprise
Operations will establish a method of reporting actual performance achievements,
relative to agreed-upon performance metric requirements, to business unit
application owners in a format that is useful for customer verification of the EOps
organization’s quality of service.
The Internal Revenue Service May Incur Unnecessary Software
Licensing Costs Related to Mainframe Hardware Capacity
The ITIL focuses on the key service management principles pertaining to service strategy, service design, service transition, service operation, and continual service improvement. Within the service design principle, supplier management is a key process. The supplier management process ensures that suppliers and the services they provide are managed to support IT service targets and business expectations. Two of the main objectives of the supplier management process are: 1) to obtain value for the money spent with suppliers and on contracts and 2) to negotiate and agree to contracts with suppliers and manage them through their life cycle.
The IRS measures the capacity of its mainframes in terms of MIPS. MIPS charts can be used for various purposes, including: 1) to provide a metric against which capacity consumption can be measured, 2) to provide capacity guidance when performing hardware upgrades within a processor family, and 3) as a means to provide independent software vendors a machine capacity with which to license and charge for software maintenance and one-time upgrade charges.
The
license costs for the software products residing on the IRS mainframes are tied
to mainframe capacity, or the number of MIPS allocated to the machines. The IRS whitepaper titled
z/196 – Gartner MIPS and Capacity Upgrade Issues, issued by the IBM Capacity Management Section, notes that there is an
opportunity for the IRS to reduce its software license costs by changing the
measure used to calculate the capacity of its mainframes. While the IRS currently uses MIPS, the IRS
Capacity Management Branch recommends changing the measurement to Millions of
Service Units, or MSUs. The whitepaper
indicates that IBM has reduced the MSU rating for each family of z-series
processors by about 10 percent for machines of equivalent capacities.
In October 2010, the IRS upgraded its Martinsburg Computing Center mainframe computers from the IBM z/9-series mainframes to the IBM z/196-series mainframes. Concerns over IBM mainframe software licensing costs raised by the IRS Capacity Management Branch in its whitepaper may not have been communicated to the stakeholders with the authority and responsibility to renegotiate the relevant hardware and software contracts. Had the IRS made the conversion from MIPS to MSUs as a basis for determining mainframe capacity prior to the IBM mainframe upgrade, the IRS could have realized a cost savings of $580,358 in its software licensing costs, using the 10 percent reduction estimate in the IRS whitepaper. Figure 2 shows the software products where the IRS could have potentially realized savings in the licensing agreements.
Figure 2: Potential
Mainframe Software License Cost Savings
|
Software \Vendor |
Software |
Potential Savings |
|
IBM Software Relationship Offering |
$41,608 |
$4,161 |
|
ASG Software Solutions |
$44,746 |
$4,475 |
|
Catalog Recovery Plus |
$129,725 |
$12,973 |
|
Computer Associates |
$398,706 |
$39,871 |
|
Perfman |
$14,619 |
|
|
SAS |
$644,813 |
$64,481 |
|
Vanguard |
$178,807 |
$17,881 |
|
Address Hygiene Software |
$1,757,276 |
$175,728 |
|
Mainframe Peripheral and Software Maintenance |
$2,461,692 |
$246,169 |
|
Total |
$5,803,558 |
$580,358 |
Note: The Total Potential Savings is greater than
10 percent due to rounding.
Source:
IRS contract documentation.
Approximately 8,000 MIPS currently allocated to the CADE will need to be repurposed when the IRS transitions from the CADE to the CADE 2 in January 2012. The IRS indicated it would establish a working group to coordinate decisions regarding managing the excess hardware capacity. In addition, the IRS has started to make decisions regarding the renegotiation of software license agreements that are dependent on hardware capacity. If these hardware resources are not fully utilized, or actions are not taken to fully renegotiate capacity-dependent software contracts, the IRS could incur unnecessary hardware and software costs.
Recommendations
Recommendation 3: The Chief Technology Officer should change the basis for determining the capacity of its IBM mainframe computers from MIPS to MSUs for future hardware upgrades.
Management’s Response: The IRS agreed with this recommendation. The EOps organization will partner with the Strategy and Planning Division to change the basis for measuring the capacity of its IBM mainframe computers and converting future software upgrades from MIPS to MSUs.
Recommendation 4: The Chief Technology Officer should review the mainframe computer capacity-dependent software agreements and attempt to renegotiate these contracts to more favorable terms for the IRS to potentially realize cost savings in mainframe software contracts.
Management’s Response: The IRS agreed with this recommendation. Over the past 2 years, the IRS has revised or restructured every mainframe software contract for maximum efficiency. The IRS has converted them to Treasury-wide vehicles where possible with the exception of the IBM Software Relationship Offering which will be restructured next fiscal year.
Office
of Audit Comment: The IRS did
not agree with the outcome measure related to the use of alternative capacity
measures in software negotiations for licensing costs. In its response, the IRS stated that it agrees
there is an opportunity to use MSUs in place of MIPS to more effectively track
capacity requirements in support of contract negotiations for software
licensing costs and to reduce software licensing costs. The IRS also stated it does not believe there
is any evidence that the vendors involved in past software licensing agreements
would have agreed to use the MSU method in determining licensing costs. However, the TIGTA maintains the validity of
this potential cost savings based on an estimate provided by the IRS Whitepaper
titled z/196 – Gartner MIPS and Capacity Upgrade Issues, dated October 2010 and prepared by the IBM Capacity
Management Section.
Appendix I
Detailed
Objective, Scope, and Methodology
Our overall objective was to evaluate the efficiency and effectiveness of the capacity and performance management of the IRS mainframe[2] environment. To accomplish our objective, we:
I. Evaluated mainframe capacity and performance management policies, standards, and procedures.
A. Searched for Federal guidance for capacity and performance management of mainframes.
B. Reviewed industry best practices (ITIL and COBIT) for capacity and performance management of mainframes.
C. Reviewed IBM and Unisys vendor guidance for capacity and performance management of mainframes.
D. Determined whether IRS capacity and performance guidance reflects Federal, industry, and vendor best practice guidance.
E. Determined whether IRS policies and procedures identify and define responsibilities for capacity and performance management.
F. Determined whether the roles and responsibilities have been formally assigned and communicated.
II. Evaluated mainframe service-level requirements.
A. Reviewed service-level agreements for reasonableness and completeness for applications processed in the IRS mainframe production environment to determine whether:
1. Definitive units of measure for mainframe capacity and performance have been established.
2. Minimum levels of service or performance have been identified and agreed to by stakeholders.
B. Reviewed current capacity and performance metrics regarding how well processors and applications are able to achieve stated mainframe service-level requirements.
III. Evaluated the monitoring, reporting, and management of mainframe capacity and performance trends.
A. Reviewed Calendar Years 2009, 2010, and 2011 Executive Steering Committee minutes and Business Performance Review reports relating to mainframe capacity and performance.
B. Reviewed Calendar Years 2009, 2010, and 2011 annual capacity and performance plans.
C. Reviewed Calendar Years 2010 and 2011 summary reports generated from IRS capacity and performance management reviews.
D. Reviewed the current status of the IBM mainframe processor upgrade (zAAP/zIIP project).
Internal
controls methodology
Internal controls relate to management’s plans, methods, and procedures used to meet their mission, goals, and objectives. Internal controls include the processes and procedures for planning, organizing, directing, and controlling program operations. They include the systems for measuring, reporting, and monitoring program performance. We determined the following internal controls were relevant to our audit objective: the MITS organization’s policies and procedures for effectively managing the performance and capacity of its mainframe computers. We evaluated these controls by interviewing management and reviewing the EOps organization’s policies and procedures and industry best practices such as COBIT and ITIL.
Appendix II
Major
Contributors to This Report
Alan R. Duncan, Assistant Inspector General for Audit
(Security and Information Technology Services)
Danny Verneuille, Director
Carol Taylor, Audit Manager
Myron Gulley, Lead Auditor
Mary Jankowski, Senior Auditor
Daniel Oakley, Information Technology
Specialist
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Deputy Chief Information Officer for Operations OS:CTO
Associate Chief Information Officer, Enterprise Operations OS:CTO:EO
Associate Chief Information Officer, Strategy and Planning OS:CTO:SP
Director, Enterprise Computing Centers OS:CTO:EO:EC
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis RAS:O
Office of Internal Control OS:CFO:CPIC:IC
Audit Liaison: Director, Risk Management Division OS:CTO:SP:RM
Appendix IV
This appendix presents detailed information on the measurable impact that our recommended corrective action will have on tax administration. This benefit will be incorporated into our Semiannual Report to Congress.
Type and Value of Outcome Measure:
· Inefficient Use of Resources – Potential; $580,358 (see page 6).
Methodology Used to Measure the Reported Benefit:
The software license costs for software products residing on the IRS mainframes[3] are tied to the mainframe capacity, or the number of MIPS allocated to the machines. The IRS whitepaper titled z/196 – Gartner MIPS and Capacity Upgrade Issues, dated October 2010 and prepared by the IBM Capacity Management Section, notes that there is an opportunity for the IRS to reduce its software costs by changing the measure it uses to calculate the capacity of its mainframes. While the IRS currently uses MIPS, the IBM Capacity Management Branch recommends changing the measurement to MSUs. The whitepaper indicates that IBM has reduced the MSU rating for each family of IBM z-series processors by about 10 percent for machines of equivalent capacities.
In October 2010, the IRS upgraded its Martinsburg Computing Center mainframe computers from the IBM z/9-series mainframes to the IBM z/196-series mainframes. Had the IRS made the conversion from MIPS to MSUs as a basis for determining mainframe capacity prior to its mainframe upgrade, the IRS could have realized a cost savings of $580,358 in its software licensing costs using the 10‑percent reduction estimate in the IRS whitepaper. Figure 1 provides the calculation for the amount of potential software cost savings.
Figure 1: Calculation of Potential Software Cost Savings
|
Software \Vendor |
Potential Savings |
|
IBM Software
Relationship Offering |
$4,161 |
|
ASG |
$4,475 |
|
Catalog
Recovery Plus |
$12,973 |
|
Computer
Associates |
$39,871 |
|
Perfman |
$14,619 |
|
SAS |
$64,481 |
|
Vanguard |
$17,881 |
|
Address
Hygiene Software |
$175,728 |
|
Mainframe
Peripheral and Software Maintenance |
$246,169 |
|
Total |
$580,358 |
Source: IRS contract documentation.
Appendix V
Internal Revenue
Service Mainframe Computer Performance
The following figures show how the IRS IBM mainframes[4] performed from January 2009 to June 2011 (the horizontal axis on each chart) with the total number of MIPS available to the machine (the vertical axis on each chart) as the basis for measurement. Figure 1 shows that in Calendar Year 2009, the Martinsburg Computing Center IBM mainframe #1 was initially allocated 5,200[5] MIPS, with upgrades to 6,400. It had a Calendar Year 2009 average weekly utilization of 62 percent of capacity. In Calendar Year 2010, the MIPS were upgraded to 7,500 MIPS. It had a Calendar Year 2010 average weekly utilization of 57 percent of capacity. In Calendar Year 2011, it was allocated 8,800 MIPS and had an average utilization of 63 percent of capacity.
Figure 1: Martinsburg IBM Mainframe #1 – Capacity and Performance Trends
Figure 1 was removed due to its size. To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.
Figure 2 shows that in Calendar Year 2009, the IBM Martinsburg Computing Center mainframe #2 was initially allocated 5,200 MIPS, with upgrades to 6,100 MIPS. It had a Calendar Year 2009 average weekly utilization of 60 percent of capacity. In Calendar Year 2010, the MIPS were upgraded to 7,500 MIPS. It had a Calendar Year 2010 average weekly utilization of 51 percent of capacity. In Calendar Year 2011, 8,000 MIPS were allocated, and had an average weekly utilization of 61 percent of capacity.
Figure 2: Martinsburg IBM Mainframe #2 – Capacity and Performance Trends
Figure 2 was removed due to its size. To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page
Figure 3 shows that in Calendar Year 2009, the IBM mainframe
at the Memphis Computing Center was allocated 900 MIPS and had an average weekly
utilization of 54 percent of capacity.
It had an average weekly utilization of 43 percent of capacity in Calendar
Years 2010 and 49 percent in 2011. The Memphis
Computing Center mainframe processes the production workload for the Integrated
Collection System and is the disaster recovery computer for the Martinsburg
Computing Center IBM mainframes.
Figure 3: Memphis IBM
Mainframe – Capacity and Performance
Trends
Figure 3 was removed due to its size. To see Figure 3, please go to the Adobe PDF version of the report on the TIGTA Public Web Page
Figures 4 and 5 provide a snapshot view of the Unisys production mainframes’ capacity and performance during one processing cycle of the 2011 Filing Season.
Figure 4: Martinsburg Unisys
Mainframe Utilization, February 14–20, 2011
Figure 4 was removed due to its size. To see Figure 4, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page
Figure 5: Memphis Unisys Mainframe Utilization, February 14–20, 2011
Figure
5 was removed due to its size. To see
Figure 5, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page
Appendix VI
|
Term |
Definition |
|
Application |
A
software program hosted by an information system. |
|
Automated Collection
System |
A
telephone contact system through which telephone assistors collect unpaid
taxes and secure tax returns from delinquent taxpayers who have not complied
with previous notices. |
|
Business Master File |
The IRS
database that consists of Federal tax-related transactions and accounts for
businesses. These include employment
taxes, income taxes on businesses, and excise taxes. |
|
Business Taxpayer
Information File |
Area of
the IRS tax information database that contains information from business
taxpayer accounts. |
|
Campus |
The data
processing arm of the IRS. The
campuses process paper and electronic submissions, correct errors, and
forward data to the Computing Centers for analysis and posting to taxpayer
accounts. |
|
Capacity management |
The
discipline that ensures IT infrastructure is provided at the right time in
the right volume at the right price, and ensuring that IT is used in the most
efficient manner. This involves input
from many areas of the business to identify what services are (or will be)
required, what information technology infrastructure is required to support
these services, what level of contingency will be needed, and what the cost
of this infrastructure will be. |
|
Customer Account Data
Engine |
The
foundation for managing taxpayer accounts in the IRS modernization plan. It will consist of databases and related
applications that will replace the IRS’s existing Master File processing
system and will include applications for daily posting, settlement,
maintenance, refund processing, and issue detection for taxpayer tax account
and return data. |
|
Customer
Account Data Engine 2 |
An IRS
application that will replace the existing Individual Master File and CADE
applications. CADE 2 is designed to
provide state-of-the-art individual taxpayer account processing and
technologies to improve service to taxpayers and enhance IRS tax
administration. |
|
Federal
Information Security Management Act |
United
States legislation that defines a comprehensive framework to protect
government information, operations, and assets against natural or man-made
threats. |
|
Filing
Season |
The
period from January 1 through mid-April when most individual income tax
returns are filed. |
|
Fiscal
Year |
A
12-consecutive-month period ending on the last day of any month, except
December. The Federal Government’s
fiscal year begins on October 1 and ends on September 30. |
|
Hardware |
The
physical components of an information system. |
|
Individual Master File |
The IRS
database that maintains transactions or records of individual tax accounts. |
|
Individual Taxpayer
Information File |
Area of
the Individual Master File database that contains information from individual
taxpayer accounts. |
|
Integrated Collection
System |
An
information management system designed to improve revenue collections by
providing revenue officers access to the most current taxpayer information,
while in the field, using laptop computers for quicker case resolution and
improved customer service. |
|
Integrated Data Retrieval
System |
IRS
computer system capable of retrieving or updating stored information. It works in conjunction with a taxpayer’s
account records. |
|
Logical Partition |
The
division of a computer’s memory and storage into multiple sets of resources
so that each set of resources can be operated independently with its own
operating system and applications. |
|
Mainframe |
A
powerful, multiuser computer capable of supporting many hundreds of thousands
of users simultaneously. |
|
Platform |
The
computer architecture and equipment using a particular operating system. |
|
Processor |
A part
of a computer, such as the central processing unit, that performs
calculations of data and other logical functions. |
|
Software |
Computer
programs and associated data that may be dynamically written or modified
during execution. |
|
Web Currency and Banking
Retrieval System |
An
online database that contains Bank Secrecy Act information. IRS field agents as well as local, State,
and Federal law enforcement agencies access the database for research in tax
cases, tracking money-laundering activities, investigative leads, intelligence for the tracking of currency flows,
corroborating information, and probative evidence. |
Appendix VII
Management’s
Response to the Draft Report
DEPARTMENT OF
THE TREASURY
INTERNAL
REVENUE SERVICE
WASHINGTON,
D.C. 20224
CHIEF
TECHNOLOGY OFFICER
SEPTEMBER 13, 2011
MEMORANDUM
FOR DEPUTY INSPECTOR GENERAL FOR AUDIT
FROM: Terence V. Milholland
/s/ Terence V. Milholland
Chief Technology Officer
SUBJECT: Draft Audit Report -
Mainframe Computer Performance Is Being Actively Monitored, but Defined-Service
Agreements and Software Licensing Can Be Improved
(Audit #
201120015) (e-trak #
2011-24391)
Thank you for
the opportunity to review and respond to the subject audit report. We agree
with the recommendations outlined in the report and the attachment details our
planned actions to implement them.
Based on our
analysis of alternative capacity measures, we believe there is an opportunity
to use Millions of Service Units (MSU) in lieu of Million Instructions Per Second (MIPS) to more effectively track capacity
requirements in support of contract negotiations for software licensing costs.
We therefore anticipate that cost savings may result from our conversion from
MIPS to MSU. However, there is no evidence vendors involved in past software
licensing agreements would have agreed to use the MSU method in determine
licensing costs. Therefore, any assertion of missed savings opportunity is, at
best, speculative, thereby we disagree with the
Outcome Measure.
We value your
continued support and the guidance your team provides. If you have any
questions, please contact me at (202) 622-6800 or Andrea Greene-Horace, Senior
Manager of Program Oversight, at (202) 283-3427.
Attachment
RECOMMENDATION #1: The Associate Chief Information Officer,
Enterprise Operations, should include specific and measurable qualitative and
quantitative metric requirements in Defined Service Agreements that can be used
to define the quality of service required by EOps
organization customers. In this effort, quantitative metrics such as
availability, reliability, performance, capacity for growth, levels of support,
continuity planning, security, and demand constraints should be considered.
CORRECTIVE ACTION #1: We agree with the recommendation. We are
working to develop measures as they relate to our Mainframe Computing Defined
Services. In accordance with the recommendation we will consider the categories
mentioned in our defined services design.
IMPLEMENTATION DATE: October 2, 2012
RESPONSIBLE OFFICIAL: Associate Chief Information Officer,
Enterprise Operations
CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into
the Joint Audit Management Enterprise System (JAMES) and monitor them on a
monthly basis until completion.
RECOMMENDATION #2: The Associate Chief Information Officer,
Enterprise Operations, should establish a method of reporting actual
performance achievements, relative to agreed-upon performance metric
requirements, to business unit application owners in a format that is useful
for customer verification of the EOps organization's
quality of services.
CORRECTIVE ACTION #2: We agree with the recommendation. Enterprise
Operations (EOps) will establish a method of
reporting actual performance achievements, relative to agreed-upon performance
metric requirements, to business unit application owners in a format that is
useful for customer verification of the EOps
organization's quality of services.
IMPLEMENTATION DATE: November 1, 2012
RESPONSIBLE OFFICIAL: Associate Chief Information Officer,
Enterprise Operations
CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into
the Joint Audit Management Enterprise System (JAMES) and monitor them on a
monthly basis until completion.
RECOMMENDATION #3: The Chief Technology Officer should change
the basis for determining the capacity of its IBM mainframe computers from MIPS
to MSUs for future hardware upgrades.
CORRECTIVE ACTION #3: We agree with the recommendation. Enterprise
Operations will partner with Strategy and Planning to
change the basis for measuring the capacity of its IBM mainframe computers and
converting future software upgrades from MIPS to MSUs.
IMPLEMENTATION DATE: October 2, 2012
RESPONSIBLE OFFICIAL: Associate Chief Information Officer,
Enterprise Operations
CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into
the Joint Audit Management Enterprise System (JAMES) and monitor them on a
monthly basis until completion.
RECOMMENDATION #4: The Chief Technology Officer should review
the mainframe computer capacity-dependent software agreements and attempt to
renegotiate these contracts to more favorable terms for the IRS to potentially
realize cost savings in mainframe software contracts.
CORRECTIVE ACTION #4: We agree with the recommendation. Over the
past 2 years we have revised or restructured every mainframe software contract
for maximum efficiency. We have converted them to Treasury-wide vehicles where
possible with the exception of the IBM Software Relationship Offering (SRO)
which we will be restructuring next fiscal year.
IMPLEMENTATION DATE: October 2, 2012
RESPONSIBLE OFFICIAL: Associate Chief Information Officer,
Strategy and Planning
CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into
the Joint Audit Management Enterprise System (JAMES) and monitor them on a
monthly basis until completion.