TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

Mainframe Computer Performance Is Being Actively Monitored, but Defined-Service Agreements and Software Licensing Can Be Improved

 

 

 

September 23, 2011

 

Reference Number:  2011-20-074

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

Phone Number   |  202-622-6500

Email Address   |  TIGTACommunications@tigta.treas.gov

Web Site           |  http://www.tigta.gov

 

 

HIGHLIGHTS

 

MAINFRAME COMPUTER PERFORMANCE IS BEING ACTIVELY MONITORED, BUT DEFINED-SERVICE AGREEMENTS AND SOFTWARE LICENSING CAN BE IMPROVED

 

Highlights

Final Report issued on September 23, 2011  

Highlights of Reference Number:  2011-20-074 to the Internal Revenue Service Chief Technology Officer.

IMPACT ON TAXPAYERS

The Internal Revenue Service (IRS) mainframe computing environment provides the processing for mission-critical tax processing systems.  The IRS can improve management of the capacity and performance of its mainframe computers by formalizing performance measures in its defined-service agreements.  There is also an opportunity for the IRS to realize cost savings in mainframe software contracts that are dependent upon the capacity of the International Business Machines Corporation (IBM) mainframe computers.  If actions are not taken to renegotiate capacity‑dependent software contracts, the IRS could incur unnecessary software costs.

WHY TIGTA DID THE AUDIT

This review is included in our Fiscal Year 2011 Annual Audit Plan and addresses the major management challenge of Modernization.  The overall objective of this review was to evaluate the efficiency and effectiveness of the capacity and performance management of the IRS mainframe environment. 

WHAT TIGTA FOUND

The IRS has incorporated Information Technology Infrastructure Library best practice principles into its mainframe capacity management policies and procedures.  However, only four of 20 defined-service agreements included measurable performance metrics.  Without a structure in place to measure and report actual performance relative to performance metric requirements, the IRS will be unable to verify and ensure that the quality of service provided by its mission-critical tax processing systems will meet the expectations of its customers and stakeholders.

TIGTA also determined that the IRS has an opportunity to realize cost savings in its software license costs that are dependent upon the capacity of its mainframe computers.  If the IRS had changed its basis for measuring the capacity of its IBM mainframes prior to a hardware upgrade in October 2010, it potentially could have saved more than $580,000 in software license costs. 

WHAT TIGTA RECOMMENDED

TIGTA recommended that the Associate Chief Information Officer, Enterprise Operations, include specific and measurable qualitative and quantitative metric measurements in the defined-service agreements and establish a method of reporting actual performance, relative to agreed-upon performance metric requirements, to business unit application owners.  TIGTA also recommended that, to realize cost savings in mainframe software contracts, the Chief Technology Officer should change the method by which mainframe capacity is measured and attempt to renegotiate capacity-dependent contracts to achieve more favorable terms for the IRS.

In its response to the report, the IRS agreed with TIGTA’s recommendations.  The IRS plans to 1) develop and incorporate performance measures into the defined services design, 2) establish a method of reporting actual performance achievements relative to agreed-upon performance metric requirements, 3) change the basis for measuring capacity of its IBM mainframe computers in future software upgrades, and 4) attempt to revise or restructure mainframe computer capacity-dependent software agreements for maximum efficiency.  The IRS disagreed with TIGTA’s $580,358 outcome measure.  TIGTA maintains the appropriateness of this measure.

 

September 23, 2011

 

 

 

MEMORANDUM FOR CHIEF TECHNOLOGY OFFICER

 

FROM:                            Michael R. Phillips /s/ Michael R. Phillips

Deputy Inspector General for Audit

 

SUBJECT:                    Final Audit Report – Mainframe Computer Performance Is Being Actively Monitored, but Defined-Service Agreements and Software Licensing Can Be Improved (Audit #201120015)

 

This report presents the results of our review of the efficiency and effectiveness of the capacity and performance management of the Internal Revenue Service’s (IRS) mainframe environment.  This audit is included in our Fiscal Year 2011 Annual Audit Plan and addresses the major management challenge of Modernization.

Management’s complete response to the draft report is included as Appendix VII.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services) at (202) 622-5894.

 

 

Table of Contents

 

Background

Results of Review

Information Technology Infrastructure Library Best Practice Principles Have Been Incorporated Into the Capacity Management Policies and Procedures

Performance Measurement Requirements in Defined-Service Agreements Are Not Formally Established to Facilitate the Management and Reporting of Mainframe Performance

Recommendations 1 and 2:

The Internal Revenue Service May Incur Unnecessary Software Licensing Costs Related to Mainframe Hardware Capacity

Recommendations 3 and 4:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measure

Appendix V – Internal Revenue Service Mainframe Computer Performance

Appendix VI – Glossary of Terms

Appendix VII – Management’s Response to the Draft Report

 

 

Abbreviations

 

CADE

Customer Account Data Engine

COBIT® 

Control Objectives for Information Technology

DSA

Defined-Service Agreements

EOps

Enterprise Operations

IBM

International Business Machines Corporation

IRS

Internal Revenue Service

IT

Information Technology

ITIL® 

Information Technology Infrastructure Library

MIPS

Millions of Instructions Per Second

MITS

Modernization and Information Technology Services

MSU

Millions of Service Units

SLA

Service-Level Agreement

 

Background

 

The Internal Revenue Service (IRS) mainframe[1] computing environment provides the processing for mission-critical tax processing systems, including the Business Master File, the Customer Account Data Engine (CADE), the Integrated Data Retrieval System, and the Individual Master File.  The Enterprise Storage and Capacity Planning Branch in the Large Systems and Storage Infrastructure Division of the Enterprise Operations (EOps) organization primarily manages capacity and performance of the IRS mainframe computer environment.  The Enterprise Storage and Capacity Planning Branch has two mainframe sections, the International Business Machines Corporation (IBM) platform section and the Unisys Corporation platform section.

The IBM mainframes reside at the Enterprise Computing Centers in Martinsburg, West Virginia (Martinsburg Computing Center), and Memphis, Tennessee (Memphis Computing Center).  The mainframe platform at the Martinsburg Computing Center includes two IBM z/196 machines that are organized into logical partitions that are paired and connected so they can cooperate as one unit to provide workload balancing.  The IBM environment at the Memphis Computing Center consists of one IBM z/9 mainframe.  The Memphis Computing Center z/9 mainframe, scheduled to be upgraded to an IBM z/196 machine in Fiscal Year 2011, will provide 100 percent disaster recovery capability for the Martinsburg Computing Center mainframes, as needed. 

IBM mainframe processing costs are billed to the IRS based on the amount of processing capacity, measured in Millions of Instructions per Second (MIPS), allocated to the mainframes.  To reduce software costs, the IRS has implemented IBM specialty processors, called zIIPs.  Work typically performed by the mainframe general processors is off-loaded to the specialty processors, thus reducing the number of MIPS allocated to processing capacity and lowering costs.  

The IRS Unisys mainframe environment contains two Dorado 280 mainframe computers, with one located at the Martinsburg Computing Center and one at the Memphis Computing Center.  Because vendor hardware support will be discontinued as of December 31, 2011, the IRS has decided to upgrade the Dorado 280s to Dorado 780s during Fiscal Year 2011.  While the new machines have been purchased, an implementation date for the production workload has not been announced.

IBM platform:  workload, capacity allocation, and capacity utilization

The first Martinsburg Computing Center IBM mainframe processes the production workloads for the Automated Collection System, the CADE, the Individual Master File, and the Business Master File.  It also provides development, test, and disaster recovery environments for other applications.  The second Martinsburg Computing Center IBM mainframe processes the production workloads for the Business Master File, the CADE, the Individual Master File, and the Web Currency and Banking Retrieval System.  The Memphis Computing Center IBM mainframe processes the production workload for the Integrated Collection System. 

The capacity of the IRS mainframes is measured in MIPS.  While the IRS mainframes run at nearly 100 percent capacity during periods of peak processing, as the charts in Appendix V demonstrate, this is not an area of concern.  Best practices indicate that modern mainframe systems are capable of running at high levels of utilization and that it is the performance of higher priority workloads that should be managed, not utilization.   

Figure 1 shows how the IBM mainframes performed from January 2009 to June 2011.  For each IBM mainframe (Martinsburg Computing Center #1, Martinsburg Computing Center #2, and Memphis Computing Center) the number of MIPS allocated at the end of the year and the average machine utilization for the year are shown. 

Figure 1:  IBM Mainframe Computer Performance
January 2009 to June 2011

 

2009 MIPS

Allocated

2009 Average

Utilization

2010  MIPS

Allocated

2010 Average

Utilization

2011 MIPS

Allocated

2011 Average

Utilization

Martinsburg Computing Center #1

6,400

62%

7,500

57%

8,800

63%

Martinsburg Computing Center #2

6,100

60%

7,500

51%

8,000

61%

Memphis Computing Center

900

54%

900

43%

900

49%

Source:  Obtained from IRS IBM Weekly Utilization Reports for Calendar Years 2009 through 2011.

A more detailed presentation of the IBM mainframe performance over the last 3 years is located in Appendix V.

Unisys platform:  workload, capacity allocation, and capacity utilization

The Unisys mainframe at the Martinsburg Computing Center processes all of the centralized Individual Taxpayer Information File workload for the 10 IRS campuses.  The Unisys mainframe at the Memphis Computing Center processes all of the centralized Business Taxpayer Information File workload for the 10 campuses.

The Unisys mainframe at the Martinsburg Computing Center is configured to run with 950 MIPS for normal weekday processing.  For weekend processing, the Unisys mainframe at the Martinsburg Computing Center borrows MIPS from the development and test environments to increase capacity to 1,200 MIPS for managing the increased workload.  The borrowed MIPS are returned to their respective systems on Monday mornings to support weekday processing.  The Unisys mainframe at the Memphis Computing Center is configured to run with 675 MIPS for normal weekday processing. 

The Unisys mainframe environment is mature and stable although the transaction processing workload has steadily increased.  No significant changes have been made in processing capacity for several years.  Each of the Annual Capacity Reports for the Unisys mainframe environment for Calendar Years 2008 through 2010 states there is sufficient capacity to manage the processing workloads for the foreseeable future. 

The IRS currently executes two primary production systems that process individual taxpayer account data, respectively known as the Individual Master File and the CADE.  The IRS is currently undergoing a major development effort to develop and deploy CADE 2 in January 2012 to replace the existing Individual Master File and CADE applications.  The CADE 2 is designed to provide state-of-the-art individual taxpayer account processing and technologies to improve service to taxpayers and enhance IRS tax administration.  Once completed, the new modernization environment should allow the IRS to more effectively and efficiently update taxpayer accounts, support account settlement and maintenance, and process refunds on a daily basis, which will contribute to improved service to taxpayers.  With the transition from the CADE to the CADE 2, it is expected that 48 percent of the MIPS (approximately 8,000) will be available to be repurposed to other applications.

This review was performed at the Modernization and Information Technology Services (MITS) organization offices in New Carrollton, Maryland, and the Enterprise Computing Center in Martinsburg, West Virginia, during the period January through June 2011.  We conducted this performance audit in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective.  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.  

 

 

Results of Review

 

Information Technology Infrastructure Library Best Practice Principles Have Been Incorporated Into the Capacity Management Policies and Procedures

The Information Technology Infrastructure Library (ITIL®) is a set of concepts and practices for information technology (IT) service management.  The ITIL focuses on the key service management principles pertaining to service strategy, service design, service transition, service operation, and continual service improvement.  Within the service design principle, capacity management is a key process. 

The purpose of capacity management is to provide a point of focus and management for all capacity and performance-related issues, relating to both services and resources, and to match the capacity of the IT to the agreed-upon requirements of the business.  A key success factor in capacity management is ensuring that it is considered during an application’s design.

In September 2010, the Chief Technology Officer outlined a goal to have the MITS organization implement ITIL best practices over the next several years.  The MITS Process Re-Engineering Executive Steering Committee governs the implementation of the ITIL.  Responsibility for implementing key ITIL concepts has been assigned to EOps executives, with an implementation plan due in September 2011.  We reviewed the capacity management policies and procedures and determined that the EOps organization has incorporated ITIL best practice principles. 

Performance Measurement Requirements in Defined-Service Agreements Are Not Formally Established to Facilitate the Management and Reporting of Mainframe Performance

The ITIL states that the primary purpose of the capacity management process is to identify and understand IT services and their use of resources, working patterns, and peaks and troughs and to ensure that the services meet their Service-Level Agreement (SLA) targets, i.e., to ensure that IT services are performed as required.  The focus is on managing service performance, as determined by the performance requirements contained in the SLAs that were agreed upon with their IRS business unit customers.  

The Information Systems Audit and Control Association’s Control Objectives for Information Technology (COBIT®) represents an authoritative, internationally accepted IT governance control framework for use by business managers, IT professionals, and IT audit professionals.  The COBIT states that an organization should define and agree to SLAs for all critical IT services based on customer requirements and IT capabilities.  The SLAs should cover customer commitments, service support requirements, quantitative and qualitative metrics for measuring the service signed off on by the stakeholders, funding and commercial arrangements (if applicable), and roles and responsibilities, including oversight of the SLA.  Metrics that should be considered include system availability, reliability, performance, capacity for growth, levels of support, continuity planning, security, and demand constraints.  The COBIT also recommends that organizations continuously monitor specified service-level performance criteria and provide reports on achievement of service levels in a format that is meaningful to the customers.  The monitoring statistics should be analyzed and acted upon to identify negative and positive trends for individual services, as well as for services overall.  

The Customer Relationships and Integration Division within the EOps organization is responsible for managing and administering all EOps organization SLAs.  Its stated mission is to serve as a focal point for establishing relationships between the IRS business units (customers) and the appropriate EOps division (service providers).  The EOps organization eliminated the need for individual applications to have SLAs.  The SLAs for individual applications that were previously in use were converted into 20 Defined-Service Agreements (DSA) or were merged into a larger document called the EOps Service Standards Document.  The EOps organization uses the term DSA to refer to the industry generic term, SLA.  The DSA describes the services received from the service provider, and the EOps Service Standards Document describes how the EOps organization (the service provider) will provide the services.  The conversion of SLAs to DSAs and a Service Standards Document was completed and will undergo an annual review. 

During our review of the 20 DSAs, we found that the EOps organization is not consistently including measurable performance metrics such as availability, reliability, performance, and capacity in these agreements.  Only 4 (20 percent) of the 20 DSAs contained any measurable performance metrics.  The remaining 16 (80 percent) documents contained no measurable metrics.  More specifically, one of the four DSAs that had measurable metrics was the Federal Information Security Management Act Reportable Application Support Service DSA.  It contained a list of 160 additional applications, and only 51 (32 percent) of the applications contained the measurable performance metrics for response times.  EOps management explained that the EOps organization has not included qualitative and quantitative metrics in its DSAs because IRS business unit customers have not asked it to do so.  Further, EOps management indicated that their Customer Relationships and Integration Division does not have the technical expertise needed to provide performance metric requirements. 

Our review of mainframe performance monitoring determined the EOps personnel responsible for capacity management of the IBM and Unisys mainframe environments are actively monitoring mainframe performance against their own informal measures.  The IBM capacity managers create an annual capacity report, as well as various day-to-day application-specific reports.  The Unisys capacity managers create periodic reports on daily, weekly, and weekend transaction processing. 

In addition, the EOps Service Standards Document states that the EOps Customer Relationships and Integration Division will provide measurable performance metrics for customer verification of its quality of service.  However, the Customer Relationships and Integration Division is not reporting actual performance achievements against specific DSA performance requirements to the EOps organization’s business unit customers. 

If specific and measurable performance metric requirements are not documented and agreed upon in the DSAs, then these documents cannot be effectively used to hold service providers to the minimum level of service required by critical applications.  Without a structure in place to report actual performance achievements relative to performance metric requirements to application owners, IRS management will be unable to verify and ensure that the quality of service provided to its mission-critical tax processing systems will meet the expectations of its customers and stakeholders.   

Recommendations

Recommendation 1:  The Associate Chief Information Officer, Enterprise Operations, should include specific and measurable qualitative and quantitative metric requirements in DSAs that can be used to define the quality of service required by EOps organization customers.  In this effort, quantitative metrics such as availability, reliability, performance, capacity for growth, levels of support, continuity planning, security, and demand constraints should be considered.

Management’s Response:  The IRS agreed with this recommendation.  The IRS is working to develop measures as they relate to the Mainframe Computing Defined Services.  In accordance with the recommendation, the IRS will consider the categories mentioned in the defined services design.

Recommendation 2:  The Associate Chief Information Officer, Enterprise Operations, should establish a method of reporting actual performance achievements, relative to agreed-upon performance metric requirements, to business unit application owners in a format that is useful for customer verification of the EOps organization’s quality of service.

Management’s Response:  The IRS agreed with this recommendation.  Enterprise Operations will establish a method of reporting actual performance achievements, relative to agreed-upon performance metric requirements, to business unit application owners in a format that is useful for customer verification of the EOps organization’s quality of service.   

The Internal Revenue Service May Incur Unnecessary Software Licensing Costs Related to Mainframe Hardware Capacity

The ITIL focuses on the key service management principles pertaining to service strategy, service design, service transition, service operation, and continual service improvement.  Within the service design principle, supplier management is a key process.  The supplier management process ensures that suppliers and the services they provide are managed to support IT service targets and business expectations.  Two of the main objectives of the supplier management process are:  1) to obtain value for the money spent with suppliers and on contracts and 2) to negotiate and agree to contracts with suppliers and manage them through their life cycle.

The IRS measures the capacity of its mainframes in terms of MIPS.  MIPS charts can be used for various purposes, including:  1) to provide a metric against which capacity consumption can be measured, 2) to provide capacity guidance when performing hardware upgrades within a processor family, and 3) as a means to provide independent software vendors a machine capacity with which to license and charge for software maintenance and one-time upgrade charges.  

The license costs for the software products residing on the IRS mainframes are tied to mainframe capacity, or the number of MIPS allocated to the machines.  The IRS whitepaper titled
z/196 Gartner MIPS and Capacity Upgrade Issues, issued by the IBM Capacity Management Section, notes that there is an opportunity for the IRS to reduce its software license costs by changing the measure used to calculate the capacity of its mainframes.  While the IRS currently uses MIPS, the IRS Capacity Management Branch recommends changing the measurement to Millions of Service Units, or MSUs.  The whitepaper indicates that IBM has reduced the MSU rating for each family of z-series processors by about 10 percent for machines of equivalent capacities. 

In October 2010, the IRS upgraded its Martinsburg Computing Center mainframe computers from the IBM z/9-series mainframes to the IBM z/196-series mainframes.  Concerns over IBM mainframe software licensing costs raised by the IRS Capacity Management Branch in its whitepaper may not have been communicated to the stakeholders with the authority and responsibility to renegotiate the relevant hardware and software contracts.  Had the IRS made the conversion from MIPS to MSUs as a basis for determining mainframe capacity prior to the IBM mainframe upgrade, the IRS could have realized a cost savings of $580,358 in its software licensing costs, using the 10 percent reduction estimate in the IRS whitepaper.  Figure 2 shows the software products where the IRS could have potentially realized savings in the licensing agreements.

Figure 2:  Potential Mainframe Software License Cost Savings

Software \Vendor

Software
Upgrade Costs

Potential Savings
(10% of Upgrade Costs)

IBM Software Relationship Offering

$41,608

$4,161

ASG Software Solutions

$44,746

$4,475

Catalog Recovery Plus

$129,725

$12,973

Computer Associates

$398,706

$39,871

Perfman

$146,185

$14,619

SAS

$644,813

$64,481

Vanguard

$178,807

$17,881

Address Hygiene Software

$1,757,276

$175,728

Mainframe Peripheral and Software Maintenance

$2,461,692

$246,169

Total

$5,803,558

$580,358

Note:  The Total Potential Savings is greater than 10 percent due to rounding.
Source:  IRS contract documentation.

Approximately 8,000 MIPS currently allocated to the CADE will need to be repurposed when the IRS transitions from the CADE to the CADE 2 in January 2012.  The IRS indicated it would establish a working group to coordinate decisions regarding managing the excess hardware capacity.  In addition, the IRS has started to make decisions regarding the renegotiation of software license agreements that are dependent on hardware capacity.  If these hardware resources are not fully utilized, or actions are not taken to fully renegotiate capacity-dependent software contracts, the IRS could incur unnecessary hardware and software costs.

Recommendations

Recommendation 3:  The Chief Technology Officer should change the basis for determining the capacity of its IBM mainframe computers from MIPS to MSUs for future hardware upgrades.

Management’s Response:  The IRS agreed with this recommendation.  The EOps organization will partner with the Strategy and Planning Division to change the basis for measuring the capacity of its IBM mainframe computers and converting future software upgrades from MIPS to MSUs.

Recommendation 4:  The Chief Technology Officer should review the mainframe computer capacity-dependent software agreements and attempt to renegotiate these contracts to more favorable terms for the IRS to potentially realize cost savings in mainframe software contracts.

Management’s Response:  The IRS agreed with this recommendation.  Over the past 2 years, the IRS has revised or restructured every mainframe software contract for maximum efficiency.  The IRS has converted them to Treasury-wide vehicles where possible with the exception of the IBM Software Relationship Offering which will be restructured next fiscal year.

Office of Audit Comment:  The IRS did not agree with the outcome measure related to the use of alternative capacity measures in software negotiations for licensing costs.  In its response, the IRS stated that it agrees there is an opportunity to use MSUs in place of MIPS to more effectively track capacity requirements in support of contract negotiations for software licensing costs and to reduce software licensing costs.  The IRS also stated it does not believe there is any evidence that the vendors involved in past software licensing agreements would have agreed to use the MSU method in determining licensing costs.  However, the TIGTA maintains the validity of this potential cost savings based on an estimate provided by the IRS Whitepaper titled z/196 – Gartner MIPS and Capacity Upgrade Issues, dated October 2010 and prepared by the IBM Capacity Management Section.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

Our overall objective was to evaluate the efficiency and effectiveness of the capacity and performance management of the IRS mainframe[2] environment.  To accomplish our objective, we:

I.                   Evaluated mainframe capacity and performance management policies, standards, and procedures.  

A.    Searched for Federal guidance for capacity and performance management of mainframes.  

B.     Reviewed industry best practices (ITIL and COBIT) for capacity and performance management of mainframes.

C.     Reviewed IBM and Unisys vendor guidance for capacity and performance management of mainframes.

D.    Determined whether IRS capacity and performance guidance reflects Federal, industry, and vendor best practice guidance.

E.     Determined whether IRS policies and procedures identify and define responsibilities for capacity and performance management. 

F.      Determined whether the roles and responsibilities have been formally assigned and communicated. 

II.                Evaluated mainframe service-level requirements.

A.    Reviewed service-level agreements for reasonableness and completeness for applications processed in the IRS mainframe production environment to determine whether:

1.      Definitive units of measure for mainframe capacity and performance have been established. 

2.      Minimum levels of service or performance have been identified and agreed to by stakeholders.

B.     Reviewed current capacity and performance metrics regarding how well processors and applications are able to achieve stated mainframe service-level requirements. 

III.             Evaluated the monitoring, reporting, and management of mainframe capacity and performance trends.

A.    Reviewed Calendar Years 2009, 2010, and 2011 Executive Steering Committee minutes and Business Performance Review reports relating to mainframe capacity and performance. 

B.     Reviewed Calendar Years 2009, 2010, and 2011 annual capacity and performance plans. 

C.     Reviewed Calendar Years 2010 and 2011 summary reports generated from IRS capacity and performance management reviews.

D.    Reviewed the current status of the IBM mainframe processor upgrade (zAAP/zIIP project).

Internal controls methodology

Internal controls relate to management’s plans, methods, and procedures used to meet their mission, goals, and objectives.  Internal controls include the processes and procedures for planning, organizing, directing, and controlling program operations.  They include the systems for measuring, reporting, and monitoring program performance.  We determined the following internal controls were relevant to our audit objective:  the MITS organization’s policies and procedures for effectively managing the performance and capacity of its mainframe computers.  We evaluated these controls by interviewing management and reviewing the EOps organization’s policies and procedures and industry best practices such as COBIT and ITIL.

 

Appendix II

 

Major Contributors to This Report

 

Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services)

Danny Verneuille, Director

Carol Taylor, Audit Manager

Myron Gulley, Lead Auditor

Mary Jankowski, Senior Auditor

Daniel Oakley, Information Technology Specialist

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn: Chief of Staff C

Deputy Commissioner for Operations Support  OS

Deputy Chief Information Officer for Operations  OS:CTO

Associate Chief Information Officer, Enterprise Operations  OS:CTO:EO

Associate Chief Information Officer, Strategy and Planning  OS:CTO:SP

Director, Enterprise Computing Centers  OS:CTO:EO:EC

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Internal Control  OS:CFO:CPIC:IC

Audit Liaison:  Director, Risk Management Division  OS:CTO:SP:RM

 

Appendix IV

 

Outcome Measure

 

This appendix presents detailed information on the measurable impact that our recommended corrective action will have on tax administration.  This benefit will be incorporated into our Semiannual Report to Congress.

Type and Value of Outcome Measure:

·         Inefficient Use of Resources – Potential; $580,358 (see page 6).

Methodology Used to Measure the Reported Benefit:

The software license costs for software products residing on the IRS mainframes[3] are tied to the mainframe capacity, or the number of MIPS allocated to the machines.  The IRS whitepaper titled z/196 Gartner MIPS and Capacity Upgrade Issues, dated October 2010 and prepared by the IBM Capacity Management Section, notes that there is an opportunity for the IRS to reduce its software costs by changing the measure it uses to calculate the capacity of its mainframes.  While the IRS currently uses MIPS, the IBM Capacity Management Branch recommends changing the measurement to MSUs.  The whitepaper indicates that IBM has reduced the MSU rating for each family of IBM z-series processors by about 10 percent for machines of equivalent capacities. 

In October 2010, the IRS upgraded its Martinsburg Computing Center mainframe computers from the IBM z/9-series mainframes to the IBM z/196-series mainframes.  Had the IRS made the conversion from MIPS to MSUs as a basis for determining mainframe capacity prior to its mainframe upgrade, the IRS could have realized a cost savings of $580,358 in its software licensing costs using the 10‑percent reduction estimate in the IRS whitepaper.  Figure 1 provides the calculation for the amount of potential software cost savings.

 

Figure 1:  Calculation of Potential Software Cost Savings

Software \Vendor

Potential Savings
(10% of Upgrade Costs)

IBM Software Relationship Offering

$4,161

ASG

$4,475

Catalog Recovery Plus

$12,973

Computer Associates

$39,871

Perfman

$14,619

SAS

$64,481

Vanguard

$17,881

Address Hygiene Software

$175,728

Mainframe Peripheral and Software Maintenance

$246,169

Total

$580,358

Source:  IRS contract documentation.

 

Appendix V

 

Internal Revenue Service Mainframe Computer Performance

 

The following figures show how the IRS IBM mainframes[4] performed from January 2009 to June 2011 (the horizontal axis on each chart) with the total number of MIPS available to the machine (the vertical axis on each chart) as the basis for measurement.  Figure 1 shows that in Calendar Year 2009, the Martinsburg Computing Center IBM mainframe #1 was initially allocated 5,200[5] MIPS, with upgrades to 6,400.  It had a Calendar Year 2009 average weekly utilization of 62 percent of capacity.  In Calendar Year 2010, the MIPS were upgraded to 7,500 MIPS.  It had a Calendar Year 2010 average weekly utilization of 57 percent of capacity.  In Calendar Year 2011, it was allocated 8,800 MIPS and had an average utilization of 63 percent of capacity.

Figure 1:  Martinsburg IBM Mainframe #1 Capacity and Performance Trends

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Figure 2 shows that in Calendar Year 2009, the IBM Martinsburg Computing Center mainframe #2 was initially allocated 5,200 MIPS, with upgrades to 6,100 MIPS.  It had a Calendar Year 2009 average weekly utilization of 60 percent of capacity.  In Calendar Year 2010, the MIPS were upgraded to 7,500 MIPS.  It had a Calendar Year 2010 average weekly utilization of 51 percent of capacity.  In Calendar Year 2011, 8,000 MIPS were allocated, and had an average weekly utilization of 61 percent of capacity.

Figure 2:  Martinsburg IBM Mainframe #2 Capacity and Performance Trends

Figure 2 was removed due to its size.  To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page

Figure 3 shows that in Calendar Year 2009, the IBM mainframe at the Memphis Computing Center was allocated 900 MIPS and had an average weekly utilization of 54 percent of capacity.  It had an average weekly utilization of 43 percent of capacity in Calendar Years 2010 and 49 percent in 2011.  The Memphis Computing Center mainframe processes the production workload for the Integrated Collection System and is the disaster recovery computer for the Martinsburg Computing Center IBM mainframes.   

Figure 3:  Memphis IBM Mainframe Capacity and Performance Trends

Figure 3 was removed due to its size.  To see Figure 3, please go to the Adobe PDF version of the report on the TIGTA Public Web Page

Figures 4 and 5 provide a snapshot view of the Unisys production mainframes’ capacity and performance during one processing cycle of the 2011 Filing Season. 

Figure 4:  Martinsburg Unisys Mainframe Utilization, February 1420, 2011

Figure 4 was removed due to its size.  To see Figure 4, please go to the Adobe PDF version of the report on the TIGTA Public Web Page

Figure 5:  Memphis Unisys Mainframe Utilization, February 1420, 2011

Figure 5 was removed due to its size.  To see Figure 5, please go to the Adobe PDF version of the report on the TIGTA Public Web Page

 

Appendix VI

 

Glossary of Terms

 

Term

Definition

Application

A software program hosted by an information system.

Automated Collection System

A telephone contact system through which telephone assistors collect unpaid taxes and secure tax returns from delinquent taxpayers who have not complied with previous notices.

Business Master File

The IRS database that consists of Federal tax-related transactions and accounts for businesses.  These include employment taxes, income taxes on businesses, and excise taxes.

Business Taxpayer Information File

Area of the IRS tax information database that contains information from business taxpayer accounts.

Campus

The data processing arm of the IRS.  The campuses process paper and electronic submissions, correct errors, and forward data to the Computing Centers for analysis and posting to taxpayer accounts.

Capacity management

The discipline that ensures IT infrastructure is provided at the right time in the right volume at the right price, and ensuring that IT is used in the most efficient manner.  This involves input from many areas of the business to identify what services are (or will be) required, what information technology infrastructure is required to support these services, what level of contingency will be needed, and what the cost of this infrastructure will be.

Customer Account Data Engine

The foundation for managing taxpayer accounts in the IRS modernization plan.  It will consist of databases and related applications that will replace the IRS’s existing Master File processing system and will include applications for daily posting, settlement, maintenance, refund processing, and issue detection for taxpayer tax account and return data. 

Customer Account Data Engine 2

An IRS application that will replace the existing Individual Master File and CADE applications.  CADE 2 is designed to provide state-of-the-art individual taxpayer account processing and technologies to improve service to taxpayers and enhance IRS tax administration. 

Federal Information Security Management Act

United States legislation that defines a comprehensive framework to protect government information, operations, and assets against natural or man-made threats.

Filing Season

The period from January 1 through mid-April when most individual income tax returns are filed.

Fiscal Year

A 12-consecutive-month period ending on the last day of any month, except December.  The Federal Government’s fiscal year begins on October 1 and ends on September 30.

Hardware

The physical components of an information system.

Individual Master File

The IRS database that maintains transactions or records of individual tax accounts.

Individual Taxpayer Information File

Area of the Individual Master File database that contains information from individual taxpayer accounts.

Integrated Collection System

An information management system designed to improve revenue collections by providing revenue officers access to the most current taxpayer information, while in the field, using laptop computers for quicker case resolution and improved customer service.

Integrated Data Retrieval System

IRS computer system capable of retrieving or updating stored information.  It works in conjunction with a taxpayer’s account records.

Logical Partition

The division of a computer’s memory and storage into multiple sets of resources so that each set of resources can be operated independently with its own operating system and applications.

Mainframe

A powerful, multiuser computer capable of supporting many hundreds of thousands of users simultaneously.

Platform

The computer architecture and equipment using a particular operating system.

Processor

A part of a computer, such as the central processing unit, that performs calculations of data and other logical functions.

Software

Computer programs and associated data that may be dynamically written or modified during execution.

Web Currency and Banking Retrieval System

An online database that contains Bank Secrecy Act information.  IRS field agents as well as local, State, and Federal law enforcement agencies access the database for research in tax cases, tracking money-laundering activities, investigative leads, intelligence for the tracking of currency flows, corroborating information, and probative evidence.

 

 

Appendix VII

 

Management’s Response to the Draft Report

 

DEPARTMENT OF THE TREASURY

INTERNAL REVENUE SERVICE

WASHINGTON, D.C. 20224

 

 

CHIEF TECHNOLOGY OFFICER

 

SEPTEMBER 13, 2011

 

 

MEMORANDUM FOR DEPUTY INSPECTOR GENERAL FOR AUDIT

 

FROM:                             Terence V. Milholland /s/ Terence V. Milholland

     Chief Technology Officer

 

SUBJECT:                       Draft Audit Report - Mainframe Computer Performance Is Being Actively Monitored, but Defined-Service Agreements and Software Licensing Can Be Improved

(Audit # 201120015) (e-trak # 2011-24391)

 

Thank you for the opportunity to review and respond to the subject audit report. We agree with the recommendations outlined in the report and the attachment details our planned actions to implement them.

 

Based on our analysis of alternative capacity measures, we believe there is an opportunity to use Millions of Service Units (MSU) in lieu of Million Instructions Per Second (MIPS) to more effectively track capacity requirements in support of contract negotiations for software licensing costs. We therefore anticipate that cost savings may result from our conversion from MIPS to MSU. However, there is no evidence vendors involved in past software licensing agreements would have agreed to use the MSU method in determine licensing costs. Therefore, any assertion of missed savings opportunity is, at best, speculative, thereby we disagree with the Outcome Measure.

 

We value your continued support and the guidance your team provides. If you have any questions, please contact me at (202) 622-6800 or Andrea Greene-Horace, Senior Manager of Program Oversight, at (202) 283-3427.

 

Attachment

 

RECOMMENDATION #1: The Associate Chief Information Officer, Enterprise Operations, should include specific and measurable qualitative and quantitative metric requirements in Defined Service Agreements that can be used to define the quality of service required by EOps organization customers. In this effort, quantitative metrics such as availability, reliability, performance, capacity for growth, levels of support, continuity planning, security, and demand constraints should be considered.

 

CORRECTIVE ACTION #1: We agree with the recommendation. We are working to develop measures as they relate to our Mainframe Computing Defined Services. In accordance with the recommendation we will consider the categories mentioned in our defined services design.

 

IMPLEMENTATION DATE: October 2, 2012

 

RESPONSIBLE OFFICIAL: Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #2: The Associate Chief Information Officer, Enterprise Operations, should establish a method of reporting actual performance achievements, relative to agreed-upon performance metric requirements, to business unit application owners in a format that is useful for customer verification of the EOps organization's quality of services.

 

CORRECTIVE ACTION #2: We agree with the recommendation. Enterprise Operations (EOps) will establish a method of reporting actual performance achievements, relative to agreed-upon performance metric requirements, to business unit application owners in a format that is useful for customer verification of the EOps organization's quality of services.

 

IMPLEMENTATION DATE: November 1, 2012

 

RESPONSIBLE OFFICIAL: Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #3: The Chief Technology Officer should change the basis for determining the capacity of its IBM mainframe computers from MIPS to MSUs for future hardware upgrades.

 

CORRECTIVE ACTION #3: We agree with the recommendation. Enterprise Operations will partner with Strategy and Planning to change the basis for measuring the capacity of its IBM mainframe computers and converting future software upgrades from MIPS to MSUs.

 

IMPLEMENTATION DATE: October 2, 2012

 

RESPONSIBLE OFFICIAL: Associate Chief Information Officer, Enterprise Operations

CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #4: The Chief Technology Officer should review the mainframe computer capacity-dependent software agreements and attempt to renegotiate these contracts to more favorable terms for the IRS to potentially realize cost savings in mainframe software contracts.

 

CORRECTIVE ACTION #4: We agree with the recommendation. Over the past 2 years we have revised or restructured every mainframe software contract for maximum efficiency. We have converted them to Treasury-wide vehicles where possible with the exception of the IBM Software Relationship Offering (SRO) which we will be restructuring next fiscal year.

 

IMPLEMENTATION DATE: October 2, 2012

 

RESPONSIBLE OFFICIAL: Associate Chief Information Officer, Strategy and Planning

 

CORRECTIVE ACTION MONITORING PLAN: We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.



[1] See Appendix VI for a glossary of terms.

[2] See Appendix VI for a glossary of terms.

[3] See Appendix VI for a glossary of terms.

[4] See Appendix VI for a glossary of terms.

[5] For reporting purposes, all MIPS allocations are rounded to the nearest 100 MIPS.