TREASURY INSPECTOR GENERAL FOR TAX
ADMINISTRATION
Improvements in Mainframe
Computer Storage Management Practices and Reporting Are Needed to Promote Effective
and Efficient Utilization of Disk Resources
May 2006
Reference Number: 2006-20-056
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone Number |
202-927-7037
Email Address | Bonnie.Heald@tigta.treas.gov
Web Site | http://www.tigta.gov
May 10, 2006
MEMORANDUM FOR CHIEF
INFORMATION OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Improvements in Mainframe Computer Storage Management Practices and Reporting Are Needed to Promote Effective and Efficient Utilization of Disk Resources (Audit # 200520024)
This report presents the results of our review of the Internal Revenue Service’s (IRS) mainframe computer storage management. The overall objective of this review was to assess the effectiveness and efficiency of the IRS’ mainframe computer storage management practices.
Synopsis
The IRS requires a large and
complex computer environment to process and store taxpayer, financial, and administrative
data. A significant portion of the
computer processing workload is performed by mainframe computer systems. These systems are managed by the
Effective October 2005, the
Enterprise Operations organization, including the
Current disk storage
capacity exceeds needs, but additional capacity is being purchased. In
our opinion, additional storage capacity continues to be purchased based on
storage allocation instead of actual usage, as evidenced by the fact that
actual storage usage information is not included in storage purchase
justifications.
Based on the storage usage rate, the $626,866 spent to purchase
21 terabytes of additional disk storage in August 2005 represents an
inefficient use of resources, and the $1,423,140 planned expense to lease 38.4
terabytes of additional disk storage beginning in Fiscal Year 2006 represents
funds that could be put to better use.
The IRS
purchased additional disk storage capacity although only 41 percent to 69
percent of existing disk storage capacity was being used. Procedures for monitoring actual storage
usage could improve overall utilization, postpone the purchasing of unnecessary
capacity, and save resources for other needs.
Purchases
of disk storage capacity also continue because Enterprise Operations organization
management allows project owners to determine how the storage capacity is to be
allocated and how long the data will remain on the disks. In addition, disk storage usage information
is not regularly shared with the project owners, and
Disk data retention procedures for
removing unused data are not standardized and consistently implemented. Our analysis of 18 judgmentally selected applications
that run on the IBM - Master File, IBM - Other, and Unisys systems showed
inconsistencies in data retention practices for the same file types. In addition, the applications running on IRS
mainframe computers include batch processes and, as a result, receive and
process data in cycles (e.g., daily, weekly, monthly) that the Computing Centers
are storing on disk. The storage
administrators manage the disk storage by the number of cycles kept on disk for
each file. We believe storing data on
disk after the cycles are processed is an inefficient use of disk storage.
File transfer procedures require
duplicate disk storage capacity. The
IRS’ target Enterprise Architecture requires storage virtualization, which is
the consolidation of multiple network storage devices into what appears to be a
single storage unit (i.e., a virtual storage pool). However, the IRS’ mainframe disk storage
devices are each connected to a mainframe computer system and are used by only
that system. Consequently, files created
and stored on one system must be transferred (i.e., copied) to another system
for additional processing, tax account research, or other use, and storage
capacity for the files must be provided on the originating and receiving
storage devices. The Associate
Chief Information Officer (CIO), Enterprise Operations, advised virtualization is something the IRS wants to implement and
she has tried for several years to obtain funding. However, funding is not available and
virtualization across systems cannot be implemented.
In Fiscal Year 2005, the Capacity Management Branch
published reports that included high-level disk storage capacity and assignment
charts and future capacity requirements based on assignments. However, the reports did not analyze disk
storage usage and base future capacity requirements on the historic usage
trends because IRS management gave greater priority to assessing computer
processor needs. In addition, storage
usage assessment responsibilities were moved to another organizational unit. We also found inconsistencies in
Recommendations
We recommended the CIO ensure the Enterprise Operations organization develops and implements standardized procedures to operate and manage disk storage capacity as a corporate asset, includes storage usage information in disk storage purchase justifications, removes unused copies of files and database records from disk storage, conducts annual capacity planning assessments including the assessment of disk storage usage, and develops standardized storage allocation and usage reports that are shared with MITS organization management and project owners. We also recommended the CIO ensure the Business Systems Development organization effectively sets application data retention parameters based on an application’s processing cycles, to improve mainframe computer disk storage efficiency.
Response
IRS management agreed with all of the recommendations in the report and will implement several corrective actions, including standardizing procedures to manage disk storage capacity as a corporate asset and data retention. The storage request process will be amended to include a capacity analysis to prevent spending on unneeded disk storage. Reports will be generated by the Enterprise Operations organization and analyzed by the Applications Development organization[2] to confirm current data retention periods or provide new retention periods, revalidate projected storage need, and/or release storage for use by other projects. The Enterprise Operations organization will take necessary actions to release any unneeded disk storage space and will conduct annual capacity planning assessments for all mainframe computers based on standardized disk storage reports. The reports will be shared with MITS organization management and project owners.
IRS management disagreed with our cost savings estimates
identified in Appendix IV based on concerns that consideration was not given to
all cost-related factors in arriving at the savings estimates. Management’s complete response to the draft
report is included as Appendix VIII.
Office of Audit
Comment
IRS management
disagreed with the outcome measures included in Appendix IV and stated we did
not include in our calculations all cost-related factors, such as maintenance
and/or environmental factors including power and air conditioning. We discussed the cost savings and the methodology
for calculating the savings with IRS management and revised the audit report
based on management’s comments. In
addition, we revised Appendix IV based on Management’s Response to the draft report
to show the consideration of the maintenance cost savings from the replaced
storage.
Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
Current Disk Storage Capacity
Exceeds Needs, but Additional Capacity Is Being Purchased
Management
Actions Are Needed to Improve Disk Storage Utilization for Mainframe Computers
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – Outcome Measures
Appendix VI
– Unisys Disk Storage Capacity Assignment and Usage Metrics
Appendix
VII – Glossary of Terms
Appendix
VIII – Management’s Response to the Draft Report
The Internal Revenue Service (IRS) requires a large and
complex computer environment to process and store taxpayer, financial, and
administrative data. The computer
environment includes mainframe,[3]
mid-range, and end user computer systems, disk and tape storage devices, and
data communication networks. A
significant portion of the computer processing workload is performed by the
IRS’ mainframe computer systems managed by the
The mainframe computer systems support the administration of the internal revenue laws and provide services and products to the taxpaying public and the IRS’ business operating divisions. The systems generate a significant amount of data that are stored on disk and tape storage devices. In Fiscal Years (FY) 2003 – 2005, the IRS spent $9,006,745 to replace older disk storage devices and to purchase and lease 56.4 terabytes of additional disk storage capacity. Between October 2001 and September 2005, the IRS’ mainframe computer disk storage capacity increased from 79 terabytes to 168.5 terabytes. Figure 1 highlights the significant growth in mainframe computer disk storage capacity for FYs 2001 – 2005. Figure 2 lists the International Business Machines (IBM) and Unisys mainframe computer systems and the disk storage capacity by location.
Figure 1: Mainframe Computer Disk Storage Capacity
Figure 1 was removed due to its
size. To see Figure 1, please go to the
Adobe PDF version of the report on the TIGTA Public Web Page.
Figure 2: Mainframe Computer System Locations and Disk Storage Capacity
|
Systems |
Capacity (in terabytes) |
|||
|
|
Martinsburg |
|
Total by
System |
|
|
IBM - Master File |
|
128.4 |
|
128.4 |
|
IBM - Other [includes the Integrated
Collection System, Automated Collection System, Printer
Replacement to Integrate New Tools System, Security and Communications System, and Administrative systems] |
15.6 |
9.4 |
8.8 |
33.8 |
|
Unisys
- |
|
3.4 |
2.9 |
6.3 |
|
Total by Location |
15.6 |
141.2 |
11.7 |
168.5 |
Source: The IRS’ As-Built Architecture dated October 25,
2005, discussions with
This review was performed in the Enterprise Operations organization
at the IRS National Headquarters in New Carrollton, Maryland, and in
The Competency Based Organization Provides
Management the Opportunity to Create a Uniform Disk Storage Operating
Environment With Standardized Management Practices
The Clinger-Cohen Act of
1996[4] states the Chief Information Officer (CIO) is
responsible for promoting the effective and efficient design and
operation of all major information resources management processes for the
executive agency. Prior consultant
studies and Treasury Inspector General for Tax Administration reviews[5]
identified opportunities to improve management processes and reduce operating
costs. The opportunities and declining
budgets caused the Enterprise Operations organization, including the
Prior to implementation of the competency based
organizational structure, the three Computing Centers were independent
organizations with mainframe computer storage management roles and
responsibilities distributed among several
The Internal
Revenue Manual states the Associate CIO, Enterprise Operations, is responsible
for ensuring the effective and efficient use of the IRS automated information
processing environment.
Our review of the January 1, 2005, January
3, 2005, and September 12, 2005, usage reports for the IBM - Master File and/or
Unisys disk storage devices showed that, compared to the Industry - Average and
Industry - Best Practice metrics used by the
The IRS purchased additional disk storage capacity although only 41 percent to 69 percent of existing disk storage capacity was being used. Procedures for monitoring actual storage usage could improve overall utilization, postpone the purchasing of unnecessary capacity, and save resources for other needs.
However, assigned storage is not effectively used. The Industry - Average for the storage in use as a percentage of capacity assigned metric is 81.8 percent, and the Industry - Best Practice is 93.7 percent. As of September 12, 2005, IRS data:
Comparable disk storage capacity and usage reports for the IBM - Other systems were not available during the review.
During FYs 2003 – 2005, the IRS spent $9,006,745 to replace older
disk storage devices and to purchase and lease 56.4 terabytes of additional
disk storage capacity, including software and maintenance. Justifications
supporting the $5,500,416 spent to purchase 51.7 terabytes of new storage
capacity for the Master File stated that additional capacity was needed to meet
capacity requirements.
·
In August
2005, 40.2 terabytes of disk storage were purchased for $1,200,000, including 19.2
terabytes of replacement disk and optical storage and 21 terabytes of additional
disk storage capacity. We prorated the
total cost and estimate the additional disk storage capacity cost $626,866. The justification for this purchase
identified $369,451 annual maintenance costs; therefore, we estimate the
$1,200,000 purchase cost would be repaid in 39 months. However, the
additional disk storage capacity purchase may not have been necessary because,
as of September 12, 2005, only 69.3 percent of the 53.5 terabytes of Current
Processing Environment-assigned disk storage capacity was used and 14.2
terabytes were being held in reserve.
·
In FY
2006, IRS management plans to lease 69.9 – 116.5 terabytes of disk storage for 4
– 5 years that will cost $560,400 – $1,000,560 per year, including 31.5
terabytes of replacement disk storage and 38.4 – 85 terabytes of additional
disk storage capacity. As of the end of
our audit work, the capacity and cost details had not been completed. Therefore, we prorated the total lease costs
(annual cost times lease period) and estimate the additional disk storage
capacity will cost $1,423,140 – $3,144,927.
However, the lease may not be necessary
because, as of September 12, 2005, only 47.7 percent of the 51.8 terabytes of Modernization-assigned
disk storage capacity was used and 9 terabytes were being held in reserve.
Management did not
provide documentation of alternate proposals that would have provided the maintenance,
performance, and
In our opinion, additional
storage capacity continues to be purchased based on storage allocation instead
of actual usage, as evidenced by the fact that actual storage usage information
is not included in storage purchase justifications. For example, the purchase of 12
terabytes for the IBM - Master File Current Processing Environment was approved
March 22, 2005. The justification stated
the purchase was needed because “. . . in 2004 [the Master File] will
experience a 20 percent growth in all of its established workloads . . .” and was
“. . . currently experiencing [disk
storage] resource shortages and . . . space reserves are at all time low levels.” As of January 1, 2005, the IBM - Master File
usage report indicated 2.2 terabytes of the total 58.5 terabytes of capacity assigned
to the Current Processing Environment was reserved for growth, while 69.4
percent of the assigned Current Processing Environment capacity was in use. By September 12, 2005, the reserve had
increased to 14.2 terabytes, while usage decreased to 69.3 percent.
Purchases of disk storage capacity also continue because Enterprise
Operations organization management has viewed disk storage capacity as owned by
the projects and allows project owners to determine how the storage capacity is
to be allocated (e.g., development, test, or production assignments) and how
long the data will remain on the disks. Enterprise
Operations organization management advised that the Information Technology
Investment Account funds the purchase of disk storage for Modernization
projects and limits the
In addition, disk storage usage information is not regularly
shared with the project owners, and
The above conditions indicate the Enterprise Operations organization
has not yet achieved its goal of managing the disk storage capacity as a
corporate asset because storage administrators are not allowed to reassign
unused or underused disk capacity to applications that need additional capacity. Procedures for managing the disk storage as a
corporate asset and monitoring actual storage usage could improve overall
utilization, postpone the purchasing of unnecessary capacity, and save
resources for other needs.
Recommendations
Recommendation
1: The CIO should
ensure the Enterprise Operations organization develops and implements standardized
procedures to operate and manage disk storage capacity as a corporate asset,
including disk storage capacity purchased with Information Technology
Investment Account funds and used in production, and
ensures the procedures give storage administrators the authority to increase or
decrease the capacity assigned to applications based on storage usage to
improve utilization.
Management’s Response: IRS management agreed with the recommendation. The Enterprise Operations organization has developed and implemented standardized procedures to operate and manage disk storage capacity as a corporate asset in the Master File Legacy[6] and Modernization environments and has initiated plans and assigned staff to standardize procedures for the remaining mainframe systems. All disk storage capacity purchased with Information Technology Investment Account funds is being reviewed from a corporate perspective for the most efficient utilization of resources.
Recommendation 2:
The CIO should ensure
Management’s Response: IRS management agreed with the recommendation. The Enterprise Operations organization will amend the storage request process to include a capacity analysis to prevent spending on unneeded disk storage. The updated process will be communicated to customers to ensure justifications adequately address the business need for additional capacity.
Office of Audit Comment:
IRS
management disagreed with the outcome measures included in Appendix IV and stated
we did not include in our calculations all cost-related factors, such as
maintenance and/or environmental factors including power and air
conditioning. We discussed the cost
savings and the methodology for calculating the savings with IRS management and
revised the audit report based on management’s comments.
In calculating the first outcome measure, we considered the maintenance
cost savings for the replaced storage in our calculation of the cost of the
replacement storage. We revised Appendix
IV to show the consideration of the maintenance cost savings. We claimed the cost of the additional storage
as an inefficient use of resources because, as of September 12, 2005, only 69.3
percent of the 53.5 terabytes of Current Processing Environment-assigned disk
storage capacity was used and 14.2 terabytes were being held in reserve. Because much of the existing storage was not
being used, the purchase of additional disk storage may have not been necessary.
In calculating the second outcome measure, we based the cost of the planned additional storage capacity on the information provided to us during the audit. We were provided with a cost of $2,590,560 for acquiring future disk storage and calculated $1,423,140 as the cost of additional storage capacity. Similar to the calculation for the first outcome measure, the maintenance cost savings from the replaced storage lowered the cost of the replacement storage. As of the end of our audit work, the capacity and cost details for the storage purchase had not been completed. In addition, no costing information was provided to us for the environmental factors such as power and air conditioning mentioned in management’s comment.
Office of Management and Budget Circular
A-130, Management of Federal Information
Resources, dated November 28, 2000, requires the CIO to ensure the
implementation and enforcement of applicable records management policies and
procedures, including requirements for archiving information maintained in
electronic format, particularly in the planning, design, and operation of
information systems. In
addition, the Internal Revenue Manual states
the CIO and the MITS organization have the responsibility for evaluating and
monitoring data administration.
One of the benefits anticipated from the Competency Based
Organization
for the
Disk data retention procedures for
removing unused data are not standardized and consistently implemented
Data management best practices indicate
data have a life cycle and are not permanent.
Therefore, an organization should determine how important the data are to
its business customers (i.e., availability and access frequency) and establish
procedures to move data that do not require immediate access from online disk
storage to near-line or offline storage.
The
The Systems Standards Manual,
which was developed for Master File system users, lists factors to consider in
determining if and how long data should reside on disk and includes procedures
to automate disk management (e.g., establishing expiration dates for data
files). However, comparable reference
manuals for other IBM and Unisys system users do not exist. Our analysis of 18 judgmentally selected applications
that run on the IBM - Master File (6 applications), IBM - Other (5
applications), and Unisys (7 applications) systems showed inconsistencies in
data retention practices for the same file types (see Appendix I for a list of
the sampled applications).
Our review of 11 IBM - Master File
and IBM - Other applications showed:
While the Systems Standards Manual
procedures for establishing expiration dates on data files do not apply to the
other IBM systems, the storage management software is the same; therefore,
storage administrators could follow similar procedures to improve storage
management.
Procedures followed by Unisys
system storage administrators and application developers were also not
standardized. The applications running
on the Unisys mainframe computers include batch processes and, as a result, receive
and process data in cycles (e.g., daily, weekly, monthly) that the Computing Centers
are storing on disk. The storage
administrators manage the disk storage by the number of cycles kept on disk for
each file. The Martinsburg and
In addition, application
developers can set the maximum number of cycles that will be stored on disk for
each file before the earliest cycle is removed to tape, with 32 cycles being
the default setting. Our review of the
stored data files for seven Unisys applications indicated efficient disk
storage usage might be hampered by the number of cycles retained on disk. For example:
The Systems Standards Manual
requires application owners to review database and file access activity to
ensure databases do not remain on disk past their intended life cycles. However, the procedure may not be
effective. After we requested file
retention data, application owners stated 1 Master File application had
obsolete files and the programmers deleted 90 (63 percent) of the 144 database files
listed for this application.
File transfer procedures require duplicate
disk storage capacity
Data files transferred from other
systems must be stored initially on disk for additional processing, tax account
research, or other use and archived to tape storage when no longer needed for
immediate access. Our review of file
transfer logs maintained for 2-week periods in mid-2005[7] identified the following:
·
The file transfer logs
listed 77,044 transfers for 27,127 file names.
·
The total size of the
77,044 transfers was 713.7 gigabytes, which represents the capacity needed if
all the files are retained for at least 2 weeks.
The data reviewed did not identify
how long these files would remain on disk; however, the results indicate the
disk capacity used could be substantial if unused files are retained on disk.
Data retention rules have not been
standardized because, prior to the IRS’ October 2005 restructuring to a Competency
Based Organization, the three
As a long-term solution to this
problem, the IRS’ target Enterprise Architecture requires computer data storage to be centrally
managed through a concept called storage virtualization, where multiple
network storage devices are consolidated into what appears to be a single
storage unit (i.e., a virtual storage pool). Virtualization should result in the
optimal placement of files to achieve the highest performance and availability. However, the IRS’
mainframe disk storage devices are currently connected to specific mainframe
computer systems (e.g., the IBM - Master File, IBM - Other, or Unisys system) and
are used by only that system. Consequently,
files created and stored on one system must be transferred to another system
for additional processing, tax account research, or other use. As a result, storage capacity must be
provided on both the originating storage device and the receiving storage
device to store the duplicate copies of the file. By implementing storage virtualization and
reducing the need to transfer files between systems, the IRS could save
resources by eliminating the creation of duplicate files.
The Associate CIO, Enterprise
Operations, advised virtualization is
something the IRS wants to implement and she has tried for several years to obtain
funding for a virtual tape solution to resolve disaster recovery issues. However, funding is not available and
virtualization across systems cannot be implemented.
We understand the difficulties in
obtaining the necessary funds to implement various information technology
initiatives and, therefore, are making no recommendations to address this issue. However, if the IRS had a business case
documenting the potential return on investment that could be realized from the
implementation of virtualization, funding support might be justified.
Disk storage management information is not consistent and complete
Prior to the recent reorganization, the Capacity Management Branch was responsible for the
application of capacity planning and performance management tools, techniques,
and analytical methods to improve the efficiency and productivity of existing
and planned computer systems and applications. Effective
October 2005, the Enterprise Operations organization’s Information Technology
Infrastructure Division has these responsibilities. The Capacity Management Branch
published annual capacity assessment reports for each of the mainframe computer
systems. However, the Branch did not
complete assessments for the IBM - Master File and IBM - Other (Administrative)
systems in FY 2005 because Capacity Management Branch staff that left the
organization were not being replaced, resulting in insufficient staff to
complete the assessments. The Associate
CIO, Enterprise Operations, advised that the Capacity Management Branch staff
positions were not filled because the duties and responsibilities of those
positions were moved to the Enterprise Operations organization’s Information
Technology Infrastructure Division effective October 2005 as part of the
implementation of the Competency Based Organization. As of December 14, 2005, only two staff
positions remained to be filled.
The published capacity reports for the other IBM systems
and the Unisys systems included high-level disk storage capacity and assignment
charts and future capacity requirements based on the assignments. However, the assessments did not analyze disk
storage usage and based future capacity requirements on the historic usage
trends. Although disk storage usage data
are collected by the Computing Centers, a Capacity Management Branch manager
explained that usage was not assessed because IRS management had given greater
priority to assessing computer processor needs because processor costs are significantly
more expensive than storage costs and storage costs have historically declined.
We also found
inconsistencies in
Figure 3: Inconsistencies
Among Management Reports
|
|
IBM - Master File |
IBM - Administrative |
Unisys |
|
Report Frequency |
Daily |
Monthly |
Weekly |
|
Measurement Unit |
Byte |
Byte |
Track |
|
Capacity and Usage Categorized By |
Project (e.g.,
Production, Test Groups, Non-Assigned) Subgroup |
Group (e.g.,
Production, Development, Available) Application |
Production, Pool (storage groups |
|
Capacity and Usage Measured By |
Assigned (i.e., disk
space allocated to each Subgroup) In Use (i.e., database
and file sizes defined) Available (i.e., not “In
Use”) Percentage In Use
(i.e., “In Use” divided by “Assigned”) |
Disk Space (i.e., assigned
to each application) Space Used (i.e.,
database and file sizes defined) Free Space (i.e., assigned
space not used) |
Allocated (i.e., assigned
to pools) In Use (i.e., tracks
containing data) Available (i.e.,
tracks not “In Use”) |
Source:
In addition, usage reports for the
The incomplete, untimely, and inconsistent disk storage
usage and capacity planning reports resulted in MITS organization management
not having corporate disk usage information to monitor and ensure the resources
were used effectively.
Recommendations
Recommendation
3: The CIO should
ensure the Enterprise Operations organization develops and implements standardized data retention procedures
requiring unused copies of files and database records to be removed from disk
storage, to improve mainframe computer disk storage efficiency.
Management’s Response: IRS management agreed with this recommendation. The Enterprise Operations organization uses standardized data retention procedures for Master File Legacy and Modernization systems and will implement this process on the other mainframe systems.
Recommendation
4: The CIO should
ensure the Business Systems Development organization effectively sets
application data retention parameters based on an application’s processing
cycles, to improve
mainframe computer disk storage efficiency.
Management’s Response: IRS management agreed with this recommendation. The Enterprise Operations organization will implement a process to generate reports across all mainframe systems including project data retention and usage information. These reports will be analyzed quarterly with the Applications Development organization.[8]
Based upon customer data retention and retrieval requirements and application processing timeframes, the Applications Development organization will confirm the correctness of current retention periods or provide new retention periods. If the reports show significant amounts of dedicated storage that remain unused by a project, the Applications Development organization will review the project’s needs and revalidate the future need or release appropriate amounts of storage for use by other projects. The Enterprise Operations organization will then take all necessary actions to release unneeded disk storage.
Recommendation 5:
The CIO should ensure the Enterprise Operations
organization conducts annual capacity planning assessments (and the assessments
measure disk storage usage) that provides MITS organization management
objective information for disk storage investment planning.
Management’s Response: IRS management agreed with this recommendation. The Enterprise Operations organization will conduct annual capacity planning assessments for all mainframe systems. The assessments will be based on standardized disk storage reports that indicate the amount of storage allocated to and actually used by applications. These reports will be used for disk storage investment planning.
Recommendation 6:
The CIO should ensure the Enterprise Operations
organization disk storage reports are standardized and indicate the amount of
storage allocated to and actually used by applications and the reports are
shared with MITS organization management and project owners to ensure complete,
consistent, and corporate-level storage capacity usage reporting and management.
Management’s Response: IRS management agreed with this recommendation. The Enterprise Operations organization will standardize disk storage reports across all corporate assets to indicate the amount of storage allocated to and actually used by applications. The Enterprise Operations organization will share the reports with MITS organization management and project owners to ensure complete, consistent, and corporate-level storage capacity usage reporting and management. The reports will be rolled up annually for a yearly assessment.
Appendix I
Detailed Objective,
Scope, and Methodology
The overall objective of this review was to assess the effectiveness and efficiency of the Internal Revenue Service’s (IRS) mainframe[9] computer storage management practices. To accomplish this objective, we:
I. Determined whether mainframe computer storage management information (such as capacity, usage, and growth projections) was accurate, complete, consistent, and timely.
A. Reviewed Federal Government and IRS guidelines to identify storage management information reporting requirements.
B. Interviewed
C. Determined the effect that inaccurate, incomplete, inconsistent, or untimely storage management information reports had on management decisions.
II. Determined whether mainframe storage management resources were allocated and operated effectively.
A. Reviewed IRS guidelines to identify storage
management procedures and standards.
B.
Interviewed
C.
Reviewed documents for storage capacity purchased
in FYs 2003 – 2005 and requested for FYs 2006 – 2007 to determine the effect of
ineffective storage allocations and operations.
III.
Determined
whether mainframe storage management resources were used efficiently.
A. Reviewed IRS storage management guidelines to
identify file retention, migration, and transfer procedures.
B.
Interviewed
IBM - Master File:
1)
Business
Master File On-Line Processing.
2)
Business
Master File Returns Transaction File Processing.
3)
Customer
Account Data Engine - Individual.
4)
Individual
Master File On-Line.
5)
Individual
Master File Returns Transaction File.
6)
National
Account Profile.
IBM - Other:
7)
Automated
Collection System.
8)
Automated
Financial System.
9)
Currency
and Banking Query System.
10)
Integrated
Collection System.
11)
Security
and Communications System.
Unisys:
12)
Daily
Taxpayer Information File Update.
13)
Electronic
Filing System.
14)
Electronic
Funds Transfer.
15)
Error
Resolution System.
16)
Generalized
Mainline Framework.
17)
Generalized
Unpostable Framework.
18)
Weekly
Taxpayer Information File Update.
For each of the 18 applications, we
identified the stored file names and reviewed data extracts listing each file’s
disk storage retention and migration parameters to determine whether copies of
the files were removed when no longer used.
C.
Interviewed
IV.
Determined
the validity and reliability of data from computer-based systems. We used computer-based data for background
information to determine the amount spent and planned to be spent on
replacement and additional disk storage capacity and to assess data retention
and file transfer procedures. We
reviewed documentation supporting the disk storage purchases. We also assessed the completeness and accuracy
of the data relating to data retention and file transfers and compared file
transfer data obtained from different computer systems to determine the data’s
reliability as they pertain to the objectives of the audit.
Appendix II
Major Contributors
to This Report
Margaret E. Begg, Assistant Inspector
General for Audit (Information Systems Programs)
Gary
Hinkle, Director
Danny
Verneuille, Audit Manager
Van
Warmke, Acting Audit Manager
Frank
Greene, Lead Auditor
Mark
Carder, Senior Auditor
Beverly
Tamanaha, Senior Auditor
Kim
McManis, Auditor
Robert Carpenter, Information Technology
Specialist
Judith Harrald, Information Technology
Specialist
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Deputy Chief Information Officer OS:CIO
Associate Chief Information Officer, Business Systems Development OS:CIO:BSD
Associate Chief Information Officer, Business Systems Modernization OS:CIO:B
Associate
Chief Information Officer,
Director, Stakeholder Management OS:CIO:SM
Director, Enterprise Computing Center OS:CIO:EO:EC
Chief Counsel CC
National Taxpayer Advocate
TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk
Analysis RAS:O
Office of Management Controls OS:CFO:AR:M
Audit Liaisons:
Deputy
Commissioner for Operations Support OS
Manager, Program Oversight Office OS:CIO:SM:PO
Appendix IV
This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration. These benefits will be incorporated into our Semiannual Report to Congress.
Type and Value of Outcome
Measure:
·
Inefficient Use of Resources – Actual; $626,866
(see page 3).
Methodology Used to
Measure the Reported Benefit:
In August 2005, the Internal
Revenue Service (IRS) spent $1,200,000 to purchase 40.2 terabytes[13] of new disk storage capacity for the International
Business Machines (IBM) - Master File system, including software and
maintenance. The storage was purchased
to add additional storage and to replace existing storage. The cost per terabyte of the additional and
replacement storage was calculated as follows:
·
Cost of
new storage – 40.2 terabytes purchased for $1,200,000 (average cost of
$29,850.75 per terabyte).
However, the purchase of the
additional storage may not have been necessary because, as of September 12,
2005, only 69.3 percent of the 53.5 terabytes of Current Processing Environment-assigned
disk storage capacity was used and 14.2 terabytes were being held in reserve.
Type and Value of Outcome
Measure:
·
Funds to Be Put to Better Use – Potential; $1,423,140 (see page 3).
Methodology Used to
Measure the Reported Benefit:
Starting in Fiscal
Year 2006 and continuing 4 years, the IRS plans to spend at least $1,423,140 to
lease 38.4 terabytes or more of additional disk storage capacity. Figure 1 shows the methodology used to
determine the cost of the additional storage.
Figure 1:
Cost of Planned Additional Storage Capacity
|
Funded Amount for
Disk Storage |
$2,590,560 |
|
Planned Leased Storage
Capacity |
69.9 terabytes |
|
Cost per Terabyte
($2,590,560 divided by 69.9 terabytes) |
$37,060.94 |
|
Planned
Replacement Capacity |
31.5 terabytes |
|
Planned Additional
Capacity |
38.4 terabytes |
|
Cost of Additional Capacity ($37,060.94
multiplied by 38.4 terabytes) |
$1,423,140 |
Source:
The maintenance cost savings of $288,000 from
the replaced storage lowered the cost of the replacement storage. As of the end of our audit work, the capacity
and cost details had not been completed.
However, the lease may not be necessary
because, as of September 12, 2005, only 47.7 percent of the 51.8 terabytes of Modernization-assigned
disk storage capacity was used and 9 terabytes were being held in reserve.
Appendix
V
International Business Machines - Master File Disk Storage Capacity
Assignment and Usage Metrics
|
|
January 1, 2005 |
September 12, 2005 |
||||
|
Projects |
Capacity Assigned (in terabytes[14]) |
In Use[15] (in terabytes) |
In Use As a Percentage of Capacity Assigned |
Capacity Assigned (in terabytes) |
In Use (in terabytes) |
In Use As a Percentage of Capacity Assigned[16] |
|
Internal
Revenue Service (IRS) Modernization Projects |
||||||
|
Production |
37.0 |
17.3 |
46.8% |
34.5 |
16.2 |
47.1% |
|
Other
(Temporary, System, Test, etc.) |
12.2 |
6.8 |
55.7% |
17.3 |
8.5 |
49.2% |
|
Totals |
49.2 |
24.1 |
49.0% |
51.8 |
24.7 |
47.7% |
|
Reserve |
7.0 |
|
9.0 |
|
||
|
Totals |
56.2 |
60.8 |
||||
|
|
Assigned As a Percentage of Capacity |
In Use As a Percentage of Capacity Assigned |
Assigned As a Percentage of Capacity |
In Use As a Percentage of Capacity Assigned |
||
|
Modernization
Projects |
87.5% |
49.0% |
85.2% |
47.7% |
||
|
Industry
- Average |
64.5% |
81.8% |
64.5% |
81.8% |
||
|
Industry
- Best Practice |
86.1% |
93.7% |
86.1% |
93.7% |
||
Source:
International Business Machines - Master File
Disk Storage Capacity Assignment and Usage Metrics – (Continued)
|
|
January 1, 2005 |
September
12, 2005 |
||||
|
Projects |
Capacity Assigned (in
terabytes) |
In
Use[17] (in
terabytes) |
In
Use As a Percentage of Capacity Assigned |
Capacity Assigned (in
terabytes) |
In
Use (in
terabytes) |
In
Use As a Percentage of Capacity Assigned[18] |
|
IRS
Current Processing Environment |
||||||
|
Production |
30.2 |
24.7 |
81.8% |
31.9 |
22.5 |
70.6% |
|
Other (Temporary, System,
Test, etc.) |
26.1 |
14.4 |
55.2% |
21.6 |
14.6 |
67.6% |
|
Totals |
56.3 |
39.1 |
69.4% |
53.5 |
37.1 |
69.3% |
|
Reserve |
2.2 |
|
14.2 |
|
||
|
Totals |
58.5 |
67.7 |
||||
|
|
Assigned As a Percentage of Capacity |
In Use As a Percentage of Capacity
Assigned |
Assigned As a Percentage of Capacity |
In Use As a Percentage of Capacity
Assigned |
||
|
Current Processing
Environment |
96.2% |
69.4% |
79.0% |
69.3% |
||
|
Industry - Average |
64.5% |
81.8% |
64.5% |
81.8% |
||
|
Industry - Best Practice |
86.1% |
93.7% |
86.1% |
93.7% |
||
|
Total Capacity |
114.7 |
|
|
128.5 |
|
|
Source:
Appendix VI
Unisys Disk Storage Capacity Assignment and Usage Metrics
|
|
January 3,
2005 |
September
12, 2005 |
||||
|
|
Capacity Assigned |
In Use[19] (in
terabytes) |
In Use As
a Percentage of Capacity Assigned |
Capacity Assigned |
In Use |
In Use As
a Percentage of Capacity Assigned |
|
Martinsburg |
||||||
|
Production |
1.8 |
0.9 |
50.0% |
1.7 |
1.0 |
58.8% |
|
Reserve |
0.4 |
|
0.5 |
|
||
|
Total Capacity |
2.2 |
2.2 |
||||
|
|
Assigned As a Percentage of Capacity |
In Use As
a Percentage of Capacity Assigned |
Assigned As
a Percentage of Capacity |
In Use As
a Percentage of Capacity Assigned |
||
|
Martinsburg |
81.8% |
50.0% |
77.3% |
58.8% |
||
|
|
||||||
|
Production |
1.7 |
0.7 |
41.2% |
1.7 |
0.7 |
41.2% |
|
Reserve |
0.0 |
|
0.0 |
|
||
|
Total Capacity |
1.7 |
1.7 |
||||
|
|
Assigned As a Percentage of Capacity |
In Use As
a Percentage of Capacity Assigned |
Assigned As
a Percentage of Capacity |
In Use As
a Percentage of Capacity Assigned |
||
|
Memphis |
100% |
41.2% |
100% |
41.2% |
||
|
Industry - Average |
64.5% |
81.8% |
64.5% |
81.8% |
||
|
Industry - Best Practice |
86.1% |
93.7% |
86.1% |
93.7% |
||
|
Martinsburg and |
||||||
|
Development and Test |
2.3 |
1.2 |
52.2% |
2.2 |
1.2 |
54.5% |
|
In Use As a Percentage of Capacity
Assigned |
|
|
|
|
||
|
Industry - Average |
|
81.8% |
|
|
81.8% |
|
|
Industry - Best Practice |
|
93.7% |
|
|
93.7% |
|
|
|
|
|
|
|
|
|
|
Total
Capacity |
6.2 |
|
|
6.1 |
|
|
Source:
Capacity Management Branch Mass Storage Reports and
Appendix VII
|
Administrative |
The Administrative systems include human resources, financial
reporting, currency transaction reporting, and other applications. |
|
As-Built Architecture |
The As-Built Architecture presents an enterprise view of the Internal
Revenue Service’s (IRS) Information Technology and Business environments and
is an integral part of the IRS’ Enterprise Architecture. |
|
Byte |
A byte is one data character. |
|
Campus |
The IRS campuses are the data processing
arm of the IRS. |
|
Computing Center |
The Computing Centers support tax processing and information management
through a data processing and telecommunications infrastructure. |
|
|
The Enterprise Architecture defines the target business practices,
the systems that enable these practices, and the technology that will support
it. |
|
|
The ECC is responsible for
maintaining and operating the enterprise information technology systems which
support the nation’s tax administration.
The ECC is organized to operate under the “corporate utility” concept (i.e.,
one organization with a uniform operating environment which employs standard
policies, practices, and procedures to manage three physical computing
centers). |
|
Gigabyte |
One gigabyte equals approximately 1 billion bytes or 500,000
typewritten pages. |
|
Integrated Collection
System, Automated Collection System, Printer Replacement to
Integrate New Tools System |
The Integrated Collection
System, Automated Collection System, and Printer Replacement to
Integrate New Tools System support
the balance-due and nonfiler tax enforcement operations. |
Glossary of Terms -
Continued
|
Integrated Data Retrieval
System (IDRS) |
The IDRS supports the
retrieval and updating of stored information, including taxpayer account
records. |
|
International Business Machines (IBM) |
The IRS uses IBM mainframe computers to
host Master File processing, collection processing, and other nationwide
applications and data processing. |
|
Mainframe |
A mainframe is a powerful multiuser computer capable of supporting
many hundreds or thousands of users simultaneously. |
|
Master File |
The Master File system supports the databases that store individual, business,
and employee plans and exempt organizations taxpayer account information and
includes Current Processing Environment applications and Modernization
Projects. |
|
Security and Communications
System |
The Security and Communications System provides security,
communications, and terminal management for the IDRS. |
|
|
The Service Center Replacement System supports receiving, capturing, and perfecting tax payment and |
|
Terabyte |
One terabyte equals approximately 1 trillion bytes or 500 million
typewritten pages. |
|
Track |
A track is the part of a disk that passes under one read/write head
while the head is stationary. A Unisys
disk storage track contains 7,168 bytes of data. |
|
Unisys |
The IRS uses Unisys mainframe computers to host applications and
specific data for the campuses. |
Appendix VIII
Management’s
Response to the Draft Report
The response was removed due to its size. To see the response, please go to the Adobe
PDF version of the report on the TIGTA Public Web Page.
[1] See Appendix VII for a Glossary of Terms.
[2] The Business Systems Development and the Business Systems Modernization organizations are being merged to create a new Applications Development organization.
[3] See Appendix VII for a Glossary of Terms.
[4] Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).
[5] Mainframe Computer Operations Efficiency and Effectiveness Should Be Improved (Reference Number 2003-20-117, dated May 2003) and Mid-range Computer Storage Resources Need Better Administration to Ensure Effective and Efficient Utilization and Accurate Reporting (Reference Number 2005-20-098, dated July 2005).
[6] Information systems and technology using older generation design approaches and technology.
[7] The review included file transfer log data for June 26,
2005, through July 9, 2005, except for File Transfer Protocol data received for
the Detroit IBM systems reviewed for the period July 31, 2005, through August
13, 2005 (see Appendix I).
[8] The Business Systems Development and the Business Systems Modernization organizations are being merged to create a new Applications Development organization.
[9] See Appendix VII for a Glossary of Terms.
[10] File Transfer Protocol is a specification designed to facilitate electronic data transfer between computers.
[11] CONNECT:Direct® is secure file transfer software published by Sterling Commerce.
[12] The exception to this period was for File
Transfer Protocol transfers at one
[13] See Appendix VII for a Glossary of Terms.
[14] See Appendix VII for a Glossary of Terms.
[15] For purposes of the disk storage reports, “in use” means the assigned maximum database and file sizes.
[16] The percentages reflected in the table may not exactly equal the result from dividing the two numbers because the numbers in the table are rounded.
[17] For purposes of the disk storage reports, “in use” means the assigned maximum database and file sizes.
[18] The percentages reflected in the table may not exactly equal the result from dividing the two numbers because the numbers in the table are rounded.
[19] For purposes of the disk storage reports, “in use” means the assigned maximum database and file sizes.