TREASURY
INSPECTOR GENERAL FOR TAX ADMINISTRATION
Management Practices Over End-user Computer Server Storage Need Improvement to Ensure Effective and Efficient Storage Utilization
July 3, 2007
Reference Number: 2007-20-103
This
report has cleared the Treasury Inspector General for Tax Administration
disclosure review process and information determined to be restricted from
public release has been redacted from this document.
Phone Number |
202-927-7037
Email Address | Bonnie.Heald@tigta.treas.gov
Web Site |
http://www.tigta.gov
July 3, 2007
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: (for) Michael R. Phillips /s/ Michael E. McKenney
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Management Practices Over End-user Computer Server Storage Need Improvement to Ensure Effective and Efficient Storage Utilization (Audit # 200620023)
This report presents the results of our review of the Internal
Revenue Service’s (IRS) end-user[1]
computer server storage management. The
overall objective of this review was to assess the effectiveness and efficiency
of the IRS management practices over the end-user computer server storage
environment for employees using the IRS Microsoft Windows computer networks.
Impact on the Taxpayer
The IRS needs to improve management practices over end-user
computer server storage to ensure effective and efficient utilization of
storage and budget resources. In Fiscal
Years 2003 through 2006, the Modernization and Information Technology Services (MITS)
organization spent $19.9 million to purchase server storage capacity and
support. However, only 27 percent of the
available storage was being used on the 2,253 servers we reviewed. In addition, the IRS had not established
policies and procedures for monitoring end-user computer server storage usage,
including both end-user infrastructure and application server storage. As a result, it may not have most efficiently
used its resources.
Synopsis
The IRS requires a large and complex computer environment,
which includes computer servers managed by several functions within the MITS
organization to process and store taxpayer, financial, and administrative data. On May 14, 2006, the MITS organization began
centralizing under the Enterprise Operations organization the management of all
servers in both the Enterprise Operations and the End User Equipment and
Services organizations. Other MITS organizations
continue to manage their own servers. In
Fiscal Years 2003 through 2006, the MITS organization spent $19.9 million to
purchase server storage capacity and support.
It planned to spend an additional $9.3 million in Fiscal Years 2006 and
2007; however, the $9.3 million was redirected and put to better use on higher
priority information technology infrastructure needs (see Appendix IV).
Although the MITS organization began centralizing server management and reallocated budgeted storage funds, policies and procedures did not ensure effective and efficient storage utilization. In Fiscal Years 2003 through 2006, the MITS organization spent $19.9 million to purchase server storage and storage support. However, only 27 percent of storage capacity was being used.
We identified storage management practices that could be
improved to provide more effective and efficient
server storage utilization and better informed storage purchasing decisions.
IRS management advised us that automated inventory tools should be used to create reports that link the Information Technology Asset Management System data to the data gathered from those tools. In addition, the contractor assisting the IRS in the Sustaining Infrastructure Initiative is recommending the IRS implement an Asset Discovery Tool to provide a more complete set of data by recognizing all assets through network addresses. The contractor is also recommending a modification of the current Information Technology Asset Management System infrastructure to support more robust data fields. Therefore, it appears various IRS organizations are pursuing different computer inventory solutions that may be inconsistent and duplicative.
In addition, the Server Consolidation and Virtualization project was not following required project management procedures because a business case and cost-benefit analysis had not been prepared. Following required project management procedures will help minimize the risks of cost overruns, schedule delays, and inadequate executive governance and oversight.
Recommendations
We recommended the Chief Information Officer (1) centralize management of all MITS organization storage servers, where appropriate, to improve the effectiveness and efficiency of storage management; (2) periodically analyze storage utilization to ensure storage space is efficiently used prior to purchasing any additional storage capacity; (3) ensure the MITS organization establishes formal policies and procedures over the assignment and monitoring of end-user computer server storage space; (4) resolve server mismatches between records on the Tivoli® inventory system and the Information Technology Asset Management System and reopen the prior corrective action regarding resolution of such mismatches until it is completed; (5) define the future inventory system design requirements to meet all user needs and implement one reliable computer inventory system; and (6) ensure required project management procedures, such as preparing a formal business case, feasibility study, and cost-benefit analysis, are followed by the Server Consolidation and Virtualization project.
Response
IRS management agreed with all recommendations except the first part of Recommendation 3 and plans to take an alternative corrective action to the second part of Recommendation 4. The corrective actions to be implemented include centralizing management of storage servers where appropriate; analyzing storage utilization quarterly; and defining policies and procedures to enforce end-user storage limits, govern file retention for exiting employees, and standardize file management processes. Also, operational databases will be synchronized with the Information Technology Asset Management System as much as possible, and Information Technology Asset Management System functionality will be improved and automated efficiencies introduced. The Server Consolidation and Virtualization project will follow management procedures.
For Recommendation 3, management did not agree to assign storage space to users based on industry best practices because the IRS standard, which assigns shared storage space based on employees’ business needs, increases employee efficiency. For Recommendation 4, management plans to open a new corrective action based on the closing action to the prior corrective action. Management’s complete response to the draft report is included as Appendix VI.
Office of Audit Comment
We continue to believe management could reduce storage costs by using the industry best practice of 200 megabytes for most employees and allowing exceptions to this policy for employees who have a business need for additional storage based on their assigned duties. We concur with not reopening the prior corrective action because the proposed actions should reduce the mismatches between records on the Tivoli® inventory system and the Information Technology Asset Management System.
Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
Storage on Servers Was
Not Effectively Managed to Ensure Efficient Use of Resources
Management Did Not Have an Accurate Inventory
of All Available Server Storage Space
A Storage
Management Improvement Initiative Was Not Following Required Project Management
Procedures
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – Outcome Measures
Appendix
V – Glossary of Terms
Appendix
VI – Management’s Response to the Draft Report
Abbreviations
|
EUES |
End User Equipment and Services |
|
IRS |
Internal Revenue Service |
|
ITAMS |
Information Technology Asset Management System |
|
MITS |
Modernization and Information Technology
Services |
The Internal Revenue Service (IRS) requires a large and complex computer environment to process and store taxpayer, financial, and administrative data. A significant portion of the computer processing workload is performed by the IRS’ mid-range[5] and mainframe computer systems. The workload is also processed by the end-user computing environment, which consists of desktop and laptop computers and network servers. The end-user computer server storage environment supports tax return processing, compliance applications, internal management applications, and enterprise storage and retrieval. We have previously reported on the storage management practices of the mid-range[6] and mainframe computers.[7] The focus of this review was on the end-user computer server storage environment for employees using the IRS Microsoft Windows computer networks, including both end-user infrastructure and application servers.
In Fiscal Years 2003 through 2006, the Modernization and Information Technology Services (MITS) organization spent $19.9 million to purchase end-user server storage and storage support for the Enterprise Operations, End User Equipment and Services (EUES), and Enterprise Services organizations. Figure 1 presents the yearly expenditures.
Figure 1:
End-user Server Storage Expenditures
|
Fiscal Year |
Amount
Expended |
|
2003 |
$1,241,061 |
|
2004 |
$9,909,778 |
|
2005 |
$8,727,492 |
|
2006 |
$58,485 |
|
Total |
$19,936,816 |
Source: The
This review was performed in MITS organization offices
at the IRS National Headquarters in New Carrollton, Maryland, and the
Management of the End User Equipment and Services Organization’s Servers Was Centralized Under the Enterprise Operations Organization
The Clinger-Cohen Act of 1996[8] states the Chief Information Officer is responsible for promoting the effective and efficient design and operation of all major information resources management processes for the executive agency. The Internal Revenue Manual states the Chief Information Officer and the Director, Enterprise Operations, within the MITS organization are responsible for ensuring the effective and efficient use of the IRS automated information processing environment.
Prior to May 14, 2006, MITS organization end-user servers
were managed by several MITS organizations (e.g., Enterprise Operations, EUES, Applications
Development [formerly Business Systems Development], and Enterprise Services). The IRS determined that managing servers
across multiple organizations was ineffective and, in May 2006, centralized
server management of all Enterprise Operations and EUES organization servers
under the Enterprise Operations organization.
However, some MITS organizations, such as the Applications Development (54 servers) and
Recommendation
Recommendation 1: The Chief Information Officer should centralize management of all MITS organization storage servers into the Enterprise Operations organization, where appropriate, to improve the effectiveness and efficiency of storage management.
Management’s Response: IRS management agreed with the recommendation. The MITS organization will centralize management of storage servers into the Enterprise Operations organization, where appropriate, to improve the effectiveness and efficiency of storage management.
Storage on Servers Was Not Effectively Managed to Ensure Efficient Use of Resources
The Internal Revenue Manual requires electronic copies of
records, including copies maintained by individuals in personal files or
directories, that have no further administrative value to be deleted within 180
calendar days after the recordkeeping copy has been produced. The Department of the Treasury Information System Life Cycle Manual (TD
P 84-01, dated March 2002) states one of the benefits of data management is
that it avoids the cost of redundant data collection and storage. File (data) management includes controlling
the creation, deletion, access, and use of data files and programs.
We received storage utilization information on 2,292 (48
percent) of 4,783 servers shown in the Information
Technology Asset Management System (ITAMS) as of September 29, 2006. Storage usage information for the remaining
servers was not readily available and obtained due to the impact the data
gathering might have had on storage management personnel. Our analysis of the information for 2,253
servers (2,292 servers less 39 newly installed servers) determined only 27
percent of the 484.1 terabytes of storage space on the servers was used. Based on the unused storage rate of 73
percent, $14.6 million of the $19.9 million[9] spent on the purchase of storage capacity and support during
Fiscal Years 2003 through 2006 represents an
inefficient use of resources (see Appendix IV).
In the
two prior audits of the IRS’ storage management practices, we reported the IRS
needed to improve the efficiency of its storage utilization and should consider
storage utilization when assessing the need to purchase additional
storage. In Fiscal Years 2006 and 2007,
the IRS budgeted $9.3 million to purchase end-user server storage and storage support for the
Enterprise Operations, EUES, and Enterprise Services organizations.
However, the MITS organization decided
not to spend the budgeted funds. These
funds were redirected and put to better use on higher priority information
technology infrastructure needs (see Appendix IV).
The storage utilization
rate is comparable to industry results reported by Gartner, Inc.,[10] which showed
that only 28 percent to 45 percent of installed storage was actually holding
data. The IRS storage utilization rate
was low, in part, because storage utilization rates were not a determining
factor in deciding whether to buy more storage space.
In addition, our review of the MITS organization’s end-user server storage management controls determined that existing policies and procedures do not:
The IRS established a policy that end-users can be assigned 500 megabytes of server storage space (called home directory space). However, industry best practices recommend 200 megabytes of storage for each user. As a result, the IRS is incurring additional costs to maintain unneeded storage space. In addition, while the IRS assigns 500 megabytes of storage space to each end-user, no formal policy is in place to ensure storage limits are enforced.
Use of storage resources is not considered for managing files and identifying needs.
A total of 2,098 former employees had files using approximately 0.138 terabytes (equivalent to 70,000,000 pages of text) of storage on the end-user servers managed by the MITS organization. Decentralized management of end-user server storage contributed to the lack of procedures for removing the files of former employees.
a) Certain files and file types are removed from servers.
b) Files that have not been accessed or modified for a specified period of time (e.g., more than 1 year) are removed and archived.
c) A review of the server storage space is completed at regular intervals (e.g., every 6 months) to determine whether only those files with a business purpose and immediate need are being stored.
IRS management indicated use of one set of standards for file management and the enforcement of users’ storage limits were currently not possible because different software platforms (e.g., Microsoft Windows 2003 Server Operating System, Microsoft Windows NT) exist. However, IRS management also stated uniform standards and policies for managing files and the enforcement of storage limits could be established after an ongoing server operating system upgrade is completed.
Management indicated system administrators are also hampered in monitoring files because they do not have the authority to look for or remove nonwork-related files or adequate software to identify old, unused, and nonwork-related files. The IRS developed draft policies and procedures for file management of storage shared by multiple users and issued guidelines to educate employees on “good housekeeping” practices for file management of personal storage. Management commented that these procedures and guidelines are targeted for inclusion in the Internal Revenue Manual; however, they did not provide us with a scheduled completion date.
Without efficient and effective storage management controls
to review storage utilization, implement standardized policies and procedures
for managing files, and enforce users’ storage limits, the IRS runs the risk of
not effectively using current storage capacity and unnecessarily purchasing
additional storage capacity.
Recommendations
Recommendation 2: The Chief Information Officer should periodically analyze storage utilization to ensure storage space is efficiently used prior to purchasing additional storage capacity.
Management’s Response:
IRS management agreed with the recommendation. The MITS organization will develop a process
to analyze storage utilization quarterly to ensure storage space is efficiently
used prior to purchasing additional capacity.
Recommendation 3: The Chief Information Officer should ensure the MITS organization establishes formal policies and procedures to:
· Assign storage space to users based on industry best practices.
· Use the new operating system capability to enforce end-user storage limits.
· Define a reasonable time period for determining the business need of files being stored for former employees. Once the business need of the files is determined, archive, transfer, or remove the files.
· Standardize file management processes.
Management’s Response: The Chief Information Officer disagreed with the recommendation that storage space be assigned to users based on industry best practices, stating that, while the industry best practice recommendation is for 200 megabytes, the IRS has found it necessary to assign 500 megabytes per user. The IRS standard assigns shared storage space based on employees’ business needs. This standard was determined by reviewing industry best practices, assessing available (current and projected) storage space, monitoring current user usage, and working to meet business requirements. The Chief Information Officer also stated this standard increases employee efficiency.
Management agreed with the three remaining parts of this recommendation and indicated the Enterprise Operations organization will:
· Implement defined policies and procedures regarding shared storage policies.
· Define a process for business organizations to identify additional requirements of file retention for exiting staff. System administrators will implement appropriate archival and removal of those files.
· Review, update, and distribute the current standardized file management process document.
Office of Audit Comment:
A 2004 study conducted by the IRS to determine the amount of storage
space to assign to users indicated that only 4 percent of the employees were
using more than 500 megabytes of storage.
The study did not indicate the percentage of employees using more than
the industry best practice of 200 megabytes.
In addition, the study identified
music, picture, and other files dating back to 1993 that were using storage
space. We recognize that some employees
may need more than 200 megabytes of storage space to accomplish their assigned
duties; however, establishing a standard 2.5 times the industry best practice
will not lead to the most efficient use of storage resources. Therefore,
we continue to believe management could reduce storage costs by using the industry
best practice of 200 megabytes for most employees and allowing exceptions to
this policy for employees who have a business need for additional storage based
on their assigned duties.
Management Did Not Have an
Accurate Inventory of All Available Server Storage Space
The Government Accountability Office Standards for Internal Control in the Federal Government state information should be recorded and communicated to management and others within the entity who need it and in a form and within a time period that enables them to carry out their responsibilities.
Management provided us with server inventory numbers from three sources.
Figure 2 shows the server inventory numbers varied depending on the source of the information.
Figure
2: Server Inventories
|
Server Inventory System |
Number of Servers |
|
ITAMS |
4,783 |
|
Tivoli®
Inventory System |
2,256 |
|
|
3,478 |
Source: MITS organization storage
management personnel.
Management
cannot timely and effectively identify the server storage environment. Therefore, the IRS may overestimate storage
needs and unnecessarily purchase storage hardware, software, maintenance, and
other support.
In a prior audit report,[11] we recommended the Chief Information Officer ensure the EUES organization resolves mismatches between records on the Tivoli® inventory system and the ITAMS. The IRS’ planned corrective action was to include developing appropriate procedures by March 1, 2006, to resolve the mismatches. The IRS closed the corrective action on March 3, 2006. However, during this audit, only 2,256 (47 percent) of the 4,783 MITS organization-managed servers in the ITAMS inventory were identified in the Tivoli® inventory system records. The Tivoli® inventory system was not complete and accurate because not all servers had Tivoli® software installed on them or servers with Tivoli® software installed were not properly configured to be recognized as servers. Storage management personnel advised us that all devices are not always labeled properly and consistently in the ITAMS.
In addition, the
Enterprise Systems Management organization was manually maintaining the Master
Server Database to record information the ITAMS does not record. As of October 4, 2006, 158 people could access
and modify the Master Server Database. Based
on the information in Figure 2, the database does not contain information on
all computer equipment in the ITAMS.
Manually maintaining a database outside of the official computer
inventory database (i.e., the ITAMS) may be an inefficient use of resources.
IRS
management advised us the ITAMS is the official equipment inventory and that
other automated inventory tools should be used to create reports that link the
ITAMS data to the data gathered from those tools. In addition, the contractor assisting the IRS
in the Sustaining Infrastructure Initiative is recommending the IRS implement
an Asset Discovery Tool to provide a more complete set of data by recognizing
all assets through network
addresses. The contractor is also
recommending a modification of the current ITAMS infrastructure to support more
robust data fields. Therefore, it
appears various IRS organizations are pursuing different computer inventory
solutions that may be inconsistent and duplicative.
Without an accurate inventory
of all available storage and the effective coordination of computer inventory
initiatives, the IRS may not know how much storage it has; be unable to make
reliable investment and resource decisions; and purchase unnecessary storage hardware, software,
maintenance, and other support.
Recommendations
Recommendation 4: The Chief Information Officer should resolve the end-user server mismatches between records on the Tivoli® inventory system and the ITAMS and reopen the prior corrective action regarding resolution of such mismatches until it is actually completed.
Management’s Response: IRS management agreed with the recommendation. The ITAMS inventory database should be the authoritative source for updates to operational databases such as the Tivoli® and Server databases. Since the audit, the IRS has taken several steps to ensure these operational databases are synchronous with the ITAMS as much as possible. The IRS is also developing perpetual inventory controls that will involve new software and processes. The Chief Information Officer plans to open a new corrective action because the closing action to the prior corrective action stated the EUES organization would develop a process to identify systems that were not being mapped by Tivoli® software and indicated that additional enhancements were needed for the data exchange between the Tivoli® inventory system and the ITAMS.
Office of Audit Comment: We concur with not reopening the prior corrective action. The corrective actions proposed for Recommendation 4 should reduce the mismatches between records on the Tivoli® inventory system and the ITAMS.
Recommendation 5: The Chief Information Officer should assess the current computer inventory
systems and initiatives and define the future inventory system design
requirements to meet all user needs. Once
the requirements are defined, one reliable computer inventory system should be
implemented.
Management’s Response: IRS management agreed with the
recommendation. The current ITAMS is the
one standard inventory system for IRS computer equipment; however, the IRS is taking
steps to improve its functionality and introduce automated efficiencies. Two
initiatives underway will result in significant improvements to the ITAMS: (1) implementation of the “One Inventory”
concept and process enhancements toward perpetual inventory controls and (2) initiation
of a Business Process Reengineering group to develop best practice asset management
processes. There are also other general
actions in progress to facilitate this goal.
A Storage Management Improvement Initiative Was Not Following Required Project Management Procedures
The
Department of the Treasury Information System Life Cycle Manual states
general standardization of life cycle management ensures systems are developed,
acquired, evaluated, and operated efficiently, within prescribed budget and
schedule constraints, and are responsive to mission requirements. In addition, the
IRS system development guidelines (currently, the Enterprise Life Cycle – Lite) stipulate that, as part of the information system
life cycle management process, project management should identify project risks
early and manage them before they become problems.
The Clinger-Cohen Act of 1996 requires agencies to use a disciplined Capital Planning and Investment Control process to acquire, use, maintain, and dispose of information technology. In addition, Office of Management and Budget Circular A-11, Preparation, Execution, and Submission of the Budget, requires each agency to include an Agency Information Technology Investment Portfolio (Exhibit 53) with its annual budget submission to the Office of Management and Budget. The IRS classifies a project as a major information technology investment if the annual investment exceeds $5 million or if the total life cycle cost exceeds $50 million. Major investments require increased executive oversight and preparation of a detailed Capital Asset Plan and Business Case (Exhibit 300).
In April 2004, IRS management
prepared a Tier
III Master Server Consolidation Plan outlining a server consolidation
strategy. Management indicated that
additional servers were to be purchased following the principles of this Plan. However, the Plan did not assess
the existing end-user server storage environment or the proposed network
storage design and did not provide cost estimates for the new storage design, which
was to include storage virtualization. In addition, a formal business case, feasibility
study, and cost-benefit analysis were not prepared.
Therefore, the server consolidation
initiative was not following required project management procedures, although it
met the Capital Planning and Investment Control cost thresholds for
a major information technology investment. Specifically, the MITS organization spent $9.9
million in Fiscal Year 2004 and $8.7 million in Fiscal Year 2005 on end-user server
storage.
The IRS considered the server consolidation strategy to be a plan of action to achieve its goals of reducing the number of servers in inventory and consolidating the location of servers to the campus sites. In July 2006, management advised us that a formal Server Consolidation and Virtualization project had been approved. The Enterprise Operations organization is sponsoring the project and a Project Manager has been assigned.
When information technology initiatives do not follow required project management procedures, the IRS increases the risks of cost overruns and schedule delays. In addition, projects may not have adequate executive governance and oversight.
Recommendation
Recommendation 6: The Chief
Information Officer should ensure required
project management procedures, such as preparing a formal business case,
feasibility study, and cost-benefit analysis, are followed by the Server Consolidation and Virtualization project.
Management’s Response: IRS management agreed with the recommendation. The MITS organization has assigned a project manager and will ensure completion of management procedures, including a business case, feasibility study, and cost-benefit analysis.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to assess the effectiveness and efficiency of the IRS management practices over the end-user[12] computer server storage environment for employees using the IRS Microsoft Windows computer networks. Specifically, we:
I.
Determined
whether the IRS developed and implemented policies and procedures for managing end-user server storage.
A. Reviewed Treasury Directives, Office
of Management and Budget Circulars, the
Internal Revenue Manual, and
other guidelines governing storage management.
B.
Obtained and evaluated the policies and
procedures for establishing and managing the size of home directories. We also reviewed the IRS process and best
practices for establishing home directory size and compared best practices to
the IRS policy and procedures. We
interviewed IRS personnel to obtain procedures for managing storage size.
C.
Obtained and evaluated the procedures and
guidelines to remove and/or archive duplicate, old, unused, and nonwork-related
files.
D. Obtained and evaluated the IRS data retention
requirements for home directories of separated and inactive employees.
E.
Identified the amounts the MITS organization
expended to add or replace capacity for, maintain, and support end-user server storage space during Fiscal Years
2003 through 2006 and the amount included in the budget request for Fiscal Year
2007.
F.
Determined whether a design flowchart for the
end-user network server storage
environment was prepared.
II. Determined the status
of the end-user server storage consolidation initiatives and whether
selected project management controls were implemented.
A. Identified the end-user server
migrations made between Fiscal Years 2003 and 2006 and determined their status.
B.
Determined whether a business case,
consolidation plan, or other related plans were developed for the end-user server storage consolidation effort and whether
a feasibility study was prepared.
C.
Evaluated the adequacy of all consolidation
plans, justifications, and need assessments, including whether a cost-benefit
or impact analysis was performed.
III. Determined whether end-user server
storage utilization was effectively and efficiently managed.
A. Interviewed
storage management personnel and
B. Analyzed the data obtained in Step III.A. to
determine whether the IRS was effectively managing end-user server storage and whether excess storage was allocated to servers.
Validity and reliability
of data from computer-based systems
We used computer-based data to determine the amounts spent and planned to be spent on replacement and additional disk storage capacity. The IRS provided Integrated Financial System information regarding the end-user server storage expenditures in Fiscal Years 2004 through 2006. We relied on the Government Accountability Office’s assessment of the reliability of the computer-processed data from the Integrated Financial System. During a review of the IRS’ financial statements,[13] the Government Accountability Office concluded the expense and reimbursable revenue information processed through the System for Fiscal Years 2005 and 2006 was reliable in all material respects. We also used computer-based data to analyze end-user server storage utilization. The storage data were validated by working with IRS storage personnel to ensure completeness and accuracy. IRS personnel provided storage utilization-related information identified through a Tivoli® scan of the servers; identified end-user servers installed from July 9, 2006, through August 9, 2006; and identified retired (out-of-service) servers.
The data used to determine the number of separated/inactive employees with home directories residing on end-user servers were validated by comparing a list of authorized storage server users to the IRS personnel system. We attempted to validate the end-user server inventory by comparing the information received from three different inventory systems (the ITAMS, the Tivoli® inventory system, and the Enterprise Systems Management Organization Master Server Database). For the identified differences in the server inventory counts, we conducted discussions with storage management personnel that had expertise with the inventory systems. Our attempt to validate the server inventory was unsuccessful, and the results are presented in detail in the Results of Review section of the report.
Appendix II
Major Contributors to This Report
Margaret
E. Begg, Assistant Inspector General for Audit (Information Systems Programs)
Gary
Hinkle, Director
Danny
Verneuille, Audit Manager
Mark
Carder, Lead Auditor
Louis
Zullo, Senior Auditor
Linda Screws, Auditor
Appendix III
Acting
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy
Commissioner for Operations Support OS
Deputy Chief Information Officer OS:CIO
Associate Chief Information Officer, End User Equipment and Services OS:CIO:EU
Associate
Chief Information Officer,
Director Stakeholder Management OS:CIO:SM
Chief
Counsel CC
National
Taxpayer Advocate TA
Director,
Office of Legislative Affairs CL:LA
Director,
Office of Program Evaluation and Risk Analysis
RAS:O
Office of
Internal Control OS:CFO:CPIC:IC
Audit Liaisons:
Deputy Commissioner for Operations Support OS
Associate Chief Information Officer, End User Equipment and Services OS:CIO:EU
Associate Chief Information Officer,
Director, Program Oversight
Office OS:CIO:SM:
Appendix IV
This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration. These benefits will be incorporated into our Semiannual Report to Congress.
Type and Value of Outcome Measure:
· Inefficient Use of Resources – Potential; $14,553,876 spent on the purchase of storage capacity and support (see page 4).
Methodology Used to Measure the Reported Benefit:
In Fiscal Years 2003 through 2006, the MITS organization spent $19.9 million to purchase end-user[14] server storage (including both end-user infrastructure and application server storage and storage support) for the Enterprise Operations, EUES, and Enterprise Services organizations. We received storage utilization information on 2,292 (48 percent) of 4,783 servers shown in the ITAMS as of September 29, 2006. Storage usage information for the remaining servers was not readily available and obtained due to the impact the data gathering might have had on storage management personnel. Our analysis of the information for 2,253 servers (2,292 servers less 39 newly installed servers) determined only 27 percent of the 484.1 terabytes of storage space was used. Based on the unused storage rate of 73 percent, $14.6 million ($19,936,816 storage costs x 73 percent) spent on the purchase of storage capacity and support during Fiscal Years 2003 through 2006 represents an inefficient use of resources.
IRS storage management personnel were unable to provide information regarding the specific amount of end-user server storage capacity purchased in Fiscal Years 2003 through 2006. As a result, we could not determine whether the entire $19.9 million spent to purchase end-user server storage was unnecessary. Therefore, we estimated the amount of unnecessary storage (i.e., inefficient use of resources) by multiplying the $19.9 million spent on storage by the 73 percent unused storage rate. Figure 1 shows the cost of unused end-user server storage capacity.
Figure 1: Cost of Unused End-user Server Storage Capacity
|
Category |
Fiscal Years 2003-2006 Totals |
Fiscal Year 2003 |
Fiscal Year |
Fiscal Year |
Fiscal
Year 2006 |
|
Total server storage and storage support costs |
$19,936,816 |
$1,241,061 |
$9,909,778 |
$8,727,492 |
$58,485 |
|
Total server infrastructure costs |
$33,898,492 |
$2,815,578 |
$14,043,529 |
$16,980,900 |
$58,485 Budgeted funds
totaling $5,734,025 were withdrawn by the
IRS and spent on other information technology needs. |
|
Percentage the server storage and storage support
costs are of server infrastructure costs |
58.8% |
44.1% |
70.6% |
51.4% |
Unknown (but
immaterial). |
|
Unused storage space rate |
73% |
|
|
|
|
|
Inefficient use of resources |
$14,553,876 |
|
|
|
|
Source: MITS
organization-provided expenditure and budget information for Fiscal Years 2003 through
2006 and storage utilization information. If MITS organizations other than the
Type and Value of Outcome Measure:
· Funds Put to Better Use – Actual; $9,345,125 (see page 4).
Methodology Used to Measure the Reported Benefit:
In our two prior audits of the
IRS’ storage management practices, we reported the IRS needed to improve the
efficiency of its storage utilization and should consider storage utilization
when assessing the need to purchase additional storage. In Fiscal Years 2006 and 2007, the IRS
budgeted $9.3 million to purchase end-user server storage and storage support
for the Enterprise Operations, EUES, and Enterprise Services
organizations. However, the MITS
organization decided not to spend the budgeted funds. These funds were redirected and put to better
use on higher priority information technology infrastructure needs. Figure 2 shows the budgeted storage funds
withdrawn and reassigned.
Figure 2: End-user Server Storage
Capacity and Support Costs
Withdrawn and Reassigned
|
Category |
Amounts |
Total Amounts |
|
Fiscal Year 2006 end-user server infrastructure funded amount
withdrawn |
$10,742,500 |
|
|
Fiscal Year 2006 infrastructure funded amount withdrawn
representative of server storage and storage support costs |
|
$5,734,025 |
|
|
|
|
|
Fiscal Year 2007 end-user server infrastructure funded amount
withdrawn |
$6,734,187 |
|
|
Fiscal Year 2007 infrastructure funded amount withdrawn
representative of server storage and storage support costs |
|
$3,611,100 |
|
Funds put to better use |
|
$9,345,125 |
Source: MITS organization-provided budget information
for Fiscal Years 2006 and 2007.
Appendix V
|
Agency Information
Technology Investment Portfolio (Exhibit 53) |
A document
covering information technology investments for the agency as a whole that is
to be submitted to the Office of Management and Budget if the agency (1) is
subject to executive branch review and is requesting information technology
funding via a Capital Asset Plan and Business Case (Exhibit 300) or (2) has
budget authority of $500,000 or more for financial management systems. |
|
Campuses |
The data
processing arm of the IRS; they process paper and electronic submissions,
correct errors, and forward data to the Computing Centers for analysis and
posting to taxpayer accounts. |
|
Capital Asset Plan
and Business Case (Exhibit 300) |
Used as a one-stop
document for a myriad of information technology management issues such as
business cases for investments, Clinger Cohen Act of 1996[15] implementation, agency’s modernization
efforts, and overall project (investment) management. |
|
Computing Centers |
Support tax
processing and information management through a data processing and
telecommunications infrastructure. |
|
End-user |
For this audit report, defined as an employee who uses the
IRS Microsoft Windows computer network to accomplish assigned duties. |
|
|
A system
development methodology for all nonmodernization projects. |
|
|
An information source for the Enterprise Operations organization server
support functions; it is populated with data that are related specifically to
server systems support, enhance support of the server environment, and are not
required in the ITAMS. |
|
Home Directory |
A directory that contains the personal files of a particular user of the system. |
|
Information Technology Asset Management System |
The official IRS computer equipment database used to
record all computer inventories. |
|
Integrated
Financial System |
The system intended to address administrative financial management weaknesses. The first release of the system will include the Accounts Payable, Accounts Receivable, General Ledger, Budget Execution, Cost Management, and Financial Reporting activities. A future release will be needed to fully resolve all administrative financial management weaknesses. |
|
Mainframe |
A powerful, multiuser computer capable of supporting many
hundreds of thousands of users simultaneously. |
|
Megabyte |
A unit of measure used to describe memory and disk
capacity that is equal to approximately 1 million bytes or 1,024
kilobytes. It is also equivalent to
approximately |
|
Microsoft Windows NT |
A family of operating systems produced by Microsoft, the first version of which was released in July 1993. |
|
Microsoft Windows 2003 Server Operating System |
The Windows server software version used by the IRS. |
|
Mid-range Computers |
File servers and related hardware, software, maintenance,
and services; they support enterprise application systems and are located at
the Computing Centers. |
|
Server |
A computer that carries out specific functions (e.g., a file
server stores files, a print server manages printers, and a network server
stores and manages network traffic). |
|
Server Virtualization |
The masking of server resources, including the number and
identity of individual physical servers, processors, and operating systems,
from server users. The server
administrator uses a software application to divide one physical server into
multiple, isolated virtual environments. |
|
Storage Device |
A device (e.g., server) capable of storing data. |
|
Terabyte |
A unit of measure used to describe memory and disk
capacity that is equal to approximately 1 trillion bytes or 1,024
gigabytes. It is also equivalent to
approximately 500,000,000 pages of text. |
|
Tier III Computing
Environment |
The IRS computing environment that consists of desktop and laptop computers as well as end-user servers. Specifically, these devices support tax return processing, postfiling, compliance, and internal management applications; infrastructure servers supporting network access; enterprise storage and retrieval; enterprise infrastructure; and Commercial Off-the-Shelf software. |
|
Tivoli® |
Tivoli® is a registered trademark owned by
IBM. The implementation of Tivoli®
is part of the IRS Enterprise Systems Management project encompassing help
desk operations, network and systems management, software distribution, asset
management, and performance measures analysis and reporting. |
|
Treasury Integrated Management
Information System |
The official automated personnel and payroll system for storing and tracking all employee personnel and payroll data. |
|
Virtualization |
Transforms physical hardware–servers, hard drives, and networks–into an infinitely flexible pool of computing resources (e.g., storage) that businesses can expand, reallocate, and use as needed. |
Appendix VI
Management’s Response to the Draft Report
The response was
removed due to its size. To see the
response, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
[1] See Appendix V for a Glossary of Terms.
[2] Total storage costs of $19,936,816 x 73 percent unused storage rate = $14,553,876 (rounded).
[3] Progress Has Been Made in Using the Tivoli®
Software, Although Enhancements Are Needed to Better Distribute Software
Updates and Reconcile Computer Inventories (Reference Number 2006-20-021,
dated December 2005).
[4] The 2,256 servers in the Tivoli® inventory system/4,783 servers in the Information Technology Asset Management System = 47 percent (rounded).
[5] See Appendix V for a Glossary of Terms.
[6] Mid-range Computer Storage Resources Need Better Administration to Ensure Effective and Efficient Utilization and Accurate Reporting (Reference Number 2005-20-098, dated July 2005).
[7] Improvements in Mainframe Computer Storage Management Practices and Reporting Are Needed to Promote Effective and Efficient Utilization of Disk Resources (Reference Number 2006-20-056, dated May 2006).
[8] Federal Acquisition Reform Act of 1996 (Information Technology Management Reform Act of 1996), Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).
[9] Total storage costs of $19,936,816 x 73 percent unused storage rate = $14,553,876 (rounded).
[10] Gartner, Inc. is the leading provider of research and analysis to the global information technology industry. [Source: SANZ (smarter about storage)® industry white paper entitled, “Building a Tiered Storage Architecture-Because Not All Data is Created Equal”.]
[11] Progress Has Been Made in Using the Tivoli®
Software, Although Enhancements Are Needed to Better Distribute Software
Updates and Reconcile Computer Inventories (Reference Number 2006-20-021,
dated December 2005).
[12] See Appendix V for a Glossary of Terms.
[13] Financial Audit: IRS’s Fiscal Years 2006 and 2005 Financial Statements (GAO-07-136, dated November 2006).
[14] See Appendix V for a Glossary of Terms.
[15] Federal Acquisition Reform Act of 1996 (Information Technology Management Reform Act of 1996), Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).