Virtual
Server Technology Has Been Successfully Implemented, but Additional Actions Are
Needed to Further Reduce the Number of Servers and Increase Savings
March 30, 2012
Reference Number: 2012-20-029
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone
Number | 202-622-6500
E-mail Address |
TIGTACommunications@tigta.treas.gov
Website | http://www.tigta.gov
HIGHLIGHTS
VIRTUAL SERVER TECHNOLOGY HAS BEEN
SUCCESSFULLY IMPLEMENTED, BUT ADDITIONAL ACTIONS ARE NEEDED TO FURTHER REDUCE THE
NUMBER OF SERVERS AND INCREASE SAVINGS
Highlights
Final Report issued on March 30, 2012
Highlights of Reference Number:
2012-20-029 to the Internal Revenue Service Chief Technology Officer.
IMPACT ON TAXPAYERS
The
IRS implemented server virtualization technology to reduce the number of Wintel
servers needed for tax administration and lower operational costs to the
taxpayers. As of the end of Fiscal Year
2011, the IRS estimated that server virtualization had saved approximately $10.2
million in equipment costs, and it expects to save approximately $1.3 million
annually in decreased electrical costs beginning in Fiscal Year 2013. The number of Wintel servers could be further
reduced for increased efficiency and cost savings of approximately $7.73
million.
WHY TIGTA DID THE AUDIT
This
review was included in our Fiscal Year 2011 Annual Audit Plan and addresses the
major management challenge of Modernization.
The overall objective of this review was to evaluate the effectiveness and efficiency of the IRS’s
efforts to consolidate and virtualize its servers. In addition, the Federal Data Center
Consolidation Initiative was passed in February 2010, mandating a reduction in the
number of Federal data centers.
WHAT
TIGTA FOUND
The IRS successfully implemented server virtualization
technology to improve server efficiency and realize cost savings. The Server Consolidation and Virtualization
Project focused on establishing a virtual server infrastructure and moving
approximately 2,500 physical Wintel servers at 13 data center locations to the
virtual environment. As of the end of Fiscal Year 2011, the IRS had
approximately 1,800 virtual servers operating on 234 physical host servers in
the virtual environment, resulting in the previously stated decrease in
equipment and electrical costs.
The IRS does not have a plan for
virtualizing Wintel servers at its field offices outside of the 13 data center locations. The IRS estimates there are approximately 650
Wintel servers in its field locations that can be decommissioned and added to
the virtual server environment. By virtualizing these servers, the IRS estimates it could
realize additional savings of approximately $7.73 million ($7.26 million in equipment
savings and $0.47 million in electrical savings over five years). Aside from the cost savings, these actions also
help the IRS to meet Federal Data Center Consolidation Initiative goals.
WHAT TIGTA RECOMMENDED
TIGTA
recommended that the Chief Technology Officer 1) develop and implement a
process to identify servers currently located in field offices that can be
virtualized and 2) create and implement a plan to virtualize those servers.
In
their response to the report, IRS management concurred with both
recommendations. The IRS plans to 1) develop
and implement a process to identify servers located in field offices that can
be virtualized and 2) create and implement a plan to virtualize those servers.
March 30, 2012
MEMORANDUM FOR CHIEF TECHNOLOGY OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Virtual Server Technology Has Been Successfully Implemented, but Additional Actions Are Needed to Further Reduce the Number of Servers and Increase Savings (Audit # 201120014)
This report presents the results of our review of the Internal Revenue Service’s efforts to consolidate and virtualize its servers. This review was included in the Treasury Inspector General for Tax Administration’s Fiscal Year 2011 Annual Audit Plan and addresses the major management challenge of Modernization.
Management’s complete response to the draft report is included as Appendix V.
Copies of this report are also being sent to
the Internal Revenue Service managers
affected by the report recommendations.
Please contact me at (202) 622-6510 if you have questions or Alan R. Duncan,
Assistant Inspector General for Audit (Security and Information Technology
Services), at (202) 622-5894.
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – Outcome Measures
Appendix
V – Management’s Response to the Draft Report
Abbreviations
|
FDCCI |
Federal Data Center Consolidation Initiative |
|
IRS |
Internal Revenue Service |
In 2007, the Internal Revenue Service’s (IRS) Infrastructure Roadmap Initiative concluded that the IRS’s diverse and widely deployed server infrastructure would benefit from the results of a consolidation and virtualization project. The initiative recommended a highly centralized, consolidated, and virtualized future state for the IRS’s computing environment. In 2007, the Virtualization Project Office was established to design and implement an enterprise-wide virtualization environment. The Server Consolidation and Virtualization Project was initiated in February 2007 and closed in December 2010.
Server
virtualization is a technology that allows a number of virtual servers[1] to run on one physical
host. The technology enables improved
hardware utilization, electrical savings, and reduced server replacement
costs. Virtualization helps an information
technology organization improve efficiency and productivity by simplifying
server deployment and administration. For
example, the impact of any hardware issues on a virtualized server can be minimized
because the applications and data on the affected server can be switched to
other servers in the pool. This allows
for decreased server downtime due to hardware failure because the system can continue
running while the hardware is replaced.
Implementing
virtualization technology at the IRS also supports the Federal Data Center
Consolidation Initiative (FDCCI).
In February 2010,
the Federal Chief Information Officer launched the FDCCI, mandating a reduction
in the number of Federal data centers.
The FDCCI aims to:
·
Encourage the use of Green Information Technology[2] by reducing the overall energy
and real estate footprint of Federal Government data centers.
·
Reduce the cost of data center hardware, software, and operations.
·
Increase the overall information technology security posture of
the Federal Government.
·
Shift information technology investments to more efficient
computing platforms and technologies.
The IRS has prepared a Data Center
Consolidation Plan in response to direction from the Office of Management and
Budget and the FDCCI. The IRS’s
approach to increase the efficiency of its Wintel[3]
server resources is to use server consolidation and centralization to reduce the
total number of required physical servers and server locations. Wintel servers are located at IRS Enterprise Computing
Centers,[4]
campuses,[5]
and many other locations across the IRS.
A primary objective of the Server Consolidation and Virtualization Project
was to reduce the number of physical Wintel servers. The Project created a virtual infrastructure
at 13 targeted locations to virtualize approximately 2,500 physical servers managed
by the IRS’s Enterprise Operations organization. Wintel servers at other IRS field locations
were not included in the scope of the Server Consolidation and Virtualization
Project. The virtual
environment consists of a series of high-end physical servers capable of
hosting multiple virtual servers on each physical host. As of the end of Fiscal Year
2011,[6] approximately 1,800 virtual servers were operating on 234
physical host servers in the virtualized infrastructure.
We obtained Server Consolidation and Virtualization Project records and interviewed Project management located in Portsmouth, New Hampshire; Hartford, Connecticut; and Providence, Rhode Island, during the period March 2011 through January 2012. We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The Internal Revenue Service Has Successfully Deployed the Technology and Infrastructure to Consolidate and Virtualize Wintel Servers at Select Data Centers
The Virtualization Project Office teamed with the Hewlett-Packard Company to aid the Server
Consolidation and Virtualization Project’s design and installation. The collaboration helped ensure the IRS
applied industry best practices to the project and allowed the IRS to prototype
the installation at the vendor’s facility to ensure the system would work
properly. Through the IRS’s Concept of
Operations document, an Enterprise Life Cycle[7] Plan was developed specifically
for an infrastructure deployment effort.
At the project’s conclusion, project management teamed with the
contractor to produce a document of lessons learned, including recommendations
and approaches that can be used for similar future infrastructure
deployment projects.
The Virtualization Project Office began installing the virtual infrastructure at selected
data centers in 2009. Monthly progress
reports were sent to the Associate Chief Information Officer, Enterprise
Operations. The IRS required the
Hewlett-Packard Company to track issues during the project. The IRS Executive Steering Committee minutes
and Business Performance Reports show the project moved successfully to its
completion in December 2010.
A process was set up to continue
to virtualize servers at the target locations after the project completion
date. As of the end of Fiscal Year 2011,
the IRS has virtualized all targeted servers at the Andover, Massachusetts;
Brookhaven, New York; and Philadelphia, Pennsylvania, Campuses. Approximately 1,500 remaining Wintel servers at
the targeted locations are expected to be virtualized by the end of Fiscal Year
2012.
The
goals of the Server Consolidation and Virtualization Project were successfully
achieved on time and within budget. The
goal of the Project was to deploy the virtualization infrastructure and begin
converting physical servers to virtual machines. By the end of Fiscal Year 2011, the project
team had succeeded in establishing a virtual server environment with
approximately 1,800 virtual servers running on 234 physical host servers
at 13 data center locations (nine campuses, three computing
centers, and the New Carrollton Federal Building). This conversion of
physical servers to virtual servers is helping to reduce energy and server
replacement costs and improve operational efficiency. Reducing the number of physical servers also
supports the goals of the FDCCI.
Reducing
the number of physical servers has also resulted in significant cost savings
associated with lower electrical output for fewer servers and hardware savings
over one-for-one server replacement. As
of the end of Fiscal Year 2011, the IRS estimated that server virtualization had
saved approximately $10.2 million in equipment costs. The IRS also expects to save approximately $1.3
million annually in decreased electrical costs beginning in Fiscal Year 2013. The virtualized servers help lower
operational costs through standardization, making it easier to load or remove a
server from the operating environment. Other benefits of
virtualization technology include decreased server hardware downtime and
automatic load balancing.[8]
Virtualization technology enables the system to move data and processing
before a server fails. Automatic failover involves automatically
moving an application to a standby server during a failure or service event to prevent
application downtime. The IRS realized failover benefits on two separate occasions since it
deployed its virtual infrastructure.
Plans Are Needed to Study and
Address the Challenges of Virtualizing the Remaining
Wintel Servers and to Maximize the Benefits of Server Virtualization
The Server Consolidation and
Virtualization Project did not include Wintel servers
in IRS field offices outside of the 13 targeted locations. The Project has been closed for one year;
however, the IRS does not have a plan for virtualizing Wintel servers at its
field office locations spread out across the country. There are approximately 1,000 Wintel servers
at remote locations that have not been virtualized.
The objective of the Server
Consolidation and Virtualization Project was to virtualize physical servers at
13 targeted locations: nine campuses, three
computing centers, and the New Carrollton Federal Building. All of these locations have similar network
characteristics, enabling a standardized installation at each site. Servers outside of the targeted locations
were eliminated from the scope of the project due to network limitations that
prevent these servers from being virtualized and moved from the remote
locations to one of the targeted locations.
The Windows Server Strategy Solutions and Standards Report from the
Infrastructure Roadmap Initiative states:
“The capability of the
enterprise network is a potential constraint when addressing consolidation or
virtualization. By centralizing all of
the computing resources to a few locations, the application traffic will change
and could impact the network. There must
be enough bandwidth and throughput on the network to handle the new traffic
patterns at each location. The network
latency between users and servers must be minimized to provide the response
times required by users. This must be
taken into account when designing a consolidation and virtualization plan.”
The
IRS’s Data Center Consolidation Plan includes a goal to virtualize physical servers
from field locations into an appropriate data center virtual environment. The IRS cannot realize maximum
cost savings until it virtualizes the Wintel servers at its field locations and
moves these virtual machines onto host servers at the computing centers. IRS management estimates approximately
650 of 1,000 Wintel servers in its field locations can be decommissioned and
added to the virtual server environment.
By virtualizing the remote servers, IRS management estimates it could
realize additional savings of approximately $7.73 million ($7.26 million in
equipment savings and $0.47 million in electrical savings over five
years). These actions would also help
the IRS reduce data center space and meet FDCCI
goals.
The
longer the IRS waits to virtualize the physical Wintel servers in its field locations, the
older these servers become. The IRS’s
server inventory showed that 39 percent of the Wintel servers in the non-targeted field offices reached end of
life[9] status during Fiscal Year
2011. For Fiscal Year 2012, the
percentage of aged Wintel servers will reach 48 percent. By effectively and efficiently virtualizing the aged and remaining
Wintel servers, the IRS will ensure it continues to support the FDCCI and meet
its own Data Center Consolidation Plan goals.
Recommendations
Recommendation 1: The Chief Technology Officer should develop and implement a process to identify servers currently located in field offices that can be virtualized.
Management’s Response: The IRS agreed with the recommendation. Enterprise Operations will develop and implement a process to identify servers currently located in field offices that can be virtualized.
Recommendation 2: The Chief Technology Officer should create and implement a plan to virtualize the servers located in field offices that can be virtualized.
Management’s Response: The IRS agreed
with the recommendation. Enterprise
Operations will create and implement a plan to virtualize the servers located
in field offices that can be virtualized.
Appendix I
Detailed Objective, Scope, and
Methodology
Our overall
objective was to evaluate
the effectiveness and efficiency of the IRS’s efforts to consolidate and
virtualize its servers. We evaluated
servers managed by the Enterprise Operations organization, as those were the
servers included in the scope of the IRS’s Server Consolidation and Virtualization
Project. To accomplish our objective,
we:
I.
Evaluated
Server Consolidation and Virtualization Project management and results.
A.
Determined
if project management for the Server Consolidation and Virtualization Project
complied with the IRS Enterprise Life Cycle[10] requirements and industry best practices.
1.
Reviewed
Internal Revenue Manual requirements for preparing an Enterprise Life
Cycle project plan.
2.
Obtained
the Enterprise Life Cycle plan,
tailored for an infrastructure deployment effort, for the project. We reviewed the project explanation and
justification for how the Enterprise Life Cycle was adapted to meet the specific needs of the project.
3.
Compared
the Enterprise Life Cycle plan
obtained in I.A.2. to the project plan requirements
obtained in I.A.1. to identify any plan requirements
not met.
4.
Identified
industry best practices for Windows server virtualization.
5.
Evaluated
the IRS’s server virtualization project to determine if industry best practices
were followed.
B.
Determined
if the Server Consolidation and Virtualization Project was effectively
implemented.
1.
Reviewed
project documentation to identify the population of Wintel servers[11] in the IRS and the number of Wintel servers
that were identified for virtualization during the project.
2.
Determined
if the project maximized the number of servers virtualized.
3.
Evaluated
the decision to close the Server Consolidation and Virtualization Project with
only 37 percent of the Wintel servers virtualized.
4.
Evaluated
procedures in place to continue virtualizing the remaining Wintel servers.
C.
Evaluated
the accuracy of calculations to support project costs, cost savings, and return
on investment.
1.
Obtained
and evaluated for accuracy and reliability all documentation related to project
costs and projected savings.
2.
Obtained
and evaluated for accuracy and reliability all documentation to support actual
cost savings claimed.
D.
Identified
and evaluated actions that have been taken to address the lessons learned
during the Wintel Server Consolidation and Virtualization Project.
1.
Identified
lessons learned and documented in the Server
Consolidation and Virtualization Initiative, Post Implementation Review Report,
dated November 17, 2010.
2.
Identified
actions taken by the IRS to address the problems in the lessons learned
documentation and evaluated the sufficiency of the actions.
II.
Evaluated the monitoring, management, and reporting of
server virtualization results and performance trends.
A.
Reviewed
Executive Steering Committee minutes and Business Performance Review reports
relating to server virtualization for Fiscal Years 2009, 2010, and 2011.[12]
1.
Identified
problematic trends in virtualized systems noted in these documents.
2.
Obtained
evidence that problems identified were effectively addressed.
B.
Interviewed
project management to determine how they monitor and measure program
performance.
1.
Identified
the measures (i.e., key performance
indicators) established for Fiscal Years 2010 and 2011 such as application and
server performance, downtime, and availability.
2.
Identified
tools used to monitor performance.
3.
Identified the performance reports management used to evaluate
strengths of virtualization performance (i.e.,
success) and gaps or weaknesses (i.e.,
nonsuccess).
4.
Determined what load balancing and failover measures are in place
to prevent or respond to failure of the host platform.
Internal controls methodology
Internal controls relate to management’s plans, methods, and procedures
used to meet their mission, goals, and objectives. Internal controls include the processes and
procedures for planning, organizing, directing, and controlling program
operations. They include the systems for
measuring, reporting, and monitoring program performance. We determined the following internal controls
were relevant to our audit objective: the
IRS policies and procedures for deploying an information technology
project. We
evaluated these controls by interviewing management and reviewing supporting
documentation.
Appendix II
Major Contributors to This Report
Alan R. Duncan, Assistant Inspector General for Audit (Security and
Information Technology Services)
Danny Verneuille, Director
Carol Taylor, Audit Manager
Myron Gulley, Acting Audit Manager
Joan Bonomi, Lead Auditor
Mark Carder, Senior Auditor
Kasey Koontz, Auditor
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Deputy Chief
Information Officer, Operations OS:CTO
Associate Chief Information Officer, Enterprise Operations OS:CTO:EO
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis RAS:O
Office of Internal Controls OS:CFO:CPIC:IC
Audit
Liaison: Director,
Risk Management Division
OS:CTO:SP:RM
Appendix IV
This appendix presents detailed information on the measurable impact that our recommended corrective action will have on tax administration. These benefits will be incorporated into our Semiannual Report to Congress.
Type and Value of Outcome Measures:
· Funds Put to Better Use – Potential; approximately $7.26 million in equipment savings (see page 4).
·
Funds Put to Better
Use – Potential; approximately $94,000 annually ($470,000 over five years) in
electrical savings (see page 4).
Methodology Used to Measure the Reported Benefit:
The reported potential benefits are based on estimates calculated by and provided to us by IRS management. There are approximately 1,000 physical Wintel servers in IRS field offices that have not been virtualized. The IRS estimates that approximately 650 of those servers could be candidates for replacement with virtual servers. If the servers are added to the virtual environment and not replaced with new physical servers, the IRS estimates it would save approximately $7.26 million in equipment costs. The IRS also estimates that decommissioning the 650 physical servers in its field offices would generate annual energy savings of approximately $94,000 ($470,000 over five years) based on the electrical cost savings of the servers less the electrical cost of the additional hardware that would need to be added to the virtual infrastructure environment.
Appendix V
Management’s Response to the Draft Report
DEPARTMENT OFTHE TREASURY
INTERNAL REVENUE SERVICE
WASHINGTON, D.C. 20224
CHIEF
TECHNOLOGY OFFICER
March 22, 2012
MEMORANDUM FOR DEPUTY
INSPECTOR GENERAL FOR AUDIT
FROM: Terence V. Milholland
/s/ Terence V. Milholland
Chief Technology
Officer
SUBJECT: Draft Audit Report- Virtual
Server Technology Has Been Successfully Implemented, but Additional Actions Are
Needed to Further Reduce the Number of Servers and Increase Savings
(Audit# 201120014) (e-trak # 2012-29743)
Thank you for the opportunity to review and respond to the
subject audit report.
We
appreciate your comments on how we implemented server virtualization technology
to improve server efficiency and realize cost savings. Thereby, our considerable efforts to reduce
equipment and associated electrical costs have been successful.
As indicated in the report, the IRS's Data Center Consolidation
Plan includes a goal to virtualize the remaining
physical servers from field locations into an appropriate data center virtual
environment. We recognize that our
existing plan requires further development to
realize full benefits of our virtualization program, therefore, we agree with
the recommendations. We also agree with the potential Outcome Measures and upon
implementation of the planned corrective actions the equipment and electrical
cost savings will be further realized.
We value your continued
support and the guidance your team provides. If you
have any questions, please contact me at (202) 622-6800 or Andrea
Greene-Horace, Senior Manager of Program Oversight, at (202) 283-3427.
Attachment
RECOMMENDATION
#1: The Chief Technology Officer should develop and implement a
process to identify servers currently located in field offices which can be
virtualized.
CORRECTIVE ACTION #1: We agree with the recommendation. Enterprise Operations will develop and
implement a process to identify servers currently located in field offices
which can be virtualized.
IMPLEMENTATION DATE: September 1, 2013
RESPONSIBLE OFFICIAL: Associate Chief Information Officer, Enterprise Operations
CORRECTIVE ACTION
MONITORING PLAN: We enter
accepted Corrective Actions into the Joint Audit Management Enterprise System
(JAMES) and monitor them on a monthly basis until completion.
RECOMMENDATION #2: The Chief Technology
Officer should create and implement a plan to virtualize
the servers located in field offices that can be virtualized.
CORRECTIVE
ACTION #2: We agree with the recommendation. Enterprise
Operations will create and implement a plan to virtualize
the servers located in field offices that can be virtualized.
IMPLEMENTATION DATE: September 1, 2014
RESPONSIBLE OFFICIAL: Associate Chief Information Officer, Enterprise Operations
CORRECTIVE ACTION
MONITORING PLAN: We enter accepted
Corrective Actions into the Joint Audit Management Enterprise System (JAMES)
and monitor them on a monthly basis until completion.
[1] A virtual server is not a physical machine. It co-resides and shares computer resources with other virtual servers on a physical computer or host.
[2] Green Information Technology, also known as Green Computing, is the movement toward a more environmentally friendly and cost-effective use of power and production in technology.
[3] A Wintel server is a server running a Microsoft Windows operating system with an Intel microprocessor.
[4] IRS Computing Centers support tax processing and information management through a data processing and telecommunications infrastructure.
[5] The data processing arm of the IRS. The campuses process paper and electronic submissions, correct errors, and forward data to the Computing Centers for analysis and posting to taxpayer accounts.
[6] A 12-consecutive-month period ending on the last day of any month, except December. The Federal Government’s fiscal year begins on October 1 and ends on September 30.
[7] Enterprise Life Cycle is the approach used by the IRS to manage and effect business change. It provides the direction, processes, tools, and assets for accomplishing business change in a repeatable and reliable manner.
[8] Automatic load balancing refers to the even distribution of processing across available resources such as servers in a network.
[9] End of life status describes a server that is aged and eligible for replacement after five years in operation.
[10] Enterprise Life Cycle is the approach used by the IRS to manage and effect business change. It provides the direction, processes, tools, and assets for accomplishing business change in a repeatable and reliable manner.
[11] A Wintel server is a server running a Microsoft Windows operating system with an Intel microprocessor.
[12] A 12-consecutive-month period ending on the last day of any month, except December. The Federal Government’s fiscal year begins on October 1 and ends on September 30.