Implementing Best Practices and Additional Controls Can
Improve
May 7, 2010
Reference Number: 2010-20-044
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone Number |
202-622-6500
Email Address | inquiries@tigta.treas.gov
Web Site |
http://www.tigta.gov
HIGHLIGHTS
IMPLEMENTING
BEST PRACTICES AND ADDITIONAL CONTROLS CAN IMPROVE
Highlights
Final
Report issued on May 7, 2010
Highlights of Reference
Number: 2010-20-044 to the Internal
Revenue Service Chief, Agency-Wide Shared Services.
IMPACT ON TAXPAYERS
Federal
agencies have been directed to improve their energy efficiency and reduce
greenhouse gas emissions. Internal
Revenue Service (IRS) management indicated the agency’s overall energy program
has consistently met standards set forth in all pertinent Executive Orders and
legislative mandates. However, while
data centers typically consume more energy than other types of buildings, the
IRS has not yet fully implemented most data center best practices. As a result, the IRS has not maximized the
energy efficiency and use of taxpayer funds resulting from decreased energy
consumption in its data centers.
WHY TIGTA DID THE AUDIT
This audit was initiated to determine whether the Real
Estate and Facilities Management organization has established and implemented an effective
environmental and energy program to ensure the improvement of data center
energy efficiency.
In January 2007, Executive Order 13423, Strengthening Federal Environmental, Energy, and Transportation
Management, was enacted to strengthen the environmental, energy, and transportation
management of Federal agencies.
WHAT
TIGTA FOUND
The
Real Estate and Facilities Management organization has implemented programs and
processes to improve energy efficiency.
For example, some facilities fully implemented an Environmental Management System
and/or were awarded the ENERGY STAR®
designation. In addition, the Real Estate and Facilities Management organization
has a training program to ensure employees requiring training have access to applicable courses and completion of the
training is documented.
However, the IRS does not have policies and procedures for improving
energy efficiency in the data centers or for implementing data center energy
efficiency best practices. We estimate
that the IRS could potentially realize savings of $3,172,872 over 4 years, at 2
sites, by implementing best practices to improve airflow management.
The IRS
also had not developed adequate policies and procedures for controlling and
monitoring energy efficiency improvement projects.
WHAT TIGTA RECOMMENDED
TIGTA
recommended that the Chief, Agency-Wide Shared Services, work with the Chief
Technology Officer to ensure 1) policies and
procedures are established to evaluate and determine which best practices to
implement, 2) information technology equipment energy use is measured (i.e.,
sub-metered), 3) employee workstations, equipment, and
furniture that are no longer needed are removed from the data centers, 4)
current and future data center space needs are identified and plans developed to consolidate or reduce excess data
center space, and 5) energy audits (including an assessment of best practices)
are performed at the data centers. In
addition, the Chief, Agency-Wide Shared Services, should ensure 6) a
governance process is established and 7) a database of all recommendations and
projects is established.
In their response to the report, IRS management agreed with all of the
recommendations but questioned the outcome measure contained in the audit
report. The IRS plans to take many
corrective actions. After considering
the IRS’ comments, TIGTA maintains that the outcome measure in the report is
valid.
May 7, 2010
MEMORANDUM FOR CHIEF, AGENCY-WIDE SHARED SERVICES
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final
Audit Report – Implementing Best Practices and Additional Controls Can Improve
This report presents the results of our review to determine whether the Real Estate and
Facilities Management organization has established and implemented an effective environmental and energy
program to ensure the improvement of data center[1] energy
efficiency. This review was included in our Fiscal
Year 2009 Annual Audit Plan and
addresses the major management challenge of Modernization of the Internal
Revenue Service (IRS).
Management’s
complete response to the draft report is included as Appendix VII.
Copies of
this report are also being sent to the IRS managers affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Alan R. Duncan, Assistant Inspector
General for Audit (Security and Information Technology Services), at (202) 622-5894.
Programs and
Processes Have Been Implemented to Improve Facility Energy Efficiency
Data Center Energy
Efficiency Best Practices Need to Be Implemented
Governance Over
Environmental and Energy Efficiency Projects Should Be Enhanced
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix
V – Locations of Facilities Selected to Implement an Environmental Management
System
Appendix VI – Glossary of Terms
Appendix
VII – Management’s Response to the Draft Report
Abbreviations
AWSS |
Agency-Wide Shared Services |
DCiE |
|
ECC |
|
EISA |
Energy |
IRS |
Internal Revenue Service |
MITS |
Modernization and Information Technology
Services |
REFM |
Real Estate and Facilities
Management |
A data center consists of computer hardware and software used for data processing (servers), data storage (storage equipment), and communications (network equipment). Data centers also usually contain specialized power conversion and backup equipment to maintain reliable, high-quality power, as well as environmental control equipment to maintain the proper temperature and humidity for the computer equipment. Most data centers are more energy intensive than other buildings due to the high power requirements of the computer equipment and the power and cooling infrastructure needed to support this equipment. In fact, data centers can be more than 40 times as energy intensive as conventional office buildings,[2] meaning that large data centers more closely resemble industrial facilities than commercial buildings with respect to energy use.
Federal
agencies have been tasked to strengthen the environmental, energy, and
transportation management of their agencies.
In August 2007, the Environmental Protection Agency reported[3] that the energy used by the nation’s servers and data centers is significant. In 2006, electricity consumed by servers and data centers (including cooling and auxiliary infrastructure) represented about 1.5 percent of national electricity use at an estimated $4.5 billion in electrical cost. This estimated level of electricity consumption is more than the electricity consumed by the nation’s color televisions and similar to the amount of electricity consumed by approximately 5.8 million average households. Federal servers and data centers alone account for approximately 10 percent of this electricity use, for a total electricity cost of about $450 million annually.
Based on 2006 efficiency trends,
national energy consumption by servers and data centers could nearly double by
2011, representing a $7.4 billion annual electricity cost. Forecasts
indicate that unless energy efficiency is improved beyond current trends, the Federal
Government’s electricity cost for servers and data centers could be nearly $740
million annually by 2011. These estimates of data center energy use should be
considered approximate because limited data are available on current data
center energy use, and there is significant uncertainty about the effects of
future technology trends, such as server consolidation and developments in
network and storage technologies. However,
these estimates and projections illustrate the magnitude of energy use in data
centers and the need for effective energy-efficiency strategies.
In May 2009, a
white paper report produced by the Eaton Corporation stated as much as 30 to 60
percent of the data center utility bill goes to support cooling systems,
including air handlers[4] and chiller
systems. Many computer room cooling
systems are inefficiently deployed or not operated under recommended
conditions. Poor airflow management
reduces both the efficiency and capacity of computer room cooling equipment. Leaking floor tiles, excessive cable
openings, poorly placed overhead supplies, and inappropriately oriented rack
exhausts can reduce the usable capacity of a computer room air conditioner unit
by 50 percent or more.[5]
In January 2007, President George W. Bush signed Executive
Order 13423 Strengthening Federal
Environmental, Energy, and Transportation Management. The purpose of this policy was to strengthen
the environmental, energy, and transportation management of Federal agencies by
“conducting their environmental, transportation, and energy-related activities
under the law in support of their respective missions in an environmentally,
economically, and fiscally sound, integrated, continuously improving,
efficient, and sustainable manner.” The
Executive Order established a goal to improve energy efficiency by 3 percent annually through
the end of Fiscal Year 2015 or 30 percent by Fiscal Year 2015 compared
to the Fiscal Year 2003 baseline year. A
revised Treasury Directive 75-04, Energy
Management Program, was issued in December 2008 establishing the policies
and assigning responsibilities for maintaining the comprehensive energy
programs within the Department of the Treasury and complying with all relevant
regulations and Executive Orders.
In October 2009,
President Barack Obama signed Executive Order 13514, Federal Leadership in Environmental, Energy, and Economic Performance,
which stipulates that Federal agencies must increase energy efficiency, reduce
greenhouse gas emissions, eliminate waste, and foster conservation and
sustainability. More specifically, it
mandates that agencies implement best practices for energy-efficient management
of Federal data centers and servers.
While previous Executive Orders pertaining to energy efficiency did not
preclude agencies from instituting efficiency measures at data centers, they
also did not require such action.
This review was performed at the Internal Revenue Service’s (IRS) Real Estate and Facilities
Management (REFM) organization’s
office in
Programs and Processes Have Been Implemented
to Improve Facility Energy Efficiency
An Environmental Management System is being implemented at 11 selected facilities[6]
An Environmental Management System is a set of processes and practices that enable an
organization to increase its operating efficiency, continually improve overall
environmental performance, and better manage and reduce its environmental
impacts, including those environmental aspects related to energy and
transportation functions. The IRS plans
to integrate the Environmental Management System with existing management systems used at each facility to help achieve
environmental and economic goals. An Environmental Management System is
considered fully implemented when it has been formally audited by an
independent qualified party, audit findings have been recognized by the
appropriate level of management of the facility implementing the system, and it
has been declared by an agency executive to be in conformance to
requirements. Two of the facilities we
reviewed (Austin Campus and ECC – Martinsburg) have fully implemented an
Environmental Management System. The ECC
–
Two facilities reviewed were awarded the ENERGY STAR® designation
ENERGY STAR is a joint program of the Environmental Protection Agency and the Department of Energy to help us all save money and protect the environment through energy efficient products and practices. An ENERGY STAR qualified facility meets strict energy performance standards set by the Environmental Protection Agency and uses less energy, is less expensive to operate, and causes fewer greenhouse gas emissions than its peers. To qualify for the ENERGY STAR designation, a building or manufacturing plant must score in the top 25 percent based on the National Energy Performance Rating System. To determine the performance of a facility, the Environmental Protection Agency compares energy use among other, similar types of facilities on a scale of 1-100. Buildings that achieve a score of 75 or higher may be eligible for the ENERGY STAR designation. The rating system accounts for differences in operating conditions, regional weather data, and other important considerations.
As a result of the efforts to improve energy efficiency, the
Memphis Campus was awarded the ENERGY STAR designation in 2006 through 2009 and
the Austin Campus was awarded the designation in 2005, 2007, and 2009 when it
was reviewed and considered for the ENERGY STAR designation. In addition,
IRS management advised us that the Andover Campus also received the ENERGY STAR
designation in 2008 and 2009.
Employees completed required environmental and energy training
The IRS ensures
employees requiring training related to the environmental
and energy programs have access to applicable courses. The IRS identified which employees needed
such training and what training courses could be completed to meet the training
requirements. Completed training
is documented on certificates of completion and/or the IRS Enterprise Learning
Management System. In addition,
contractors who performed Environmental Management System and Environmental, Safety and Health audits
determined employees completed the required training.
The IRS efforts in the Environmental Management System, ENERGY STAR, and employee training areas have helped to ensure the agency complies with Executive Order 13423, Strengthening Federal Environmental, Energy, and Transportation Management, and attains its goals. For example, agencies were directed to:
• Implement environmental management systems to ensure environmental issues are addressed, including environmental aspects of energy and transportation functions.
• Improve energy efficiency of agency facilities.
• Establish environmental management training programs.
The IRS efforts will also ensure compliance with Treasury Directive 75-04, Energy Management Program, which states bureaus are responsible for establishing and maintaining comprehensive energy programs within the Department of the Treasury and complying with all relevant regulations and Executive Orders. This includes ensuring that all appropriate personnel receive training for implementing Executive Orders, including development of outreach programs that include education, training, and promotion of ENERGY STAR and other energy-efficient products for Federal purchase card users.
IRS management indicated the agency’s overall energy program has consistently met standards set forth in all pertinent Executive Orders and legislative mandates. While the IRS’ accomplishments are notable and have improved its energy efficiency, additional actions at the data centers to implement best practices will further improve the IRS’ energy efficiency and enhance governance over environmental and energy efficiency projects.
Data Center Energy Efficiency Best Practices
Need to Be Implemented
The Agency-Wide Shared Services (AWSS) organization, in conjunction with the Modernization and Information Technology Services (MITS) organization, has overall responsibility to ensure that the IRS is complying with Executive Orders 13423 and 13514 and applying best practices. However, the AWSS and MITS organizations do not have a process to consider, select, and apply best practices for improving data center energy efficiency. The AWSS organization has identified Department of the Treasury Best Energy Practices, but these do not address energy efficiency in the data centers, as they were developed prior to the issuance of Executive Order 13514.
We made the following observations regarding the
implementation of best practices when we visited the Austin and Ogden Campuses,
ECC – Martinsburg, and ECC –
Observations of the
implementation of best practices
·
There were gaps between floor tiles (i.e.,
missing tiles) allowing hot and cold air to mix which can overwork the computer
room air conditioning units. Perforated
tiles were placed ad hoc around the server room, many times cooling the hot
aisle (ECC – Martinsburg, ECC –
·
Brush
grommets in the raised floor to cover floor spaces/gaps and blanking panels in
the server racks to cover spaces in rack-mounted equipment were not used or not
used consistently to prevent air leaks (ECC – Martinsburg, ECC –
Memphis, and Ogden Campus). Austin
Campus building management sealed air leaks in the floor prior to our visit. Best
practices state that brush grommets or other barriers and blanking
panels can help optimize airflow by minimizing
the mixing of hot and cold air. Figure 1
provides an illustration of the usage of brush grommets in the raised floor and
blanking panels in the server racks.
Figure
1: Brush Grommets in the Raised Floor and
Blanking Panels in the Server Racks
Figure 1 was removed due to its
size. To see Figure 1, please go to the
Adobe PDF version of the report on the TIGTA Public Web Page.
·
Servers were not effectively placed in hot/cold
aisle configurations. Three sites (ECC –
Martinsburg, ECC –
Figure 2:
Figure 2 was removed due to its size. To see Figure 2, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
·
There were inconsistencies in row-to-row spacing
(called pitch) of the servers in the data center layout at all sites visited (Austin
and Ogden Campuses, ECC – Martinsburg, and ECC –
·
There were no occupancy sensors to control
lighting or sub-metering that
monitored power consumption at the sites visited (
·
None of the sites visited had the capability to
determine how energy use is distributed among the computer equipment and
support systems in the data center. Best practices state installing sub-meters at
key locations provides a powerful tool for measuring the energy performance of
individual systems. Providing these
sub-meters with recording capability allows an agency to monitor system
performance over time, providing evidence of degradations and improvements.
Other observations
·
Furniture, old computer command modules, and computer
and other equipment that are no longer used were stored in the data
centers. For example, the Austin Campus
data center had pallets of copy paper and old command modules that occupied up
to an estimated one-third of the data center space. The
·
The ECC – Martinsburg and the ECC –
·
There were occupied employee workstations
(cubicles) in the data center. The ECC –
Martinsburg had multiple occupied workstations surrounded by servers. ECC –
While this audit focused largely on cooling and airflow management, IRS
management indicated that the AWSS organization is also working to improve the
power conversion and distribution systems supporting the data centers, by
right-sizing and using more energy-efficient technology in their
Uninterruptible Power Supply systems.
Executive Order 13423, dated January 24, 2007, states that Federal agencies conduct their environmental, transportation, and energy-related activities in an environmentally, economically, and fiscally sound, continuously improving, efficient, and sustainable manner. Executive Order 13514, dated October 5, 2009, states the agency head shall consider reductions associated with implementing best management practices for energy-efficient management of servers and Federal data centers.
The IRS does not have policies and procedures for improving
energy efficiency in the data centers or for implementing data center energy
efficiency best practices. Not fully implementing
applicable best practices or other data center improvements affects the IRS’
ability to minimize energy consumption and
costs, resulting in the inefficient use of resources and taxpayer funds not being most effectively used. In addition,
without an effective and efficient process to implement best practices, the IRS
may not fully comply with Executive Order 13514 to improve the energy-efficient management of servers and Federal data
centers.
Industry guidelines suggest that implementing best practices
to improve airflow management can reduce data center cooling by up to 50
percent. In addition to the previously
discussed airflow management best practices, another strategy to improve
airflow management is using flexible clear plastic
barriers, such as plastic supermarket refrigeration covers, to seal the space
between the tops of the server racks and the ceiling or air return location,
thereby containing heat produced by equipment.
Fiscal Year 2009 electricity costs
for the ECC –
In addition, the Department of Energy provides an online software tool (Data Center Energy Profiler) to help industries worldwide identify how energy is being purchased and consumed by data center(s), and also identify potential energy and cost savings. The Data Center Energy Profiler checklist provides a complete list of questions for developing a profile. After entering the required information, the user is provided with a customized, printable report that shows the details of energy purchases for the data center, how energy is consumed by the data center, potential cost and energy savings, comparison of the data center energy utilization versus other data centers, and a list of next steps to follow to start saving energy.
We
asked IRS management at the four sites in our review to complete a Data Center
Energy Profiler case for the respective sites.
Based on the information entered, the software tool generates the Data Center Infrastructure Efficiency (DCiE) metric to
indicate the current efficiency rate and the potential estimated efficiency
rate if best practices are implemented.
The DCiE metric has become a commonly used metric for data center
efficiency. Figure 3 provides the
current and potential DCiE metric for the four sites visited.
Figure
3: Potential DCiE Improvement
Site |
Current Site Energy |
Potential Site Energy |
Efficiency Measure Change |
Percentage Change |
ECC – |
0.66 |
0.88 |
0.22 |
33.33% |
ECC – Martinsburg |
0.71 |
0.90 |
0.19 |
26.76% |
|
0.72 |
0.90 |
0.18 |
25.00% |
Austin Campus |
0.79 |
0.85 |
0.06 |
7.59% |
Source:
Recommendations
The Chief, AWSS, should work with
the Chief Technology Officer to ensure:
Recommendation 1: Policies and procedures are established to evaluate and determine which best practices to implement to improve data center energy efficiency.
Management’s Response: IRS management agreed with the
recommendation. The AWSS organization will
convene a Data Center Working Group, co-chaired by the
REFM and MITS organizations, to assess best practices to implement in IRS data
centers, communicate best practices with campus contacts, establish methods for
tracking progress toward implementation, and research new opportunities for
improving data center energy efficiency.
Office of Audit Comment: While management agreed with this recommendation and is planning
to take several actions to improve data center energy efficiency, they are not comfortable with the current language used
to discuss the outcome measure “Cost Savings – Funds Put to Better Use –
Potential; $793,218 per year; $3,172,872
over a 4-year period.” Management
stated that the figure does not factor in implementation costs, does not take
into consideration best practices already implemented, and is based on 50
percent energy savings, which represent the upper limit of potentially
achievable savings.
The potential savings calculations are based
on our observations of the implementation of best practices and the best
information available. The calculations do
not factor in implementation costs because the cost estimates were not
available. In addition, the potential
for energy savings is much higher because our estimates do not include the data
centers at the other nine IRS campuses, for which energy consumption
information is not available. We believe
the reported outcome measure is valid.
Recommendation 2: Information technology equipment energy use is measured (i.e., sub-metered) in order to determine the energy efficiency and savings from implementing energy improvements.
Management’s Response: IRS management agreed with the
recommendation. The REFM organization will identify one campus and one
computing center, perform an analysis examining resources and engineering
efforts needed for sub-metering, and undertake sub-metering efforts in these
locations. Based on observed costs and
benefits and any additional analysis conducted, the REFM organization will
assess the feasibility of moving forward with sub-metering at other facilities
and provide a recommendation on whether to move forward or not.
Recommendation 3: Employee
workstations, and information technology equipment and non-information
technology equipment and furniture that are no longer needed are removed from
the data centers.
Management’s Response: IRS management agreed with the
recommendation. The REFM organization will consult with the Director, ECC, in
the process of identifying information technology and non-information
technology based equipment as well as workstations and furniture no longer
needed and will develop a plan for removal, where appropriate.
Recommendation 4: Current and future data center space needs are identified and plans developed to consolidate or reduce excess data center space.
Management’s Response: IRS management agreed with the recommendation. The REFM organization will consult with the MITS Enterprise Operations organization, which will take the lead in the process of assessing the current state of data center space and developing a plan of action for space consolidation, where deemed appropriate. Through the Data Center Working Group or another venue, the MITS Enterprise Operations organization will develop a plan in accordance with the Office of Management and Budget Data Center Consolidation plan developed by the Department of the Treasury.
Recommendation 5: Energy audits (including an assessment of best practices)
are performed at the data centers.
Management’s Response: IRS management agreed with the
recommendation. The REFM organization will ensure that audits are completed
timely and tracked, as required by the Energy Independence and Security Act (EISA)
of 2007,[8] and include provisions to assess data centers with respect
to industry best practices.
Governance Over Environmental and Energy Efficiency
Projects Should Be Enhanced
The IRS has not yet developed adequate policies and
procedures for controlling and monitoring energy efficiency improvement
projects. Improvement projects
(including recommendations from environmental and energy reviews or audits) are
not recorded in a central location to facilitate efficient oversight. The Austin Campus
and the ECC – Martinsburg provided information on completed improvements. The ECC –
The IRS evaluates and implements recommendations and
improvement projects on a location by location basis rather than an
enterprise-wide basis. Discussions held to evaluate and select
which recommendations and improvements to implement are not documented.
In addition to Executive
Order 13423 requirements, Office of
Management and Budget Circular Number A-123, Management’s Responsibility for Internal Control dated December 21,
2004, states that management is responsible for
establishing and maintaining internal control to achieve the objectives of
effective and efficient operations, reliable financial reporting, and
compliance with applicable laws and regulations.
IRS resources could be put to better use if all recommendations and improvement projects were considered together and selected for implementation based on the highest return on investment, risk, and budget. However, without adequate controls to monitor and measure the benefits of improving data center energy efficiency, the REFM organization will not have assurance that actual costs and benefits are accurately tracked and reported, and that IRS efforts have yielded positive and measurable results. In addition, without an effective and efficient process to monitor energy efficiency costs and benefits, the IRS may not achieve the energy efficiency improvements included in Executive Orders 13423 and 13514, and Treasury Directive 75-04.
Recommendations
The Chief, Agency-Wide Shared
Services, should ensure that:
Recommendation
6: A governance
process is established to evaluate, select, and approve recommendations and projects
to implement. The process should include
a requirement to document all recommendations and projects, whether they were
selected for implementation, and the justification for selecting or not
selecting the recommendation and project.
Management’s Response: IRS management agreed with the recommendation. The REFM organization
will design a governance plan for evaluating and implementing projects, and
tracking them in accordance with the corrective action in
Recommendation 7. The plan will specify
the individuals or board that will oversee energy improvement projects and will
delineate project selection procedures and criteria.
Recommendation 7: A database of all recommendations and projects to improve energy efficiency and the environment is established. For each recommendation, the database should include information such as whether the recommendation was selected for implementation, projected and actual cost to implement and, if applicable, implementation start and completion dates, and projected and actual cost savings, as well as other benefits realized from the implementation.
Management’s Response: IRS management agreed with the
recommendation. All
energy conservation measures or improvement projects identified by a formal
audit performed as required by the EISA will be tracked in the Department of
Energy EISA Compliance Tracking System database, as mandated by the Act. The Compliance Tracking System database is
projected to be available by June 2010.
For projects not identified through a formal audit process, the REFM organization
will design a Microsoft Excel-based tracking tool. Information to be tracked will include
project listings, justifications for selecting or not selecting projects, start
and completion dates, projected and actual costs, and savings or other benefits
realized from implementation.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective was to determine whether the REFM organization has established and implemented
an effective environmental and energy program to ensure the improvement of data
center[9] energy
efficiency. To accomplish our objective, we:
I. Determined whether the environmental and energy program to improve the energy efficiency at data centers within IRS facilities included adequate planning and coordination.
A. Determined whether the environmental and energy program that has been developed and implemented is meeting the organizational goals and objectives.
B. Researched the Internet to identify best practices used by industry to improve the energy efficiency of data centers and identify any relevant best practices or lessons learned that could be used in improving the IRS’ environmental and energy program.
C. Identified Governmental Agency best practices that have been successfully implemented to reduce the energy consumption within their data centers.
D. Reviewed audit reports from other Inspector General organizations and reports from the Department of Energy and the Environmental Protection Agency to identify recommendations and best practices that can be successfully implemented at the IRS data centers.
II. Determined whether the environmental and energy program to improve the energy efficiency at the data centers is being effectively implemented.
A. Discussed with REFM organization management the extent of the improvements to data centers and/or IRS facilities to make them energy efficient.
B. Obtained and reviewed REFM organization reports documenting projects planned or in progress for each data center and/or IRS facility energy efficiency.
C. Obtained for each Environmental Management System conforming data center and/or IRS facility any initial or subsequent audits performed and verified that the data center and/or IRS facility properly met the requirements.
D. Obtained for each applicable data center and/or IRS facility any environmental compliance, energy, or building audits performed to determine the issues identified that require action.
E. Reviewed any REFM organization reports documenting for each data center and/or IRS facility energy efficiency projects planned or in progress.
III. Determined whether efforts to improve energy efficiency at the data centers are being effectively monitored, measured, and reported.
A. Reviewed the REFM organization action logs or other control logs (Environmental Management System Status Report) used to list the recommendations and corrective actions for energy efficiency improvements planned for each of the data centers and/or IRS facilities to determine whether recommendations from the different audits were included in the logs.
B. Reviewed annual/quarterly reports and scorecards prepared by the REFM organization to report its overall status of the data center and/or IRS facility energy efficiency program. We verified a limited portion of the Environmental Stewardship Scorecard for accuracy.
C. Reviewed the REFM organization energy efficiency action plans and the reports summarizing the completed or in process energy efficiency improvements to determine whether the improvements will increase energy efficiency at the affected data centers and/or IRS facilities.
1. Based on the research performed on industry and Government best practices regarding data center energy efficiency, identified energy efficiency improvements that the IRS could consider to implement in the data centers.
2. Performed visitations to selected IRS facilities with the following attributes: 1) facilities with a computing or data center present (ECC – Martinsburg and ECC – Memphis), 2) an IRS facility that has received the most significant energy consumption improvements or leads in energy efficiency (Austin Campus), and 3) an IRS facility performing large or significant computer operations (Ogden Campus).
3. Applied the industry and Government best practices tools and calculated the potential energy savings if energy efficient improvements were directly made to the data centers.
Internal
controls methodology
Internal controls relate to management’s plans, methods, and procedures used to meet their mission, goals, and objectives. Internal controls include the processes and procedures for planning, organizing, directing, and controlling program operations. They include the systems for measuring, reporting, and monitoring program performance. We determined the following internal controls were relevant to our audit objective: REFM organization’s policies and procedures for implementing an effective environmental and energy program to ensure the improvement of data center energy efficiency. We evaluated these controls by interviewing management, reviewing data center best practices, reviewing policies and procedures such as the Internal Revenue Manuals and Executive Orders, and performing site inspections of the facilities selected for this audit.
Appendix II
Major Contributors to This Report
Alan
R. Duncan, Assistant Inspector General for Audit (Security and Information
Technology Services)
Margaret
E. Begg, Acting Assistant Inspector General for Audit (Security and Information
Technology Services)
Scott
Macfarlane, Director
Danny
Verneuille, Audit Manager
Mike Garcia,
Senior Auditor
Beverly
Tamanaha, Senior Auditor
Tina
Wong, Senior Auditor
Ashley
Guthrie, Auditor
Anthony
Morrison, Program Analyst
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Chief Technology Officer OS:CTO
Director, Real Estate and Facilities Management OS:A:RE
Director, Stakeholder Management OS:CTO:SM
Chief
Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis RAS:O
Office of Internal Control OS:CFO:CPIC:IC
Audit Liaisons:
Chief, Agency-Wide Shared Services OS:A
Director, Program Oversight Office
OS:CTO:SM:
Appendix IV
This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration. This benefit will be incorporated into our Semiannual Report to Congress.
Type and Value of Outcome Measure:
· Cost Savings – Funds Put to Better Use – Potential; $793,218 per year; $3,172,872 over a 4-year period (see page 5).
Methodology Used to Measure the Reported Benefit:
Industry guidelines suggest that
implementing best practices to improve airflow management can reduce data
center[10] cooling by up to 50 percent. From the Tennessee Valley Authority
energy report, Detailed Energy Study
Report, dated March 20, 2009, for the Memphis Campus, we determined that 31
percent of the electricity cost was attributable to airflow management in the
data center. REFM organization
management provided the 2009 electricity costs for ECC –
Figure
1: Estimated Potential Savings From
Implementing Best
Practices at Two of the ECCs
Row |
|
|
Martinsburg |
Sum of the Prior Columns |
1 |
Fiscal Year 2009 total electricity cost for the ECC
campus. |
$3,060,291 |
$2,057,243 |
$5,117,534 |
2 |
Percentage of the electricity cost attributable to the
data center
(i.e., cooling, computer room air conditioner fans, uninterruptible power
supply, and batteries). |
31% |
31%* |
— |
3 |
Estimated Fiscal Year 2009 data center electricity cost.
(Row 1 x Row 2) |
$948,690 |
$637,745 |
$1,586,435 |
4 |
Potential savings based
on industry estimate from implementing airflow management best practices. |
Up to 50% |
Up to 50% |
— |
5 |
Total potential annual decrease in energy usage by
improving airflow management, before costs to implement best practices. (Row 3 x Row 4) |
$474,345 |
$318,873 |
$793,218 |
6 |
Costs to implement best practices (The information needed
to compute the costs was not available.) |
Unknown |
Unknown |
Unknown |
7 |
Potential 4-year decrease in energy usage cost before
costs to implement best practices.
(Row 5 x 4 years) |
$1,897,380 |
$1,275,492 |
$3,172,872 |
Source: Our calculations based on IRS utility costs;
the
Sourcebook;[11]
and “Is your data center running out of power or cooling?”[12]
·
We did not have the percentage of electricity costs
attributable to the ECC – Martinsburg data center; therefore, we used the ECC –
Appendix V
Locations of
Facilities Selected to Implement an Environmental Management System
Appendix VI
Air Handlers |
A device
used to condition and circulate air as part of a heating, ventilating, and
air-conditioning system. |
Blanking Panel |
A device used to fill empty vertical spaces in
the rack to maintain proper airflow. |
Brush Grommet |
A device
used to prevent cold air from escaping through cable holes in raised
floor environments. By blocking bypass
airflow, brush grommets significantly increase the cooling effectiveness of
the data center because cold supply no longer mixes with hot exhaust air
through unsealed cable holes. |
Campus |
The data
processing arm of the IRS. The
campuses process paper and electronic submissions, correct errors, and
forward data to the Computing Centers for analysis and posting to taxpayer
accounts. |
Chiller Systems |
A cooling system used in a data center to remove heat
from one element and deposit it into another element. Chillers are used to cool the water used in
their heating, ventilation, and air-conditioning units. Round-the-clock operation of chillers is
crucial to the data center, given the considerable heat produced by many
servers operating in close proximity to one another. Without them, temperatures would quickly
rise to levels that would corrupt mission-critical data and destroy hardware. |
|
A building or portion of a building whose primary
function is to house a computer room and its support areas; data centers
typically contain high-end servers and storage products with mission-critical
functions. |
|
A measure used to determine the energy efficiency of a
data center. The measure, which is
expressed as a percentage, is calculated by dividing information technology
equipment power by total facility power. |
|
Supports
tax processing and information management through a data processing and
telecommunications infrastructure. |
|
An
application that provides training, administration, and training resource
management (instructors, classroom, and all web resources for learning). |
Rack |
An enclosed cabinet housing computer equipment. |
Sub-metering
(intelligent monitoring) |
Allows power
consumption to be monitored at every level in the power distribution
hierarchy, giving visibility into current operational parameters and the
impact of other energy-saving steps. |
Uninterruptible
Power Supply |
An
emergency power source to provide power to data center facilities until
emergency generators come on. |
White Paper |
In information
technology, a white paper is often a paper written by a lead product designer
to explain the philosophy and operation of a product in a marketplace or
technology context. |
Appendix VII
Management’s Response to the Draft Report
DEPARTMENT OF THE TREASURY
INTERNAL REVENUE SERV1CE
WASH1NIGTON, D. C. 20224
CHIEF
AGENCY-WIDE
SHARED
SERVICES
APR 12, 2010
MEMORANDUM FOR MICHAEL R. PHILLIPS
DEPUTY INSPECTOR GENERAL FOR AUDIT
FROM: David A. Grant /s/ David A. Grant
Chief, Agency-Wide Shared Services
SUBJECT: Draft Audit Report - Implementing Best Practices
and Additional Controls Carl Improve
Thank you for the opportunity to comment on the draft audit report. The overall objective of this audit was to assess whether the Internal Revenue Service has established an effective energy and environmental program to ensure improvements in energy efficiency at IRS data centers.
We appreciate that the Treasury Inspector General for Tax
Administration recognized our accomplishments related to energy and
environmental management of our facilities. Two major IRS facilities have
implemented an Environmental Management System (EMS), and we are planning to
implement an
Since 2008, the IRS has been engaged in a data center consolidation effort to right-size computing environments in new buildings, as well in reconfiguring existing space to reduce the footprint of computer rooms enterprise-wide. Along with this effort, newer technology being deployed also requires less space, thus further reducing the overall footprint. In addition, we are working with Treasury on the Office of Management and Budget Data Center Consolidation effort. As we move forward with these efforts, we will repurpose computer room space to improve overall efficiency. While there are many challenges ahead for the IRS energy program, we continue to meet all relevant goals as required by executive orders and legislation.
Al though we have taken steps 10 promote energy efficiency at IRS data centers, we recognize that the IRS has potential for greater energy savings and improved efficiency, and our corrective actions are attached. We will convene a working group to oversee many of the recommended improvements to meet IRS' goals for energy reduction. We plan to assess which best practices are most appropriate for IRS data centers, and to implement and track improvement projects in a systematic manner. We believe the implementation of recommended best practices will lead to improved energy efficiency at IRS data centers.
Thank you for your guidance in this effort. If you have any questions, please contact me at (202) 622-7500, or a member of your staff may contact Stuart Burns, Director, Real Estate and Facilities Management, at (202) 435-6300. For matters concerning audit follow-up, please contact Larry Pugh, Office of Strategy and Finance, Agency-Wide Shared Services, at (202) 622-4541.
Attachment
RECOMMENDATION 1:
Policies and procedures are established to evaluate and determine which best practices to implement to improve data center energy efficiency.
CORRECTIVE ACTION:
Agency-Wide Shared Services (AWSS) will convene a Data Center Working Group co-chaired by Real Estate and Facilities Management (REFM) and Modernization & Information Technology Services (MITS), to assess best practices to implement in IRS data centers, communicate best practices with campus contacts, establish methods for tracking progress toward implementation, and research new opportunities for improving data center energy efficiency.
IMPLEMENTATION DATE:
June 30, 2010 - Convene Working Group
June 30, 2011 - Document best practices, make recommendations, and document progress.
RESPONSIBLE OFFICIAL:
Director, Real Estate and Facilities Management, Agency-Wide Shared Services
Recommendation 2:
Information technology equipment energy use is measured (Le. sub-metered) in order to determine the energy efficiency and savings from implementing energy improvements.
CORRECTIVE ACTION:
Due to the complexity and scale of effort needed for data center sub-metering, REFM will identify one campus and one computing center, perform an analysis examining resources and engineering effort s needel1 for sub-metering, and undertake sub-metering efforts at these locations, REFM will analyze costs, benefits and other considerations surrounding sub-metering at the specified locations. Based on observed costs and benefits, and any additional analysis conducted, REFM will assess the feasibility of moving forward will, sub-metering at other facilities and provide a recommendation as to moving forward or not. REFM will subsequently provide the Treasury Inspector General for Tax Administration (TIGTA) with a report and recommendations detailing the analysis and findings.
IMPLEMENTATION DATE:
October 31, 2011 - Study provided to TIGTA
RESPONSIBLE OFFICIAL:
Director, Real Estate and Facilities Management, Agency-Wide Shared Services
RECOMMENDATION 3:
Employee workstations and information technology equipment and non-information technology equipment and furniture that are no longer needed are removed from the data centers.
CORRECTIVE ACTION:
REFM will consult with the Director, Enterprise Computing Center (ECG) within MITS in the process of identifying information technology and non-information technology based equipment, workstations and furniture no longer needed, and will develop a plan for removal, where appropriate.
IMPLEMENTATION DATE:
November 30. 2010 - Development of equipmef1t1fumiture removal plan
June 30, 2011 - Implementation of removal plan
RESPONSIBLE OFFICIAL:
Director, Real Estate and Facilities Management, Agency-Wide Shared Services
RECOMMENDATION
4:
Current and future data center space needs are identified and plans developed to consolidate or reduce excess data center space.
CORRECTIVE ACTION:
REFM will consult with MITS Enterprise Operations, who will take the lead in the process of assessing the current state of data center space and developing a plan of action for space consolidation, where deemed appropriate. Through the Data Center Working Group or another venue, MITS Enterprise Operations will develop a plan in accordance with the Office of Management and Budget Data Center Consolidation plan developed by Treasury.
IMPLEMENTATION DATE:
March 1, 2012
RESPONSIBLE OFFICIAL:
Associate
Chief Information Officer,
RECOMMENDATION 5:
Energy audits (including an assessment of best practices) are performed at the data centers.
CORRECTIVE ACTION:
REFM will ensure that audits are completed timely and tracked, as required by the Energy Independence and Security Act of 2007 (EISA 2007), and include provisions to assess data centers with regard to industry best practices.
IMPLEMENTATION DATE:
October 31, 2014 (25 percent each year, on a 4·yearcycle, as required by EISA 2007)
RESPONSIBLE OFFICIAL:
Director, Real Estate and Facilities Management, Agency-Wide Shared Services
RECOMMENDATION 6:
Governance process is established to evaluate, select, and approve recommendations and projects to implement. The process should include a requirement to document all recommendations and projects, whether they were selected for implementation, and the justification for selecting or not selecting the recommendation and project.
CORRECTIVE ACTION:
REFM will design a governance plan for evaluating find implementing projects, and tracking them in accordance with the corrective action in Recommendation 7. The plan will specify the individuals or board that will oversee energy improvement projects and will delineate project selection procedures and criteria.
IMPLEMENTATION DATE:
September 30, 2010 - Development of governance plan
March 31, 2011 - Demonstration of plan implementation
RESPONSIBLE OFFICIAL:
Director, Real Estate and Facilities Management, Agency-Wide Shared Services
RECOMMENDATION 7:
Database of all recommendations and projects to improve energy efficiency and the environment is established, For each recommendation, the database should include information such as whether the recommendation was selected for implementation, projected and actual cost to implement and, if applicable, implementation start and completion dates, and projected and actual cost savings, as well as other benefits realized from the implementation.
CORRECTIVE ACTION:
All energy conservation measures or improvement projects identified by a formal .audit performed as required by EISA 2007 will be tracked in the Department of Energy EISA 432 Compliance Tracking System (CTS) database, as mandated by the Act. The CTS database, which is being created as a result of EISA 2007, is projected to be available by June, 2010. For projects not identified through a formal audit process, REFM will design an MS Excel-based tracking tool. Information to be tracked will include project listings, justifications for selecting or not selecting projects, start and completion dates, projected and actual costs, and savings or other benefits realized from implementation.
IMPLEMENTATION DATE:
September 30, 2010 - Development of draft Excel-based tool
March 31, 2011 - Implementation project tracking through Excel-based tool
RESPONSIBLE OFFICIAL:
Director, Real Estate and Facilities Management, Agency-Wide Shared Services
AUDIT RESPONSE POINT (from page 4, paragraph 21:
TIGTA calculated that by implementing better airflow management practices at the ECC-Memphis and the ECC-Martinsburg, IRS could save $3,172,872 over the next four years. TIGTA plans to include this in the Semiannual Report to Congress, and asked for IRS concurrence or non-concurrence with this assessed benefit.
IRS RESPONSE:
While airflow management improvements could certainly lead to energy-related savings for IRS, we believe the estimated figure in the outcome measure may overstate savings that are likely to accrue from such improvements_ Specifically, the figure does not factor in implementation costs, does not take into consideration best practices already implemented, and is based on 50% energy savings, which represent the upper limit of potentially achievable savings, As a result, we are not comfortable with the current language used to discuss the outcome measure, but we would concur with the outcome measure with the following language revisions:
"Implementing better airflow management practices at the ECC-Memphis and the ECC-Martinsburg may require a significant investment on the part of IRS. As part of the corrective actions resulting from this audit, IRS will analyze data center best practices. analyze sub-metering costs and benefits and implement sub-metering at two locations, and examine return on investment for improvement projects undertaken, If, based on the results from these analyses and the assessed costs and benefits of potential improvement projects, IRS opts to invest in the recommended airflow management initiatives, these improvements could potentially lead to gross savings for IRS of up to $3,172,872 over the next four years. However, this figure does not factor in implementation costs IRS would bear in executing the improvement initiatives."
[1] See Appendix VI for a glossary of terms.
[2] Greenberg et al. 2006.
[3] Report to Congress on Server and Data Center Energy Efficiency Public Law 109-431, dated August 2, 2007, U.S. Environmental Protection Agency Energy Star Program.
[4] See Appendix VI for a glossary of terms.
[5] Spears, Ed, “Is your data center running out of power or cooling?” dated May 30, 2009, Eaton Corporation, http://powerquality.eaton.com/about-us/news-events/whitepapers/data-center-power-cooling.asp.
[6] See Appendix V for the locations of the 11 facilities selected to implement an Environmental Management System.
[7] See Appendix IV for the calculations.
[8] Pub. L. 110-140, 121 Stat. 1492.
[9] See Appendix VI for a glossary of terms.
[10] See Appendix VI for a glossary of terms.
[11] Pacific Gas and Electric Company, dated January 2006.
[12]
Spears, Ed, dated May 30, 2009, Eaton
Corporation, http://powerquality.eaton.com/about-us/news-events/whitepapers/data-center-power-cooling.asp.