TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

Implementing Best Practices and Additional Controls Can Improve Data Center Energy Efficiency and the Environmental and Energy Program

May 7, 2010

Reference Number: 2010-20-044

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

Phone Number | 202-622-6500

Email Address | inquiries@tigta.treas.gov

Web Site | http://www.tigta.gov

HIGHLIGHTS

IMPLEMENTING BEST PRACTICES AND ADDITIONAL CONTROLS CAN IMPROVE DATA CENTER ENERGY EFFICIENCY AND THE ENVIRONMENTAL AND ENERGY PROGRAM

Highlights

Final Report issued on May 7, 2010

Highlights of Reference Number: 2010-20-044 to the Internal Revenue Service Chief, Agency-Wide Shared Services.

IMPACT ON TAXPAYERS

Federal agencies have been directed to improve their energy efficiency and reduce greenhouse gas emissions. Internal Revenue Service (IRS) management indicated the agency’s overall energy program has consistently met standards set forth in all pertinent Executive Orders and legislative mandates. However, while data centers typically consume more energy than other types of buildings, the IRS has not yet fully implemented most data center best practices. As a result, the IRS has not maximized the energy efficiency and use of taxpayer funds resulting from decreased energy consumption in its data centers.

WHY TIGTA DID THE AUDIT

This audit was initiated to determine whether the Real Estate and Facilities Management organization has established and implemented an effective environmental and energy program to ensure the improvement of data center energy efficiency.

In January 2007, Executive Order 13423, Strengthening Federal Environmental, Energy, and Transportation Management, was enacted to strengthen the environmental, energy, and transportation management of Federal agencies.

WHAT TIGTA FOUND

The Real Estate and Facilities Management organization has implemented programs and processes to improve energy efficiency. For example, some facilities fully implemented an Environmental Management System and/or were awarded the ENERGY STAR^® designation. In addition, the Real Estate and Facilities Management organization has a training program to ensure employees requiring training have access to applicable courses and completion of the training is documented.

However, the IRS does not have policies and procedures for improving energy efficiency in the data centers or for implementing data center energy efficiency best practices. We estimate that the IRS could potentially realize savings of $3,172,872 over 4 years, at 2 sites, by implementing best practices to improve airflow management.

The IRS also had not developed adequate policies and procedures for controlling and monitoring energy efficiency improvement projects.

WHAT TIGTA RECOMMENDED

TIGTA recommended that the Chief, Agency-Wide Shared Services, work with the Chief Technology Officer to ensure 1) policies and procedures are established to evaluate and determine which best practices to implement, 2) information technology equipment energy use is measured (i.e., sub-metered), 3) employee workstations, equipment, and furniture that are no longer needed are removed from the data centers, 4) current and future data center space needs are identified and plans developed to consolidate or reduce excess data center space, and 5) energy audits (including an assessment of best practices) are performed at the data centers. In addition, the Chief, Agency-Wide Shared Services, should ensure 6) a governance process is established and 7) a database of all recommendations and projects is established.

In their response to the report, IRS management agreed with all of the recommendations but questioned the outcome measure contained in the audit report. The IRS plans to take many corrective actions. After considering the IRS’ comments, TIGTA maintains that the outcome measure in the report is valid.

May 7, 2010

MEMORANDUM FOR CHIEF, AGENCY-WIDE SHARED SERVICES

FROM: Michael R. Phillips /s/ Michael R. Phillips

Deputy Inspector General for Audit

SUBJECT: Final Audit Report – Implementing Best Practices and Additional Controls Can Improve Data Center Energy Efficiency and the Environmental and Energy Program (Audit # 200920025)

This report presents the results of our review to determine whether the Real Estate and Facilities Management organization has established and implemented an effective environmental and energy program to ensure the improvement of data center[1] energy efficiency. This review was included in our Fiscal Year 2009 Annual Audit Plan and addresses the major management challenge of Modernization of the Internal Revenue Service (IRS).

Management’s complete response to the draft report is included as Appendix VII.

Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services), at (202) 622-5894.

Table of Contents

Background

Results of Review

Programs and Processes Have Been Implemented to Improve Facility Energy Efficiency

Data Center Energy Efficiency Best Practices Need to Be Implemented

Recommendation 1:

Recommendations 2 through 5:

Governance Over Environmental and Energy Efficiency Projects Should Be Enhanced

Recommendation 6:

Recommendation 7:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measure

Appendix V – Locations of Facilities Selected to Implement an Environmental Management System

Appendix VI – Glossary of Terms

Appendix VII – Management’s Response to the Draft Report

Abbreviations

AWSS	Agency-Wide Shared Services
DCiE	Data Center Infrastructure Efficiency
ECC	Enterprise Computing Center
EISA	Energy Independence and Security Act
IRS	Internal Revenue Service
MITS	Modernization and Information Technology Services
REFM	Real Estate and Facilities Management

Background

A data center consists of computer hardware and software used for data processing (servers), data storage (storage equipment), and communications (network equipment). Data centers also usually contain specialized power conversion and backup equipment to maintain reliable, high-quality power, as well as environmental control equipment to maintain the proper temperature and humidity for the computer equipment. Most data centers are more energy intensive than other buildings due to the high power requirements of the computer equipment and the power and cooling infrastructure needed to support this equipment. In fact, data centers can be more than 40 times as energy intensive as conventional office buildings,[2] meaning that large data centers more closely resemble industrial facilities than commercial buildings with respect to energy use.

Federal agencies have been tasked to strengthen the environmental, energy, and transportation management of their agencies.

In August 2007, the Environmental Protection Agency reported[3] that the energy used by the nation’s servers and data centers is significant. In 2006, electricity consumed by servers and data centers (including cooling and auxiliary infrastructure) represented about 1.5 percent of national electricity use at an estimated $4.5 billion in electrical cost. This estimated level of electricity consumption is more than the electricity consumed by the nation’s color televisions and similar to the amount of electricity consumed by approximately 5.8 million average households. Federal servers and data centers alone account for approximately 10 percent of this electricity use, for a total electricity cost of about $450 million annually.

Based on 2006 efficiency trends, national energy consumption by servers and data centers could nearly double by 2011, representing a $7.4 billion annual electricity cost. Forecasts indicate that unless energy efficiency is improved beyond current trends, the Federal Government’s electricity cost for servers and data centers could be nearly $740 million annually by 2011. These estimates of data center energy use should be considered approximate because limited data are available on current data center energy use, and there is significant uncertainty about the effects of future technology trends, such as server consolidation and developments in network and storage technologies. However, these estimates and projections illustrate the magnitude of energy use in data centers and the need for effective energy-efficiency strategies.

In May 2009, a white paper report produced by the Eaton Corporation stated as much as 30 to 60 percent of the data center utility bill goes to support cooling systems, including air handlers[4] and chiller systems. Many computer room cooling systems are inefficiently deployed or not operated under recommended conditions. Poor airflow management reduces both the efficiency and capacity of computer room cooling equipment. Leaking floor tiles, excessive cable openings, poorly placed overhead supplies, and inappropriately oriented rack exhausts can reduce the usable capacity of a computer room air conditioner unit by 50 percent or more.[5]

In January 2007, President George W. Bush signed Executive Order 13423 Strengthening Federal Environmental, Energy, and Transportation Management. The purpose of this policy was to strengthen the environmental, energy, and transportation management of Federal agencies by “conducting their environmental, transportation, and energy-related activities under the law in support of their respective missions in an environmentally, economically, and fiscally sound, integrated, continuously improving, efficient, and sustainable manner.” The Executive Order established a goal to improve energy efficiency by 3 percent annually through the end of Fiscal Year 2015 or 30 percent by Fiscal Year 2015 compared to the Fiscal Year 2003 baseline year. A revised Treasury Directive 75-04, Energy Management Program, was issued in December 2008 establishing the policies and assigning responsibilities for maintaining the comprehensive energy programs within the Department of the Treasury and complying with all relevant regulations and Executive Orders.

In October 2009, President Barack Obama signed Executive Order 13514, Federal Leadership in Environmental, Energy, and Economic Performance, which stipulates that Federal agencies must increase energy efficiency, reduce greenhouse gas emissions, eliminate waste, and foster conservation and sustainability. More specifically, it mandates that agencies implement best practices for energy-efficient management of Federal data centers and servers. While previous Executive Orders pertaining to energy efficiency did not preclude agencies from instituting efficiency measures at data centers, they also did not require such action.

This review was performed at the Internal Revenue Service’s (IRS) Real Estate and Facilities Management (REFM) organization’s office in Crystal City, Virginia; the Enterprise Computing Centers (ECC) in Memphis, Tennessee, and Martinsburg, West Virginia; and the IRS Campuses in Austin, Texas, and Ogden, Utah, during the period June 2009 through January 2010. We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.

Results of Review

Programs and Processes Have Been Implemented to Improve Facility Energy Efficiency

An Environmental Management System is being implemented at 11 selected facilities[6]

An Environmental Management System is a set of processes and practices that enable an organization to increase its operating efficiency, continually improve overall environmental performance, and better manage and reduce its environmental impacts, including those environmental aspects related to energy and transportation functions. The IRS plans to integrate the Environmental Management System with existing management systems used at each facility to help achieve environmental and economic goals. An Environmental Management System is considered fully implemented when it has been formally audited by an independent qualified party, audit findings have been recognized by the appropriate level of management of the facility implementing the system, and it has been declared by an agency executive to be in conformance to requirements. Two of the facilities we reviewed (Austin Campus and ECC – Martinsburg) have fully implemented an Environmental Management System. The ECC – Memphis and Ogden Campus are scheduled to have their systems fully implemented by 2011 and 2010, respectively.

Two facilities reviewed were awarded the ENERGY STAR^® designation

ENERGY STAR is a joint program of the Environmental Protection Agency and the Department of Energy to help us all save money and protect the environment through energy efficient products and practices. An ENERGY STAR qualified facility meets strict energy performance standards set by the Environmental Protection Agency and uses less energy, is less expensive to operate, and causes fewer greenhouse gas emissions than its peers. To qualify for the ENERGY STAR designation, a building or manufacturing plant must score in the top 25 percent based on the National Energy Performance Rating System. To determine the performance of a facility, the Environmental Protection Agency compares energy use among other, similar types of facilities on a scale of 1-100. Buildings that achieve a score of 75 or higher may be eligible for the ENERGY STAR designation. The rating system accounts for differences in operating conditions, regional weather data, and other important considerations.

As a result of the efforts to improve energy efficiency, the Memphis Campus was awarded the ENERGY STAR designation in 2006 through 2009 and the Austin Campus was awarded the designation in 2005, 2007, and 2009 when it was reviewed and considered for the ENERGY STAR designation. In addition, IRS management advised us that the Andover Campus also received the ENERGY STAR designation in 2008 and 2009.

Employees completed required environmental and energy training

The IRS ensures employees requiring training related to the environmental and energy programs have access to applicable courses. The IRS identified which employees needed such training and what training courses could be completed to meet the training requirements. Completed training is documented on certificates of completion and/or the IRS Enterprise Learning Management System. In addition, contractors who performed Environmental Management System and Environmental, Safety and Health audits determined employees completed the required training.

The IRS efforts in the Environmental Management System, ENERGY STAR, and employee training areas have helped to ensure the agency complies with Executive Order 13423, Strengthening Federal Environmental, Energy, and Transportation Management, and attains its goals. For example, agencies were directed to:

• Implement environmental management systems to ensure environmental issues are addressed, including environmental aspects of energy and transportation functions.

• Improve energy efficiency of agency facilities.

• Establish environmental management training programs.

The IRS efforts will also ensure compliance with Treasury Directive 75-04, Energy Management Program, which states bureaus are responsible for establishing and maintaining comprehensive energy programs within the Department of the Treasury and complying with all relevant regulations and Executive Orders. This includes ensuring that all appropriate personnel receive training for implementing Executive Orders, including development of outreach programs that include education, training, and promotion of ENERGY STAR and other energy-efficient products for Federal purchase card users.

IRS management indicated the agency’s overall energy program has consistently met standards set forth in all pertinent Executive Orders and legislative mandates. While the IRS’ accomplishments are notable and have improved its energy efficiency, additional actions at the data centers to implement best practices will further improve the IRS’ energy efficiency and enhance governance over environmental and energy efficiency projects.

Data Center Energy Efficiency Best Practices Need to Be Implemented

The Agency-Wide Shared Services (AWSS) organization, in conjunction with the Modernization and Information Technology Services (MITS) organization, has overall responsibility to ensure that the IRS is complying with Executive Orders 13423 and 13514 and applying best practices. However, the AWSS and MITS organizations do not have a process to consider, select, and apply best practices for improving data center energy efficiency. The AWSS organization has identified Department of the Treasury Best Energy Practices, but these do not address energy efficiency in the data centers, as they were developed prior to the issuance of Executive Order 13514.

We made the following observations regarding the implementation of best practices when we visited the Austin and Ogden Campuses, ECC – Martinsburg, and ECC – Memphis. Prior to our visit to the Austin Campus, Austin Campus building management was briefed by AWSS organization management on the results obtained at the earlier sites visited. The Austin Campus building management took the initiative and implemented several corrective actions. Because of this, we cannot comment on the original state of the Austin Campus data center.

Observations of the implementation of best practices

· There were gaps between floor tiles (i.e., missing tiles) allowing hot and cold air to mix which can overwork the computer room air conditioning units. Perforated tiles were placed ad hoc around the server room, many times cooling the hot aisle (ECC – Martinsburg, ECC – Memphis, and Ogden Campus). Austin Campus building management commented it moved or replaced 50 floor tiles prior to our visit.

· Brush grommets in the raised floor to cover floor spaces/gaps and blanking panels in the server racks to cover spaces in rack-mounted equipment were not used or not used consistently to prevent air leaks (ECC – Martinsburg, ECC – Memphis, and Ogden Campus). Austin Campus building management sealed air leaks in the floor prior to our visit. Best practices state that brush grommets or other barriers and blanking panels can help optimize airflow by minimizing the mixing of hot and cold air. Figure 1 provides an illustration of the usage of brush grommets in the raised floor and blanking panels in the server racks.

Figure 1: Brush Grommets in the Raised Floor and Blanking Panels in the Server Racks

Figure 1 was removed due to its size. To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

· Servers were not effectively placed in hot/cold aisle configurations. Three sites (ECC – Martinsburg, ECC – Memphis, and Ogden Campus) had at least one server rack aligned where the hot air from the servers blew into the cold aisle. Austin Campus building management commented that it rotated one server rack to ensure proper alignment. In addition, servers at the first three locations were not consistently placed in rows (i.e., some servers were placed in an ad hoc manner around the computer room). Figure 2 provides an illustration of a best practice hot/cold aisle configuration.

Figure 2: Data Center Hot/Cold Aisle Configuration

Figure 2 was removed due to its size. To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

· There were inconsistencies in row-to-row spacing (called pitch) of the servers in the data center layout at all sites visited (Austin and Ogden Campuses, ECC – Martinsburg, and ECC – Memphis). For example, ECC – Martinsburg’s row-to-row spacing ranged from 7 tiles (14 ft.) to 2 tiles (4 ft.).

· There were no occupancy sensors to control lighting or sub-metering that monitored power consumption at the sites visited (Austin and Ogden Campuses, ECC – Martinsburg, and ECC – Memphis). However, the ECC – Martinsburg and ECC – Memphis can control their lighting with their computer systems. Best practices state that, generally, using occupancy sensors could save between 35 and 45 percent in energy consumption.

· None of the sites visited had the capability to determine how energy use is distributed among the computer equipment and support systems in the data center. Best practices state installing sub-meters at key locations provides a powerful tool for measuring the energy performance of individual systems. Providing these sub-meters with recording capability allows an agency to monitor system performance over time, providing evidence of degradations and improvements.

Other observations

· Furniture, old computer command modules, and computer and other equipment that are no longer used were stored in the data centers. For example, the Austin Campus data center had pallets of copy paper and old command modules that occupied up to an estimated one-third of the data center space. The Austin and Ogden Campuses’ data centers also contained computer equipment that needed to be discarded and/or removed. Unnecessary items/objects can block airflow causing the room to be inefficiently cooled. The items can also hinder plans to consolidate the computer rooms.

· The ECC – Martinsburg and the ECC – Memphis had an abundance of unused space in their data centers. For example, the ECC – Martinsburg has three different areas for servers but only one-half of the space in each area was being used. The unused space results in unnecessary cooling expenses which could be reduced by consolidating the computer rooms and/or partitioning the unused space. Also, in April 2006, an energy audit at the ECC – Martinsburg, performed by a contractor, determined the data center has 10 times the required air conditioning needed to cool the current environment. The audit report stated that some computer room air conditioners could be shut off and restarted as equipment is added, saving fan power and maintenance. The Austin Campus and Ogden Campus will also have an abundance of unused space once the unused computer and other equipment are removed from the data centers.

· There were occupied employee workstations (cubicles) in the data center. The ECC – Martinsburg had multiple occupied workstations surrounded by servers. ECC – Memphis had one occupied workstation in the data center.

While this audit focused largely on cooling and airflow management, IRS management indicated that the AWSS organization is also working to improve the power conversion and distribution systems supporting the data centers, by right-sizing and using more energy-efficient technology in their Uninterruptible Power Supply systems.

Executive Order 13423, dated January 24, 2007, states that Federal agencies conduct their environmental, transportation, and energy-related activities in an environmentally, economically, and fiscally sound, continuously improving, efficient, and sustainable manner. Executive Order 13514, dated October 5, 2009, states the agency head shall consider reductions associated with implementing best management practices for energy-efficient management of servers and Federal data centers.

The IRS does not have policies and procedures for improving energy efficiency in the data centers or for implementing data center energy efficiency best practices. Not fully implementing applicable best practices or other data center improvements affects the IRS’ ability to minimize energy consumption and costs, resulting in the inefficient use of resources and taxpayer funds not being most effectively used. In addition, without an effective and efficient process to implement best practices, the IRS may not fully comply with Executive Order 13514 to improve the energy-efficient management of servers and Federal data centers.

Industry guidelines suggest that implementing best practices to improve airflow management can reduce data center cooling by up to 50 percent. In addition to the previously discussed airflow management best practices, another strategy to improve airflow management is using flexible clear plastic barriers, such as plastic supermarket refrigeration covers, to seal the space between the tops of the server racks and the ceiling or air return location, thereby containing heat produced by equipment.

Fiscal Year 2009 electricity costs for the ECC – Memphis and the ECC – Martinsburg (Main and Annex buildings) facilities were $3,060,291 and $2,057,243, respectively. The Tennessee Valley Authority Detailed Energy Study Report, dated March 20, 2009, for the Memphis Campus shows that 31 percent of the total electrical usage was attributed to the data center (excluding lights and data processing). Based on the electricity costs data for the ECC – Martinsburg and ECC – Memphis, we estimate that the IRS could potentially realize savings of $793,218 per year and $3,172,872 over 4 years for both sites if data center best practices were implemented at the 2 facilities. The calculations[7] of estimated potential savings do not factor in implementation costs, since the potential cost estimates have not yet been calculated due to the complexity and diversity of potential improvement initiatives. In addition, the potential for energy savings is much higher because our estimates do not include the data centers at the other nine IRS campuses, for which energy consumption information is not available. The implementation of best practices at these additional sites could result in significant additional cost savings to the IRS.

In addition, the Department of Energy provides an online software tool (Data Center Energy Profiler) to help industries worldwide identify how energy is being purchased and consumed by data center(s), and also identify potential energy and cost savings. The Data Center Energy Profiler checklist provides a complete list of questions for developing a profile. After entering the required information, the user is provided with a customized, printable report that shows the details of energy purchases for the data center, how energy is consumed by the data center, potential cost and energy savings, comparison of the data center energy utilization versus other data centers, and a list of next steps to follow to start saving energy.

We asked IRS management at the four sites in our review to complete a Data Center Energy Profiler case for the respective sites. Based on the information entered, the software tool generates the Data Center Infrastructure Efficiency (DCiE) metric to indicate the current efficiency rate and the potential estimated efficiency rate if best practices are implemented. The DCiE metric has become a commonly used metric for data center efficiency. Figure 3 provides the current and potential DCiE metric for the four sites visited.

Figure 3: Potential DCiE Improvement

Site	Current Site Energy Use-DCiE	Potential Site Energy Use- DCiE	Efficiency Measure Change	Percentage Change
ECC – Memphis	0.66	0.88	0.22	33.33%
ECC – Martinsburg	0.71	0.90	0.19	26.76%
Ogden Campus	0.72	0.90	0.18	25.00%
Austin Campus	0.79	0.85	0.06	7.59%

Source: Data Center Energy Profiler tool assessments completed by the IRS.

Recommendations

The Chief, AWSS, should work with the Chief Technology Officer to ensure:

Recommendation 1: Policies and procedures are established to evaluate and determine which best practices to implement to improve data center energy efficiency.

Management’s Response: IRS management agreed with the recommendation. The AWSS organization will convene a Data Center Working Group, co-chaired by the REFM and MITS organizations, to assess best practices to implement in IRS data centers, communicate best practices with campus contacts, establish methods for tracking progress toward implementation, and research new opportunities for improving data center energy efficiency.

Office of Audit Comment: While management agreed with this recommendation and is planning to take several actions to improve data center energy efficiency, they are not comfortable with the current language used to discuss the outcome measure “Cost Savings – Funds Put to Better Use – Potential; $793,218 per year; $3,172,872 over a 4-year period.” Management stated that the figure does not factor in implementation costs, does not take into consideration best practices already implemented, and is based on 50 percent energy savings, which represent the upper limit of potentially achievable savings.

The potential savings calculations are based on our observations of the implementation of best practices and the best information available. The calculations do not factor in implementation costs because the cost estimates were not available. In addition, the potential for energy savings is much higher because our estimates do not include the data centers at the other nine IRS campuses, for which energy consumption information is not available. We believe the reported outcome measure is valid.

Recommendation 2: Information technology equipment energy use is measured (i.e., sub-metered) in order to determine the energy efficiency and savings from implementing energy improvements.

Management’s Response: IRS management agreed with the recommendation. The REFM organization will identify one campus and one computing center, perform an analysis examining resources and engineering efforts needed for sub-metering, and undertake sub-metering efforts in these locations. Based on observed costs and benefits and any additional analysis conducted, the REFM organization will assess the feasibility of moving forward with sub-metering at other facilities and provide a recommendation on whether to move forward or not.

Recommendation 3: Employee workstations, and information technology equipment and non-information technology equipment and furniture that are no longer needed are removed from the data centers.

Management’s Response: IRS management agreed with the recommendation. The REFM organization will consult with the Director, ECC, in the process of identifying information technology and non-information technology based equipment as well as workstations and furniture no longer needed and will develop a plan for removal, where appropriate.

Recommendation 4: Current and future data center space needs are identified and plans developed to consolidate or reduce excess data center space.

Management’s Response: IRS management agreed with the recommendation. The REFM organization will consult with the MITS Enterprise Operations organization, which will take the lead in the process of assessing the current state of data center space and developing a plan of action for space consolidation, where deemed appropriate. Through the Data Center Working Group or another venue, the MITS Enterprise Operations organization will develop a plan in accordance with the Office of Management and Budget Data Center Consolidation plan developed by the Department of the Treasury.

Recommendation 5: Energy audits (including an assessment of best practices) are performed at the data centers.

Management’s Response: IRS management agreed with the recommendation. The REFM organization will ensure that audits are completed timely and tracked, as required by the Energy Independence and Security Act (EISA) of 2007,[8] and include provisions to assess data centers with respect to industry best practices.

Governance Over Environmental and Energy Efficiency Projects Should Be Enhanced

The IRS has not yet developed adequate policies and procedures for controlling and monitoring energy efficiency improvement projects. Improvement projects (including recommendations from environmental and energy reviews or audits) are not recorded in a central location to facilitate efficient oversight. The Austin Campus and the ECC – Martinsburg provided information on completed improvements. The ECC – Memphis and Ogden Campus could not provide a list of improvements. In addition, two IRS Headquarters program owners are the only individuals aware of all energy efficiency projects.

The IRS evaluates and implements recommendations and improvement projects on a location by location basis rather than an enterprise-wide basis. Discussions held to evaluate and select which recommendations and improvements to implement are not documented.

In addition to Executive Order 13423 requirements, Office of Management and Budget Circular Number A-123, Management’s Responsibility for Internal Control dated December 21, 2004, states that management is responsible for establishing and maintaining internal control to achieve the objectives of effective and efficient operations, reliable financial reporting, and compliance with applicable laws and regulations.

IRS resources could be put to better use if all recommendations and improvement projects were considered together and selected for implementation based on the highest return on investment, risk, and budget. However, without adequate controls to monitor and measure the benefits of improving data center energy efficiency, the REFM organization will not have assurance that actual costs and benefits are accurately tracked and reported, and that IRS efforts have yielded positive and measurable results. In addition, without an effective and efficient process to monitor energy efficiency costs and benefits, the IRS may not achieve the energy efficiency improvements included in Executive Orders 13423 and 13514, and Treasury Directive 75-04.

Recommendations

The Chief, Agency-Wide Shared Services, should ensure that:

Recommendation 6: A governance process is established to evaluate, select, and approve recommendations and projects to implement. The process should include a requirement to document all recommendations and projects, whether they were selected for implementation, and the justification for selecting or not selecting the recommendation and project.

Management’s Response: IRS management agreed with the recommendation. The REFM organization will design a governance plan for evaluating and implementing projects, and tracking them in accordance with the corrective action in
Recommendation 7. The plan will specify the individuals or board that will oversee energy improvement projects and will delineate project selection procedures and criteria.

Recommendation 7: A database of all recommendations and projects to improve energy efficiency and the environment is established. For each recommendation, the database should include information such as whether the recommendation was selected for implementation, projected and actual cost to implement and, if applicable, implementation start and completion dates, and projected and actual cost savings, as well as other benefits realized from the implementation.

Management’s Response: IRS management agreed with the recommendation. All energy conservation measures or improvement projects identified by a formal audit performed as required by the EISA will be tracked in the Department of Energy EISA Compliance Tracking System database, as mandated by the Act. The Compliance Tracking System database is projected to be available by June 2010. For projects not identified through a formal audit process, the REFM organization will design a Microsoft Excel-based tracking tool. Information to be tracked will include project listings, justifications for selecting or not selecting projects, start and completion dates, projected and actual costs, and savings or other benefits realized from implementation.

Appendix I

Detailed Objective, Scope, and Methodology

The overall objective was to determine whether the REFM organization has established and implemented an effective environmental and energy program to ensure the improvement of data center[9] energy efficiency. To accomplish our objective, we:

I. Determined whether the environmental and energy program to improve the energy efficiency at data centers within IRS facilities included adequate planning and coordination.

A. Determined whether the environmental and energy program that has been developed and implemented is meeting the organizational goals and objectives.

B. Researched the Internet to identify best practices used by industry to improve the energy efficiency of data centers and identify any relevant best practices or lessons learned that could be used in improving the IRS’ environmental and energy program.

C. Identified Governmental Agency best practices that have been successfully implemented to reduce the energy consumption within their data centers.

D. Reviewed audit reports from other Inspector General organizations and reports from the Department of Energy and the Environmental Protection Agency to identify recommendations and best practices that can be successfully implemented at the IRS data centers.

II. Determined whether the environmental and energy program to improve the energy efficiency at the data centers is being effectively implemented.

A. Discussed with REFM organization management the extent of the improvements to data centers and/or IRS facilities to make them energy efficient.

B. Obtained and reviewed REFM organization reports documenting projects planned or in progress for each data center and/or IRS facility energy efficiency.

C. Obtained for each Environmental Management System conforming data center and/or IRS facility any initial or subsequent audits performed and verified that the data center and/or IRS facility properly met the requirements.

D. Obtained for each applicable data center and/or IRS facility any environmental compliance, energy, or building audits performed to determine the issues identified that require action.

E. Reviewed any REFM organization reports documenting for each data center and/or IRS facility energy efficiency projects planned or in progress.

III. Determined whether efforts to improve energy efficiency at the data centers are being effectively monitored, measured, and reported.

A. Reviewed the REFM organization action logs or other control logs (Environmental Management System Status Report) used to list the recommendations and corrective actions for energy efficiency improvements planned for each of the data centers and/or IRS facilities to determine whether recommendations from the different audits were included in the logs.

B. Reviewed annual/quarterly reports and scorecards prepared by the REFM organization to report its overall status of the data center and/or IRS facility energy efficiency program. We verified a limited portion of the Environmental Stewardship Scorecard for accuracy.

C. Reviewed the REFM organization energy efficiency action plans and the reports summarizing the completed or in process energy efficiency improvements to determine whether the improvements will increase energy efficiency at the affected data centers and/or IRS facilities.

1. Based on the research performed on industry and Government best practices regarding data center energy efficiency, identified energy efficiency improvements that the IRS could consider to implement in the data centers.

2. Performed visitations to selected IRS facilities with the following attributes: 1) facilities with a computing or data center present (ECC – Martinsburg and ECC – Memphis), 2) an IRS facility that has received the most significant energy consumption improvements or leads in energy efficiency (Austin Campus), and 3) an IRS facility performing large or significant computer operations (Ogden Campus).

3. Applied the industry and Government best practices tools and calculated the potential energy savings if energy efficient improvements were directly made to the data centers.

Internal controls methodology

Internal controls relate to management’s plans, methods, and procedures used to meet their mission, goals, and objectives. Internal controls include the processes and procedures for planning, organizing, directing, and controlling program operations. They include the systems for measuring, reporting, and monitoring program performance. We determined the following internal controls were relevant to our audit objective: REFM organization’s policies and procedures for implementing an effective environmental and energy program to ensure the improvement of data center energy efficiency. We evaluated these controls by interviewing management, reviewing data center best practices, reviewing policies and procedures such as the Internal Revenue Manuals and Executive Orders, and performing site inspections of the facilities selected for this audit.

Appendix II

Major Contributors to This Report

Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services)

Margaret E. Begg, Acting Assistant Inspector General for Audit (Security and Information Technology Services)

Scott Macfarlane, Director

Danny Verneuille, Audit Manager

Mike Garcia, Senior Auditor

Beverly Tamanaha, Senior Auditor

Tina Wong, Senior Auditor

Ashley Guthrie, Auditor

Anthony Morrison, Program Analyst

Appendix III

Report Distribution List

Commissioner C

Office of the Commissioner – Attn: Chief of Staff C

Deputy Commissioner for Operations Support OS

Chief Technology Officer OS:CTO

Director, Real Estate and Facilities Management OS:A:RE

Director, Stakeholder Management OS:CTO:SM

Chief Counsel CC

National Taxpayer Advocate TA

Director, Office of Legislative Affairs CL:LA

Director, Office of Program Evaluation and Risk Analysis RAS:O

Office of Internal Control OS:CFO:CPIC:IC

Audit Liaisons:

Chief, Agency-Wide Shared Services OS:A

Director, Program Oversight Office OS:CTO:SM:PO

Appendix IV

Outcome Measure

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration. This benefit will be incorporated into our Semiannual Report to Congress.

Type and Value of Outcome Measure:

· Cost Savings – Funds Put to Better Use – Potential; $793,218 per year; $3,172,872 over a 4-year period (see page 5).

Methodology Used to Measure the Reported Benefit:

Industry guidelines suggest that implementing best practices to improve airflow management can reduce data center[10] cooling by up to 50 percent. From the Tennessee Valley Authority energy report, Detailed Energy Study Report, dated March 20, 2009, for the Memphis Campus, we determined that 31 percent of the electricity cost was attributable to airflow management in the data center. REFM organization management provided the 2009 electricity costs for ECC – Memphis and ECC – Martinsburg. Based on the information obtained, Figure 1 provides the estimated potential savings from implementing best practices at these two ECCs.

Figure 1: Estimated Potential Savings From Implementing Best
Practices at Two of the ECCs

Row		Memphis	Martinsburg	Sum of the Prior Columns
1	Fiscal Year 2009 total electricity cost for the ECC campus.	$3,060,291	$2,057,243	$5,117,534
2	Percentage of the electricity cost attributable to the data center (i.e., cooling, computer room air conditioner fans, uninterruptible power supply, and batteries).	31%	31%*	—
3	Estimated Fiscal Year 2009 data center electricity cost. (Row 1 x Row 2)	$948,690	$637,745	$1,586,435
4	Potential savings based on industry estimate from implementing airflow management best practices.	Up to 50%	Up to 50%	—
5	Total potential annual decrease in energy usage by improving airflow management, before costs to implement best practices. (Row 3 x Row 4)	$474,345	$318,873	$793,218
6	Costs to implement best practices (The information needed to compute the costs was not available.)	Unknown	Unknown	Unknown
7	Potential 4-year decrease in energy usage cost before costs to implement best practices. (Row 5 x 4 years)	$1,897,380	$1,275,492	$3,172,872

Source: Our calculations based on IRS utility costs; the Tennessee Valley Authority report cited previously; A Design Guidelines
Sourcebook;[11] and “Is your data center running out of power or cooling?”[12]

· We did not have the percentage of electricity costs attributable to the ECC – Martinsburg data center; therefore, we used the ECC – Memphis percentage. However, the ECC – Martinsburg data center percentage may be higher because the ratio of office space to data center space is lower at ECC – Martinsburg.

Appendix V

Locations of Facilities Selected to Implement an Environmental Management System

Fresno, California

Washington, District of Columbia

Atlanta, Georgia

Covington, Kentucky

Andover, Massachusetts

Brookhaven, New York

Philadelphia, Pennsylvania

Memphis, Tennessee

Austin, Texas

Ogden, Utah

Martinsburg, West Virginia

Appendix VI

Glossary of Terms

Air Handlers	A device used to condition and circulate air as part of a heating, ventilating, and air-conditioning system.
Blanking Panel	A device used to fill empty vertical spaces in the rack to maintain proper airflow.
Brush Grommet	A device used to prevent cold air from escaping through cable holes in raised floor environments. By blocking bypass airflow, brush grommets significantly increase the cooling effectiveness of the data center because cold supply no longer mixes with hot exhaust air through unsealed cable holes.
Campus	The data processing arm of the IRS. The campuses process paper and electronic submissions, correct errors, and forward data to the Computing Centers for analysis and posting to taxpayer accounts.
Chiller Systems	A cooling system used in a data center to remove heat from one element and deposit it into another element. Chillers are used to cool the water used in their heating, ventilation, and air-conditioning units. Round-the-clock operation of chillers is crucial to the data center, given the considerable heat produced by many servers operating in close proximity to one another. Without them, temperatures would quickly rise to levels that would corrupt mission-critical data and destroy hardware.
Data Center	A building or portion of a building whose primary function is to house a computer room and its support areas; data centers typically contain high-end servers and storage products with mission-critical functions.
Data Center Infrastructure Efficiency	A measure used to determine the energy efficiency of a data center. The measure, which is expressed as a percentage, is calculated by dividing information technology equipment power by total facility power.
Enterprise Computing Center	Supports tax processing and information management through a data processing and telecommunications infrastructure.
Enterprise Learning Management System	An application that provides training, administration, and training resource management (instructors, classroom, and all web resources for learning).
Rack	An enclosed cabinet housing computer equipment.
Sub-metering (intelligent monitoring)	Allows power consumption to be monitored at every level in the power distribution hierarchy, giving visibility into current operational parameters and the impact of other energy-saving steps.
Uninterruptible Power Supply	An emergency power source to provide power to data center facilities until emergency generators come on.
White Paper	In information technology, a white paper is often a paper written by a lead product designer to explain the philosophy and operation of a product in a marketplace or technology context.

Appendix VII

Management’s Response to the Draft Report

DEPARTMENT OF THE TREASURY

INTERNAL REVENUE SERV1CE

WASH1NIGTON, D. C. 20224

CHIEF

AGENCY-WIDE

SHARED SERVICES

APR 12, 2010

MEMORANDUM FOR MICHAEL R. PHILLIPS

DEPUTY INSPECTOR GENERAL FOR AUDIT

FROM: David A. Grant /s/ David A. Grant

Chief, Agency-Wide Shared Services

SUBJECT: Draft Audit Report - Implementing Best Practices and Additional Controls Carl Improve Data Center Energy Efficiency and the Environmental and Energy Program (Audit # 200920025)

Thank you for the opportunity to comment on the draft audit report. The overall objective of this audit was to assess whether the Internal Revenue Service has established an effective energy and environmental program to ensure improvements in energy efficiency at IRS data centers.

We appreciate that the Treasury Inspector General for Tax Administration recognized our accomplishments related to energy and environmental management of our facilities. Two major IRS facilities have implemented an Environmental Management System (EMS), and we are planning to implement an EMS at nine other facilities. In addition, two of the campuses studied in the audit received the ENERGY STAR designation each time they were reviewed since 2006. Moreover, IRS ensures, as appropriate, that employees complete al l necessary training related to environmental and energy management.

Since 2008, the IRS has been engaged in a data center consolidation effort to right-size computing environments in new buildings, as well in reconfiguring existing space to reduce the footprint of computer rooms enterprise-wide. Along with this effort, newer technology being deployed also requires less space, thus further reducing the overall footprint. In addition, we are working with Treasury on the Office of Management and Budget Data Center Consolidation effort. As we move forward with these efforts, we will repurpose computer room space to improve overall efficiency. While there are many challenges ahead for the IRS energy program, we continue to meet all relevant goals as required by executive orders and legislation.

Al though we have taken steps 10 promote energy efficiency at IRS data centers, we recognize that the IRS has potential for greater energy savings and improved efficiency, and our corrective actions are attached. We will convene a working group to oversee many of the recommended improvements to meet IRS' goals for energy reduction. We plan to assess which best practices are most appropriate for IRS data centers, and to implement and track improvement projects in a systematic manner. We believe the implementation of recommended best practices will lead to improved energy efficiency at IRS data centers.

Thank you for your guidance in this effort. If you have any questions, please contact me at (202) 622-7500, or a member of your staff may contact Stuart Burns, Director, Real Estate and Facilities Management, at (202) 435-6300. For matters concerning audit follow-up, please contact Larry Pugh, Office of Strategy and Finance, Agency-Wide Shared Services, at (202) 622-4541.

Attachment

RECOMMENDATION 1:

Policies and procedures are established to evaluate and determine which best practices to implement to improve data center energy efficiency.

CORRECTIVE ACTION:

Agency-Wide Shared Services (AWSS) will convene a Data Center Working Group co-chaired by Real Estate and Facilities Management (REFM) and Modernization & Information Technology Services (MITS), to assess best practices to implement in IRS data centers, communicate best practices with campus contacts, establish methods for tracking progress toward implementation, and research new opportunities for improving data center energy efficiency.

IMPLEMENTATION DATE:

June 30, 2010 - Convene Working Group

June 30, 2011 - Document best practices, make recommendations, and document progress.

RESPONSIBLE OFFICIAL:

Director, Real Estate and Facilities Management, Agency-Wide Shared Services

Recommendation 2:

Information technology equipment energy use is measured (Le. sub-metered) in order to determine the energy efficiency and savings from implementing energy improvements.

CORRECTIVE ACTION:

Due to the complexity and scale of effort needed for data center sub-metering, REFM will identify one campus and one computing center, perform an analysis examining resources and engineering effort s needel1 for sub-metering, and undertake sub-metering efforts at these locations, REFM will analyze costs, benefits and other considerations surrounding sub-metering at the specified locations. Based on observed costs and benefits, and any additional analysis conducted, REFM will assess the feasibility of moving forward will, sub-metering at other facilities and provide a recommendation as to moving forward or not. REFM will subsequently provide the Treasury Inspector General for Tax Administration (TIGTA) with a report and recommendations detailing the analysis and findings.

IMPLEMENTATION DATE:

October 31, 2011 - Study provided to TIGTA

RESPONSIBLE OFFICIAL:

Director, Real Estate and Facilities Management, Agency-Wide Shared Services

RECOMMENDATION 3:

Employee workstations and information technology equipment and non-information technology equipment and furniture that are no longer needed are removed from the data centers.

CORRECTIVE ACTION:

REFM will consult with the Director, Enterprise Computing Center (ECG) within MITS in the process of identifying information technology and non-information technology based equipment, workstations and furniture no longer needed, and will develop a plan for removal, where appropriate.

IMPLEMENTATION DATE:

November 30. 2010 - Development of equipmef1t1fumiture removal plan

June 30, 2011 - Implementation of removal plan

RESPONSIBLE OFFICIAL:

Director, Real Estate and Facilities Management, Agency-Wide Shared Services

RECOMMENDATION 4:

Current and future data center space needs are identified and plans developed to consolidate or reduce excess data center space.

CORRECTIVE ACTION:

REFM will consult with MITS Enterprise Operations, who will take the lead in the process of assessing the current state of data center space and developing a plan of action for space consolidation, where deemed appropriate. Through the Data Center Working Group or another venue, MITS Enterprise Operations will develop a plan in accordance with the Office of Management and Budget Data Center Consolidation plan developed by Treasury.

IMPLEMENTATION DATE:

March 1, 2012

RESPONSIBLE OFFICIAL:

Associate Chief Information Officer, Enterprise Operations

RECOMMENDATION 5:

Energy audits (including an assessment of best practices) are performed at the data centers.

CORRECTIVE ACTION:

REFM will ensure that audits are completed timely and tracked, as required by the Energy Independence and Security Act of 2007 (EISA 2007), and include provisions to assess data centers with regard to industry best practices.

IMPLEMENTATION DATE:

October 31, 2014 (25 percent each year, on a 4·yearcycle, as required by EISA 2007)

RESPONSIBLE OFFICIAL:

Director, Real Estate and Facilities Management, Agency-Wide Shared Services

RECOMMENDATION 6:

Governance process is established to evaluate, select, and approve recommendations and projects to implement. The process should include a requirement to document all recommendations and projects, whether they were selected for implementation, and the justification for selecting or not selecting the recommendation and project.

CORRECTIVE ACTION:

REFM will design a governance plan for evaluating find implementing projects, and tracking them in accordance with the corrective action in Recommendation 7. The plan will specify the individuals or board that will oversee energy improvement projects and will delineate project selection procedures and criteria.

IMPLEMENTATION DATE:

September 30, 2010 - Development of governance plan

March 31, 2011 - Demonstration of plan implementation

RESPONSIBLE OFFICIAL:

Director, Real Estate and Facilities Management, Agency-Wide Shared Services

RECOMMENDATION 7:

Database of all recommendations and projects to improve energy efficiency and the environment is established, For each recommendation, the database should include information such as whether the recommendation was selected for implementation, projected and actual cost to implement and, if applicable, implementation start and completion dates, and projected and actual cost savings, as well as other benefits realized from the implementation.

CORRECTIVE ACTION:

All energy conservation measures or improvement projects identified by a formal .audit performed as required by EISA 2007 will be tracked in the Department of Energy EISA 432 Compliance Tracking System (CTS) database, as mandated by the Act. The CTS database, which is being created as a result of EISA 2007, is projected to be available by June, 2010. For projects not identified through a formal audit process, REFM will design an MS Excel-based tracking tool. Information to be tracked will include project listings, justifications for selecting or not selecting projects, start and completion dates, projected and actual costs, and savings or other benefits realized from implementation.

IMPLEMENTATION DATE:

September 30, 2010 - Development of draft Excel-based tool

March 31, 2011 - Implementation project tracking through Excel-based tool

RESPONSIBLE OFFICIAL:

Director, Real Estate and Facilities Management, Agency-Wide Shared Services

AUDIT RESPONSE POINT (from page 4, paragraph 21:

TIGTA calculated that by implementing better airflow management practices at the ECC-Memphis and the ECC-Martinsburg, IRS could save $3,172,872 over the next four years. TIGTA plans to include this in the Semiannual Report to Congress, and asked for IRS concurrence or non-concurrence with this assessed benefit.

IRS RESPONSE:

While airflow management improvements could certainly lead to energy-related savings for IRS, we believe the estimated figure in the outcome measure may overstate savings that are likely to accrue from such improvements_ Specifically, the figure does not factor in implementation costs, does not take into consideration best practices already implemented, and is based on 50% energy savings, which represent the upper limit of potentially achievable savings, As a result, we are not comfortable with the current language used to discuss the outcome measure, but we would concur with the outcome measure with the following language revisions:

"Implementing better airflow management practices at the ECC-Memphis and the ECC-Martinsburg may require a significant investment on the part of IRS. As part of the corrective actions resulting from this audit, IRS will analyze data center best practices. analyze sub-metering costs and benefits and implement sub-metering at two locations, and examine return on investment for improvement projects undertaken, If, based on the results from these analyses and the assessed costs and benefits of potential improvement projects, IRS opts to invest in the recommended airflow management initiatives, these improvements could potentially lead to gross savings for IRS of up to $3,172,872 over the next four years. However, this figure does not factor in implementation costs IRS would bear in executing the improvement initiatives."

[3] Report to Congress on Server and Data Center Energy Efficiency Public Law 109-431, dated August 2, 2007, U.S. Environmental Protection Agency Energy Star Program.

[5] Spears, Ed, “Is your data center running out of power or cooling?” dated May 30, 2009, Eaton Corporation, http://powerquality.eaton.com/about-us/news-events/whitepapers/data-center-power-cooling.asp.

[6] See Appendix V for the locations of the 11 facilities selected to implement an Environmental Management System.

[11] Pacific Gas and Electric Company, dated January 2006.

[12] Spears, Ed, dated May 30, 2009, Eaton Corporation, http://powerquality.eaton.com/about-us/news-events/whitepapers/data-center-power-cooling.asp.