TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

The Data Center Consolidation Initiative Has Made Significant Progress, but Program Management Should Be Improved to Ensure That Goals Are Achieved

 

 

 

June 10, 2013

 

Reference Number:† 2013-20-013

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

 

Phone Number† /† 202-622-6500

E-mail Address /TIGTACommunications@tigta.treas.gov

Website†††††††††† /http://www.treasury.gov/tigta

 

 

HIGHLIGHTS

THE DATA CENTER CONSOLIDATION INITIATIVE HAS MADE SIGNIFICANT PROGRESS, BUT PROGRAM MANAGEMENT SHOULD BE IMPROVED TO ENSURE THAT GOALS ARE ACHIEVED

Highlights

Final Report issued on June 10, 2013

Highlights of Reference Number:† 2013-20-013 to the Internal Revenue Service Chief Technology Officer.

IMPACT ON TAXPAYERS

In February 2010, the Office of Management and Budget established the Federal Data Center Consolidation Initiative mandating that Federal agencies reduce costs and save energy by reducing the number of Federal data centers.† Decreasing redundant or unnecessary data centers, as well as reducing operational costs and energy consumption, is a significant part of Federal efforts to reduce costs and make more efficient use of taxpayer funds.

WHY TIGTA DID THE AUDIT

In response to the Office of Management and Budget mandate, the IRS established a Data Center Consolidation Initiative (DCCI) Program Management Office (PMO) in 2010 and set a goal to reduce its data center space by 50 percent by the end of Fiscal Year 2015.† TIGTA initiated this audit to determine whether the IRSís DCCI would effectively reduce data center space and increase energy efficiency.

WHAT TIGTA FOUND

While the IRS has exceeded its first two yearsí goals for reducing data center space and improving the energy efficiency of its data centers, management of the DCCI needs to be improved to ensure that the IRS meets its future DCCI goals.† Two years of the IRSís five-year DCCI have elapsed without a clear plan for how the data center space reduction goals will be accomplished.† The IRSís DCCI PMO has not formalized a project management plan that addresses challenges and details decisions, milestones, and time frames related to how the IRS will meet its data center consolidation goals.† In addition, the IRS has not identified the optimal footprint for its data centers.† TIGTA observed significant empty space at the IRSís Enterprise Computing Center in Detroit, Michigan.† TIGTA was informed by IRS management that the IRSís lease for this space will expire in April 2015 and that the systems at this location could be moved to other IRS data centers.† During our audit, the IRS announced plans to close the Enterprise Computing Center in Detroit, estimating the IRS will save approximately $15 million annually.

Further, the IRSís DCCI does not include all of its data centers that meet the standard Federal definition of a data center within its baseline inventory.† The IRS has an additional 115,343 square feet of data center space in 61 buildings.

WHAT TIGTA RECOMMENDED

TIGTA recommended that the Chief Technology Officer provide, as appropriate, additional resources to the DCCI PMO, require the PMO to develop a project management plan that addresses challenges to accomplishing project goals and solutions, and ensure that the Enterprise Computing Center in Detroit is consolidated into the Martinsburg and Memphis Enterprise Computing Centers.

TIGTA also recommended that the Chief Technology Officer work with the Real Estate and Facilities Management organization to correct the inventory, update the inventory submitted to the Department of the Treasury, formalize plans to develop and apply a baseline footprint to the IRSís small data centers, and coordinate with the Chief, Agency-Wide Shared Services, to shut down computer room air conditioners no longer needed and adjust the set point temperature, as appropriate, on the remaining computer room air conditioners in accordance with industry best practices.

In their response to the report, IRS management agreed with seven of the report recommendations and partially agreed with one recommendation. †The IRS has taken or plans to take appropriate corrective actions in response to all of the report recommendations.

 

June 10, 2013

 

 

MEMORANDUM FOR CHIEF TECHNOLOGY OFFICER

 

FROM:†††††††††††††††††††††† Michael E. McKenney /s/ Michael E. McKenney

††††††††††††††††††††††††††††††††† Acting Deputy Inspector General for Audit

 

SUBJECT:††††††††††††††††† Final Audit Report Ė The Data Center Consolidation Initiative Has Made Significant Progress, but Program Management Should Be Improved to Ensure That Goals Are Achieved (Audit # 201220021)

 

This report presents the results of our review of the Internal Revenue Serviceís (IRS) Data Center Consolidation Project.† The overall objective of this review was to determine whether the IRSís Data Center Consolidation Initiative would effectively reduce data center space and increase energy efficiency.† This review is included in our Fiscal Year 2013 Annual Audit Plan and addresses the major management challenge of Achieving Program Efficiencies and Cost Savings.

Managementís complete response to the draft report is included as Appendix VI.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.† If you have any questions, please contact me or Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services).

 

 

Table of Contents

 

Background

Results of Review

The Data Center Consolidation Initiative Program Management Office Has Made Significant Progress in Reducing Data Center Space and Improving Energy Efficiency, but Lacks a Clear Plan for Accomplishing Its Goals

Recommendations 1 through 3:

The Internal Revenue Serviceís Data Center Consolidation Initiative Does Not Include All Data Centers

Recommendations 4 through 7:

Recommendation 8:

Appendices

Appendix I Ė Detailed Objective, Scope, and Methodology

Appendix II Ė Major Contributors to This Report

Appendix III Ė Report Distribution List

Appendix IV Ė Outcome Measure

Appendix V Ė Glossary of Terms

Appendix VI Ė Managementís Response to the Draft Report

 

 

Abbreviations

 

CIO

Chief Information Officer

CTO

Chief Technology Officer

DCCI

Data Center Consolidation Initiative

ECC

Enterprise Computing Center

FDCCI

Federal Data Center Consolidation Initiative

FY

Fiscal Year

IRS

Internal Revenue Service

IT

Information Technology

OMB

Office of Management and Budget

PMO

Program Management Office

TIGTA

Treasury Inspector General for Tax Administration

 

 

Background

 

In February 2010, the Office of Management and Budget (OMB) established the Federal Data Center Consolidation Initiative (FDCCI) as a Governmentwide initiative designed to reduce the energy and real estate footprint of Federal data centers while increasing efficiency, strengthening the overall Government security posture, and promoting Green Information Technology[1] by reducing the total number of Federal data centers.† The FDCCI guidance required agencies to inventory their data center assets, develop consolidation plans, and integrate those plans into agency Fiscal Year (FY) 2012 budget submissions.† The FDCCI would reduce the number of data centers across the Government and assist agencies in applying best practices from the public and private sector, with goals to:

       Promote the use of Green Information Technology by reducing the overall energy and real estate footprint of Government data centers.

       Reduce the cost of data center hardware, software, and operations.

       Increase the overall information technology security posture of the Government.

       Shift information technology investments to more efficient computing platforms and technologies.

The OMB provided updated guidance in July 2011.† This guidance required agencies and all subordinate organizations to complete all missing elements of their respective consolidation plans and submit them to the General Services Administration.† The plans were required to include a full master plan schedule which identifies, by quarter, the data centers to be closed through FY 2015.† The plans were also to reflect challenges experienced to date and integrate lessons learned.

In response to the OMB directive to consolidate Federal data centers, the Department of the Treasury and its bureaus, including the Internal Revenue Service (IRS), developed data center consolidation plans.† In October 2010, the IRS created the Data Center Consolidation Initiative (DCCI) Program Management Office (PMO).† The IRSís DCCI goal is to reduce its data center space by 50 percent, 10 percent per year, from FY 2011 through FY 2015.† The IRS also has a goal to reduce the number of data centers from 15 primary data centers to eight by the end of FY 2015.† The IRSís DCCI plan states that the approach is to centralize, virtualize, and decommission server resources to meet the following goals:† consolidate the governance of all servers, reduce the IRSís data center footprint, enhance the IRSís disaster recovery capabilities, and streamline the IRSís processes to reach a more mature level of the Information Technology Infrastructure Libraryģ.

There are currently three bills in Congress with potential legislation that may affect the FDCCI.† Two of the three bills would specifically require agencies to submit complete data center consolidation plans to the OMB.[2]

This review was performed at the IRS Enterprise Computing Centers (ECC) in Detroit, Michigan; Memphis, Tennessee; and Martinsburg, West Virginia, and the IRS data centers in Atlanta, Georgia; St. Louis, Missouri; New York, New York; Oklahoma City, Oklahoma; and Dallas, Texas, during the period June through November 2012.† We conducted this performance audit in accordance with generally accepted government auditing standards.† Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective.† We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.† Detailed information on our audit objective, scope, and methodology is presented in Appendix I.† Major contributors to the report are listed in Appendix II.

 

 

Results of Review

 

The Data Center Consolidation Initiative Program Management Office Has Made Significant Progress in Reducing Data Center Space and Improving Energy Efficiency, but Lacks a Clear Plan for Accomplishing Its Goals

The IRSís Chief Technology Officer (CTO) set a DCCI goal for the IRS to reduce its baseline data center space (reported at approximately 392,000 square feet) by 50 percent (or approximately 196,000 square feet) by the end of FY 2015.† The PMO plans to meet the goal by reducing the footprint by 10 percent, or approximately 39,000 square feet, annually during FYs 2011 through 2015.† The OMB did not set data center space savings goals or requirements for Federal agencies.† Similarly, the Department of the Treasury did not set specific DCCI goals for its bureaus.

Project management best practices have established that a project management plan is a key project planning document.† The project management plan documents planning assumptions and decisions for the project and provides a framework for managing project activities and for completing the project successfully.† The objective of a project management plan is to define the scope of work to be accomplished and the approach to be used by the project team to deliver the solution.† In September 2010, the IRS submitted its DCCI plan to the Department of the Treasury.† The plan stated that a project management discipline will be applied to the effort, including development of a comprehensive master program schedule, detailed implementation schedules, and spending plans.

While the IRS has exceeded its goals in the first two years for reducing data center space and improving the energy efficiency of its data centers, management of the project needs to be improved to ensure that the IRS meets its DCCI goals by the end of FY 2015.† Two years of the IRSís five-year DCCI have elapsed without a clear plan for how the data center space reduction goals will be accomplished.† In addition, challenges remain that, if not addressed in a formal plan, could result in the IRS not meeting its future DCCI goals.

Significant progress has been made in the first two years of the DCCI

The IRSís DCCI PMO, in partnership with the IRSís Real Estate and Facilities Management organization, has exceeded its first two yearsí goals in reducing the IRSís data center footprint and improving data center energy efficiency.† As a result of consolidation efforts, by the end of FY 2012 (the second year of the five-year project) the IRS had closed two data centers and successfully reduced 76,560 square feet of the baseline data center square footage at five other data centers.† In doing so, the IRS has exceeded its plan of reducing 10 percent per year of the baseline square footage for the first two years of the five-year project period.† The IRS has worked with the Department of the Treasury and made progress in relocating Department of the Treasury systems into the IRSís data center space.

The IRS has also taken steps to improve the energy efficiency of its data centers.† These steps, taken in response to a prior Treasury Inspector General for Tax Administration (TIGTA) audit report,[3] include implementing ďhotĒ and ďcoldĒ aisles for servers, installing motion sensors to control overhead lighting, and shutting down redundant computer room air conditioners.† The IRS is in the process of installing power metering in two data centers located in Kansas City, Missouri, and Memphis, Tennessee.† This will enable the IRS to determine the exact amount of electricity used in these data centers.† Currently, the IRS is only able to estimate the power usage of its data centers.† Following implementation of metering at these locations, the IRS will assess the costs, benefits, and efforts required to install metering and determine whether metering will be recommended for other IRS data centers.

A project management plan has not been developed

The DCCI PMO prepared an operating charter that established the PMO and set forth its objectives, governance responsibilities, membership composition, and high-level operating framework.† While the DCCI PMO was formed in October 2010, the operating charter was not developed until July 2012, nearly two years after the start of the DCCI PMO

A project management plan is needed to address challenges to accomplishing DCCI goals.† The IRSís DCCI PMO has not formalized a project management plan that addresses challenges and details decisions, milestones, and time frames related to how the IRS will meet its goal of reducing data center space by approximately 196,000 square feet by the end of FY 2015.† The IRS uses quarterly CTO briefing document presentations for managing the DCCI.† Specific plans are developed for each local site as the IRS begins to address specific campuses, computing centers, or other site locations.

In addition, while the IRS can perform DCCI planning and analysis work during the filing season, it cannot make any major information technology production environment changes or relocate hardware.[4] †This further shortens the time frame and provides additional challenges which the IRS has to address in the next three fiscal years to accomplish its DCCI goals.

The IRS lacks a strategy for the ECCs

In a prior report, we recommended that current and future IRS data center space needs be identified and plans developed to consolidate or reduce excess data center space.† IRS management agreed with this recommendation and stated that the Enterprise Operations organization would take the lead in the process of assessing the current state of data center space and developing a plan of action for space consolidation, where appropriate.

Two years after the start of the DCCI, the IRS has not identified the optimal footprint or configuration for its ECCs.† In August 2012, nearly two years after the start of the DCCI, the IRS prepared a statement of work to engage a contractor to study the three ECCs and make recommendations for how the IRS should create world-class data center facilities to meet its current and future information technology needs.

We observed significant empty data center space at the ECC-Detroit.† We were informed by IRS management that the IRSís lease for this space will expire in April 2015 and that the systems that remain at this location could be moved to other IRS data centers.† Based on this, it was our opinion that the IRS should close the data center at the ECC-Detroit.† In July 2012, during our audit, the CTO announced the closure of the ECC-Detroit.† The basis for this decision has not been documented as part of a project management plan.† In September 2011, a draft business case was developed for the ECC-Detroit that assumed the data center would remain open.† The business case approach was then changed and replaced with a document, ďStrategic Overview of the ECCs and Their Role in Supporting the IRS Data Center Consolidation Initiative,Ē dated November 2011.† The strategic overview document also includes future plans for the ECC-Detroit.† The document states three computing centers are necessary to meet the IRSís information technology needs.† In 2012, the DCCI PMO began to prepare a statement of work for a contractor to study the ECCs and make recommendations on how the IRS can implement a world-class data center organization.† In August 2012, the draft statement of work was revised to reflect the IRSís plans to close the ECC-Detroit.† The CTO estimates the IRS will save approximately $15 million annually by closing the ECC-Detroit.† We agree with this decision as it is consistent with a prior TIGTA recommendation, and we have also raised this issue during this audit. †The IRS expects to receive the results of the contractorís world-class data center study in August 2013.

Closing the ECC-Detroit presents another issue not addressed in a project management plan.† The IRSís telecommunications gateway infrastructure, BlackBerry enterprise servers, and e-mail archiving are configured to be located in three sites.† The systems are currently located at the three ECC locations.† With the closure of the ECC-Detroit, the IRS needs to identify and prepare a third location to support these systems.

A decision has not been made on how the Annex building at the ECC-Martinsburg, a large space of approximately 36,000 square feet, will contribute to the 195,000 square foot reduction in data center space.† This building contains the command center for the ECC-Martinsburg, the Department of the Treasuryís continuity of operations equipment, and the IRSís Research, Analysis, and Statistics organization equipment.† The DCCI managers informed us that there may be adequate unused space within either of the two ECC-Martinsburg computer rooms to accommodate these items and there may be a need for additional office space at the ECC‑Martinsburg.† To complicate this decision, the Annex building is a privately owned building leased by the IRS, whereas the ECC-Martinsburg facility is a Government-owned building.

The DCCI PMOís decision to reduce space at the ECC-Martinsburg and the ECC-Memphis facilities was also not documented in a project management plan detailing how the space reduction will be accomplished.† The IRS briefing documents currently show that data center space at the two computing centers will be reduced by 50,000 square feet.† In addition, the decision has been made without knowing the results of the data center study, which has not yet been completed.† One of the key focuses of the study is to project the future state of the IRSís data centers; it would be beneficial to await the results of the study prior to making a decision to reduce space at the remaining computing centers.† The DCCI PMO has indicated that the results of the data center study will improve decisions at the Memphis and Martinsburg ECCs regarding potential space reductions, space to be made available for other Department of the Treasury bureaus, or conversion to office space.

The DCCI PMO program manager plans to turn over responsibility for implementing the World Class Data Center Study recommendations to the ECC executives.† The DCCI PMO responsibilities related to the data center study should be included in a project management plan and should require a continued role to ensure that the recommendations are implemented.† Without such a role, the ECC improvements may not be realized.

At the beginning of the project, the IRSís Information Technology (IT) organization management did not provide sufficient resources to the DCCI PMO for project planning and to ensure that the team could efficiently identify the best way to accomplish the DCCI goals.† The program manager estimated that he is only able to devote 25 percent of his time to the DCCI.† In March 2012, 18 months after the start of the DCCI, the DCCI PMO did not have a full-time senior project manager.† The DCCI PMO informed the CTO that due to all the interconnected activities affecting consolidation, virtualization, post-of-duty consolidation, and coordination with the IT organization and local Real Estate and Facility Management organization staffs as well as the business units, an additional full-time employee with project management expertise was needed.† During our fieldwork, the DCCI PMO received a full-time senior project manager and an additional staff member to assist the senior project manager.

Significant challenges remain, including a strategy for the ECCs that have not been sufficiently addressed in a project management plan.† If not addressed, the IRS may not fully achieve its DCCI goals by the end of FY 2015 as planned to reduce the real estate footprint and energy consumption of its data centers.

Recommendations

To ensure that the IRS accomplishes its DCCI goals in the final three years, the CTO should:

Recommendation 1: †Provide, as appropriate, additional resources to support the DCCI PMO to ensure that the future project goals are accomplished and ensure that the DCCI PMO can maintain a role in guiding the implementation of the World Class Data Center Study recommendations.

Managementís Response:† The IRS agreed with this recommendation.† In November 2012, the Associate Chief Information Officer (CIO), Enterprise Operations, added appropriate resources to the DCCI.† The IRS took this action to ensure that project goals would be accomplished and to ensure that the DCCI PMO maintained a role in guiding the implementation of the World Class Data Center Study recommendations in the computing centers.

Recommendation 2: †Require the DCCI PMO to develop a project management plan following project management best practices.† The plan should include the development of a comprehensive master program schedule, detailed implementation schedules, and spending plans and should address challenges to accomplishing project goals and solutions.

Managementís Response:† The IRS agreed with this recommendation.† As a result of discussions during this audit, the IRS DCCI PMO consolidated the existing strategy and planning documents and began developing a master project plan that will address challenges, detail decisions, milestones, and time frames related to how the IRS will meet its remaining data center consolidation goals.

Recommendation 3:† Ensure that the ECC-Detroit is consolidated into the Martinsburg and Memphis ECCs.

††††††††††† Managementís Response:† The IRS partially agreed with this recommendation.† The IRS responded that citing specific locations in the recommendation limits the IRSís ability to make the best decision as to where to locate the ECC-Detroit computing assets. †IRS management stated they will ensure that the ECC-Detroit computing assets are consolidated into an appropriate facility.

The Internal Revenue Serviceís Data Center Consolidation Initiative Does Not Include All Data Centers

At the start of the FDCCI in February 2010, the agenciesí data center inventories included a diverse collection of facilities.† As a result, in October 2010, the Federal CIO announced a common baseline definition for a Federal Government data center, developed by the FDCCI Task Force.† The Task Force defined a data center as any room greater than 500 square feet in area, devoted to data processing, and meeting a tier (I, II, III, or IV) classification defined by the Uptime Institute.[5]

After more than a year of work under the FDCCI, the Task Force updated its standard data center definition.† In March 2012, the Federal CIO issued a memorandum formalizing an updated definition of a data center.† The memorandum states, ďÖa data center is now defined as a closet, room, floor or building for the storage, management, and dissemination of data and informationĒ and ďUnder this revised definition, neither square footage nor the Uptime Institute tier classifications are required to define a facility as a data center.Ē† The memorandum also states, ďThe Task Force has made it clear that facilities below 500 square feet in area that do not meet an Uptime Institute classification also consume significant amounts of resources.Ē† The Federal CIO explained that expanding the scope of the FDCCI effort to include even the smallest spaces will better assist agencies in locating inefficient information technology assets and consolidation opportunities.† Consolidation encompasses not just large facilities but also small server clusters and wiring closets.

The IRSís detailed DCCI plan, submitted to the Department of the Treasury in September 2010 and updated in June 2011, states that the IRSís five-year plan for data center consolidation will focus on locations hosting more than six computing assets, or roughly 124 IRS locations.† The plan also states that the IRS is committed to policies and practices that reduce the environmental impact of IRS operations.† The IRS has committed to such best practices as implementing computer room air-conditioner temperature and humidity set points that are consistent with industry best practices, and shutting down unnecessary computer room air conditioners, as appropriate.

The IRSís DCCI does not include all data centers that meet the standard Federal definition of a data center within its baseline inventory.† The IRSís five-year DCCI is focused on achieving data center space reduction goals at 17 locations, rather than 124 locations, as stated in the IRSís detailed DCCI plan.† In its DCCI plan, submitted to and approved by the Department of the Treasury in September 2010, the IRS identified 15 data centers as primary data center locations in the data center baseline inventory (three ECCs, one Annex building at the ECC-Martinsburg, nine campuses, and two Washington, D.C., area data centers).† In June 2012, the IRS submitted to the Department of the Treasury an updated baseline data center inventory of 17 data centers (including the 15 data centers reported in September 2010 plus one Criminal Investigation data center and one additional Washington, D.C., area data center).† Given limited resources and time to accomplish data center consolidation goals, the IRSís CTO decided to limit the IRSís focus to the 17 selected data centers.

The IRSís annual inventory updates submitted to the Department of the Treasury in FYs 2011 and 2012 did not include the data centers meeting the standard larger than 500-square-feet definition of 2011 or the expanded 2012 definition that eliminated the square footage requirement.† In the February 2010 memorandum launching the start of the FDCCI, the OMB required agencies to update their asset inventories annually by the end of the third quarter of each fiscal year.† The March 2012 memorandum reminded agencies of this requirement.

In addition to the 17 data centers included in the DCCI, the IRS has 115,343 square feet of data center space in 61 buildings in 31 states, Washington, D.C., and Puerto Rico.† These additional data centers meet the OMBís initial definition of any room greater than 500 square feet in area.† The IRS also has an additional 25,244 square feet of space in 122 rooms that meet the OMBís expanded definition that eliminates area as a requirement.

We visited eight data centers greater than 500 square feet and not included in the IRSís DCCI, which were located in five cities.† The data centers ranged from 1,437 square feet to 8,399 square feet and accounted for 33,825 (29 percent) of the additional 115,343 square feet not included in the DCCI.† Our review of the sites found that opportunities exist for space reduction and energy savings at all sites visited.† Specifically, we found:

       Computer room air conditioners were powered on and cooling rooms with little or no computer equipment.

       Set point temperatures on computer room air conditioners were not consistently set to save energy and lower energy costs, as the IRS recently did at its ECCs and campuses.† There were a total of 28 computer room air conditioners at the eight data centers we visited.† Temperatures for 21 of the 28 units were set lower than 72 degrees, ranging from 68 to 71 degrees.† Only seven units at one location were set at 72 degrees.†

       The IRS data center inventory is not accurate.† For example, a room at one location was not in the inventory, while the inventory at another location included a room that no longer exists.† One large computer room was listed as three separate smaller rooms in the inventory.

During our audit, the DCCI PMO informed us that it had been asked by senior IT organization management to study such smaller data centers to identify consolidation and energy savings opportunities and that they plan to develop standard footprints for small data centers.

The IRSís DCCI PMO also informed us that data center consolidation for the smaller data centers not included in the DCCI are being addressed as building leases expire and also based on the Real Estate and Facilities Management organization move or renovation projects currently taking place.† Building leases for 38 (62 percent) of the 61 buildings that house 79,380 square feet of data center space will expire before the FY 2015 DCCI end date.† Building leases for 23 (38 percent) of the 61 buildings housing 35,963 square feet will expire after the DCCI end date.

When the IRSís data center inventory does not include all of its data centers, the Department of the Treasury cannot provide an accurate data center inventory to the OMB.† In turn, the OMB then does not have an accurate count of data centers within the scope of the FDCCI.† In addition, if there are too many data centers (no matter their size) supporting the organization, they lead to power inefficiencies and add unnecessary cost, taking resources that could be used for other IT organization purposes.† The IRS continues to incur energy costs at those locations where leases will be renewed or where many years remain before lease expiration.† We could not determine the additional cost the IRS is incurring to continue operating the data centers not included in the IRSís DCCI.† The IRS does not have metering technology in place to measure electric use at its data centers.† As a result, the IRS could not provide data center energy usage and cost information.

Recommendations

To ensure the most energy and space-efficient data center configurations, the CTO should:

Recommendation 4: †Work with the Real Estate and Facilities Management organization to correct the inventory when the DCCI PMO identifies discrepancies during local site visits.

Managementís Response: †The IRS agreed with this recommendation.† The Enterprise Operations organization will work with the Real Estate and Facilities Management organization within Agency-Wide Shared Services to develop a formal process to correct the data center inventory.

Recommendation 5: †Formalize plans to develop and apply a baseline footprint to the IRSís small data centers.

Managementís Response: †The IRS agreed with this recommendation.† The Enterprise Operations organization, in coordination with the Real Estate and Facilities Management organization, will develop a baseline footprint for the IRSís small data centers.† This footprint will be applied to future build out and/or space reduction projects in small data centers, where cost effective.

Recommendation 6:Update the annual inventory submitted to the Department of the Treasury to include all data centers, regardless of size.

Managementís Response: †The IRS agreed with this recommendation.† The IRS will update the annual inventory submitted to the Department of the Treasury to include all data centers.

To ensure the most efficient use of computer room air conditioners at the IRSís data centers, the CTO, in conjunction with the Chief, Agency-Wide Shared Services, should:

Recommendation 7:Identify and shut down computer room air conditioners no longer needed.

Managementís Response: †The IRS agreed with this recommendation.† The Enterprise Operations organization, in coordination with the Real Estate and Facilities Management organization, will formalize a process for identifying and shutting down computer room air conditioners that are no longer needed.

Recommendation 8:Adjust the set point temperatures as appropriate on the remaining computer room air conditioners in accordance with industry best practices.

Managementís Response: †The IRS agreed with this recommendation.† The Enterprise Operations organization, in coordination with the Real Estate and Facilities Management organization, will formalize a process for ensuring that computer room air conditioners are adjusted to industry standard air conditioner set point temperatures.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this audit was to determine whether the IRSís DCCI would effectively reduce data center space and increase energy efficiency.† To accomplish this objective, we:

I.                 Identified all OMB and Department of the Treasury data center consolidation requirements and criteria, as well as industry best practices, which would be used to evaluate the IRSís DCCI.

A.    Researched the Federal CIO website to identify and obtain OMB and Federal CIO FDCCI requirements and criteria.

B.    Researched the Department of the Treasury website to identify and obtain any DCCI requirements.

C.    Researched the Internet, including the Federal CIO website, to identify FDCCI best practices.

D.    Scheduled and summarized criteria and best practices.

II.               Determined if the IRSís DCCI meets Federal requirements and includes appropriate governance, goals, and metrics to effectively achieve data center consolidation.

A.    Evaluated the governance and oversight of the IRSís DCCI to determine if executives are sufficiently informed of the results.

B.    Determined if IRS data center consolidation plans are consistent with Department of the Treasury and OMB guidelines as well as industry best practices.†

C.    Determined if goals have been established for the DCCI.

D.    Determined if the IRS has established metrics to monitor progress toward accomplishing DCCI goals.

E.     Determined if the IRSís DCCI plans include all of its data centers, regardless of size, and assessed any related consolidation plans.† We selected a judgmental[6] sample of eight of 107 data centers, greater than 500 square feet, not included in the scope of the IRSís DCCI.† The sample selection was based on data center square footage and type of systems located at each data center.† We chose a judgmental sample to include the largest of the data centers not included in the scope of the IRSís DCCI as well as two data centers requested by DCCI PMO management.

F.     Determined if the IRS has considered multiple consolidation alternatives at the ECCs.

III.             Determined if the IRSís DCCI plans include all costs, savings, funding, and return on investment considerations to effectively measure and accomplish DCCI goals.

A.    Determined if the IRS captures and tracks the costs of operating its data centers.

B.    Determined if the IRS can identify all costs associated with its DCCI plans and activities.

C.    Determined if the IRS has computed expected savings to be realized from its DCCI efforts.

D.    Determined if the IRS has calculated the return on investment for its DCCI.

E.     Determined if the DCCI plans address how the efforts will be funded.

IV.            Determined if the IRSís DCCI plans are being effectively implemented to reduce data center space and reduce energy costs.

A.    Determined if the IRS has been using metrics to demonstrate its progress towards meeting DCCI goals.

B.    Determined if the IRS is using actual incurred cost data for tracking the effectiveness of its DCCI efforts.

C.    Identified DCCI savings achieved.

D.    Determined if the IRS is meeting its DCCI goals to reduce energy costs and consumption.

Internal controls methodology

Internal controls relate to managementís plans, methods, and procedures used to meet their mission, goals, and objectives.† Internal controls include the processes and procedures for planning, organizing, directing, and controlling program operations.† They include the systems for measuring, reporting, and monitoring program performance.† We determined the following internal controls were relevant to our audit objective: †the Enterprise Operations and Real Estate and Facilities Management organizationsí plans, policies, and processes to manage, monitor, and report on the status and progress of the IRSís DCCI.† We evaluated these controls by interviewing the DCCI management, analyzing the DCCI documentation, and visiting selected data centers.

 

Appendix II

 

Major Contributors to This Report

 

Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services)

Danny Verneuille, Director

John Ledford, Audit Manager

Joan Bonomi, Lead Auditor

Myron Gulley, Senior Auditor

Curtis Kirschner, Senior Auditor

Ryan Perry, Senior Auditor

 

Appendix III

 

Report Distribution List

 

Principal Deputy Commissioner

Office of the Commissioner Ė Attn:† Chief of Staff† C

Deputy Commissioner for Operations Support† OS

Chief, Agency-Wide Shared Services† OS:A

Associate Chief Information Officer, Enterprise Operations† OS:CTO:EO

Director, Enterprise Computing Center† OS:CTO:EO:EC

Director, Real Estate and Facilities Management† OS:A:RE

Chief Counsel† CC

National Taxpayer Advocate† TA

Director, Office of Legislative Affairs† CL:LA

Director, Office of Program Evaluation and Risk Analysis† RAS:O

Office of Internal Control† OS:CFO:CPIC:IC

Audit Liaison:† Director, Risk Management Division† OS:CTO:SP:RM

 

Appendix IV

 

Outcome Measure

 

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration.† This benefit will be incorporated into our Semiannual Report to Congress

Type and Value of Outcome Measure:

       Cost Savings Ė Funds Put to Better Use Ė Potential; $15 million per year; $60 million over a four-year period (see page 3).

Methodology Used to Measure the Reported Benefit:

In a prior audit report[7] dated May 7, 2010, TIGTA recommended that current and future IRS data center space needs be identified and plans developed to consolidate or reduce excess data center space.† IRS management agreed with this recommendation and stated that the Enterprise Operations organization would take the lead in the process of assessing the current state of data center space and developing a plan of action for space consolidation, where appropriate.

In meetings with IRS managers in March 2012 during the planning of this audit, we inquired about the status and plans for the ECC-Detroit and were informed that no decision had been made.† During this audit, we observed significant empty data center space at the ECC-Detroit.† We were informed by IRS management that the IRSís lease for this space will expire in April 2015 and that the systems that remain at this location could be moved to other IRS data centers.† Therefore, it is our opinion that the IRS should close the data center at the ECC-Detroit. †In July 2012, during this audit, the IRSís CTO announced the closure of the ECC-Detroit. †Prior to the announcement, all of the DCCI documentation indicated that the IRS would maintain a data center at the ECC-Detroit.† We believe the decision to close the ECC-Detroit was due, in part, to this audit as well as the TIGTAís prior report recommendation.†

The IRSís CTO estimates that the IRS will save approximately $15 million annually by closing the ECC-Detroit.† The $15 million estimated annual savings for the closing of the ECC-Detroit was calculated in the following manner.† Currently, the IRS occupies 484,000 square feet of space in the Detroit facility, at a fully loaded rental rate of $53.23 per square foot.† This equates to approximately $25 million in annual costs.† When the IRS ceases data center operations in Detroit, the IRS will no longer require the facility to have Level 5 security, but rather a standard level, and will only require traditional office space.† As a result, the IRS will seek to obtain a rental agreement that is more commensurate with the current Detroit market rate of approximately $29 per square foot for (nonĖdata center) office space.† In addition, after the data center function leaves the facility, the IRS can eliminate the square footage currently needed for this function and can right-size the office space as well; the estimated space required for the IRS office function will be approximately 354,000 square feet.† If the IRS is able to obtain a rental rate approximating the $29 per square foot market rate and reduce the amount of space leased to approximately 354,000 square feet, the new estimated annual costs would be approximately $10 million, representing an estimated annual rent and operating cost savings of approximately $15 million ($25 million in current costs minus $10 million in new estimated costs). †While this benefit will extend beyond one year for the IRS, we are conservatively estimating the savings over four years.

 

Appendix V

 

Glossary of Terms

 

Term

Definition

Campus

The data processing arm of the IRS.† The campuses process paper and electronic submissions, correct errors, and forward data to the Computing Centers for analysis and posting to taxpayer accounts.

Consolidation

An approach to reducing data center space that involves moving servers to a few selected data centers or moving small data centers to larger centers.

Decommission

An approach to accomplishing data center consolidation that involves turning off servers that are not being used or are used infrequently.

Governance

A set of processes, guidelines, and policies that guide and affect the direction of an organizationís behavior or assets.

Green Information Technology

Optimal use of information and communication technology for managing the environmental sustainability of enterprise operations, as well as that of their products, services, and resources, throughout their life cycles.

Information Technology Infrastructure Libraryģ

A set of practices for information technology service management that focuses on aligning information technology services with the needs of business.† The Information Technology Infrastructure Libraryģ describes procedures, tasks, and checklists used by an organization for establishing a minimum level of competency.† It allows the organization to establish a baseline from which it can plan, implement, and measure.† It is used to demonstrate compliance and to measure improvement.†

Virtualization

An approach that helps to accomplish data center consolidation.† It involves moving applications and data on several physical servers onto a single virtual server.

 

Appendix VI

 

Managementís Response to the Draft Report

 

 

DEPARTMENT OF THE TREASURY

INTERNAL REVENUE SERVICE

WASHINGTON, D.C. 20224

 

 

 

CHIEF TECHNOLOGY OFFICER

 

 

May 13, 2013

 

 

MEMORANDUM FOR DEPUTY INSPECTOR GENERAL FOR AUDIT

 

FROM: ††††††††††††††† †††††††††††Terence V. Milholland /s/ Terence V. Milholland

†††††††††† Chief Technology Officer

 

 

SUBJECT: ††††††††††††††††††††† Draft Audit Report - The Data Center Consolidation Initiative Has Made Significant Progress, but Program Management Should Be Improved to Ensure Goals Are Achieved (Audit# 201220021) (e-trak 2013-38864)

 

Thank you for the opportunity to review and respond to the subject audit report.† We appreciate your flexibility in working with the IRS team in revising the draft and noting the positive accomplishments achieved by the Data Center Consolidation Initiative (DCCI).

 

The IRS is proud of the fact that in the first two years of the project, we have exceeded our goals of reducing data center space and are on track to meet our goals for the third year.† In 2011, we released 13 percent of space; in 2012, we released 12 percent of space; and this year, we are expecting to release another 13 percent of space.

 

We agree with seven of the recommendations in your report, as well as with the potential savings outlined in the Outcome Measure, and we partially agree with another recommendation.† The attachment contains our planned corrective actions to address each finding.

 

We value your continued support and the guidance your team provides.† If you have any questions, please contact me at (202) 622-6800 or a member of your staff may contact Lisa Starr, Senior Manager of Program Oversight, at (202) 283-3607.

 

Attachment

 

RECOMMENDATION #1:† To ensure that the IRS accomplishes its DCCI goals in the final three years, the CTO should provide, as appropriate, additional resources to support the DCCI PMO to ensure that the future project goals are accomplished and ensure that the DCCI PMO can maintain a role in guiding the implementation of the World-Class Data Center Study recommendations.

 

CORRECTIVE ACTION #1:† We agree with the recommendation.† In November 2012, the Associate Chief Information Officer, Enterprise Operations added appropriate resources to the DCCI project.† This action was taken to ensure that project goals would be accomplished and to ensure that the DCCI PMO maintained a role in guiding the implementation of the World Class Data Center Study recommendations in the computing centers.

 

IMPLEMENTATION DATE:† November, 2012

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† N/A

 

RECOMMENDATION #2:† To ensure that the IRS accomplishes its DCCI goals in the final three years, the CTO should require the DCCI PMO to develop a project management plan following project management best practices.† The plan should include the development of a comprehensive master program schedule, detailed implementation schedules, and spending plans and should address challenges to accomplishing project goals and solutions.

 

CORRECTIVE ACTION #2:† We agree with the recommendation.† As a result of discussions during this audit, the IRS DCCI PMO consolidated the existing strategy and planning documents and began developing a master project management plan.† The plan will address challenges, detail decisions, milestones and timeframes related to how the IRS will meet its remaining data center consolidation goals.

 

IMPLEMENTATION DATE:† September 25,2013

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #3:† To ensure that the IRS accomplishes its DCCI goals in the final three years, the CTO should ensure that the ECC-Detroit is consolidated into the Martinsburg and Memphis ECCs.

 

CORRECTIVE ACTION #3:† We partially agree with the recommendation.† By citing specific locations in the recommendation, the IRS's ability to make the best decision as to where the ECC-Detroit computing assets should be located is limited.† We will ensure the ECC-Detroit computing assets are consolidated into an appropriate facility.

 

IMPLEMENTATION DATE:† April 25, 2015

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #4:† To ensure the most energy and space-efficient data center configurations, the CTO should work with the Agency-Wide Shared Services, Real Estate and Facilities Management (REFM) organization to correct the inventory when the DCCI PMO identifies discrepancies during local site visits.

 

CORRECTIVE ACTION #4:† We agree with the recommendation.† To ensure the IRS accomplishes its DCCI goals in the final three years, Enterprise Operations will work with the Real Estate and Facilities Management (REFM) organization within Agency≠ Wide Shared Services (AWSS) to develop a formal process to correct the data center inventory.

 

IMPLEMENTATION DATE:† December 25,2013

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION# 5:† To ensure the most energy and space-efficient data center configurations, the CTO should formalize plans to develop and apply a baseline footprint to the IRS's small data centers.

 

CORRECTIVE ACTION #5:† We agree with the recommendation.† Enterprise Operations, in coordination with Agency-Wide Shared Services, Real Estate and Facilities Management (REFM), will develop a baseline footprint for IRS's small data centers.† This footprint will be applied to future build out and/or space reduction projects in our small data centers, where cost effective.

 

IMPLEMENTATION DATE:†† May 25, 2014

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #6:† To ensure the most energy and space-efficient data center configurations, the CTO should update the annual inventory submitted to the Department of the Treasury to include all data centers, regardless of size.

 

CORRECTIVE ACTION #6:† We agree with the recommendation.† The IRS will update the annual inventory submitted to the Department of the Treasury to include all data centers.

 

 

IMPLEMENTATION DATE:† June 25,2013

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion

 

RECOMMENDATION 7:† To ensure the most efficient use of computer room air conditioners at the IRS's data centers, the CTO, in coordination with the Chief, Agency≠Wide Shared Services, should identify and shut down computer room air conditioners no longer needed.

 

CORRECTIVE ACTION #7:† We agree with the recommendation.† To ensure the most efficient use of computer room air conditioners at the IRS's data centers, Enterprise Operations in coordination with Agency-Wide Shared Services, Real Estate and Facilities Management (REFM), will formalize a process for identifying and shutting down computer room air conditioners that are no longer needed.

 

IMPLEMENTATION DATE:† December 25, 2013

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.

 

RECOMMENDATION #8:† To ensure the most efficient use of computer room air conditioners at the IRS's data centers, the CTO, in coordination with the Chief, Agency≠Wide Shared Services, should adjust the set point temperatures as appropriate on the remaining computer room air conditioners in accordance with industry best practices.

 

CORRECTIVE ACTION #8:† We agree with the recommendation.† To ensure the most efficient use of computer room air conditioners at the IRS's data centers, Enterprise Operations, in coordination with Agency-Wide Shared Services Real Estate and Facilities Management (REFM), will formalize a process for ensuring computer room air conditioners are adjusted to industry standard air conditioner set point temperatures.

 

IMPLEMENTATION DATE:† December 25, 2013

 

RESPONSIBLE OFFICIAL:† Associate Chief Information Officer, Enterprise Operations

 

CORRECTIVE ACTION MONITORING PLAN:† We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES) and monitor them on a monthly basis until completion.



[1] See Appendix V for a glossary of terms.

[2] Data Center Optimization Act, H.R. 6278, 112th Cong. (2011Ė2012); Smart Energy Act, H.R. 4017, 112th Cong. (2011Ė2012); and Federal Cost Reduction Act of 2011, H.R. 1713, 112th Cong. (2011Ė2012).

[3] TIGTA, Ref. No. 2010-20-044, Implementing Best Practices and Additional Controls Can Improve Data Center Energy Efficiency and the Environmental and Energy Program (May 2010).

[4] Every year the Production Environment Control Process (formerly known as the Filing Season Moratorium) is put into place at the IRS to stabilize its production environments during peak processing times.† During this time, no changes to the production environment are allowed to be implemented without executive approval.

[5] The Uptime Institute is an unbiased, third-party data center research, education, and consulting organization focused on improving data center performance and efficiency through collaboration and innovation.

[6] A judgmental sample is a nonstatistical sample, the results of which cannot be used to project to the population.

[7] TIGTA, Ref. No. 2010-20-044, Implementing Best Practices and Additional Controls Can Improve Data Center Energy Efficiency and the Environmental and Energy Program, p. 12 (May 2010).