TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

Inadequate Early Oversight Led to Windows Upgrade Project Delays

 

 

 

September 28, 2015

 

Reference Number: 2015-20-073

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

 

Phone Number  /  202-622-6500

E-mail Address /  TIGTACommunications@tigta.treas.gov

Website           /  http://www.treasury.gov/tigta

 

 

HIGHLIGHTS

INADEQUATE EARLY OVERSIGHT LED TO WINDOWS UPGRADE PROJECT DELAYS

Highlights

Final Report issued on September 28, 2015

Highlights of Reference Number:  2015-20-073 to the Internal Revenue Service Chief Technology Officer.

IMPACT ON TAXPAYERS

Operating systems are critical software on computers that serve as a foundation to allow all other programs, software, and applications to run on the computers.  When an operating system reaches its end of life, companies such as Microsoft stop supporting the operating system, which leaves the systems vulnerable to attack.  For the IRS, the use of outdated operating systems may expose taxpayer information to unauthorized disclosure, which can lead to identity theft.  Further, network disruptions and security breaches may prevent the IRS from performing vital taxpayer services, such as processing tax returns, issuing refunds, and answering taxpayer inquiries.

WHY TIGTA DID THE AUDIT

The overall objective of this review was to review IRS efforts to upgrade the operating system on its Windows® workstations and servers.  This audit is included in TIGTA’s Fiscal Year 2015 Annual Audit Plan and addresses the major management challenge of Security for Taxpayer Data and Employees.

WHAT TIGTA FOUND

The IRS was unable to upgrade all of its Windows workstations from Windows XP and all of its Windows servers from Windows Server 2003 by the Microsoft end of life deadlines.  At the conclusion of our fieldwork, the IRS had not accounted for the location or migration status of approximately 1,300 workstations and upgraded only about one-half of its Windows servers from the 2003 software version to the 2008 release.  Since April 2011 when the IRS initially started the Windows workstation upgrade project, the IRS spent approximately $128 million to upgrade its Windows workstations and expects to spend an additional $11 million through the end of Fiscal Year 2015.  TIGTA found that the IRS did not follow established policies over project management and provided inadequate oversight and monitoring of the Windows XP upgrade early in its effort.

WHAT TIGTA RECOMMENDED

TIGTA recommended that the Chief Technology Officer:  1) ensure that all workstations have been adequately accounted for and upgraded to Windows 7; 2) ensure that enterprise-wide information technology maintenance and upgrade efforts going forward follow the Enterprise Life Cycle, as prescribed by IRS policy, to mitigate potential delays and to ensure project transparency and accountability; and 3) require appropriate Executive Steering Committees to oversee enterprise-wide information technology maintenance and upgrade efforts with regular project reviews and executive approvals.

The IRS agreed with two recommendations.  First, the IRS stated it has accounted for all workstations that need to be upgraded to Windows 7 and plans to track them until completed.  Second, the IRS plans to ensure that enterprise-wide upgrade efforts receive adequate oversight.

The IRS partially agreed with our recommendation that large-scale upgrade projects should follow the Enterprise Life Cycle.  It disagreed that all upgrade efforts should follow the Enterprise Life Cycle but agreed that large‑scale enterprise-wide efforts need to have a set of well documented minimum project documentation requirements to ensure that effective project management is adhered to for projects of this size.

 

September 28, 2015

 

 

MEMORANDUM FOR CHIEF TECHNOLOGY OFFICER

 

FROM:                       Michael E. McKenney /s/ Michael E. McKenney

                                  Deputy Inspector General for Audit

 

SUBJECT:                  Final Audit Report – Inadequate Early Oversight Led to Windows Upgrade Project Delays (Audit # 201520007)

 

This report presents the results of our review of the Internal Revenue Service (IRS) efforts to upgrade the operating system on its Windows® workstations and servers to Windows 7 and Windows Server 2012, respectively.  This audit is included in the Treasury Inspector General for Tax Administration’s Fiscal Year 2015 Annual Audit Plan and addresses the major management challenge of Security for Taxpayer Data and Employees.  This audit was also part of our statutory requirement to annually review the adequacy and security of IRS technology.

Management’s complete response to this report is included as Appendix V.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  If you have any questions, please contact me or Danny Verneuille, Acting Assistant Inspector General for Audit (Security and Information Technology Services).

 

 

Table of Contents

 

Background

Results of Review

The Windows Upgrade Project Did Not Follow Established Policy, Received Inadequate Early Oversight, and Continues to Be Delayed

Recommendation 1:

Recommendations 2 and 3:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Glossary of Terms

Appendix V – Management’s Response to the Draft Report

 

Abbreviations

 

CTO

Chief Technology Officer

ELC

Enterprise Life Cycle

ESC

Executive Steering Committee

IRS

Internal Revenue Service

 

Background

 

Windows XP for workstations and Windows Server 2003 for servers are Microsoft operating systems that have reached their end of life.

Operating systems[1] are critical software on computers that serve as a foundation to allow all other programs, software, and applications to run on the computers.  Simply put, it makes sure that different programs and users do not interfere with each other and operate as intended.  Because of its importance, operating systems must be updated on a regular basis to patch security vulnerabilities and, if necessary, upgraded completely in order to fix crucial weaknesses or to address new threats to its functionality.  The older an operating system gets, the more security vulnerabilities it has and, at some point, software companies such as Microsoft stop supporting the software with new patches.  When this occurs, vendors will offer more current versions of their operating system, to which organizations should upgrade in order to ensure that its computers are not vulnerable to attack and compromise.

Windows® XP for workstations and Windows Server 2003 for servers are Microsoft operating systems that have reached their end of life.  That means Microsoft made a business decision to stop supporting these operating systems effective April 2014 and July 2015, respectively, and encouraged customers to upgrade to more current versions of its operating systems.  Windows XP upgraded to Windows 7, and Windows Server 2003 upgraded to Windows Server 2008 and four years later Windows Server 2012.  For organizations that do not upgrade its Windows computers by the end of life deadline, Microsoft offers support for these systems on a contracted fee basis.

This review was performed with information obtained from Internal Revenue Service (IRS) Enterprise Operations officials located in IRS offices throughout the United States during the period December 2014 through June 2015.  We conducted this performance audit in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective.  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

 

Results of Review

 

The Windows Upgrade Project Did Not Follow Established Policy, Received Inadequate Early Oversight, and Continues to Be Delayed

The IRS was unable to upgrade all of its Windows workstations from Windows XP and all of its Windows servers from Windows Server 2003 by the Microsoft end of life deadlines.  We acknowledge that these Windows upgrade efforts were monumental and unprecedented for the IRS, particularly with the Windows XP upgrade due to its volume of approximately 110,000 workstations and geographical disbursement throughout the country, including Alaska, Hawaii, and Puerto Rico.  The IRS also discovered nearly 6,000 applications being used by employees to do their jobs that required an assessment of each application to determine whether it would operate on Windows 7.  In addition, budgetary constraints at the start of the Windows XP upgrade effort on April 2011 forced the IRS to upgrade old computers rather than purchase new computers, which would have made the upgrade process easier due to the compatibility of new hardware with new operating systems.  The IRS has spent almost $128 million over the past four years on its effort to upgrade Windows XP to the Windows 7 operating system and expects to spend an additional $11 million through the end of Fiscal Year 2015 for a total project cost of $139 million.

As of May 2015, the IRS has completed most of the Windows XP workstation upgrades across the country.  Approximately 1,300 workstations have yet to be located or confirmed as running the old operating system.  In addition, the IRS is about halfway through upgrading its Windows 2003 servers to the 2008 release of the Windows Server operating system and is now preparing for the 2012 software upgrade.  The IRS has not yet begun the upgrade to Windows Server 2012 and is in the initial planning stages for developing project budget estimates and other planning documents.  Approximately 3,000 Windows 2003 servers continue to be delayed for upgrade.  IRS officials informed us that they are uncertain this number is correct because when many of these servers were deployed, inventory controls were not in place and they are uncertain whether all of these servers are running the Windows 2003 version of the operating system.  Further, many of the legacy applications running on these servers cannot be upgraded to function properly on Windows Server 2008 without reengineering.  Finally, aging infrastructure has prevented some upgrades as well.

Upgrading to the new Microsoft workstation and server operating systems is critical because older versions are not supported and regularly patched for security flaws, which makes them more vulnerable to hacking.  As a result, the IRS does not have the latest ability to combat data breaches and remains at risk of hacking attempts and data loss or corruption due to malware.  When the IRS’s data and network are not secured, taxpayer information becomes vulnerable to unauthorized disclosure, which can lead to identity theft.  Further, security breaches can cause network disruptions and prevent the IRS from performing vital taxpayer services, such as processing tax returns, issuing refunds, and answering taxpayer inquiries.

Policies and established internal controls were bypassed in the Windows 7 upgrade effort

In April 2011, the IRS Information Technology organization approved the Microsoft Technologies Program Management Office operating charter, whose coverage consisted of six Microsoft products including the Windows 7 upgrade effort.  Information technology projects at the IRS are typically overseen by an executive steering committee (ESC).  The primary objective of the ESC is to ensure that information technology infrastructure investment, program, and project objectives are met; risks are managed appropriately; and the expenditure of enterprise resources is fiscally sound.  The ESC provides governance for all investments, programs, and projects within its assigned portfolio, including infrastructure investments such as hardware and software acquisitions.  The ESC conducts milestone reviews for projects and decisions are documented and signed by executive members.  In this manner, project cost and schedule overruns are clearly documented and resource adjustments are transparent.

Because the Windows 7 effort was not making sufficient progress in its completion, the Chief Technology Officer (CTO) made the decision in July 2012 to oversee the Windows 7 upgrade directly due to its complexity and magnitude.  Further, the CTO decided to bypass IRS Enterprise Life Cycle (ELC) policy.  No ELC artifacts were created or signed after the initial project charter document, which was approved in April 2011.  The IRS ELC policy outlines the repeatable processes and deliverables that IRS project managers are required to follow in order to mitigate risks when implementing information systems initiatives.

The ELC Planned Maintenance Path is a process used for planned system maintenance of a non‑emergency nature.  This path manages change in an organized manner, minimizes the disruption caused by frequent system changes, and increases the efficiency and effectiveness of the system change process.  Under the path, changes may include combinations of different types of maintenance, including corrective maintenance (e.g., fix errors, bugs, defects; replace equipment) or adaptive maintenance (e.g., conform to changed environment—an operating system upgrade or change of database management system).

IRS management contends that operating system upgrades are not required to follow ELC guidance and that the ELC policy is largely development-centric.  We agree that the policy largely addresses information technology development, but the processes, documentation, and review are equally important for large-scale enterprise-wide software upgrades.  IRS management informed us the Windows 7 project followed sound project management principles.  However, Federal agencies are held to a higher standard than other organizations especially with regard to transparency and accountability.  Without maintaining similar project documentation which provides version control and approval signatures, projects and initiatives run the risk of delays, less transparency, and accountability.  This results in difficulty assessing whether money could have been saved through alternative choices.  We concluded that management should have followed established policy for project management by using the ELC process in the Windows 7 enterprise-wide upgrade effort.

While the CTO’s decision may have been made to ensure high-level emphasis and attention, the IRS was unable to show and prove decisions were made after appropriate discussion and consideration of various factors.  In addition to artifacts like the project management plan, transition management plan, and system deployment plan that the ELC requires for the maintenance path, projects overseen by IRS ESCs must pass milestone readiness and exit reviews by participating executives who also document and sign significant project decisions.  No official meeting minutes with the CTO or decision documents were created or signed throughout the Windows 7 project.  While Information Technology officials stated that the briefings with the CTO provided sufficient oversight and transparency over project decisions and progress, we believe the IRS should have maintained official documentation for recordkeeping purposes.

Windows 7 upgrade delays continue to present security risks

Microsoft ended mainstream support for Windows XP in April 2009.  Two years later, in April 2011, the IRS signed the project charter to begin the process for upgrading to Windows 7.  However, the IRS did not actually begin upgrading workstations until September 2012.  Microsoft announced that extended support for Windows XP would be discontinued in April 2014.  IRS contracted with Microsoft to provide continued support for one additional year beyond this deadline.  Figure 1 provides the timeline of the IRS’s Windows 7 upgrade project.

 Figure 1:  Timeline for IRS Windows Workstations Upgrade

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Source:  IRS and Microsoft.

The IRS informed us that all workstations were upgraded and transitioned to the User and Network Services function as of December 2014, eight months after Microsoft’s April 2014 deadline for extended support.  However, the IRS also informed us that there are approximately 1,300 workstations that, according to their inventory records, are still running an outdated operating system.  Project managers responsible for the upgrade could not confirm this number because they cannot locate these workstations and the inventory system is not accurate.[2]  IRS management stated that with an enterprise-wide initiative this large and the complexity of its environment, it is unreasonable to expect 100 percent of the workstations to be migrated without collaboration with other key Information Technology delivery partners assisting during the clean-up period.  We believe that running workstations with outdated operating systems pose significant security risks to the IRS network and data, particularly in the environment where a chain is only as strong as its weakest link.  External hackers or malicious insiders need to locate only the one computer with security weaknesses, such as one with an outdated operating system, to exploit in order to steal data or further compromise other computers.

Despite the eventual progress made by the IRS on the Windows XP upgrade efforts, we believe the IRS provided inadequate oversight and monitoring during the early phases of this effort, starting with including it among other Microsoft product upgrades rather than making this effort its own project up to the decision made by the CTO to oversee the project himself.  In addition, after taking four years to upgrade to Windows 7, the IRS is now faced with the challenge of addressing Microsoft’s announcement to end extended support for Windows 7 in January 2020.

The IRS is only halfway through completing its upgrade of its Windows servers to an operating system that is already seven years old

Based on our discussions with management, we determined it is unlikely that the IRS will have its servers upgraded to Windows Server 2012 any time this Fiscal Year.  The Windows Server 2012 release has many security improvements including smart card two-factor authentication, access control improvements, policy-based access control management for applications, and anti-malware protection, making it less vulnerable to hacking.  The IRS only recently, in March 2015, assigned a project manager over the migration to Windows Server 2012, and basic planning documents such as budget estimates and deployment schedules are still unsigned and incomplete.

In fact, the IRS still has not fully upgraded its servers from Windows Server 2003 to the 2008 release.  Currently, the IRS has approximately 3,000 Windows servers still running the 2003 operating system.  Management informed us that they have upgraded approximately 4,100 servers to the 2008 version which is already seven years old.  The IRS currently has no servers running the 2012 operating system in production at this time.  It has three or four test servers that it is using to develop policies and expects to begin testing in June 2015.  Until testing is complete, no servers running the 2012 operating system will be deployed into production.[3]  The server upgrade project team has completed no ELC documents because they are treating this effort as a refresh or upgrade—not a development project—as directed by Enterprise Operations management.  When we met with the server upgrade team, a Microsoft contractor participated in our meeting and informed us that Microsoft would continue to support the IRS servers running the 2003 operating system version on an ongoing basis for a premium fee.  Figure 2 provides the timeline of the IRS’s Windows server upgrade project.

Figure 2:  Timeline for IRS Windows Servers Upgrade

Figure 2 was removed due to its size.  To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Source:  IRS and Microsoft.

Similar to the Windows 7 effort, the Windows server upgrade lacks sufficient oversight and accountability and delays in upgrading pose a risk of weakening the IRS security posture.  The IRS will begin paying a premium for extended service on an outdated server operating system that no longer receives critical security upgrades automatically from the vendor.  As a result, we determined the IRS has not adequately planned for the Windows server upgrade in regard to the costs, potential security implications, and amount of time necessary to complete the upgrade.

Recommendations

The Chief Technology Officer should:

Recommendation 1:  Ensure that all workstations have been adequately accounted for and upgraded to Windows 7.

Management's Response:  The IRS agreed with this recommendation and stated that it has already accounted for all workstations and associated upgrades to Windows 7 and that security mitigations are in place to ensure that the remaining workstations cannot be exploited by malware.  The IRS will track remaining upgrades until complete.


 

Recommendation 2:  Ensure that enterprise-wide information technology maintenance and upgrade efforts going forward follow the ELC, as prescribed by IRS policy, to mitigate potential delays and to ensure project transparency and accountability.

Management's Response:  The IRS partially agreed with this recommendation.  While the IRS acknowledged the ELC is principally a project management methodology to be used for large-scale software development projects and the introduction of new technology, it disagreed that this methodology must be used for infrastructure upgrades.  However, the IRS did agree that large-scale, enterprise-wide efforts such as the two Windows upgrade projects need to have a set of well documented minimum project documentation requirements to ensure that effective project management is adhered to for projects of this size.  The IRS Information Technology organization will seek our advice on representative processes in Government.

Recommendation 3:  Require appropriate ESCs to oversee enterprise-wide information technology maintenance and upgrade efforts with regular project reviews and executive approvals.  This should include the Windows server upgrade project for which an adequate plan is needed to prepare for the end of extended support for Windows Server 2008 and for the upgrade to Windows Server 2012.

Management's Response:  The IRS agreed with this recommendation.  The IRS maintains the position that this infrastructure upgrade effort had appropriate oversight and governance and successfully delivered in a manner that exceeded the intent of our recommendation.  However, the IRS will seek our advice on representative processes in Government.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

Our overall objective was to review IRS efforts to upgrade the operating system[4] on its Windows® workstations and servers to Windows 7 and Windows Server 2012, respectively.  To accomplish this objective, we:

I.          Evaluated IRS efforts to upgrade its Windows workstations to the Windows 7 operating system.

A.        Determined the overall project costs to date and estimated costs remaining.

B.        Evaluated overall project oversight.

1.         Reviewed the CTO’s oversight responsibilities because there was no direct governance board oversight.

2.         Reviewed the roles and responsibilities of the Windows 7 Project Manager to determine his or her scope of oversight and control within the IRS and over contractors.

3.         Reviewed project artifacts.

4.         Reviewed meeting minutes for appropriate attendees, content, and decisions made to determine whether participation was adequate.

C.        Evaluated IRS progress in meeting project deadlines.

II.         Evaluated IRS planning to upgrade its Windows servers to the Windows Server 2012 operating system.

A.        Reviewed the planning documentation and artifacts completed for the server upgrade for completeness.

B.        Evaluated the overall IRS Windows server upgrade project estimates for cost and schedule reasonableness.

Internal controls methodology

Internal controls relate to management’s plans, methods, and procedures used to meet their mission, goals, and objectives.  Internal controls include the processes and procedures for planning, organizing, directing, and controlling program operations.  They include the systems for measuring, reporting, and monitoring program performance.  We determined that the following internal controls were relevant to our audit objective:  IRS guidelines for information technology projects and project management best practices.  We evaluated these controls by conducting interviews and meetings with Windows 7 and Windows Server project management in the Enterprise Operations function.  We also reviewed project documentation.

 

Appendix II

 

Major Contributors to This Report

 

Alan R. Duncan, Assistant Inspector General for Audit (Security and Information Technology Services)

Kent Sagara, Director

Joseph F. Cooney, Audit Manager

Jena Whitley, Lead Auditor

George L. Franklin, Senior Auditor

Louis Lee, Senior Auditor

Midori Ohno, Senior Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Deputy Chief Information Officer for Operations  OS:CTO

Associate Chief Information Officer, Enterprise Operations  OS:CTO:EO

Associate Chief Information Officer, User and Network Services  OS:CTO:UNS

Chief Counsel  CC
National Taxpayer Advocate  TA

Director, Office of Audit Coordination  OS:PPAC:AC
Director, Office of Program Evaluation and Risk Analysis  RAS:O
Office of Internal Control  OS:CFO:CPIC:IC

Audit Liaison:  Director, Risk Management Division  OS:CTO:SP:RM

 

Appendix IV

 

Glossary of Terms

 

Term

Definition

Accountability

Ensuring that officials in an organization are answerable for their actions and that there is redress when duties and commitments are not met.

Artifact

The tangible result (output) of an activity or task performed by a project during the life cycle.

Chief Technology Officer

Leads the organization and advises the Commissioner about information technology matters, manages all IRS information system resources, and is responsible for delivering and maintaining modernized information systems throughout the IRS.

Contractor

An organization external to the IRS that supplies goods and services according to a formal contract or task order.  A contractor is a type of provider.

ELC Planned Maintenance Path

Has defined requirements, sequential progression through the phases, evolving teams, and uses a developmental technical approach for correcting solutions that are already in production.  Planned Maintenance typically addresses numerous system requirements and simultaneously implements all of the changes as a new version (or release) of the system.

Enterprise Life Cycle

The approach used by the IRS to manage and effect business change.  The ELC provides the direction, processes, tools, and assets for accomplishing business change in a repeatable and reliable manner.

Enterprise Operations

Supports the mainframe and server environment by providing efficient, cost effective, secure, and highly reliable server and mainframe services for all IRS business entities and taxpayers.  It maintains and operates the tax processing systems of the IRS to provide data and other information technology services to the Nation’s taxpayers and the IRS Business Operating Divisions.

Executive Steering Committee

The primary objective of the ESC is to ensure that information technology infrastructure investment, program, and project objectives are met; risks are managed appropriately; and the expenditure of enterprise resources is fiscally sound.  The ESC provides governance for all investments, programs, and projects within its assigned portfolio.

Governance

The decisionmaking authority over a project or program derived by a formal charter in accordance with the IRS Governance directive.

Hardware

The physical parts of a computer and related devices.  It includes motherboards, hard drives, monitors, keyboards, mice, printers, and scanners.

Information Technology

Any equipment or interconnected system or subsystem of equipment that is used in the automatic acquisition, storage, manipulation, management, movement, control, display, switching, interchange, transmission, or reception of data or information by an executive agency.

Microsoft XP Extended Support

Microsoft offers extended support for a minimum of five years from the date of a product’s general availability.  When products exit Extended support, paid custom support contracts (if offered) are the only option, and updates to the product, including security updates, should not be expected outside of the terms defined within paid custom support contracts.

Milestone

A project management decision point placed at a natural breakpoint in the life cycle where management determines whether a project can proceed to the next phase.  Milestones are points at which management requires updated cost, progress, and risk information to make project funding and decisions for project continuation.

Operating System

The software that communicates with computer hardware to allocate memory, process tasks, access disks and peripherals, and serves as the user interface.

Oversight

IRS management of project work conducted by outside contractors to assure that IRS needs and contractual terms are met.  Also, monitoring or governance of IRS projects by organizations outside the IRS.

Project Manager

Responsible for ensuring the project is progressing properly.  The Project Manager also ensures that team members have the information and tools necessary to complete their work, organizes meetings, facilitates release planning, and monitors work being done.

Project Risk

The risk that the project will not be completed on schedule or within budget.

Risk

The level of impact on agency operations (including mission, functions, image, or reputation), agency assets, or individuals, resulting from the operation of an information system given the potential impact of a threat and the likelihood of that threat occurring.

Software

A general term that describes computer programs and consists of lines of code written by computer programmers that have been compiled into a computer program.

Transparency

An organization is open in the clear disclosure of information, rules, plans, processes, and actions.  Information should be presented in plain and readily comprehensible language and formats appropriate for different stakeholders and made available in sufficient time to permit analysis, evaluation, and engagement by relevant stakeholders.

Windows 7

The seventh version of the Microsoft Windows Operating System, introduced in October 2009.

Windows XP

Introduced in October 2001, it was one of Microsoft’s most popular operating systems, and in April 2014 Microsoft ended their extended support, which is also known as “end of life.”

Workstation/Desktop

A computer that is designed to stay in a single location and can be part of a network or a standalone machine.

 

Appendix V

 

Management’s Response to the Draft Report

 

DEPARTMENT OF THE TREASURY

INTERNAL REVENUE SERVICE

WASHINGTON, D.C. 20224

 

CHIEF TECHNOLOGY OFFICER

 

 

September 9, 2015

 

 

MEMORANDUM FOR DEPUTY INSPECTOR GENERAL FOR AUDIT

 

FROM:                             Terence V. Milholland /s/ Terence V. Milholland

                                          Chief Technology Officer'

 

SUBJECT:                       Draft Audit Report - Inadequate Early Oversight Led to Windows Upgrade Project Delays (201520007)

 

Thank you for the opportunity to review your draft audit report on two critical infrastructure upgrade projects - Windows workstations and Windows serversFollowing are comments to clarify and provide additional context around the issues you raise in your report.

 

We appreciate your acknowledgement that these Windows upgrade efforts were monumental for the Internal Revenue Service, challenged by a number of factors including the sheer number of workstations (approximately 110,000), nearly 6,000 applications that required migration determinations, technology incompatibilities and resource and budgetary constraintsHowever, there are several points in your report that we believe do not accurately reflect our efforts on this deployment.

 

Project Management and Governance

 

The audit report indicates that the IRS IT organization did not follow all project management policiesIt also does not acknowledge that the approach we used, which included direct oversight by the CTO and the leadership team as the governance body, resulted in the project successfully delivering the upgradesAlthough the audit report indicates that the projects may have benefited by producing various project management artifacts, it does not identify any specific instances where critical decisions were not made timely or where impacted stakeholders were harmed as a result of the lack of full recordkeeping.

 

The Enterprise Life Cycle (ELC), that the report references, is principally a project management methodology to be used for large scale software development projects and the introduction of new technologyWe disagree that the ELC methodology must be used for infrastructure upgrades.  However, we agree that large scale, enterprise-wide efforts such as the two Windows upgrade projects need to have a minimum set of project documentation requirements to ensure that effective project management is adhered to for projects of this size.

 

The audit report states that the upgrade projects received inadequate oversight and monitoring.  Enhanced CTO level oversight and governance of this project provided equivalent or better oversight of the project then required by the ELCDuring the critical implementation phase, the project received direct CTO level oversight with IT and Business executive leadership engagement, including:  IT Enterprise Governance briefings; monthly CTO briefings with ACIOs; weekly meetings with business units' Business Systems Planners; weekly meetings with the Business Systems Modernization Executive, and weekly contract and financial status reviews with the executive sponsor.  These practices enabled the IRS to dramatically increase the velocity of upgrades while minimizing risks and costs.  They also helped to reduce complexity, mitigate delays, and improve transparency and stakeholder accountability.

 

Inventory

 

The audit incorrectly concludes that IRS has not accounted for all XP workstations.  We acknowledge there were challenges with our inventory data due to the many antiquated systems in our IT ecosystem.  In spite of this, we took extraordinary steps to identify, document and upgrade every XP workstation in the IRS.  On several occasions throughout the audit, the IRS provided information to the TIGTA team that clearly documented the number of workstations to be upgraded, where those workstations were located, and our strategy to complete the upgrades.  Although, footnoted in the report, TIGTA opted not to change their assertion that the IRS had not accounted for all XP workstations.  As of this date, only 71 Windows XP workstations remain to be migratedRisks on these workstations have been mitigated and upgrades will be completed by the end of this calendar year. ·

 

Security

 

Finally, we disagree with the audit report conclusion that the IRS did not take appropriate steps to ensure the security of the XP workstations during the upgrades.  We mitigated our risks with a number of documented tasks which we shared with TIGTA.  For example, the report does not mention that the IRS established a Microsoft Extended Support Agreement, during the period April 2014 through April 2015, that enabled IRS to continue to receive patches and security updates for the XP workstations.  The IRS also implemented a strong risk mitigation strategy for the remaining XP workstations.  We took immediate steps to develop and execute the strategy with specific threat management options to reduce the risk of exposure to exploitation through the use of malware.  Details of the XP Mitigation strategy are described in the risk based decision document dated February 19, 2015 and CTO Close out Report, both of which were made available to TIGTA at the beginning of this audit.  This strategy is in line with our security risk management program and has served to ensure that no Windows XP workstations were left open to known security exploits.

 

We are committed to continuously improving our information technology systems and processes.  We value your continued support and the assistance and guidance your team provides.  If you have any questions, please contact me at (240) 613-9373 or Karen Mayr at (202) 368-8396.

 

RECOMMENDATION #1:  Ensure that all workstations have been adequately accounted for and upgraded to Windows 7.

 

CORRECTIVE ACTION #1:  We agree with this recommendation, questioning its necessity, as the IRS already accounts for all workstations and associated upgrades to Windows 7.  Security mitigations are in place to ensure the remaining 71 XP workstations cannot be exploited by malware.  We are tracking remaining upgrades until complete.

 

IMPLEMENTATION DATE:  12/31/15.

 

RESPONSIBLE OFFICIAL:  Associate CIO, User and Network Services

 

CORRECTIVE ACTION MONITORING PLAN:  We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES).  These Corrective Actions are monitored on a monthly basis until completion.

 

RECOMMENDATION #2:  Ensure that enterprise-wide information technology maintenance and upgrade efforts going forward follow the Enterprise Life Cycle as prescribed by the IRS policy to mitigate potential delays and to ensure project transparency and accountability.

 

CORRECTIVE ACTION #2:  We partially agree with this recommendation. IRS's Enterprise Life Cycle (ELC) is principally a project management methodology to be used for large-scale software development projects and the introduction of new technology.  We disagree that this methodology must be used for infrastructure upgrades.  However, we agree that large-scale, enterprise-wide efforts such as the 2 Windows upgrade projects need to have a set of well documented minimum project documentation requirements to ensure that effective project management is adhered to for projects of this size.  The IRS IT organization will seek TIGTA's advice on representative processes in government.

 

IMPLEMENTATION DATE #2:  July 1, 2016

 

RESPONSIBLE OFFICIAL #2:  Associate CIO, Strategy and Planning

 

CORRECTIVE ACTION MONITORING PLAN:  We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES).  These Corrective Actions are monitored on a monthly basis until completion.

 

RECOMMENDATION #3:  Require appropriate Executive Steering Committees to oversee enterprise-wide information technology maintenance and upgrade efforts with regular project reviews and executive digital signatures.  This should include the Windows server upgrade project where an adequate plan is needed to prepare for the end of extended support for Windows Server 2008 and for the upgrade to Windows Server 2012.

 

CORRECTIVE ACTION #3:  We agree with this recommendation.  We believe this infrastructure upgrade effort had appropriate oversight and governance, and successfully delivered in a manner that exceeded the intent of your recommendation.   However, as stated above in Corrective Action #2, we will seek TIGTA's advice on representative processes in government.

 

IMPLEMENTATION DATE #3:  July 1, 2016

 

RESPONSIBLE OFFICIAL #3:  Associate CIO, Strategy and Planning

 

CORRECTIVE ACTION MONITORING PLAN:  We enter accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES).  These Corrective Actions are monitored on a monthly basis until completion.



[1] See Appendix IV for a glossary of terms.

[2] After the conclusion of our fieldwork, the IRS provided documentation that these workstations were located and upgraded to Windows 7, as of July 22, 2015.  We were unable to verify this information.

[3] After the conclusion of our fieldwork, the IRS provided us updated information about its Windows server upgrades.  As of July 22, 2015, the IRS has updated approximately 61 percent of its Windows Server 2003 systems to the 2008 release.  Further, there are now 42 Windows Server 2012 systems running in the test environment and none in production.  We were unable to verify this information.

[4] See Appendix IV for a glossary of terms.