Additional Actions Are Needed to Establish and Maintain Controls Over Computer Hardware and Software Changes

 

December 2003

 

Reference Number:2004-20-026

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

December 16, 2003

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:†††† Gordon C. Milbourn III /s/ Gordon C. Milbourn III

†† ††††††††††††††Acting Deputy Inspector General for Audit

 

SUBJECT:†††† Final Audit Report Ė Additional Actions Are Needed to Establish and Maintain Controls Over Computer Hardware and Software Changes (Audit # 200320015)

 

This report presents the results of our review of the Internal Revenue Serviceís (IRS) configuration management (CM) process for computer hardware and software.The overall objective of this review was to determine whether the IRSí Modernization and Information Technology Services (MITS) organization effectively implemented an enterprise-wide CM process.

The IRS is dependent on a large collection of computer systems with complex interdependencies among a network of mainframe computers, mid-range computers, individual computers, several hundred vendor-supplied software products, and millions of lines of computer code.The MITS organization is currently modernizing, consolidating, and maintaining these computer systems to support the mission of the IRS.Responsibility for managing these computer systems is divided among the MITS organizations as follows:

        The Information Technology Services (ITS) organization develops, operates, and maintains computer hardware and software that supports the production environment.

        The Business Systems Modernization Office (BSMO) acquires and delivers new computer hardware and software for the IRSí modernized business processes.

Among the disciplines needed to manage and coordinate these efforts is an integrated CM process to ensure that the integrity and consistency of the IRSí computer systems are maintained throughout their life cycles.The CM process systematically identifies and baselines the items that make up a system (identification), formally controls any modifications to those items (control), reports on the status of the CM process (status accounting), and ensures that baseline configurations are implemented (audit).

In summary, the MITS organization has made progress in defining and establishing an enterprise-wide CM process through the issuance of a CM Directive that describes the CM process to be used throughout the MITS organization, and standard operating procedures (e.g., configuration control boards, configuration items and baselines, and configuration control).The MITS organization has chartered a Configuration Management Working Group to establish, maintain, and improve the CM process.

However, the CM functions (i.e., identification, control, status accounting, and audit) have not been uniformly implemented within the MITS organization.An integrated, enterprise-wide implementation of the CM process within the MITS organization is particularly important for modernized systems that will migrate in stages or releases to ensure computer system changes are properly managed throughout their life cycles.In addition, this process provides a means to document, communicate, and coordinate system development and production CM baselines between the BSMO and the ITS organizations.For example, as a result of CM control weaknesses, the Enterprise Systems Management (ESM) project incurred additional contractor costs of approximately $216,500 and a 4-month schedule delay to rollout ESM Release 2.1.Without an integrated and uniform CM process, there is an increased potential that modernized and existing systems will require extensive rework resulting in additional costs, schedule delays, and other risks to the IRSí computer operations (e.g., system outages and data corruption).

The implementation deficiencies found in the MITS organizationís CM processes occurred because the MITS CM Directive and procedures did not establish executive level responsibility that would ensure that:

        The CM processes were implemented throughout the ITS organization and coordinated with the BSMO.

        Deficiencies identified in internal CM assessments were appropriately addressed.

        The different CM software used by the MITS organization facilitated enterprise-wide CM.

        Policies were established defining authority levels and threshold criteria to approve and control production changes in the ITS organization.

To promote the establishment of an integrated MITS organization CM process, we recommended that the Chief Information Officer (CIO) modify the MITS CM Directive and procedures to:1) assign responsibility for ensuring that MITS CM processes are implemented throughout the ITS organization and coordinated with the BSMO and that CM deficiencies are appropriately addressed; and 2) establish governance policies, similar to those used by the BSMO, for defining the authority levels and threshold criteria to approve and control changes to the production environment in the ITS organization. Additionally, we recommended that the CIO develop a transition plan to implement standardized Enterprise Architecture compliant CM software to be used throughout the MITS organization to facilitate CM on an enterprise-wide level.

Managementís Response:IRS management agreed with our recommendations.The MITS organization will revalidate its CM Directive to address organizational responsibility, governance policy, and needed improvements in the Configuration Control Board (CCB) structure.In addition, management will address the establishment of governance policies and threshold criteria to approve and control changes to the ITS production environment while establishing plans to organize separate CCBs for the ITS and BSMO organizations.Regarding CM software, the MITS organization will identify acceptable CM software and publish applicable guidance upon the completion of ongoing CM software assessments within the ITS organization.Managementís complete response to the draft report is included as Appendix VI.

Copies of this report are also being sent to the IRS managers who are affected by the report recommendations.Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

Table of Contents

Background

Progress Has Been Made in Implementing Configuration Management Processes

Additional Actions Are Needed to Establish Integrated, Enterprise-Wide Configuration Management Processes

Recommendations 1 and 2:

Appendix I Ė Detailed Objective, Scope, and Methodology

Appendix II Ė Major Contributors to This Report

Appendix III Ė Report Distribution List

Appendix IV Ė Outcome Measures

Appendix V Ė Overview of Configuration Management Functions

Appendix VI Ė Managementís Response to the Draft Report

 

Background

The Internal Revenue Service (IRS) is dependent on a large collection of computer systems with complex interdependencies among a network of mainframe computers, mid-range computers, individual computers, several hundred vendor-supplied software products, and millions of lines of computer code.The IRSí Modernization and Information Technology Services (MITS) organization is currently modernizing, consolidating, and maintaining these computer systems to support the IRSí mission.Responsibility for managing these computer systems is divided among the MITS organizations as follows:

        The Information Technology Services (ITS) organization develops, operates, and maintains computer hardware and software that supports computer systems in production.

        The Business Systems Modernization Office (BSMO) acquires and delivers new computer hardware and software for the IRSí modernized business processes.

An integrated, enterprise-wide configuration management (CM) process is essential for ensuring that the integrity and consistency of the IRSí computer systems are maintained throughout their life cycles.The purpose of CM is to systematically identify and baseline the items that make up a system (identification), formally control any modifications to those items (control), report on the status of the CM process (status accounting), and ensure that baseline configurations are implemented (audit).

Both the Treasury Inspector General for Tax Administration (TIGTA) and the General Accounting Office (GAO) have issued reports on the IRSí Business Systems Modernization efforts that commented on the MITS organizationís CM process.Our report focuses on the implementation of the CM processes throughout the MITS organization and on selected modernization projects that had migrated from the BSMO to support and maintenance within the ITS organization.Our audit work in the Office of Security Services was limited due to the anticipated restructuring of that office as part of the realignment of the IRSí management structure.Personnel from the Office of Security Services indicated that they plan to place security policy documentation under CM control; therefore, no additional fieldwork was performed in that office.

Audit work was conducted in the MITS organization at the IRS National Headquarters in New Carrollton, Maryland, from May through September 2003.The audit was conducted in accordance with Government Auditing Standards.Detailed information on the audit objective, scope, and methodology is presented in Appendix I.Major contributors to the report are listed in Appendix II.

Progress Has Been Made in Implementing Configuration Management Processes

MITS organization management has recognized the need to institutionalize an enterprise-wide CM process throughout their organization and issued a CM directive in August 2002 to support that need.Management has taken several specific actions to implement this directive as well as an enterprise-wide CM process.Specifically, the MITS organization has taken the following actions:

        Chartered a MITS Configuration Management Working Group (CMWG) to establish, maintain, and improve CM processes, procedures, and techniques to be used throughout the MITS organization.

        Issued various CM standard operating procedures (e.g., configuration control boards [CCB], configuration items [CI] and baselines, configuration control, and configuration management process compliance assessments).

        Conducted CM process compliance assessments of eight BSMO projects to evaluate whether CM policy and procedures were being followed.

        Chartered a MITS CCB as the authority for receiving, reviewing, and approving proposed system change requests and changes to system baselines that have a cost impact that exceeds the dollar threshold and authority levels established for lower level project CCBs.The MITS organization CCB is also intended to be the forum to resolve conflicts such as those resulting from request impact analysis and authority issues that occur at or among the subordinate CCBs for individual projects.

        Chartered project level CCBs within the BSMO, such as the Internet Refund/Fact of Filing (IRFOF) Project and the Infrastructure Modernization Project.

        Issued a directive that defined the BSMOís authority levels (e.g., for BSMO project level CCBs) and threshold criteria for changing BSMO project baselines for schedule, cost, or requirements.For example, the BSMO Infrastructure Modernization Project CCB has the authority for approving proposed change requests that affect infrastructure modernization projects with a cost impact threshold of less than $500,000, and those above this level would be forwarded to a higher level CCB.

        Chartered organizational level CCBs within ITS (e.g., for the Detroit, Martinsburg, and Tennessee Computing Centers).

        Established the Office of Configuration Management (OCM) within the BSMO, whose chief, as chair of the MITS CMWG, is responsible for establishing, maintaining, and improving CM processes and procedures throughout the MITS organization.

The ITS organization has an effort underway to align the existing computing center CM processes with the Triplex Strategy.Further, the BSMO has developed a CM training plan, developed CM training courses, and held initial CM classes for the MITS organization.

Our review identified that the MITS organization has made progress in implementing a CM process; however, as explained below, further actions are needed to establish and integrate uniform CM implementation processes across the MITS organization.

Additional Actions Are Needed to Establish Integrated, Enterprise-Wide Configuration Management Processes

Treasury Directive 84-01, Information Systems Life Cycle Manual, dated March 2002, requires CM to be used throughout every projectís life cycle.It also defines the four CM functions of identification, control, audit, and status accounting.The IRS has incorporated these requirements into its systems life cycle methodologies (Enterprise Life Cycle [ELC] and ELC-Lite), the Enterprise Architecture (EA), and the MITS organization CM Directive and procedures.The MITS organization CM Directive also cites the American National Standards Institute/Electronic Industries Alliance Standard 649, National Consensus Standard for Configuration Management, an industry CM best practice.Additionally, the Office of Management and Budget Circular A-123, Management Accountability and Control, dated July 1995, requires that the appropriate authority, responsibility, and accountability are defined and delegated to accomplish the implementation of the CM process and that an appropriate organizational structure is established to effectively carry out these CM responsibilities.

However, the CM functions outlined in Treasury Directive 84-01 have not been uniformly implemented within the MITS organization.Specifically, the following areas could be improved for each of the four required CM functions:

Identification:The IRS identified CIs for the current production environment that affected modernization project releases in 2002. However, not all ITS divisions have identified and baselined the CIs for their production systems.An OCM contractor was identifying the CIs; however, the effort was not completed because funding for the contractors was cut in February 2003.An effective CM process requires that CIs be identified.These items must be identified and controlled prior to establishing system baselines for production systems that will be affected by BSMO projects.

Control:The BSMO has chartered project level CCBs to control changes to the BSMO project baselines and established threshold criteria for decision-making by the project and MITS CCBs, as well as Executive Steering Committees.However, the ITS organization has not chartered lower level CCBs, except for the Enterprise Operations Servicesí (EOS) Computing Center CCBs, which are change management rather than CM oriented.The EOS plans to establish a CM process as part of its Triplex initiative.An effective CM process requires a CCB to set and control baselines.

Status Accounting and Audit:The BSMO is performing the CM status accounting and audit functions.The OCM performed CM compliance assessments of several BSMO projects and identified problems with at least one of these functions in each assessment.Although the BSMO project managers have the responsibility to correct project specific CM process issues, the OCM does not have the authority to ensure that the issues are corrected.Also, the ITS organization is not performing the status accounting and audit functions because it has not finished establishing the identification and control processes.

Without an integrated, enterprise-wide CM process, the IRS cannot adequately assure that changes to its computer system configurations are properly managed throughout their life cycles.An integrated, enterprise-wide implementation of the CM process within the MITS organization is particularly important for modernized systems that will migrate in stages or releases to ensure computer system changes are properly managed throughout their life cycles.In addition, this process provides a means to document, communicate, and coordinate system development and production CM baselines between the BSMO and the ITS organizations.

For example, the Enterprise Systems Management (ESM) project experienced schedule delays and incurred additional costs because it did not have an integrated, enterprise-wide CM process.For the ESM project, such a process is necessary since different organizations are responsible for developing and deploying ESM releases.Specifically, the ESM team from the IRSí PRIME Business Systems Modernization contractor (PRIME) is responsible for development, and the ITSí End-User Equipment and Services (EUES) organization is responsible for deployment.

In February 2003, during the ESM Release 2.1 deployment, the EUES organization, with support from the PRIME, upgraded the ESM production environment.During this upgrade, changes were introduced to the production environment without adequate testing or adherence to CM processes.As a result of these CM weaknesses, a database server experienced serious, unexpected performance problems.Resolution of the performance problems delayed the implementation of ESM Release 2.1 for 4 months and increased the PRIME contractorís cost by approximately $216,500 since this work fell outside the scope of the PRIME contractorís existing task orders.

The ESM system is just one of several modernized systems that will be migrated in stages over the next several years.Between Fiscal Year (FY) 2003 and FY 2005, the IRS scheduled nine other modernized systems to migrate to the production environment.These modernized systems include new tax administration and financial management systems.These modernized systems not only have interdependencies with each other, but also with the existing IRS systems.Consequently, delays in one project can cause delays in others.For example, the delays in implementing ESM Release 2.1 delayed full management reporting functionality for the IRFOF system.

Several factors contributed to the implementation deficiencies found in the MITS organization.First, the MITS organization CM Directive and procedures did not establish executive level responsibility that would ensure that the MITS organization CM processes are implemented throughout the ITS organization and coordinated with the BSMO, and that deficiencies identified in internal CM assessments are appropriately addressed.Second, governance policy has not been established for defining authority levels and threshold criteria to review, approve, and control production changes in the ITS organization or for elevating change requests from the ITS organization to a higher-level CCB or committee for approval or coordination such as that found in the BSMO organization.

In addition, some ITS organizations use different CM software that does not comply with the current EA and may not readily facilitate the coordination of system baseline information on an enterprise-wide CM level.The current revision of the IRSí EA, dated July 9, 2003, includes an Enterprise Standards Profile that identifies commercial off-the-shelf software products for use in various computer environments.However, prior EA versions and the MITS organization CM policies and procedures did not identify approved CM software for enterprise-wide use.As a result, some ITS organizations use CM software that does not comply with the current EA.The use of non-compliant software to automate the CM process places the ITS organizations at risk of not being able to effectively communicate and coordinate changes on the production environment with affected organizations and projects, such as BSMO projects that have production environment interdependencies.Since a variety of CM software is currently being used, a period of transition will be needed to review and establish the CM software to be used and integrated throughout the MITS organization.

For example, a contributing cause for not adhering to CM processes for the ESM project was the use of different automated software to manage changes for the project.The BSMO and the PRIME use IBMís Rational software for configuration management, and the EUES organization uses IRS developed software to control change requests.

Without an integrated and uniform CM process, there is an increased potential that modernized and existing systems will require extensive rework resulting in additional costs, schedule delays, and other risks to the IRSí computer operations (e.g., system outages and data corruption).

Recommendations

To promote the establishment of an integrated MITS organization CM process, we recommend that the Chief Information Officer:

1.      Modify the MITS organization CM Directive and procedures to:a) assign organizational responsibility for ensuring that MITS organization CM processes are implemented throughout the ITS organization and coordinated with the BSMO and that CM deficiencies are appropriately addressed; and b) establish governance policies, similar to those used by the BSMO, for defining the authority levels and threshold criteria to approve and control changes to the production environment in the ITS organization.

Managementís Response:The MITS organization will revalidate its CM Directive to address organizational responsibility, governance policy, and needed improvements in the CCB structure.In addition, management will address the establishment of governance policies and threshold criteria to approve and control changes to the ITS production environment while establishing plans to organize separate CCBs for the ITS and BSMO organizations.

2.      Develop a transition plan to implement standardized EA-compliant CM software to be used throughout the MITS organization to facilitate CM on an enterprise-wide level.

Managementís Response:The MITS organization will identify acceptable CM software and publish applicable guidance upon the completion of ongoing CM software assessments within the ITS organization.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to determine whether the Modernization and Information Technology Services (MITS) organization effectively implemented an enterprise-wide configuration management (CM) process.

As part of this review, we interviewed personnel and reviewed CM documentation throughout the MITS organizations of Business Systems Modernization Office (BSMO), Information Technology Services (ITS), Business Planning and Assurance, and Security Services.Within the BSMO, we interviewed personnel from the Office of Configuration Management and Systems Engineering & Integration Division as well as the project teams for the Enterprise Systems Management (ESM) and Internet Refund/Fact of Filing (IRFOF) projects.Within the ITS organization, we interviewed personnel from the Infrastructure Architecture and Engineering, Business Systems Development, Enterprise Operations Services, End User Equipment and Services, Enterprise Networks, and Web Services functions.

This audit assessed the CM processes throughout the MITS organization and selected projects that migrated from acquisition by the BSMO to support and maintenance by the ITS organization.The ESM and IRFOF projects were judgmentally selected from the population of BSMO projects based on one project having been fully migrated and another being migrated in stages or releases to the production environment.The IRFOF Project was selected as a project that had migrated from the BSMO environment to the production environment, which is operated and supported by the ITS organization.The ESM project was selected as a project that migrated in stages or releases since a release was already being supported in the ITSí production environment and future releases were being developed by the IRSí PRIME Business Systems Modernization contractor that is overseen by the BSMO.

To accomplish the overall objective for this audit, we:

I.

Identified applicable Federal Government standards and industry best practices that guide the CM process.This included Department of the Treasury directives, Office of Management and Budget circulars, and information technology standards organization documents.

II.

Evaluated the ITS Executive and the BSMO Office of Configuration Management roles and responsibilities for administering the enterprise-wide CM process.

III.

Evaluated the policies and procedures supporting the enterprise-wide CM process.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary V. Hinkle, Director

Theodore Grolimund, Audit Manager

Kevin Burke, Senior Auditor

Christopher Funke, Senior Auditor

Frank Greene, Senior Auditor

Michael Howard, Senior Auditor

Tina Wong, Senior Auditor

Olivia Jasper, Auditor

 

Appendix III

 

Report Distribution List

 

CommissionerC

Office of the Commissioner Ė Attn:Chief of StaffC

Deputy Commissioner for Operations SupportOS

Associate Commissioner for ModernizationOS:CIO:B

Chief, Information Technology ServicesOS:CIO:I

Chief, Mission AssuranceOS:MA

Director, Business Systems DevelopmentOS:CIO:I:BSD

Acting Director, End User Equipment and ServicesOS:CIO:I:EU

Director, Enterprise NetworksOS:CIO:I:EN

Director, Enterprise OperationsOS:CIO:I:EO

Director, Infrastructure, Architecture and EngineeringOS:CIO:I:IA

Director, Portfolio ManagementOS:CIO:R:PM

Director, Web ServicesOS:CIO:I:W

Manager, Enterprise Systems ManagementOS:CIO:I:EU:ESM

Manager, Office of Configuration ManagementOS:CIO:B:MP:CM

Chief CounselCC

National Taxpayer AdvocateTA

Director, Office of Legislative AffairsCL:LA

Director, Office of Program Evaluation and Risk AnalysisRAS:O

Office of Management ControlsOS:CFO:AR:M

Audit Liaisons:

Associate Commissioner for ModernizationOS:CIO:B

Chief, Information Technology ServicesOS:CIO:I

Chief, Mission AssuranceOS:MA

Director, Business Systems DevelopmentOS:CIO:BSD

Director, End User Equipment and ServicesOS:CIO:I:EU

Director, Enterprise Networks OS:CIO:I:EN

Director, Enterprise Operations ServicesOS:CIO:I:EOS

Director, Infrastructure, Architecture and EngineeringOS:CIO:I:IA

Director, Web ServicesOS:CIO:I:W

Program Manager, Program Oversight and CoordinationOS:CIO:R:PM:PO

 

Appendix IV

 

Outcome Measures

 

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration.This benefit will be incorporated into our Semiannual Report to the Congress.

Type and Value of Outcome Measure:

        Inefficient use of resources Ė Actual; $216,500 (see page 4).

Methodology Used to Measure the Reported Benefit:

Not having an integrated, enterprise-wide Configuration Management (CM) process was demonstrated by the schedule delays and increased costs of the Enterprise Systems Management (ESM) project.The ESM Release 2.1 was developed by the Internal Revenue Serviceís (IRS) PRIME Business Systems Modernization contractor (PRIME) and deployed by the Information Technology Servicesí End-User Equipment and Services (EUES) organization.In February 2003, during the ESM 2.1 deployment, the EUES organization, with support from the PRIME, upgraded an ESM production environment.

During this upgrade, changes were introduced to the production environment without adequate testing or adherence to CM processes and, as a result, a database server experienced serious, unexpected performance problems.A contributing cause for not adhering to CM processes was the use of different automated software to manage changes for the ESM project.The Business Systems Modernization Office and the PRIME use IBMís Rational software for configuration management and the EUES organization uses IRS-developed software to control change requests.Resolution of the performance problems delayed the implementation of ESM Release 2.1 for 4 months and increased the PRIME contractorís cost by approximately $216,500 since this work fell outside the scope of their existing task orders.This cost was documented in a notification of change letter to the PRIME contract, sent to the IRS on April 11, 2003.

 

Appendix V

 

Overview of Configuration Management Functions

 

The chart was removed due to its size.To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

 

Appendix VI

 

Managementís Response to the Draft Report

 

The response was removed due to its size.To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.