TREASURY INSPECTOR GENERAL FOR TAX
ADMINISTRATION
Enhancements to Transition
Management Guidance Will Enable Efficient and Effective Modernization Project
Implementation
April 2006
Reference Number: 2006-20-054
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone Number |
202-927-7037
Email Address | Bonnie.Heald@tigta.treas.gov
Web Site | http://www.tigta.gov
April 28, 2006
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Enhancements to Transition Management Guidance Will Enable Efficient and Effective Modernization Project Implementation (Audit # 200520020)
This report presents the results of our review of the Internal Revenue Service (IRS) Modernization and Information Technology Services organization’s Transition Management activities.[1] The overall objective of this review was to determine whether Transition Management guidance and direction are adequate to ensure modernized systems are transitioned smoothly from the contractor to the IRS support organization. This review was part of our Fiscal Year 2005 audit plan for reviews of the IRS’ Business Systems Modernization efforts.
Synopsis
Transition Management activities prepare receiving organizations to use, operate, and maintain modernized systems. IRS business operating divisions and functional organizations must be prepared to use the new capabilities. Additionally, the Information Technology Systems organization must be prepared to accept changes to its production environment and to operate and maintain new systems at the contractually established time.
In August 2002, we reported [2] the Enterprise Life Cycle[3] guidance on Transition Management activities was not clearly organized and did not address all activities recommended by the Capability Maturity Model.[4] Our reviews of current Transition Management guidance and activities found these areas are now adequately addressed with comprehensive guidance and tools developed by the Transition Management Office. The Transition Management Office is integral in helping project teams, receiving organizations, and support organizations focus on readiness in the areas of business process, staffing, training, communication, hardware, software, and budget.
The Transition Management Office guidance and tools were
successfully delivered through the Transition Management Guide, which describes
the process the IRS uses to prepare its receiving organizations for system
transitions. The Transition Management
Office also developed and implemented the Transition Management Repository, which
is designed to store
transition-related information for each project.
The
Transition Management Office has developed comprehensive guidance and tools
that prepare IRS support organizations to receive Modernization projects.
The Transition Management Office can further improve transition efforts by ensuring project gap closure activities adhere to the latest Transition Management guidance. It can also refine its performance measures to include assessments of the interim status of the Transition Management process and its resultant products.
We reviewed the Modernized e-File[5] project and the Integrated Financial System[6] project transition activities. The Modernized e-File project Release 3.2 followed existing guidance to position the project for a successful transition to the support organization. However, the Integrated Financial System project did not follow Transition Management guidance requiring validation and control of the readiness gaps to help ensure Transition Management Plan completion.
The Transition Management Office issued revised guidance in October 2005 to improve controls over readiness gaps in executing the Transition Management Plan. This new guidance included a gap closure and tracking template, which all project teams should use. When followed, this template should reduce the gap closure and control weaknesses we identified.
In May 2004, the Business Systems Modernization Office prepared a root cause analysis for Transition Management activities. One of the concerns was “Measures of transition readiness are neither identified nor used.” On December 19, 2005, the Transition Management Office finalized the Transition Management Performance Measurement Process, which provides stakeholder assessments of the Transition Management Office and Transition Management tools. Although the Transition Management Performance Measurement Process appears to be a successful effort to measure the effectiveness of the Transition Management Office and its tools, the Transition Management Office can enhance its performance measurement process by assessing the interim status of the Transition Management process and its resultant products.
Recommendation
The Chief Information Officer should direct the Transition Management Office to enhance its performance measures through use of the Capability Maturity Model guidance. Examples of performance measurements considering the Capability Maturity Model guidance could include Transition Management hours expended on a project, number of workshops held, percentage of Transition Management Plan completion, percentage of Transition Management Repository item completion, and percentage of Transition Management Plan gaps closed.
Response
IRS management agreed with our recommendation. To implement the corrective action, the Transition Management Office has augmented its Transition Management Performance Measurement Process to include measures reflective of those suggested in the Capability Maturity Model Integration. Management’s complete response to the draft report is included as Appendix V.
Copies of this
report are also being sent to the IRS managers affected by the report
recommendation. Please contact me
at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant
Inspector General for Audit (Information Systems Program), at (202) 622-8510.
The Transition Management Program Has Progressed
Significantly Since Our 2002 Audit
Transition Management Performance Measures Need Further Development
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix
IV – Enterprise Life Cycle Overview
Appendix V
– Management’s Response to the Draft Report
Transition Management focuses on preparing receiving organizations for both the organizational processes and the technology that will exist in the modernized environment.
Transition Management is defined as the management of activities associated with the smooth transfer of modernized business capabilities, including business processes and systems, from the developer to the receiving organization. Transition Management activities prepare receiving organizations to use, operate, and maintain modernized systems. This includes helping the receiving organizations understand the new system and its implications on the receiving organizations’ work and helping them address those implications. Internal Revenue Service (IRS) business operating divisions and functional organizations must be prepared to use the new capabilities. Additionally, the Information Technology Systems organization must be prepared to accept changes to its production environment and to operate and maintain new systems at the contractually established time.
The IRS selected the Computer Sciences Corporation to serve as the Prime Systems Integration Services Contractor (PRIME)[7] for the Business Systems Modernization program. The IRS and the PRIME contractor developed the Enterprise Life Cycle (ELC) to provide guidance for designing, developing, and supporting Business Systems Modernization projects. Appendix IV presents an overview of the components of the ELC.
The ELC establishes a set of repeatable processes and a system of reviews that promote delivery of promised business results. The ELC processes are in line with the Software Engineering Institute’s Capability Maturity Model.[8] The Capability Maturity Model is a widely recognized and adopted framework for improving software and system acquisition processes.
Transition Management is identified as one of the Capability Maturity Model key process areas and is included as part of the ELC. The intent of Transition Management is to provide a disciplined transition of the Business Systems Modernization projects being developed to the IRS organization responsible for their long-term support and maintenance. All Business Systems Modernization project teams need Transition Management activities to take place to successfully transfer the projects to the IRS. The goal of Transition Management activities is to ensure the support organizations are fully prepared to accept responsibility for supporting and maintaining the projects and related products.
This audit is a followup to our August 2002 report entitled Management Advisory Report: Progress Has Been Made in Developing Transition to Support Guidance for Modernization Projects.[9] This audit was performed at the Business Systems Modernization Office facilities in New Carrollton, Maryland, during the period September 2005 through January 2006. The audit was conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The Transition Management Program Has Progressed Significantly Since Our 2002 Audit
In August 2002, we reported the ELC guidance on Transition Management activities was not clearly organized and did not address all activities recommended by the Capability Maturity Model. Specifically, Transition Management guidance did not sufficiently address the following two areas:
· Responsibility for the software products is transferred only after the software support organization demonstrates its capability and capacity to modify and support the software products.
·
The project team conducts activities to ensure
support of the software products is maintained and is effective during the
transition from the contractor to the software support organization.
We also reported project team implementation weaknesses
included:
·
Project
teams did not develop Transition Management Plans early in the project life
cycle of the four modernization projects we reviewed.
·
Project
teams did not timely include Transition Management Plans and activities in task
orders.[10]
The Transition Management Office has developed comprehensive guidance and tools that prepare IRS support organizations to receive Modernization projects.
Our reviews of current Transition Management guidance and activities found these areas are now adequately addressed. The Transition Management Office is integral in helping project teams, receiving organizations, and support organizations focus on readiness in the areas of business process, staffing, training, communication, hardware, software, and budget.
The Transition Management Office has successfully delivered the Transition
Management Guide
The Transition Management Guide describes the process the IRS uses to prepare its receiving organizations for system transitions. It includes a thorough template for preparing project Transition Management Plans. It also contains templates for use in assessing and closing receiving organization readiness gaps.
The Transition Management Guide provides direction for three Transition Management entities:
· Projects – Direction to identify, document, and help implement transition requirements that will ensure organization readiness to receive the modernized system.
· IRS business operating divisions, functional organizations, and Modernization and Information Technology Services (MITS) groups – Direction to assess their readiness to accept new capabilities and to plan and perform transition-related activities.
· MITS Business Integration Transition Management Office team members – Direction to assist business operating divisions, functional organizations and MITS groups with achieving organization readiness to receive modernized systems.
The Transition Management Office developed and implemented the
Transition Management Repository
The Transition Management
Repository, which resides on the IRS intranet, is designed to store
transition-related information for each project. The Transition Management Repository includes
both a data warehouse and a document template that itemize the required format
and contents of project transition documentation.
Information in the Repository is used to generate a project’s official Transition Management Plan. The Repository can generate reports that provide an integrated view of transition impacts across projects, for instance by release[11] or by IRS receiving organization. It is also used to access the Gap Closure Action Log template.
The Transition Management Office identified the need to ensure Transition Management activities continue to improve the efficiency and effectiveness of transitioning projects from the developing to the supporting organization. The Transition Management Office can further improve transition efforts by ensuring:
· Project gap closure activities adhere to the latest Transition Management guidance.
· Transition Management performance measures are refined to include measurements of the interim status of the Transition Management process and its resultant products.
We reviewed the Modernized e-File (MeF)[12] project and the Integrated Financial System (IFS)[13] project transition activities. The MeF project Release 3.2 efforts followed existing guidance to position the project for a successful transition to the support organization. However, the IFS project did not follow Transition Management guidance in effect at the time of its transition efforts. The IFS project did not follow guidance that required it to validate and control the readiness gaps to help ensure completion of the Transition Management Plan.
We reviewed the April 29, 2005, IFS Milestone 5 Transition Management Plan. Based on documentation in this Plan and related implementation activities, we could not determine whether the IRS receiving organizations were resolving all the transition readiness gaps listed in the Transition Management Plan. Further, the IFS project team could not provide documentation about the work in process or completed to close the Transition Management Plan readiness gaps.
Without proper control over activities to close readiness gaps, neither the project team nor the Transition Management Office can be sure all gaps identified in the Transition Management Plan have been closed prior to transition of a project to the support organization.
The MITS organization and the Transition Management Office recognized the need to improve control of the Transition Management activities.
The Transition Management Office issued revised guidance in October 2005 to improve controls over readiness gaps in executing the Transition Management Plan. This new guidance included a gap closure and tracking template, which all project teams should use. When followed, this template should reduce the gap closure and control weaknesses we identified. In addition, the MITS organization convened the System Acceptance and Transition to Operations and Maintenance workgroup to define root causes hindering system acceptance and successful transition. This workgroup is in the process of addressing these root causes.
In May 2004, the Business Systems Modernization Office
prepared a root cause analysis report for Transition Management activities. The report listed the top 10 themes and
concerns of management. One of those
concerns was “Measures of transition
readiness are neither identified nor used.”
On December 19, 2005, the Transition Management Office finalized the Transition Management Performance Measurement Process. This Process provides historical stakeholder assessments of the Transition Management Office and Transition Management tools, such as the Transition Management Guide and Transition Management Repository. The measures are an accumulation of stakeholder responses about the three phases of the Transition Management process:
· Organizational alignment.
· Gap identification.
· Gap closure.
The Transition Management Performance Measurement Process appears to be a successful effort to measure the effectiveness of the Transition Management Office and its tools from the perspective of the receiving organizations. It should result in improved Transition Management Office/stakeholder relations and Transition Management tools. However, the Transition Management Office can enhance its performance measurement process by measuring the interim status of the Transition Management process and its resultant products.
The Capability Maturity Model requires measurements to determine the status of the transition activities and resultant products. The Capability Maturity Model provides the following examples of measurements to track status:
· Effort expended.
· Funds expended.
· Progress towards completion of transition to support.
· Completion of milestones.
The Transition Management Performance Measurement Process does not provide a means to measure the status of Transition Management activities. Without an avenue for measuring the status and effectiveness of Transition Management activities, the Modernization program may be unable to efficiently and effectively manage the transition of its projects from the development to support organizations.
Recommendation
Recommendation 1: The Chief Information Officer should direct the Transition Management Office to enhance its performance measures through use of the Capability Maturity Model guidance. Examples of Transition Management performance measurements considering the Capability Maturity Model guidance could include Transition Management hours expended on a project, number of workshops held, percentage of Transition Management Plan completion, percentage of Transition Management Repository item completion, and percentage of Transition Management Plan gaps closed.
Management’s Response: The IRS agreed with this recommendation. As part of its Transition Management Guide, the Transition Management Office has augmented its Transition Management Performance Measurement Process to include measures reflective of those suggested in the Capability Maturity Model Integration. The Transition Management Performance Measurement Process now includes the following proposed measures:
· Percentage of projects with a detailed Transition Management work plan by Milestone 2 Exit.
· Percentage of projects using the Transition Management Repository.
· Percentage of projects that have an approved Transition Management Plan at Milestone 2-5 Exits.
· Percentage of readiness gaps closed or successfully reduced at Milestone 4 Exit.
· Percentage of readiness gaps closed or successfully reduced at Milestone 5 Exit.
Data for the proposed measures will be collected through established Transition Management practices. Transition Management Office representatives will collect the data as part of the service they provide to Project Transition Management teams and through the monthly Transition Readiness Reporting process. This approach enables the Transition Management Office to collect the necessary data through its normal support of project Transition Management teams.
Appendix I
Detailed Objective,
Scope, and Methodology
The overall objective of our review was to determine whether Transition Management[14] guidance and direction are adequate to ensure modernized systems are transitioned smoothly from the contractor to the Internal Revenue Service (IRS) support organization. To accomplish our objective, we reviewed available documentation and interviewed Transition Management Office, Business Systems Modernization Office, and Information Technology Services organization managers and analysts, and Modernization Acquisition analysts from the Agency-Wide Shared Services organization. This review was part of our Fiscal Year 2005 audit plan for reviews of the IRS’ Business Systems Modernization efforts. Specifically, we:
I.
Determined the status and adequacy of Transition Management guidance in
use and in development.
A.
Assessed current Transition Management guidance
to determine whether it improved on weaknesses we previously identified in our
report entitled Management Advisory
Report: Progress Has Been Made in Developing Transition to Support Guidance for
Modernization Projects.[15]
B.
Interviewed the Transition Management Office Senior Manager to determine
the effective date for using current guidance and whether future guidance
updates were in development to further direct transition activities.
C.
Interviewed the Business Systems Modernization Office Acquisition Project
Manager and Information Technology Services organization Project Manager
for the Integrated Financial System (IFS)[16] and the Modernized e-File
(MeF)[17] projects to determine how the
Transition Management guidance was being communicated, controlled, and implemented.
II.
Assessed the implementation of Transition Management guidance in modernization
projects for the IFS and MeF projects.
A.
Determined whether modernization
project task
orders[18] included adequate Transition
Management Plans with necessary preparation and transitioning activities.
B.
Identified projects requiring operations and maintenance contracts due to
a project change or Transition Management deliverables or activities not timely
tasked or planned.
C.
Determined whether the IFS and MeF projects had adequate Transition
Management Plans for the IRS support organizations to receive the applications
transitioned from the developer.
III.
Determined the adequacy of Transition Management
direction and project coordination and communication among the participating
organizations such as the PRIME contractor,[19]
Business Systems Modernization Office, Information Technology Services
organization, and Transition Management Office.
A.
Interviewed
the Transition Management Office Senior Manager and reviewed Transition Management guidance to determine the status of the Transition Management Office
organizational placement within the Modernization and Information Technology
Services organization.
B.
Identified
plans for Transition Management to provide guidance to Information Technology
Services organization systems (in
addition to Business Systems Modernization Office systems) after the initiation
of the Enterprise Services organization.
C.
Determined the adequacy of Transition Management readiness reviews for
the IFS and MeF projects.
Appendix II
Major Contributors
to This Report
Margaret E. Begg, Assistant Inspector General for Audit
(Information Systems Programs)
Gary V. Hinkle, Director
Edward A.
Neuwirth, Audit Manager
Glen J.
Rhoades, Senior Auditor
Louis V. Zullo, Senior
Auditor
Perrin T.
Gleaton, Auditor
Appendix III
Commissioner C
Office of the
Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Associate Chief Information Officer, Applications Development OS:CIO:B
Associate
Chief Information Officer,
Deputy Associate Chief Information Officer, Applications Development OS:CIO:AD
Director, Procurement OS:A:P
Director, Stakeholder Management OS:CIO:SM
Deputy Associate Chief Information Officer, Business Integration OS:CIO:ES:BI
Deputy Associate Chief Information Officer, Systems Integration OS:CIO:ES:SI
Director, Test, Assurance, and Documentation OS:CIO:AD:TAD
Chief Counsel
CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and
Risk Analysis RAS:O
Office of Management Controls OS:CFO:AR:M
Audit Liaisons:
Associate Chief Information Officer, Applications Development OS:CIO:B
Director,
Program Oversight OS:CIO:SM:
Appendix IV
Enterprise Life
Cycle Overview
The Enterprise Life Cycle (ELC) is the Internal Revenue Service’s (IRS) standard approach to business change and information systems initiatives. It is a collection of program and project management best practices designed to manage business change in a successful and repeatable manner. The ELC addresses large and small projects developed internally and by contractors.
The ELC includes such requirements as:
·
Development of
and conformance to an enterprise architecture.
·
Improving
business processes prior to automation.
·
Use of
prototyping and commercial software, where possible.
·
Obtaining early
benefit by implementing solutions in multiple releases.[20]
·
Financial
justification, budgeting, and reporting of project status.
In addition, the ELC improves the IRS’ ability to manage changes to the enterprise; estimate the cost of changes; and engineer, develop, and maintain systems effectively. Figure 1 provides an overview of the layers, paths, phases, and milestones (shown as “MS” in Figure 1) within the ELC Framework.
Figure
1: ELC Framework
Figure
1 was removed due to its size. To see
Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
ELC Layers
The ELC is a
framework for organizing and using IRS directives, processes, procedures,
templates, and standards to accomplish business change. It is organized as a set of interacting
layers.
·
The Management Layer specifies how to plan
and control business change programs, projects, acquisitions, and solutions
throughout the ELC.
·
The Governance Layer specifies additional
controls imposed from outside the project or program.
·
The Solution Life Cycle Layer specifies
what should be done but not how to do it.
·
The Solution Layer manages the solution as
it is produced, including providing standards for consistent solution
specification and formal review of solution content. This Layer provides control over work
products that may be produced by multiple internal and external developers
using differing methodologies.
·
The Methodology Layer details how to do the
work and specifies a unique set of work products to be produced. Specific methodologies are not part of the
ELC Framework.
·
The Specialty Areas Layer provides
additional guidance for areas of particular importance within the IRS. These areas include Enterprise Integration,
Test, and Evaluation;[21] Business Rules Harvesting[22] and Management; Transition Management;[23] Enterprise Architecture; Capital Planning and
Investment Control;[24] Security and Privacy; Requirements Development and
Management.
A path specifies a
unique “philosophy” or orientation for performing the work. Although the ELC specifies a standard for the
work required to produce and operate business change solutions, there are
multiple ways to approach and accomplish the required work. Paths are like alternate roads, each of which
crosses different terrain, but all of which lead to the same destination. The ELC provides five distinct paths or approaches
to developing systems:
·
The Large Custom Path is for large
projects.
·
The Small Custom Path is for small
projects.
·
The Commercial-Off-the-Shelf Path is a
commercial, software-based approach.
·
The Joint Application Development/Rapid
Application Development Path is a highly accelerated, prototyping-based
approach for very small, standalone solutions or solution components.
·
The Iterative Custom Path is a hybrid
approach that combines elements of the other approaches.
A phase is a broad segment of
work encompassing activities of similar scope, nature, and detail and providing
a natural breakpoint in the life cycle.
Each phase begins with a kickoff meeting and ends with an executive
management decision point (called a milestone) at which IRS executives make
“go/no-go” decisions for continuation of a project. Project funding decisions are often
associated with milestones.
Figure
2: ELC Phases and Milestones
|
Phase |
General Nature of Work |
Concluding Milestone |
|---|---|---|
|
Vision and
Strategy/ Enterprise Architecture Phase |
High-level direction setting. This is the only phase for enterprise
planning projects. |
0 |
|
Project Initiation
Phase |
Startup of development projects. |
1 |
|
Domain Architecture
Phase |
Specification of the operating concept,
requirements, and structure of the solution.
|
2 |
|
Preliminary
Design Phase |
Preliminary design of all solution
components. |
3 |
|
Detailed
Design Phase |
Detailed design of solution components. |
4A |
|
System Development
Phase |
Coding, integration, testing, and
certification of solutions. |
4B |
|
System Deployment
Phase |
Expanding availability of the solution
to all target users. This is usually
the last phase for development projects. |
5 |
|
Operations
and Maintenance Phase |
Ongoing management of operational systems. |
System Retirement |
Source:
The ELC Guide.
Appendix V
Management’s
Response to the Draft Report
The response was
removed due to its size. To see the
response, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
[1] Transition Management is the management of activities associated with the smooth transfer of modernized business capabilities, including business processes and systems, from the developer to the receiving organization.
[2] Management Advisory Report: Progress Has Been Made in Developing Transition to Support Guidance for Modernization Projects (Reference Number 2002-20-146, dated August 2002).
[3] The Enterprise Life Cycle establishes a set of repeatable processes and a system of reviews, checkpoints, and milestones that enable delivery of promised business results. See Appendix IV for an overview of the components of the Enterprise Life Cycle.
[4] The Software Engineering Institute’s Capability Maturity Model is a widely recognized industry standard for efficiently and effectively developing computer projects and systems.
[5] The Modernized e-File project’s goal is to replace the current technology for filing IRS tax return forms with modernized, Internet-based electronic filing applications.
[6] The Integrated Financial System project includes the accounts payable, accounts receivable, general ledger, budget execution, cost management, and financial reporting activities.
[7] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.
[8] The Capability Maturity Model was developed by the Software Engineering Institute and is a widely recognized industry standard for efficiently and effectively developing computer projects and systems. Both the IRS and the Computer Sciences Corporation intend to use the Capability Maturity Model as their standard for developing Business Systems Modernization projects.
[9] Reference Number 2002-20-146, dated August 2002.
[10] A task order is an order for services placed against an established contract.
[11] A release is a specific edition of software.
[12] The MeF project’s goal is to replace the current technology for filing IRS tax return forms with modernized, Internet-based electronic filing applications.
[13] The IFS project includes the accounts payable, accounts receivable, general ledger, budget execution, cost management, and financial reporting activities.
[14] Transition Management is the management of activities associated with the smooth transfer of modernized business capabilities, including business processes and systems, from the developer to the receiving organization.
[15] Reference Number 2002-20-146, dated August 2002.
[16] The IFS project includes the accounts payable, accounts receivable, general ledger, budget execution, cost management, and financial reporting activities.
[17] The MeF project’s goal is to replace the current technology for filing IRS tax return forms with modernized, Internet-based electronic filing applications.
[18] A task order is an order for services placed against an established contract.
[19] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.
[20] A release is a specific edition of software.
[21] Enterprise Integration, Test, and Evaluation includes processes for integrating multiple components of a solution and conducting various types and levels of testing on the solution.
[22] A business rule is a statement that defines or constrains
some aspect of the business. Harvesting
is a general term used to broadly describe the entire set of activities
involved in gathering, formalizing, analyzing, and validating business rules
for a particular scope.
[23] Transition Management helps ensure personnel and organizations are prepared to receive, use, operate, and maintain the business processes and technology provided by business change solutions.
[24] The Capital Planning Investment and Control process
manages a central portfolio of information technology investments across the
IRS.