Implementation of the New Methodology for Systems Modernization Needs Increased Focus and Support
November 2000
Reference Number: 2001-20-015
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
November 6, 2000
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Implementation of the New Methodology for Systems Modernization Needs Increased Focus and Support
This report presents the results of our review of the Internal Revenue Service's (IRS) efforts to develop and implement its Enterprise Life Cycle (ELC). Our objective was to assess the IRS' processes for development and implementation of the ELC for the Business Systems Modernization Program.
In summary, we found that although the IRS is making progress in its deployment efforts, additional focus and support is needed to avoid further delays in fully implementing its systems modernization methodology. Delays in implementing the ELC thus far have impacted systems modernization projects that are currently underway.
We recommended that the IRS provide additional focus and support to the ELC Deployment Team, develop a validation process for the PRIME contractor’s ELC deliverables, implement improvements to the ELC library, and expedite development and delivery of ELC guidance and training. In addition, we recommended that the IRS require the PRIME contractor to perform a training needs-assessment for the projects and a staffing analysis, in addition to increasing the staffing in its training group.
Management's response was due on October 26, 2000. As of October 30, 2000, management had not responded to the draft report.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or your staff may call Scott Wilson, Associate Inspector General for Audit (Information Systems Programs) at (202) 622-8510.
Enterprise Life Cycle Training Has Not Been Provided in a Timely Manner
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
The Internal Revenue Service (IRS) is revamping business processes and implementing new technology, as part of its comprehensive modernization program. To assist in managing the modernization efforts for information systems, the IRS hired a contractor in December 1998. The contractor, along with the IRS, is modifying its existing business change methodology to meet the specific needs of the IRS. The modified methodology is known as the Enterprise Life Cycle (ELC). The ELC incorporates aspects of realizing a new business vision, from strategy development through system deployment and operations. The goal is to create IRS technology systems that improve taxpayer service, add new security features, and carry the agency into the new century.
The IRS has established a program management office, the Business Systems Modernization Office (BSMO), to oversee the modernization effort. Individuals within this office are responsible for the implementation of the ELC methodology. The objective of this audit was to assess the IRS’ processes for development and implementation of the ELC for the Business Systems Modernization Program.
Results
The IRS is making progress in implementing ELC processes and procedures required for effective management of the Business Systems Modernization Program. The BSMO "stood up," or was officially recognized, on July 13, 2000. Roles and responsibilities for the BSMO staff have been defined, and the IRS is in the process of ensuring these roles are effectively communicated and appropriate training is conducted.
However, documentation prepared by the PRIME contractor and accepted by the IRS indicates that completion targets for certain key ELC activities have been delayed beyond their initial deadline of September 2000, the implementation date indicated in the IRS’ latest Congressional funding request.
Increased Focus on Completing Interim Tasks Is Needed to Ensure Further Enterprise Life Cycle Implementation Delays Do Not Occur
The IRS initially received $9 million in 2 increments to complete initial ELC deployment by October 31, 1999. The IRS later revised plans to deploy the first phase of the ELC and presented in its spending plan to the Congress that it would complete ELC deployment by September 30, 2000. The IRS received an additional $5.1 million to complete the ELC implementation as a result of the approval of the latest spending plan.
Although the IRS has made progress in implementing the ELC and deployment activities have started, target dates for completion of several key deployment activities, such as delivery of educational courses and implementation of strategies to communicate ELC processes, have been delayed beyond September 30, 2000. Completion of the initial training and communications initiatives is critical to ensure that the BSMO effectively manages and oversees the modernization effort. Delays in the implementation activities have resulted in delays in the progress of modernization projects and could result in additional delays, as well as lack of quality in project deliverables. In addition, implementation delays beyond target dates could cause the Congress to question future commitments the IRS makes regarding its implementation plans.
The Current Version of the Enterprise Life Cycle Lacks Key Components Needed by the Project Developers
The IRS has not completed the development of key components of the ELC. Procedures for Investment Decision Management, a key legislative requirement for government agencies, had not been fully incorporated into the ELC. In addition, key supplements to the ELC, such as the Enterprise Architecture, Security and Privacy, and Legacy Systems Transformation, have not been finalized.
The IRS and the PRIME contractor have established an automated library to store approved polices, procedures, and other guidance for using the ELC. Although this is a worthwhile endeavor, the library currently lacks the level of detail necessary to deploy the ELC. A sample of the library’s users reported that they need more training on using the ELC and the associated library, more examples of completed work products, and better linkages and a centralized index in the automated library.
Because the ELC is a dynamic, evolving methodology, it is imperative that accurate, timely guidance be provided to project personnel. Inadequate ELC guidance, thus far, has resulted in inconsistency, inefficiency, and frustration in the preparation of project deliverables.
Enterprise Life Cycle Training Has Not Been Provided in a Timely Manner
At the time of our review, the overall Education and Development Plan for BSMO had been prepared, but individual training and lesson plans were not developed. One reason training efforts are not further along is because they were dependent on identification and definition of roles and responsibilities as well as an analysis of current skills. This identification and definition was not finalized until late June 2000.
Timely training for IRS and contractor employees working on the modernization projects is also at risk. At the time of our review, a formal needs-assessment for project training had not been conducted, and only 25 percent of the PRIME contractor staff in 2 of the IRS’ critical projects attended the initial ELC course. Many of the untrained contractor staff were employees from firms that were sub-contractors to the CSC and, thus, they had less experience using the ELC methodology.
The PRIME Program Training Group is currently staffed with only three people, including one part-time instructor. This staffing level will be insufficient to modify and deliver the 41 courses that have been identified as necessary for the modernization project personnel.
Training plays an integral part in the deployment of ELC processes and procedures. Without adequate training, the IRS can expect continued delays in project development and an overall lack of quality control over the modernization effort.
Summary of Recommendations
We recommend that the IRS’ Chief Information Officer (CIO) provide additional focus and support to the ELC Deployment Team, develop a validation process for the PRIME contractor’s ELC deliverables, implement improvements to the ELC library, and expedite development and delivery of ELC guidance and training. In addition, we recommend that the CIO require the PRIME contractor to perform a training needs-assessment for the projects and a staffing analysis, in addition to increasing the staffing in its training group.
Management’s Response: Management’s response was due on October 26, 2000. As of October 30, 2000, management had not responded to the draft report.
Our audit objective was to assess the Internal Revenue Service’s (IRS) processes for development and implementation of the Enterprise Life Cycle (ELC) for the Business Systems Modernization Program. To accomplish this objective, we evaluated the processes for developing the ELC to ensure that it is current, complete, and in a usable format and structure. We also evaluated processes established to deploy, or implement, the ELC in the IRS’ program management office and with the IRS’ key stakeholders. The scope of the audit included discussing the status of the ELC Deployment Program with key IRS and PRIME Alliance officials and reviewing available documentation.
This audit was conducted in the Business Systems Modernization Office (BSMO) in New Carrollton, Maryland, and at the Computer Sciences Corporation (CSC) in Landover, Maryland. We performed this review from January to June 2000, in accordance with Government Auditing Standards.
Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.
The IRS has been attempting to modernize its outdated tax processing system for more than a decade. After 10 years and over $3 billion spent with minimal improvement and intense scrutiny from the Congress, the IRS decided to use a contractor to help develop modernized systems.
In December 1998, the IRS selected the CSC to serve as the PRIME contractor for the Business Systems Modernization Program over the next 15 years. The first priority of the PRIME was to implement a systems life cycle environment patterned after the CSC’s Catalyst methodology. This approach incorporates aspects of realizing a new business vision, from strategy development through system deployment and operations. The goal is to create IRS technology systems that improve taxpayer service, add new security features, and carry the agency into the new century.
The IRS adopted Catalyst and expanded its scope to form an IRS-specific methodology known as the ELC. The ELC provides a process for initiating, designing, implementing, coordinating, and managing business change. The ELC establishes a set of repeatable processes and a system of reviews, checkpoints, and milestones that reduce the risks of a system development program and ensure alignment with the IRS’ overall business strategy.
The ELC is intended to be an integral part of the IRS’ business and information systems modernization efforts. All IRS and PRIME contractor personnel involved in the modernization will use the ELC to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision-making and information sharing.
The IRS has established an ELC Deployment Team to roll out the required ELC processes and procedures necessary for the BSMO to execute the degree of program management and control required by the Clinger-Cohen Act. According to the Act, the IRS must identify and adopt best practices for information technology management, including commercial-sector practices.
The ELC Deployment Plan begins by mapping and validating the tasks, roles, and interfaces of BSMO personnel and key stakeholders. The Plan continues by mapping and validating ELC processes and procedures to those tasks, roles, and interfaces. This process assessment is designed to ensure that BSMO roles and responsibilities align with ELC processes and procedures.
The IRS is making progress in implementing ELC processes and procedures required for effective management of the Business Systems Modernization Program. The IRS reached a key milestone in initiating deployment of the ELC at the program level on June 30, 2000, when the PRIME contractor delivered the Business Systems Modernization Organizational Change Framework, Organizational Transition Plan. The Plan describes the organizational transition that the BSMO must use to achieve business goals in support of the Business Systems Modernization Program.
In addition, the BSMO "stood up," or was officially recognized, on July 13, 2000. Roles and responsibilities for the BSMO staff have been defined, and the IRS is in the process of ensuring these roles are effectively communicated and appropriate training is conducted.
When deployment is complete, the ELC will address items reported by the General Accounting Office as missing from the IRS Systems Life Cycle, including how work processes will be reengineered, how business requirements will be specified, and how business cases for technology investments will be formulated and evaluated.
However, documentation prepared by the PRIME contractor and accepted by the IRS indicates that completion targets for certain key ELC activities have been delayed beyond their initial deadline of September 2000, the implementation date indicated in the IRS’ latest Congressional funding request. Our review of the IRS’ processes for developing and implementing the ELC identified that:
Further delays in implementing ELC processes and providing training to program and project employees will likely result in additional delays in completing modernization projects as well as lower quality in the deliverables provided by the projects.
Increased Focus on Completing Interim Tasks Is Needed to Ensure Further Enterprise Life Cycle Implementation Delays Do Not Occur
The IRS initially received $9 million for the implementation and deployment of the ELC in 2 separate funding measures and originally scheduled to deploy the ELC by October 31, 1999. On April 18, 2000, a third funding request of $5.1 million was approved for ELC enhancements, maintenance, and full deployment. The target date for completion of ELC implementation in the most recent spending plan provided to the Congress was September 30, 2000.
However, even with this increased emphasis and additional funding, we believe that deployment of the ELC will not be completed by the September 30, 2000, implementation date. We have these concerns because the IRS and the CSC have not yet developed the organizational vision, strategies, and objectives for the BSMO. In addition, documentation prepared by the PRIME contractor and accepted by the IRS indicated that critical deployment activities, such as the delivery of ELC education to program management employees and the implementation of strategies to communicate ELC processes, are not scheduled for completion until December 2000.
The timely completion of these initiatives is critical because the success or failure of systems modernization projects is dependent on the ability of the BSMO to manage and oversee the modernization effort.
There are several causes for the delays in deploying the ELC. First, and most critical, has been the instability of the BSMO. Numerous changes in organizational structure occurred in late 1999 and early 2000. Also, several high-level management personnel in the modernization office have either left the IRS or have moved to other positions. An organizational structure was officially approved, and the BSMO "stood up" and was formally recognized on July 13, 2000.
This climate of instability has resulted in delays in establishing the roles and responsibilities of modernization personnel and in implementing the ELC. Another Treasury Inspector General for Tax Administration (TIGTA) report detailed this issue and made recommendations to improve the program management of the modernization effort.
Second, individuals responsible for deploying the ELC have been assigned concurrent responsibilities in other program activities. The staff assigned to the ELC Deployment Team is limited, and when their focus has been shifted to other priorities, delays have occurred.
Third, individuals in the BSMO have not been fully trained in deploying the ELC. The ELC is a complex methodology to incorporate, and training is a key element to enable effective deployment. Training is part of the deployment plan but is not scheduled to be completed until December 2000. This report details the delays in providing BSMO personnel ELC training in a later section.
Delays in deploying the ELC have contributed to weak or non-existent quality reviews of two of the IRS’ critical systems modernization projects, e-Services and Customer Communications. Consequently, incomplete or inadequate work products have been delivered, and delays have been experienced in delivering systems that will improve customer service to taxpayers.
Reviews by the BSMO are critical tools required by the ELC to ensure products meet ELC requirements. For example, milestone reviews on three of the IRS’ critical projects were not conducted until they were approaching the third milestone. This review identified that over 40 percent of the work products were either missing or were still in the planning stages and other products were incomplete.
Further delays in implementing the ELC methodology beyond target due dates could cause the Congress to further limit the IRS’ funding and to question future commitments the IRS makes regarding its implementation plans.
The IRS has noted the risk of allowing modernization projects to continue prior to the deployment of the ELC and is placing intense emphasis on implementing it. To mitigate the risk, the IRS is revising PRIME contract work requests to enable enhanced monitoring and accountability, implementing additional processes (i.e., Milestone Readiness Reviews), and rescoping/ rescheduling certain projects.
Recommendation
In addition to recommendations made in the TIGTA audit report mentioned earlier, we recommend that the Chief Information Officer (CIO):
1. Increase focus and support to the ELC deployment staff to ensure that interim tasks are completed and approved by scheduled due dates.
Management’s Response: Management’s response was due on October 26, 2000. As of October 30, 2000, management had not responded to the draft report.
The Current Version of the Enterprise Life Cycle Lacks Key Components Needed By the Project Developers
The IRS has adopted the CSC’s Catalyst methodology as a basis for the ELC. Although this methodology is extensive, supplemental materials specific to the IRS’ Business Systems Modernization Program have to be developed to address government requirements and user needs. Several of these critical supplemental materials needed for ELC implementation have not been timely prepared. In addition, users need more detailed guidance as well as training, examples, and indexing to make the ELC library more useful.
Critical ELC supplements and procedures have not been timely prepared
Several key supplements to the ELC, such as the Enterprise Architecture, Security and Privacy, and Legacy Systems Transformation, have not been finalized. A draft of the Security and Privacy document has been available to projects, but the draft has not gone through the approval process. These three supplements provide critical structure and framework for the projects to follow in designing modernized systems.
At the time of our audit, critical procedures for the Investment Decision Management (IDM) process had not been fully incorporated into the ELC. The IDM process guides the IRS in prioritizing, approving, funding, monitoring, and evaluating investment initiatives. The Clinger-Cohen Act requires agencies to have specific qualitative and quantitative criteria for projects to maximize value. These procedures were completed in late June 2000 and, according to management, would be incorporated into the user library in July 2000.
The PRIME contractor delivered the ELC Guide to the IRS on November 12, 1999. The ELC Guide provides an introduction and overview of the processes, products, reviews, techniques, participants, and interfaces of the ELC. IRS program-specific details are to be included as supplemental materials.
The PRIME contractor did not foresee the implication of excluding these materials until several modernization projects were well underway. As a result, the omission of the Enterprise Architecture supplement contributed to delays in meeting design and development target dates for the Customer Communications Project.
The ELC user library does not have the level of detail necessary for deployment
The ELC has been deployed to the projects in the form of an automated data repository -- an automated library. This library, called the Process Asset Library, is a navigable web site covering ELC processes with links to the ELC Guide, Catalyst, and additional items, such as procedures, templates, and examples of work products. This library is updated quarterly with new information and examples.
The ELC Process Asset Library, version 2000.2, as implemented in May 2000, lacks the level of detail necessary to fully deploy the ELC. We sampled 36 (20 percent) of the 180 work products in the ELC library. Of the 36 work products, we found that 18 (50 percent) did not have examples or instructions to guide project teams in design and development activities. For example, the product decision model is a work product that did not have an example in the ELC library. This document assists the IRS in the approval of the system concept for a modernization initiative and the authorization of funds for design of that system.
On May 11, 2000, the IRS reported 69 ELC directives, process descriptions, procedures, and templates still needed to be written or modified. These ELC gaps were identified in mid-April, but the ELC Deployment Team had other tasks to address and was not able to complete these deliverables. A quarterly update to the ELC library to incorporate this guidance was scheduled for July 15, 2000.
Users need additional training, examples, and indexing to make the library more useful
To determine whether improvements are needed to make the library more usable, we surveyed a sample of 87 (32 percent) of the 275 IRS employees and PRIME contractor personnel at the project and program levels assigned to IRS modernization projects. Of the 87 surveys issued, we received 46 responses. Suggestions for improvements were focused in three primary areas:
The ELC is a dynamic and evolving methodology. It is imperative that accurate, complete, and timely information be provided to project personnel. Delays in providing this data result in project deliverables that do not meet ELC requirements. Inadequate ELC guidance, thus far, has resulted in inconsistency, inefficiency, and frustration in the preparation of project deliverables.
Recommendations
We recommend that the CIO:
2. Develop a validation process to ensure that the ELC material provided by the PRIME contractor meets the IRS’ needs.
Enterprise Life Cycle Training Has Not Been Provided in a Timely Manner
The PRIME Program Training Group was established in November 1999 to provide training on ELC processes and procedures for contractors and IRS employees working on systems development projects. The ELC Deployment Team is currently responsible for identifying ELC training courses for the BSMO and stakeholders.
ELC training for program management personnel has thus far been limited to overview courses
The PRIME contractor performed a competency assessment for 34 of the 67 (51 percent) employees in the BSMO. The PRIME contractor identified 234 underlying skills and knowledge components that were needed to implement program management business processes. The BSMO had an overall competency assessment rate (on a scale of 1-5) of 2.7, versus the target assessment rate of 4.3.
Results of the assessment for the 34 employees were used to establish a competency baseline and develop the Education and Development Plan for the entire BSMO. The training needs identified in the Plan allow for program level oversight, as required by the ELC.
One reason that training efforts are not further along is because they were dependent on the identification and definition of roles and responsibilities as well as an analysis of current skills. This identification and definition was not finalized until late June 2000.
Although roles and responsibilities of the BSMO have been defined, BSMO employees and stakeholders have not had training for their specific duties. This staff has been offered only two general ELC training courses thus far: The ELC Overview and The ELC in Action. In addition, the IRS has not prepared individual training and lesson plans to execute specific roles and responsibilities.
According to plans prepared by the PRIME contractor and approved by IRS management, the delivery of this role-based training and coaching is not scheduled for completion until December 2000. Until this training is completed, the ability of the employees in the BSMO to effectively carry out current quality review and monitoring of assignments in accordance with the ELC will be limited. As a result, the timeliness and quality of project delivery could suffer.
Contractor personnel developing modernized systems have not had adequate training on using ELC processes and procedures
The deployment strategy for the PRIME contract project training involves a just-in-time training approach. As the IRS’ modernization projects enter into the different phases of the ELC, training for that phase is deployed at the beginning of the project phase.
A needs-assessment is used to target specific skill areas in need of development; it defines expected performance results. Part of this assessment includes a gap analysis to assess where performance is currently, versus where it needs to be in the future. At the time of our audit, a formal training needs-assessment had not been performed for the project participants.
The ELC Overview is the first course in the education series for employees learning Catalyst and the ELC. In reviewing training records, we found that only 16 of the 63 (25 percent) PRIME contractor personnel in the Customer Communications Project, and 3 of the 13 (23 percent) PRIME contractor personnel in the e-Services Project, had attended The ELC Overview course. Of the 57 employees who had not taken the training class, 45 (79 percent) were employees of
sub-contracting firms and not employees of the CSC; thus, they had limited, if any, prior exposure to the Catalyst methodology. Training of these employees is critical to ensure that the IRS receives adequate guidance in developing projects.
The PRIME Program Training Group is currently staffed with only three people, including one part-time instructor. These limited training resources contributed to the absence of a training needs-assessment for both PRIME contractor personnel and IRS employees assigned to modernization projects. Without additional instructors, the Training Group will have difficulty with just-in-time delivery of the 41 Catalyst courses identified as necessary for training the modernization project employees.
IRS employees working with the contractor to develop modernized systems have expressed concern that although PRIME contractor personnel are experienced professionals, many do not have the necessary knowledge or experience in using the ELC. Training plays an integral part in the deployment of the ELC processes and procedures. Providing the necessary ELC training to managers and employees will equip them with the knowledge and skills required to carry out their tasks. Without adequate training, the IRS can expect to experience continued delays in development of key customer service projects and inconsistency in the level of quality of the systems that are developed.
Recommendations
We recommend that the CIO:
Thus far, the IRS has received $14.1 million from the Congress for development and implementation of the ELC. The IRS has experienced problems in building the program management capability required to deploy the ELC. Modernization project development has struggled in its early efforts due, in part, to a lack of ELC training and guidance.
Although progress in deploying the ELC is being made, additional focus and support is needed to ensure that the ELC Deployment Team meets deliverable due dates and effectively fills the gaps in ELC processes, procedures, and guidance. Critical training needs must be addressed quickly, so that the BSMO personnel can begin fulfilling their new roles and responsibilities and provide the oversight required by the ELC. In addition, the IRS should ensure that the PRIME contractor provides additional focus on training project personnel in required ELC processes, so that projects produce quality deliverables that meet milestone requirements.
Appendix I
Detailed Objective, Scope, and Methodology
Our overall objective was to assess the Internal Revenue Service’s (IRS) processes for development and implementation of the Enterprise Life Cycle (ELC) for Business Systems Modernization.
I. To evaluate the processes for developing the ELC to ensure that it is current, complete, and in a usable format and structure, we:
Staffing Level
Funding Level
Appendix II
Major Contributors to This Report
Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)
Scott A. Macfarlane, Director
Tammy L. Whitcomb, Audit Manager
Eulala Davis, Senior Auditor
Michelle Griffin, Senior Auditor
Jimmie Johnson, Sr., Senior Auditor
Appendix III
Deputy Commissioner Modernization C:DM
Chief Information Officer IS
Director, Business Systems Modernization B
Deputy Director, Business Systems Modernization B:E
Director of Program Management B:PM
Director, Legislative Affairs CL:LA
Office of Management Controls CFO:A:M
Office of Chief Counsel CC
Director, Office of Program Evaluation and Risk Analysis M:O
National Taxpayer Advocate TA
Audit Liaisons:
Chief Information Officer IS
Director, Business Systems Modernization B:E