To Ensure the Customer Account Data Engine’s Success, Prescribed Management Practices Need to Be Followed

 

November 2004

 

Reference Number:  2005-20-005

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

November 22, 2004

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:     Gordon C. Milbourn III /s/ Gordon C. Milbourn III

                 Acting Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - To Ensure the Customer Account Data Engine’s Success, Prescribed Management Practices Need to Be Followed (Audit # 200420015)

 

This report presents the results of our review of the delivery of the Customer Account Data Engine (CADE) Release 1.  The overall objective of this review was to determine whether the Internal Revenue Service (IRS) and PRIME contractor would timely and effectively deliver the CADE Release 1 requirements – processing Forms 1040EZ for single taxpayers with refunds due or zero balances – in compliance with existing Enterprise Life Cycle (ELC) guidelines.  This review is part of our Fiscal Year 2004 audit plan for reviews of the IRS’ Business Systems Modernization efforts.

In summary, the CADE is the foundation for managing taxpayer accounts in the IRS modernization plan.  The CADE will consist of databases and related applications to replace the IRS’ existing Master File processing systems.  The Master Files are the IRS’ central and official repository of taxpayer information.  While the CADE program has experienced a series of setbacks and postponements since its beginning in September 1999, the IRS and the PRIME contractor have positioned the program for success.  Specifically, improvement plans have been identified for CADE program management, requirements have been developed to meet CADE Release 1.1 objectives, the CADE Release 1.1 contract task orders adequately reflected the project work performed, and the 2004 Pilot successfully simulated actual processes and procedures.

As part of the deployment strategy to demonstrate a working CADE, the 2004 Pilot included operating the new system in a simulation of the actual production environment.  However, a major upgrade to the CADE Release 1.1 application was installed after the completion of the 2004 Pilot.  Although this upgrade went through system integration testing and deployment site readiness testing, it did not receive the level and volume of performance testing to simulate the high-volume processing a pilot test offers.  Because the upgrade was not included in the high-volume 2004 Pilot testing, the CADE is at risk to experience taxpayer account and refund processing problems during high volume periods of the 2005 Filing Season.

The CADE Operator’s Guide was needed for the 2004 Pilot, as well as for live production that began in July 2004.  The documentation within the CADE Operator’s Guide and computer handbooks was incomplete and not always consistent with actual CADE processing.  Without adequate documentation describing the steps and resources needed to operate the CADE, efficient and effective processing cannot be assured.

The CADE currently includes manual processes that could be automated to achieve processing efficiencies.  These processes were not automated because they were not included as requirements to operate the CADE.  Automation of these processes would help ensure consistency in operations, avoid human error, and reduce staff time needed to perform the manual operations.

In October 2003, the Software Engineering Institute reported the CADE program did not have a dedicated system architect to provide technical leadership.  The report concluded a dedicated system architect is critical to the success of the CADE program.  Without a dedicated system architect’s oversight and management, architecture elements selected by CADE technical specialists may not effectively interoperate with the IRS’ other highly specialized information systems elements.

The IRS needs to be able to fully restore CADE functionality in the event of a disaster after system implementation.  While the CADE Disaster Recovery Plan adequately addressed disaster recovery requirements, these requirements were not tested prior to implementation.  Without full disaster recovery testing, the IRS may have difficulty identifying the risks the CADE faces to be able to fully recover in the event of a disaster.  As a result, taxpayer data contained on the CADE may be lost, requiring the IRS to contact taxpayers and/or their representatives to reacquire the lost information.

To help ensure the success of future CADE releases, we recommended the Chief Information Officer (CIO) direct the Business Systems Modernization Office (BSMO) to work with the PRIME contractor to ensure project development changes undergo appropriate performance testing before the system is deployed; documentation for future CADE releases is reviewed, tested, and approved before pilot testing and/or live production are allowed to proceed; and inefficient manual processes are automated in future CADE releases.

While plans to dedicate a system architect for the CADE program are under consideration, the CIO needs to ensure the BSMO makes the system architecture resources being acquired available on a full-time basis to the CADE program.  Further, to ensure the ability of the IRS to restore the CADE after a disaster with the least disruption to the IRS mission, the CIO needs to ensure the IRS tests all aspects of the CADE disaster recovery capabilities during the Annual IRS Disaster Recovery Test and tests disaster recovery capabilities for future CADE releases prior to implementation.

Management’s Response:  IRS management agreed with the report recommendations and corrective actions are underway to address them.  The CIO is planning to have semiannual CADE releases in July and January.  The July delivery will have the higher risk and more complex functionality, while the January component will have the filing season changes and any additional changes that capacity permits.  Because the IRS will have the returns from earlier in the filing season available for testing the July release, it will conduct performance testing on that release using the highest volume periods.  Whether or not performance testing is also done for the January release will be based on the specific nature of the changes that the IRS makes for that specific delivery.  Planned corrective actions also provide for having full operational documentation available at the time of implementation for future releases and implementing a number of changes to address known operational inefficiencies.  The CIO is pursuing the hiring of qualified engineering resources to further support the CADE project.  Lastly, the CIO is taking steps to make CADE a part of the annual disaster recovery testing.  Management’s complete response to the draft report is included as Appendix VII.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

Table of Contents

Background

After Several Postponements, the Customer Account Data Engine Program Has Positioned Itself for Success

Significant Software Changes Were Added to the Customer Account Data Engine Release 1.1

Recommendation 1:

Improvements to the Customer Account Data Engine Operator’s Guide Need to Be Completed

Recommendation 2:

Manual Processes Within the Customer Account Data Engine Release 1.1 Need to Be Automated for Future Releases

Recommendation 3:

The Customer Account Data Engine Program Does Not Have a Dedicated System Architect

Recommendation 4:

Disaster Recovery Capabilities Were Not Tested Prior to Customer Account Data Engine Release 1.1 Implementation

Recommendations 5 and 6:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Business System Modernization Challenge Issues Relating to the Customer Account Data Engine

Appendix V – Customer Account Data Engine Costs

Appendix VI – Enterprise Life Cycle Overview

Appendix VII – Management’s Response to the Draft Report

 

Background

The Customer Account Data Engine (CADE) is the foundation for managing taxpayer accounts in the Internal Revenue Service’s (IRS) modernization plan.  The CADE will consist of databases and related applications to replace the IRS’ existing Master File processing systems.  The Master Files are the IRS’ central and official repository of taxpayer information. 

The CADE project will include two phases.  The first phase involves individual tax accounts and will be incrementally deployed in five releases, each related to a specific taxpayer segment, over several years.  For example, Release 1 will process electronic and paper income tax returns for single filers with no dependents (Form 1040EZ).  Release 2 will add the U.S. Individual Income Tax Returns (Form 1040) for families and taxpayers filing as head of household.  At the conclusion of Release 5, the CADE will have replaced the Individual Master File (IMF), and the IRS will conduct business much differently than it does today.  The second phase of the CADE will convert the Business Master File tax accounts; however, development and implementation dates for the second phase have not been established.

The IRS initiated the CADE project in September 1999.  The CADE Release 1 was initially planned for delivery in January 2002.  The IRS and PRIME contractor postponed the planned delivery date of Release 1 several times, from January 2002 until August 2003.

In July 2003, the IRS acknowledged Release 1 would not meet its August 31, 2003, delivery date.  This latest setback occurred for several reasons.  First, the Release 1 did not have consistent, well-understood requirements.  For example, in January 2003, 4 years after the initial contract award, the subcontractor, IBM, began a 4-month effort to confirm the accuracy and completeness of Release 1 requirements.  Other reasons include low computer program productivity levels, IRS/PRIME contractor role confusion, and noncompliance with management processes.

In 2003, the IRS launched a comprehensive review consisting of three studies and a benchmarking analysis of the Business Systems Modernization (BSM) program.  In addition, the PRIME contractor launched an internal study.  The studies confirmed the IRS modernization effort is a massive, highly complex, high-risk program confronted by a number of critical management and technological challenges.  To address the various assessment findings, the IRS developed 46 action items for itself and the PRIME contractor to complete at various times:  14 items within 30 days, 19 items within 90 days, and 13 items over the long term.  Two more issues were added after reviewing recommendations from the IRS Oversight Board, making the total 48 action issues.  The 10 BSM Challenge Issues related to the CADE are listed in Appendix IV.

On January 7, 2004, IRS executives approved a revised CADE project schedule.  In July, the PRIME contractor and the BSM Office (BSMO) implemented the 2003 Filing Season version of the CADE to use in the 2004 Filing Season.  Identified as Release 1.1, it will enable the IRS to launch the CADE in a low-risk setting and gain valuable operational experience.  To date, the IRS has spent about $86.8 million in developing the CADE.  This includes about $24.5 million managed under a firm fixed-price contract.  Appendix V presents the history of the contract award amounts for the CADE project.

This review is the third in a series of reviews of the CADE project development and deployment activities and was performed at the BSMO facilities in New Carrollton, Maryland, and at the IRS Martinsburg, West Virginia, Computing Center during the period April through July 2004.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.  Appendix VI presents an overview of the Enterprise Life Cycle (ELC).

 

After Several Postponements, the Customer Account Data Engine Program Has Positioned Itself for Success

 

While the CADE program has experienced a series of setbacks and postponements since its beginning in September 1999, the IRS and the PRIME contractor have positioned the program for success.  Specifically, improvement plans have been identified for CADE program management, requirements have been developed to meet CADE Release 1.1 objectives, the CADE Release 1.1 contract task orders adequately reflected the project work performed, and the 2004 Pilot successfully simulated actual processes and procedures.

The BSM Challenge Issues resulted in plans to improve CADE program management

The BSM Challenges Plan Close-Out Report, dated
May 1, 2004, provides for improved future CADE project development.  The CADE-related BSM Challenge Issues and related closeout plans are described in Appendix IV.

Requirements have been developed to meet CADE Release 1.1 objectives

One of the reasons cited for postponements and delays encountered by the CADE program was the inadequate definition of project requirements.  To address this concern, we reviewed the project requirements contained in the CADE Business System Design Report.  The design report identifies over 900 requirements for Release 1.1.  We judgmentally sampled these requirements and determined they were adequately tested and/or defined in CADE project documentation.  We determined the requirements were adequately developed by tracing them to documentation of the CADE project’s systems integration testing, deployment site readiness testing, and security testing.  These tests indicated the requirements allow the CADE Release 1.1 to achieve its objectives.

In addition, we performed reviews to determine whether all essential Form 1040EZ requirements were adequately included in the CADE.  To accomplish this we successfully traced the information from the 12 line items on the Form 1040EZ to the CADE Business System Design Report.

The CADE Release 1.1 contract task orders adequately reflected the project work planned

PRIME contract administration is a concern cited by both the IRS and the IRS Oversight Board.  We reviewed the CADE Release 1.1 task order provisions and deliverables and noted several contract administration strengths.  For example, CADE work transferred from an earlier task order included an appropriate transfer of funding to the current Release 1.1 task order.  In addition, contract negotiations secured a $2.7 million reduction in the Release 1.1 costs from the contractor due to CADE Release 1.0 work which was deferred to a later release of the CADE.

The 2004 CADE Pilot successfully simulated actual processes and procedures 

As part of the deployment strategy to demonstrate a working CADE, the 2004 CADE Pilot plan included operating the new system in a simulation of the actual production environment.  The 2004 Pilot included an initialization procedure, daily/weekly processing, timely referral to the Department of the Treasury Financial Management Service (FMS) to generate tax refunds, and the ability to update the information to taxpayer accounts.

The 2004 Pilot scenarios used previously captured transaction data from the 2004 Filing Season.  It was the first opportunity for the IRS employees to schedule and submit the CADE jobs in a shoulder-to-shoulder environment.  Unlike the 2003 Pilot, in which the PRIME contractor was primarily responsible for the scheduling and execution of the jobs, the IRS performed these activities with the PRIME team members assisting their efforts.  This shift of responsibility from the PRIME contractor to the IRS occurred gradually during the 2004 Pilot as the IRS employees became more comfortable with the CADE.

During the week of May 17, 2004, IRS employees successfully ran the 2004 Pilot and completed a week’s worth of simulated tax return processing.  Upon completion of this processing, CADE-generated data and reports were successfully validated with corresponding IMF-generated data and reports.  Additionally, on May 24, 2004, the IRS received correspondence from the FMS that all CADE test files successfully passed through the FMS internal systems.  Had the CADE been in production rather than pilot mode, the FMS stated it would have processed actual refunds and electronic transfers using data provided by the CADE.

Although the 2004 Pilot successfully operated as planned, it did not run the final version of software for CADE Release 1.1.  This report addresses areas of concern requiring IRS and PRIME contractor attention to ensure successful future deployments.

 

Significant Software Changes Were Added to the Customer Account Data Engine Release 1.1

 

The 2004 CADE Pilot testing activities included validating business expectations, processing timelines, performance assumptions, balancing and control of tax returns and refunds, refund processing, and CADE interfaces with other existing and modernized project applications.  The CADE project team completed the 2004 Pilot testing of Release 1.1 on May 26, 2004.  A major upgrade to the CADE Release 1.1 application, called the Delta Build, was installed on June 18, 2004.  The Delta Build included numerous changes to the CADE Release 1.1, including changes known since at least the start of the 2003 Pilot in July 2002.  The changes include improvements to the Job Control Language and file and naming standards, as well as modifications to address verification software used within the CADE. 

Although the Delta Build went through systems integration testing and deployment site readiness testing, it did not receive the level and volume of performance testing, simulating high-volume processing a pilot test offers.  Also, a CADE subject-matter expert advised us the changes made to the CADE with the Delta Build were broader and more significant than all the changes made to the CADE since the 2003 Pilot.  Because the issues were more complex than the PRIME contractor originally thought, the Delta Build was postponed until after the 2004 Pilot so the project could stay on schedule for delivery by the end of September 2004.

The CADE staff performed an independent verification of the data processed by the Delta Build.  However, the absence of complete testing of the CADE Release 1.1 in a production environment added to the risk of unforeseen errors occurring during actual processing.  In fact, when the CADE Release 1.1 began processing tax returns on July 1, 2004, it experienced a critical failure resulting in the CADE being taken out of production on July 3, 2004.  The failure was resolved and the CADE resumed processing on July 14, 2004.  By August 2, 2004, the CADE had processed 1,136 Forms 1040EZ.

Although tax return and refund processing was successfully reverted to the IRS’ existing Master File processing systems, additional Delta Build testing and reviews were needed prior to reinitiating CADE Release 1.1 processing.  Further, because the Delta Build was not included in the high volume 2004 Pilot testing, the CADE is at risk to experience taxpayer account and refund processing problems during high volume periods of the 2005 Filing Season.

Recommendation

1.      The Chief Information Officer (CIO) should direct the BSMO to work with the PRIME contractor to ensure future project development changes undergo appropriate performance testing, simulating high-volume processing, before deploying the system.

Management’s Response:  The CIO is planning to have semiannual CADE releases in July and January.  The July delivery will have the higher risk and more complex functionality, while the January component will have the filing season changes and any additional changes that capacity permits.  Because the IRS will have the returns from earlier in the filing season available for testing the July release, it will conduct performance testing on that release using the highest volume periods.  Whether or not performance testing is also done on the January release will be based on the specific nature of the changes that it makes for that specific delivery.

 

Improvements to the Customer Account Data Engine Operator’s Guide Need to Be Completed

 

In March 2003, we reported that improvements to the CADE Operator’s Guide, used by the IRS for testing and operating the CADE, would help ensure an effective Release 1 deployment.  The BSMO agreed and reported on April 6, 2004, that the CADE Operator’s Guide was completed and approved by the PRIME contractor’s Change Control Board as of February 23, 2004. 

The CADE 2004 Pilot plan emphasized following the documentation and procedures described in the CADE Operator’s Guide.  It also stated the pilot participants should understand and follow the delivered documentation.

The CADE Operator’s Guide was needed for the 2004 CADE Pilot, as well as for live production that began in July 2004.  However, the CADE Release 1.1 task order did not schedule delivery of the Guide until September 15, 2004. 

The CADE subject-matter expert advised us the documentation within the CADE Operator’s Guide and computer handbooks was incomplete and not always consistent with actual CADE processing.  For example, the System Flows Appendix of the CADE Operator’s Guide provides details to IRS computer operators to restore CADE processing after a tax processing error is identified.  The 2004 CADE Pilot experienced two instances in which daily cycle processing experienced problems and the CADE had to be restored so the daily processing could be rerun.  However, the CADE Operator’s Guide could not be used to initiate rerunning the CADE because the System Flows Appendix did not specify what steps to follow to identify the files or data needing deletion to fix the CADE processing problems.

Live production may be affected by the incomplete and inconsistent documentation in the CADE Operator’s Guide.  A draft version of the CADE Operator’s Guide was issued June 21, 2004, which included updated procedures to provide complete and consistent instructions to operate the CADE.  Subsequent to stopping the CADE’s live production on July 3, 2004, a CADE official from the IRS identified critical agreed to revisions that were not made to the June 21, 2004, CADE Operator’s Guide.  Without adequate documentation describing the steps and resources needed to operate the CADE, efficient and effective processing cannot be assured.

Recommendation

2.      The CIO should direct the BSMO to ensure pilot and production operational documentation for future CADE releases is reviewed, tested, and approved before both the pilot test and live production are allowed to proceed.  This review should also ensure the corrective action to our March 2003 report is incorporated into this review and the IRS’ minimum documentation standards are completed and met in current and future CADE releases.

Management’s Response:  The CIO plans to have the full operational documentation available at the time of implementation for future releases.

 

Manual Processes Within the Customer Account Data Engine Release 1.1 Need to Be Automated for Future Releases

 

The IRS engaged the PRIME contractor to design and build the various modernization projects, including the CADE.  The PRIME contractor was selected to provide the IRS with access to commercial best practices, guarantee access to viable alternative solutions, and streamline the system acquisition process. 

Although the PRIME contractor was hired to provide the qualities above, the CADE currently includes manual processes that could be automated to achieve processing efficiencies.  These processes were not automated because they were not included as requirements to operate the CADE.

CADE project officials advised us of two examples of manual processes required if normal CADE operations are interrupted and processing needs to be rerun:

·         The process used to initiate the CADE computer programs is currently performed manually although the process could be automated.

·         CADE refund processing automatically starts daily at 9 P.M. eastern standard time.  A lengthy manual process must be invoked if this processing requires initiation at a different time.

CADE project officials advised us the automation of these CADE manual processes was overlooked during the requirements development process because of less than adequate communications with the PRIME contractor.  Because these types of manual processes were not identified for automation in the project, the contract did not include this development activity.  

The efficiency of CADE processing could be improved if manual processes are automated.  Automation of these processes would help ensure consistency in operations, avoid human error, and reduce staff time needed to perform the manual operations.  

Recommendation

3.      The CIO should direct the BSMO to ensure inefficient manual processes, including the processes cited above, are automated in future CADE releases.

Management’s Response:  The CIO plans a number of specific changes for the January 2005 and July 2005 CADE release deliveries to address known operational inefficiencies.

 

The Customer Account Data Engine Program Does Not Have a Dedicated System Architect

 

As one of the three studies initiated by the IRS during the summer of 2003, the Software Engineering Institute (SEI) performed an independent assessment of the CADE program.  One of the principal findings of the SEI report was that the CADE program did not have a dedicated system architect to provide technical leadership.  The SEI reported the appointment of a dedicated system architect is critical to the success of the CADE program.  The BSM Enterprise Architecture group provided the CADE an individual to perform system architect duties only on an as needed basis.  This person was not assigned solely to the CADE program. 

Comprehensive enterprise architecture efforts reduce risk to information technology projects by providing a solid foundation upon which to build and deploy IRS business solutions.  Sound enterprise architecture practices include established infrastructure standards, clearly defined application architecture, and a body of knowledge regarding best practices in project design.  Sound enterprise architecture practices also can eliminate much of the risk involved in custom application development, as well as the deployment of commercial off-the-shelf products.  The system architect has the task of implementing and coordinating these efforts.

The CADE program has not had consistent and comprehensive communications, design, and development throughout its history.  The teamwork, communications, and relationship between the IRS and the PRIME contractor did not always promote efficiency within the CADE program.  The absence of a dedicated CADE system architect contributed to the inability of the IRS and the PRIME contractor to communicate effectively to refine and validate the CADE requirements.  This led to inconsistent, misunderstood requirements. 

For example, the PRIME contractor did not understand the standards necessary in the IRS to implement address verification software.  Several versions were used in the CADE pilot activities that failed.  The failure was realized when the IRS determined the PRIME contractor was using United States Postal Service, not IRS, standards for addressing.  An IRS system architect could have identified the necessary standard at the inception of this portion of the application development.

We also identified communication inefficiencies between BSMO executives and CADE project officials.  For example, in January 2004, BSMO executives presented CADE information to the IRS Oversight Board regarding the content and timetables for the delivery of three CADE subreleases.  One month later, we met with representatives from the CADE project and the responsible IRS business unit to discuss the three subreleases.  At that meeting, the CADE and IRS business representatives stated they did not know a third CADE subrelease was planned for delivery in 2006.

The BSM Challenge process addressed weaknesses identified in the SEI’s report on the CADE related to the need for a dedicated system architect.  In response to this Challenge, the BSMO received funds to fill 11 system architect/system engineering positions for the BSM effort.  Five positions were planned to be added during late Fiscal Year 2004 and six positions will be added in Fiscal Year 2005.  Plans to dedicate a system architect for the CADE program are under consideration. 

Without a dedicated system architect, technology choices can be inappropriate for any number of reasons.  Architecture elements selected by CADE technical specialists may not interoperate as well as expected with the IRS’ other highly specialized information systems elements.  At the project level, inappropriate application architecture can create integration difficulties or cause a system to fail under production volumes.

By dedicating a system architect, the CADE program will have an accountable resource to manage the:

·         CADE technical issues.

·         Impact of proposed solutions to technical issues on the IRS’ current and future modernization projects, including the impact on existing legacy systems.

·         Development of the CADE to ensure compliance with IRS’ Enterprise Architecture.

·         Development of the CADE to facilitate the IRS’ business processes.

·         Communication of project requirements and development with the PRIME contractor and other interested stakeholders.

Recommendation

4.      The CIO should ensure the BSMO makes the system architecture resources being acquired available to the CADE program on a full-time basis.

Management’s Response:  The CIO has been recruiting for the last year to find qualified engineering resources.  The BSMO has hired a few new employees with the appropriate experience, some of which are dedicated to the CADE project.  However, the IRS has found it difficult to find and attract qualified individuals to work for the Government.  Until the additional staffing is onboard, the BSMO is providing the CADE engineering/architecture support from the existing Enterprise Architecture team and MITRE.

 

Disaster Recovery Capabilities Were Not Tested Prior to Customer Account Data Engine Release 1.1 Implementation

 

The IRS needs to be able to fully restore CADE functionality in the event of a disaster after system implementation.  Our review of the draft CADE Release 1.1 system requirements showed that, while the CADE Disaster Recovery Plan adequately addressed disaster recovery requirements, these requirements were not tested prior to CADE Release 1.1 implementation.

According to project officials, the CADE disaster recovery capabilities will be tested in the fall of 2004 as part the Annual IRS Disaster Recovery Test.  CADE disaster recovery testing plans executed in the fall of 2004 will ensure the CADE will be able to resume processing should a disruption occur during the 2005 Filing Season when processing levels will be significantly higher.

Without full disaster recovery testing, the IRS may have difficulty identifying the risks the CADE faces to be able to fully recover in the event of a disaster.  As a result, taxpayer data contained on the CADE may be lost, requiring the IRS to contact taxpayers and/or the representatives to reacquire the lost information.

Recommendations

To ensure the ability of the IRS to restore the CADE after a disaster with the least disruption to the IRS mission, the CIO needs to ensure:

5.      All aspects of the CADE disaster recovery capabilities are tested during the Annual IRS Disaster Recovery Test.

Management’s Response:  The Director, Enterprise Computing Center (ECC), tested the recovery and restoration of key components of CADE processing in order to ensure a high confidence level in the IRS’ ability to recover the CADE in the event of a true disaster.  The test was conducted during the weeks of October 18, 2004, and October 25, 2004.  There were some issues with the CADE requiring correction.

6.      Disaster recovery capabilities for future releases of the CADE are fully tested prior to implementation.

Management’s Response:  The CIO will have the Enterprise Operations organization work with the ECC to ensure that CADE disaster recovery testing is included in the annual disaster recovery testing performed in the fall of each year.  Since most new CADE functionality will be deployed in the July release, the fall testing would be timely, i.e., prior to the start of the new filing season when the IRS will encounter high volumes.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

The overall objective of this review was to determine whether the Internal Revenue Service (IRS) and PRIME contractor would timely and effectively deliver the Customer Account Data Engine (CADE) Release 1 requirements – processing Forms 1040EZ for single taxpayers with refunds due or zero balances – in compliance with existing Enterprise Life Cycle (ELC) guidelines.  This review is part of our Fiscal Year 2004 Audit Plan for reviews of the IRS’ Business Systems Modernization (BSM) efforts.

To accomplish our objective, we reviewed available documentation and interviewed PRIME contractor, IRS Information Technology Services office, and BSM Office (BMSO) executives, managers, and analysts.  We:

I.                   Determined whether all essential Form 1040EZ current processing environment requirements were identified and included for the CADE Release 1.1 using the guidance provided in the ELC.

A.    Reviewed the System Requirements Model (Section 3) of the CADE System Engineering Model View for Release 1.0, dated February 3, 2004, which contained 933 requirements.  The requirements were divided into 12 sections.

B.     Judgmentally sampled 93 of the 933 requirements to assess whether the CADE Release 1.1 objectives would be achieved.  We used judgmental sampling because we did not plan to project results across the population.  The sample of requirements included:

1.      Reviewing one requirement from nine requirement sections (nine total requirements).  

2.      Reviewing all requirements from 3 sections (57 requirements) that did not include adequate documentation tracing the requirement to the tests of requirements. 

3.      Reviewing all Financial Management System related requirements (27 requirements).

II.                Reviewed the 2004 CADE Pilot activities to determine whether the BSMO and the PRIME contractor used these results to successfully deliver CADE taxpayer benefits.

III.             Reviewed the CADE Release 1.1 contract task order provisions to determine whether contracted requirements were delivered.

IV.             Determined the benefits the BSM Challenge Issues results provided or planned to be provided for the CADE project development.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary Hinkle, Director

Edward A. Neuwirth, Audit Manager

Bruce Polidori, Senior Auditor

Lawrence White, Senior Auditor

Louis V. Zullo, Senior Auditor

Kim McManis, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS 

Deputy Commissioner, Wage and Investment Division  SE:W 

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Associate Chief Information Officer, Modernization Management  OS:CIO:MM

Director, Stakeholder Management Division  OS:CIO:SM

Deputy Associate Chief Information Officer, Program Management  OS:CIO:B:PM
Deputy Associate Chief Information Officer, Systems Integration  OS:CIO:B:SI

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaisons: 

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Manager, Program Oversight Office  OS:CIO:SM:PO

 

Appendix IV

 

Business System Modernization Challenge Issues Relating to the Customer Account Data Engine

The Business Systems Modernization (BSM) Challenges Plan Close-Out Report, dated May 1, 2004, provides for or plans to provide for improved future Customer Account Data Engine (CADE) project development.  The CADE-related BSM Challenge Issues and related close-out plans include:

-        Make business owner and program director accountable for project success.

The BSM Office (BSMO) is restructuring its management and governance structure to institutionalize the roles of Business Leader (replacing Business Owner), Business Requirements Director, Program Director, and Program Manager.

-        Implement short duration, discrete "Tiger Teams" in the CADE, Integrated Financial System, and e-Services projects.

The CADE Tiger Team conducted a comprehensive analysis of the CADE project and provided recommendations in a report about managing defects, contracts, and program execution.

-        Designate business architects for all projects to support business insight required throughout the project life.

The role of the Business Architects (title changed to Business Requirements Director – see footnote 2) and other program/project management positions were defined as part of the larger governance structure reorganization.

-        Assign business owners and architects to validate business unit participation and accountability in projects.

Roles of the Business Leader, the Business Requirements Director, the BSM Program Director, the Acquisition Project Manager, and the Information Technology Services Coordinator have been defined and socialized among the BSM, the Modernization and Information Technology Services organization, and the business (operating division/function) executives.

-        Align critical engineering talent to most critical projects.

Key engineering talent from MITRE, the PRIME contractor, and other appropriate resources were identified; the impact of realigning this talent is being addressed.

-        Identify key productivity and quality metrics across the life cycle based on industry standards.

A joint Internal Revenue Service (IRS), PRIME contractor, and MITRE Tiger Team developed the PRIME Enterprise Life Cycle (ELC) – the key productivity and quality measures roll-out plan for effective implementation and institutionalization associated with ELC Milestone 4.

-        Move acceptance testing to be earlier in the life cycle.

The PRIME contractor’s testing and deployment office collaborated with the IRS to develop a combined Systems Integration Test/Systems Acceptance Test for the CADE Release 1.1.

-        Establish independent architecture and engineering team of IRS and contractor business architects and system engineers.

The IRS and the PRIME contractor formed the Joint Engineering Team (JET) consisting of business architects and system engineers.

-        Clearly define business requirements and tightly manage them to control scope.

The BSMO worked with the PRIME contractor and representatives from various process teams to define Milestone 4A exit criteria, obtain buy-in from key stakeholders, and re-define ELC requirements and milestone exit criteria.

-        Identify "blockers" on current contracting actions.

Blockers were identified on October 24, 2003, which centered on business integration, systems integration, and the PRIME contractor’s program management office activities.

 

Appendix V

 

Customer Account Data Engine Costs

 

Source:  CADE contract task orders from the Internal Revenue Service (IRS) Procurement Office.

 

Source:  Business Systems Modernization (BSM) Expenditure Plans from the IRS as of July 2004.

*    Scheduled for future release development.
** Estimated costs for future release development.

 

Source:  The IRS BSM Fiscal Year 2004 and 2005 Expenditure Plan.

 

Appendix VI

 

Enterprise Life Cycle Overview

 

The Enterprise Life Cycle (ELC) defines the processes, products, techniques, roles, responsibilities, policies, procedures, and standards associated with planning, executing, and managing business change.  It includes redesign of business processes; transformation of the organization; and development, integration, deployment, and maintenance of the related information technology applications and infrastructure.  Its immediate focus is the Internal Revenue Service (IRS) Business Systems Modernization (BSM) program.  Both the IRS and the PRIME contractor must follow the ELC in developing/acquiring business solutions for modernization projects.

The ELC framework is a flexible and adaptable structure within which one plans, executes, and integrates business change.  The ELC process layer was created principally from the Computer Sciences Corporation’s Catalyst^® methodology.  It is intended to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision making and information sharing.  Other components and extensions were added as needed to meet the specific needs of the IRS BSM program.

ELC Processes

A process is an ordered, interdependent set of activities established to accomplish a specific purpose.  Processes help to define what work needs to be performed.  The ELC methodology includes two major groups of processes:

Life-Cycle Processes, which are organized into phases and subphases and address all domains of business change.

Management Processes, which are organized into management areas and operate across the entire life cycle.

 

Enterprise Life-Cycle Processes

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

Life-Cycle Processes

The life-cycle processes of the ELC are divided into six phases, as described below:

·                   Vision and Strategy - This phase establishes the overall direction and priorities for business change for the enterprise.  It also identifies and prioritizes the business or system areas for further analysis.

·                     Architecture - This phase establishes the concept/vision, requirements, and design for a particular business area or target system.  It also defines the releases for the business area or system.

·                     Development - This phase includes the analysis, design, acquisition, modification, construction, and testing of the components of a business solution.  This phase also includes routine planned maintenance of applications.

·                     Integration - This phase includes the integration, testing, piloting, and acceptance of a release.  In this phase, the integration team brings together individual work packages of solution components developed or acquired separately during the Development phase.  Application and technical infrastructure components are tested to determine whether they interact properly.  If appropriate, the team conducts a pilot to ensure all elements of the business solution work together.

·                     Deployment - This phase includes preparation of a release for deployment and actual deployment of the release to the deployment sites.  During this phase, the deployment team puts the solution release into operation at target sites.

·                     Operations and Support - This phase addresses the ongoing operations and support of the system.  It begins after the business processes and system(s) have been installed and have begun performing business functions.  It encompasses all of the operations and support processes necessary to deliver the services associated with managing all or part of a computing environment.

The Operations and Support phase includes the scheduled activities, such as planned maintenance, systems backup, and production output, as well as the nonscheduled activities, such as problem resolution and service request delivery, including emergency unplanned maintenance of applications.  It also includes the support processes required to keep the system up and running at the contractually specified level.

Management Processes

Besides the life-cycle processes, the ELC also addresses the various management areas at the process level.  The management areas include:

·                     IRS Governance and Investment Decision Management - This area is responsible for managing the overall direction of the IRS, determining where to invest, and managing the investments over time.

·                     Program Management and Project Management - This area is responsible for organizing, planning, directing, and controlling the activities within the program and its subordinate projects to achieve the objectives of the program and deliver the expected business results.

·                     Architectural Engineering/Development Coordination - This area is responsible for managing the technical aspects of coordination across projects and disciplines, such as managing interfaces, controlling architectural changes, ensuring architectural compliance, maintaining standards, and resolving issues.

·                     Management Support Processes - This area includes common management processes, such as quality management and configuration management that operate across multiple levels of management.

Milestones

The ELC establishes a set of repeatable processes and a system of milestones, checkpoints, and reviews that reduce the risks of systems development, accelerate the delivery of business solutions, and ensure alignment with the overall business strategy.  The ELC defines a series of milestones in the life-cycle processes.  Milestones provide for “go/no-go” decision points in the project and are sometimes associated with funding approval to proceed.  They occur at natural breaks in the process where there is new information regarding costs, benefits, and risks and where executive authority is necessary for next phase expenditures.

There are five milestones during the project life cycle: 

·                     Milestone 1 - Business Vision and Case for Action.  In the activities leading up to Milestone 1, executive leadership identifies the direction and priorities for IRS business change.  These guide which business areas and systems development projects are funded for further analysis.  The primary decision at Milestone 1 is to select BSM projects based on both the enterprise-level Vision and Strategy and the Enterprise Architecture.

·                     Milestone 2 - Business Systems Concept and Preliminary Business Case.  The activities leading up to Milestone 2 establish the project concept, including requirements and design elements, as a solution for a specific business area or business system.  A preliminary business case is also produced.  The primary decision at Milestone 2 is to approve the solution/system concept and associated plans for a modernization initiative and to authorize funding for that solution.

·                     Milestone 3 - Business Systems Design and Baseline Business Case.  In the activities leading up to Milestone 3, the major components of the business solution are analyzed and designed.  A baseline business case is also produced.  The primary decision at Milestone 3 is to accept the logical system design and associated plans and to authorize funding for development, test, and (if chosen) pilot of that solution.

·                     Milestone 4 - Business Systems Development and Enterprise Deployment Decision.  In the activities leading up to Milestone 4, the business solution is built.  The system is integrated with other business systems and tested, piloted (usually), and prepared for deployment.  The primary decision at Milestone 4 is to authorize the release for enterprise-wide deployment and commit the necessary resources.

·                     Milestone 5 - Business Systems Deployment and Postdeployment Evaluation.  In the activities leading up to Milestone 5, the business solution is fully deployed, including delivery of training on use and maintenance.  The primary decision at Milestone 5 is to authorize the release of performance-based compensation based on actual, measured performance of the business system.

 

Appendix VII

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.