Improvements in the Customer Account Data Engine Pilot Plan
Need to Be Considered to Help Ensure the Pilot’s Success
November 2002
Reference
Number: 2003-20-018
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
November
5, 2002
MEMORANDUM FOR
DEPUTY COMMISSIONER FOR MODERNIZATION
&
CHIEF INFORMATION OFFICER
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Acting Deputy Inspector
General for Audit
SUBJECT: Final Audit Report - Improvements in the Customer Account Data
Engine Pilot Plan Need to Be Considered to Help Ensure the Pilot’s Success
(Audit # 200120043)
This
report presents the results of our review of the adequacy of the Customer Account Data Engine (CADE) Release 1 Pilot
and data conversion plans. The CADE
will replace the Internal Revenue Service’s (IRS) antiquated taxpayer account
system with a modernized database. The
new system, which is the centerpiece of the IRS’ modernization strategy, will
facilitate faster processing of taxpayer returns and refunds and improved
customer service. The conversion of
taxpayer accounts to the modernized database will happen in several
stages. The first group of taxpayer
accounts to be moved to the CADE includes single taxpayers who file an
electronic or paper Income Tax Return for Single and Joint Filers With No
Dependents (Form 1040EZ). The IRS and
its PRIME contractor are conducting a pilot of this first stage during 2002,
with plans to begin processing live tax returns through the CADE in April 2003.
In
summary, we found that the IRS’ data
conversion plans adequately document the process necessary to move taxpayer
account data to and from existing IRS systems and the modernized CADE. One key to the pilot’s success is to
carefully track the data conversion progress against the plans and timely
address any potential barriers or risks.
On February 4, 2002, the PRIME contractor prepared a draft pilot
plan to guide the execution of the pilot process. On May 2, 2002, the PRIME contractor distributed the draft pilot
plan for review, comment, and concurrence by the IRS. The PRIME contractor considered the comments and suggestions and
incorporated many of them into the pilot plan.
The IRS formally approved the plan on July 24, 2002.
The pilot plan outlines
various activities, including the use of processing scenarios and problem
management reporting. Although the plan
provides a comprehensive and organized approach to conduct the pilot, it does
not include all necessary details or procedures about performing and assessing
specific activities, as well as resolving problems and training pilot team
members. Without a complete and
accurate pilot plan, the IRS cannot adequately monitor pilot progress or ensure
that the PRIME contractor will be taking the appropriate steps to expeditiously
complete the pilot objectives.
We discussed the contents of
this report with management from the IRS’ Business Systems Modernization Office
(BSMO) on September 9, 2002. BSMO
management told us that the CADE pilot plan met their expectations and that
details not included in the pilot plan could be located in related CADE project
documents. We reviewed the related CADE
documents that were provided to us but still did not find the details we
believe are needed to effectively execute and monitor the pilot activity and to
assess the pilot results.
We
believe that to provide greater assurance that the pilot objectives will be
achieved, the BSMO should require the PRIME contractor to complete detailed
acceptance criteria, execution scenarios, defect reporting procedures, and team
training plans. The BSMO should also
ensure complete and thorough reviews of the PRIME contractor’s future project
execution plans prior to approving them.
Management’s Response: The
Deputy Commissioner for Modernization & Chief Information Officer responded
that he
found helpful several of the report observations about defect reporting and
that the IRS is actively pursuing corrective actions in those areas and
monitoring them through the risk database.
However, the IRS disagreed with recommendations about revising the CADE
pilot plan. The response stated that
the IRS added the pilot
phase to the first release of the CADE as part of a mitigation strategy to
assure itself that the PRIME contractor was building a practical and executable
design without unduly affecting taxpayers during the upcoming filing season. The response further stated that the
pilot plan establishes high-level goals, scope, structure, and activities and
that the IRS did not intend the plan to provide all of the details for running
and measuring the success of the pilot. Management’s complete response to the report
is included as Appendix V.
Office of Audit Comment: Although the IRS responded that the pilot plan met the IRS’ and the PRIME contractor’s expectations, we maintain that the plan was incomplete and did not provide the necessary guidance to effectively execute the pilot, nor was the missing information contained in the additional documents provided to us. The completion of the CADE pilot test has been delayed by several weeks, and while there are many factors contributing to the delays, we believe more detailed execution scenarios and acceptance criteria will allow the IRS and the PRIME contractor to better manage the pilot test to ensure success and minimize additional delays. We also believe that accepting incomplete plans on current and future modernization projects increases the risk of delays in providing benefits to the IRS and taxpayers. While we still believe our recommendations are valid, we do not intend to elevate our disagreement concerning management’s response to the Department of the Treasury for resolution.
Copies
of this report are also being sent to the IRS managers who are affected by the
report recommendations. Please contact
me at (202) 622-3837 if you have questions, or Scott Wilson, Assistant
Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
The Pilot
Plan Needs to Include Detailed Acceptance Criteria
The Pilot Plan Needs to Include Sufficient Detail in Execution Scenarios
The
Pilot Plan Needs to Include Adequate
Defect Reporting Procedures
The Pilot Plan Needs to Include a Detailed Training Schedule
The Data Conversion Plan Documentation Is Adequate
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix V – Management’s Response to the Draft Report
The Customer Account Data Engine (CADE) project is the foundation for managing taxpayer accounts in the Internal Revenue Service’s (IRS) modernization plan. The CADE project will incrementally design, develop, and implement the data foundation for a modernized IRS. It will accomplish this by replacing the system that processes individual taxpayer account and return information, known as the Individual Master File, and the related components with new technology, new applications, and new databases. It is a critical project in modernizing the IRS’ computer systems.
The database and applications developed by the CADE will enable the development of subsequent modernized systems. In conjunction with other applications, the CADE will allow employees to post transactions and update taxpayer account and return data from their desks. Updates will be immediately available to any employee who accesses the data and will provide a complete, timely, and accurate account of the taxpayer’s information.
Successful implementation of the CADE will change the IRS’ customer service capabilities, such as providing faster tax refunds. The CADE will change the way IRS employees do their jobs by giving them instant access to taxpayers’ records, similar to bank employees accessing account balances in real-time to determine how much money is in customer accounts.
The IRS initiated the CADE project in September 1999. In August 2000, the CADE project completed the architecture phase, and in June 2001 the project exited the design phase. The CADE Phase 1 will consist of five releases for deployment between 2002 and 2006. The CADE Release 1 deployment (Income Tax Return for Single and Joint Filers With No Dependents [Form 1040 EZ] electronic and paper tax returns for single taxpayers) is scheduled for April 2003. The four subsequent releases will eventually process all individual taxpayer accounts. The CADE Phase 2 will process business accounts; however, development and implementation dates have not been established.
The IRS postponed the deployment of CADE Release 1 from January 2002 until April 2003. This delay occurred in part because the PRIME contractor could not finalize a contract for improved software in time for the development phase of CADE Release 1. The delay also occurred because the PRIME contractor underestimated the complexity of developing balancing and reconciliation software.
To test CADE Release 1 operational abilities, the IRS initiated a Release 1 Pilot in July 2002. Using real business data and transactions, the Release 1 Pilot will allow the IRS and the PRIME contractor to assess and validate the CADE in a test environment. The Release 1 Pilot requires the ability to convert taxpayer data residing on existing IRS systems to and from the modernized CADE database.
The pilot is scheduled to be performed in three successive phases. Phase 1 is the Initial Operations, where the PRIME contractor will operate the CADE applications. Phase 2 is the Transition to Operations, where both the PRIME contractor and the IRS work “shoulder-to-shoulder” to operate the CADE applications. Phase 3 is the Mock Operations, where the IRS will operate the CADE applications.
The overall objective of this review was to determine the adequacy of the CADE Release 1 Pilot and data conversion plans. We initiated this review at the request of IRS Business Systems Modernization Office (BSMO) executives on May 29, 2002.
To accomplish our objective, we reviewed available documentation and interviewed PRIME contractor and BSMO executives, managers, and analysts located at the IRS National Headquarters and New Carrollton, Maryland, offices. We performed the audit between May and August 2002. The audit was conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The CADE project team worked with the PRIME contractor to
initiate a pilot using real business
data and transactions to assess and validate CADE Release 1 capabilities in an
operational type of environment. The pilot goals include validating CADE processing activities such as balancing,
reconciliation and controls, backup and recovery, end-to-end operations, and
interface applications.
On February 4, 2002, the PRIME
contractor prepared a draft pilot plan to guide the execution of the pilot
process. On May 2, 2002, the PRIME
contractor distributed the draft pilot plan for review, comment, and
concurrence by the IRS. The PRIME
contractor considered the comments and suggestions and incorporated many of
them into the pilot plan. The IRS
formally approved the plan on July 24, 2002.
The pilot plan outlines various activities, including the use of processing scenarios and problem management reporting. Although the plan provides a comprehensive and organized approach to conduct the pilot, it does not include all necessary details or procedures about pilot acceptance criteria, pilot execution scenarios, defect reporting, and pilot team training.
We discussed the contents of this report with management from the BSMO on September 9, 2002. BSMO management told us that the CADE pilot plan met their expectations and that details not included in the pilot plan could be located in related CADE project documents. We reviewed the related CADE documents that were provided to us but still did not find the details we believe are needed to effectively execute and monitor the pilot activity and to assess the pilot results.
The Enterprise Life Cycle (ELC) states that acceptance criteria and business performance objectives, as well as requirements and their satisfaction criteria, should be used to measure both the business and technical performance of a release. The CADE Pilot project team will use the measurement criteria presented in the pilot plan to evaluate the readiness to progress between the pilot phases and to assess the overall pilot success. However, the pilot plan does not include objective measurement criteria to allow an assessment of the acceptability of the pilot’s overall performance. The pilot plan does not include specific detail about what to measure and how to perform the measurements. For example, the following two success measurements are included in the plan. The CADE shall be able to:
·
“Consistently run one day’s worth of
taxpayer data within a single day.”
·
“Run
multiple days’ worth of CADE data within a single day to stress test the
environment from a system and software point of view.”
These pilot goals do not
include specific parameters to objectively assess performance to accept the
pilot results. For example, of what
does “one day’s worth of taxpayer data” consist? How many days does the CADE need to demonstrate that it can
consistently run to meet this success measurement? How many days make up “multiple”
days worth of data? Although the ELC
requires development of pilot acceptance measurement criteria, the IRS approved
the pilot plan without details to calculate these measurements.
Without detailed acceptance criteria to assess pilot performance, the IRS will not be able to record and monitor actual measurement results and compare them to planned results. The absence of detailed objective measurements does not allow the IRS to make a confident assessment about how well the pilot results meet the IRS’ requirements. The absence of these data could contribute to a decision to:
· Deploy the CADE Release 1 when it is not suitable for release, potentially causing faulty processing results and/or a loss of data that affect taxpayer account balances.
· Not deploy the CADE Release 1 although it may be suitable for release, depriving the IRS and taxpayers of numerous benefits.
To allow for an assessment of pilot performance with the ability to record and monitor actual measurement results and compare them to planned results, the BSMO should:
1.
Work with the PRIME contractor to develop detailed
pilot success measurements and incorporate the measurement process into the
pilot.
Management’s Response: The Deputy Commissioner for Modernization & Chief Information Officer (DCM/CIO) responded that the IRS does not plan to take any corrective action. He stated that the intent of the pilot plan was to establish goals, scope, structure, and activities that will occur during the pilot. The response also stated that the criteria established for beginning the pilot, moving from phase to phase, and measuring the pilot’s success are in the pilot go/no-go checklists, not in the pilot plan. Although the checklists themselves do not include quantitative criteria, they include key activities that the IRS and the PRIME contractor need completed in order to proceed through the sub-phases and pilot exit. The IRS and the PRIME contractor established a team of stakeholders and a documented process to ensure they completed checklist items before deciding to proceed. Though most of the checklist items are not quantitative, the IRS and the PRIME contractor believe that the team’s collective judgment provides more than adequate assurance that each phase of the pilot is successful.
The DCM/CIO stated that the IRS cannot anticipate specific, measurable outcomes until it runs records through the pilot
and compares them to the results of the current system. The pilot is successful when it achieves the
same results as the current system, or the IRS discovers and resolves the
reasons for differences.
Office of Audit Comment: Although
the IRS’ and the PRIME contractor’s “go/no-go” team’s collective judgment may
provide appropriate decisions to proceed through the pilot, this approach does
not conform to the IRS’ system development life cycle process – the ELC. As this report presents, the ELC requires
that acceptance criteria and business performance
objectives, as well as requirements and their satisfaction criteria, be
used to measure both the business and technical performance of a release. Following the ELC allows the IRS to base
go/no-go decisions on objective quantitative data.
CADE pilot processing has operational activities that outline potential processing steps or procedures (structured into specific scenarios) that the CADE needs to perform. The pilot scenarios are organized into categories based on processing cycles: Initialization, Back Office Processing, Daily Cycle Processing, Weekly Cycle Processing, and Performance Testing.
The pilot execution will be conducted by running the scenarios according to the related processing cycle. The scenarios can be run individually or as part of an end-to-end operating scenario to test the CADE’s readiness. Scenarios are weighted based on their importance, and each should have its own success criteria.
The plan includes a Scenario Layout that provides a template to document the scenarios. Significant fields in the Scenario Layout include the:
·
Description/Objective - Description of the
scenario.
·
Success Criteria - Criteria to be measured
indicating success of the scenario run.
· Scenario Steps - Step-by-step documentation of the scenario to be run.
We reviewed the pilot plan appendices that contain the pilot execution scenarios and determined that they were incomplete to the extent that they do not include sufficient detail for the pilot participants to execute and evaluate the pilot. The documentation does not always provide a description of what the scenario will accomplish, the steps to follow for the pilot team to execute the scenario, or success criteria to assist in determining if the scenarios meet business expectations.
Although the scenario layout documentation included in the pilot plan was not complete, the IRS accepted and approved the scenarios in this plan for use in executing the pilot. Without sufficient guidance provided by complete scenarios, pilot team members will have difficulty in: executing pilot steps, monitoring pilot activities because of gaps in historical documentation about the processing steps in the scenario runs, and assessing the adequacy of operational activities because of the absence of objective measurement criteria about scenario success.
The following table presents the deficiencies in completing the scenario layout documentation in the pilot plan.
Scenario
Layout Deficiencies
Scenario Category |
Total Scenarios |
Incomplete Description/ Objective |
Incomplete Success Criteria |
Incomplete Scenario Steps |
|---|---|---|---|---|
|
Initialization |
5 |
0 |
0 |
2 |
|
Back
Office Processing |
54 |
39 |
40 |
45 |
|
Daily
Cycle Processing |
25 |
22 |
25 |
25 |
|
Weekly
Cycle Processing |
2 |
0 |
2 |
2 |
|
Performance
Testing |
49 |
7 |
19 |
49 |
|
Total |
135 |
68 |
86 |
123 |
Source: CADE Release 1 Pilot Plan version 2.0.
To help ensure that the scenarios are properly executed and
evaluated, the BSMO should:
2. Require the PRIME contractor to complete the pilot plan scenarios with detailed description/objective, scenario steps, and success criteria.
Management’s Response: The DCM/CIO disagreed with this recommendation. He stated that the IRS did not intend the
scenario content to be detailed enough to run
the scenarios. The IRS’ position was
that when it approved the pilot plan, it did not expect the scenario details to
be complete.
Office of Audit Comment: Although the DCM/CIO responded that the pilot could be accomplished with incomplete and missing scenarios, the pilot plan based the pilot execution on the ability to run the scenarios according to the related processing cycle. As we stated earlier, without sufficient guidance provided by complete scenarios, pilot team members will have difficulty in executing pilot steps, monitoring pilot activities, and assessing the adequacy of operational activities. The completion of the CADE pilot test has been delayed by several weeks, and while there are many factors contributing to the delays, we believe more detailed execution scenarios and acceptance criteria would allow the IRS and the PRIME contractor to better manage the pilot test to ensure success and minimize additional delays.
Problems found in system software or hardware are referred to as defects. Typically, defects are identified during system testing or by the end users of a product.
The pilot plan includes a “Problem Management” section that provides guidance to capture and track defect reports about pilot activities through the PRIME contractor and IRS problem reporting databases. The PRIME contractor is using the ClearQuest database, while the IRS is using the Information Technology Asset Management System (ITAMS) database to track and manage reported defects.
Although the pilot plan includes some guidance for defect reporting, it does not provide sufficient direction to close defect reports and reconcile reported defects between the PRIME contractor and IRS databases, require the IRS’ approval of defect report resolution and closure, and limit the PRIME contractor defect report administrator’s access to the ITAMS database.
The pilot plan does not include adequate defect report closure and reconciliation procedures to ensure accuracy of both the PRIME contractor and IRS defect report databases
The pilot plan includes procedures to input and track defect reports for both the ClearQuest and ITAMS defect report databases. The plan indicates that the PRIME contractor defect report administrator will manually track defect reports in both databases. However, the plan procedures do not include a reconciliation process to ensure the accuracy and completeness of the databases.
Defect report resolution and closure procedures do not include the IRS’ approval
Pilot plan procedures do not require the IRS’ concurrence to approve defect report resolution and closure actions. The PRIME contractor related that it would not require IRS approval of defect resolution actions to close defect reports. The IRS’ CADE team members related that their involvement occurred when they received a report of the defect reports closed. This involvement does not provide the IRS the opportunity to reject the decision to close a defect because of an inadequate resolution action.
Controls do not limit the PRIME contractor defect report administrator’s access in the IRS defect report database
Controls are not in place to limit the PRIME contractor defect report administrator’s ITAMS defect report database privileges. The IRS has plans to allow the PRIME contractor defect report administrator access to close records from this database. Without controls to limit privileges in the IRS’ ITAMS database, inappropriate closure of defects without the IRS’ knowledge and acceptance of their resolution may occur.
Although the defect report input, closure, and reconciliation processes were not thoroughly developed in the pilot plan, the IRS accepted and approved these processes for use in executing the pilot. Without adequate controls to manage the reporting and resolution of defects, the IRS does not have assurance that defects are properly resolved.
To ensure that appropriate actions are taken to address
defects identified during the pilot, the BSMO should:
3.
Require the PRIME contractor to
provide detailed procedures for reconciling defect reports in the ClearQuest
and ITAMS databases.
Management’s Response: The DCM/CIO
agreed with this recommendation.
The PRIME contractor has drafted procedures which document the process
for interfacing defect reporting information between the ClearQuest and
ITAMS defect tracking tools. These
procedures contain specific information on the reconciliation of these reports.
4. Require the PRIME contractor to provide written procedures directing that the IRS approve defect report resolution actions prior to defect report closure.
Management’s
Response: The
DCM/CIO agreed with this recommendation.
The IRS monitors ITAMS
usage and requires the PRIME contractor to develop procedures to include defect
report resolution. Under the draft
procedures, the PRIME contractor initiates the recommendation to close defect
reports in the ITAMS and the IRS is responsible for the final closure.
5. Limit the PRIME contractor defect report administrator’s ITAMS database privileges to “Read Only.”
Management’s
Response:
The DCM/CIO disagreed that the PRIME contractor should be limited to
“Read Only” access to the ITAMS. For
Phases 1 and 2, the IRS believes that the
PRIME contractor should be allowed to close defect reports. In mock operations, Phase 3, the CADE will
operate in a near production environment, so the IRS plans to be involved in
closing defect reports. The IRS will
evaluate this process during the pilot and implement it with appropriate
changes for production. It allows contractor access to the ITAMS, which is covered under
Corporate Management Standards in the Internal Revenue Manual. Contractors have Level 2 access supporting
the modernized and legacy projects. Service Provider Level 2 allows the user to update and close
problem tickets – typical for most users.
Office of Audit Comment: The DCM/CIO agrees with the need to implement appropriate internal controls to ensure the integrity of data. The planned actions above recognize the defect report controls needed in Phase 3 mock operations. However, not limiting closure privileges to the IRS on its ITAMS database during Phases 1 and 2 does not give the IRS assurance that appropriate defect solutions took place prior to defect report closure. As the report presents, without controls to limit privileges in the IRS’ ITAMS database, inappropriate closure of defects without the IRS’ knowledge and acceptance of their resolution may occur.
The pilot plan includes a brief section on planned training activities to be conducted in preparation to execute the three CADE Release 1 Pilot phases and the initial aspects of the CADE project deployment. The plan describes five types of training:
·
Overview sessions for all individuals who
are affected by the CADE.
·
Targeted training for security
administrators, database administrators, systems administrators, and systems
operators.
·
Shoulder-to-Shoulder training for those
individuals whose roles are affected by CADE procedures.
·
Coaches Preparation training for selected
“coaches” to help with the Shoulder-to-Shoulder training.
·
Train the Trainer sessions to
prepare IRS trainers to deliver training for future deployment.
Although the pilot plan describes the general training
approach, it does not include specifics about training session dates, participants, locations, or
instructor assignments. We reviewed a
supplemental training schedule provided from the PRIME contractor that included
training events, approximate dates, resources, and materials. However it did not include a detailed and
complete schedule of sessions, participants, and instructors.
Although the training plan was not thoroughly developed in
the pilot plan, the IRS accepted and approved the approach for use in executing
the pilot. Without a detailed training schedule, all IRS
personnel needing training may not receive training timely or at all. Without this training, the IRS may not have
the knowledge necessary to independently run the last phase of the pilot as
planned. Consequently, the IRS may not
be ready to operate the deployed CADE Release 1.
To ensure that proper training
occurs for the IRS personnel involved in the pilot, the BSMO should:
6. Confirm the training dates with the PRIME contractor and request a detailed training schedule with specific dates, session locations, and complete class rosters (including instructors).
Management’s Response: The DCM/CIO disagreed with this recommendation. The response stated that the pilot plan is not the appropriate place to include
detailed training plans. The PRIME
contractor is responsible for conducting the training. The IRS stated that it provided the Treasury
Inspector General for Tax Administration (TIGTA) with a copy of the CADE
Release 1 Training Initiative plan and the training schedule, classroom
training was partially accomplished before the TIGTA’s review, and the
remainder of the training will be handled per established procedures.
Office of Audit Comment: The DCM/CIO’s response did not
address the recommendation about confirming detailed training plans with the
PRIME contractor to be sure training resources are available when needed. The pilot
plan does include a brief section on planned training, and we reviewed a supplemental training
schedule to the pilot plan provided by the PRIME contractor. Additionally, the IRS response referred to
another schedule of training called the CADE Release 1 Training Initiative plan. By referring to several different training
plan vehicles, each with limited detail about training plans and resources,
assurance that the appropriate resources will be available and that the
training is scheduled is, at best, difficult.
Detailing this information in one document, such as the pilot plan, will
minimize confusion and provide an easily accessible reference.
As presented above, the IRS approved and accepted a CADE pilot plan that we believe was incomplete and does not provide the necessary guidance to effectively execute the pilot. Before approving the pilot plan, the IRS was given sufficient opportunity to review it and notify the PRIME contractor of any problems identified. The contract that included the pilot plan, Task Order 73, states, “The Government shall review deliverables in accordance with the ELC to ensure completeness, accuracy, and compliance with the requirements in Section C of this task order.” Upon initial receipt of a deliverable, the IRS had 11 working days to submit its comments. The IRS performed two reviews of the pilot plan and provided comments regarding many concerns it had with the plan. However, those comments did not include the deficiencies presented in this report relating to the pilot plan acceptance criteria, execution scenarios, defect reporting, and training.
The IRS performed yet another review before the official “go/no-go” decision that allowed the start of the pilot. The “go/no-go” decision is based, in part, on a checklist review. The checklist review, performed by the PRIME contractor, MITRE, and IRS reviewers, specifically addressed execution scenarios. The checklist asked, “Have Pilot scenarios been completed, reviewed, and accepted by Pilot stakeholders as fit for use?” The reviewers responded that all pilot Phase 1 scenarios had been reviewed and approved.
Our analysis of Phase 1 scenarios included requesting the BSMO to define the Phase 1 scenarios and provide the status of the scenario layout completion. The BSMO could not provide us the information about which scenarios would be run during Phase 1 or to what extent the PRIME contractor completed the layouts for the relevant scenarios. Analysis of scenarios that we believe are relevant to Phase 1 shows that only 3 of 24 of these scenario layouts were completed.
In addition to these shortcomings, the plan also alluded to work that will not be performed. The plan includes an entire section on System Acceptability Testing (SAT), even though the IRS Product Assurance office related that SAT testing would not be performed during the pilot. When we discussed this with the BSMO, it did not realize that the SAT section had not been removed from the plan.
Without a complete and detailed pilot plan, the IRS could have difficulty in adequately monitoring pilot progress or ensuring that the PRIME contractor will be taking appropriate steps to expeditiously complete the CADE Release 1 pilot objectives.
As previously noted, BSMO management stated that the pilot plan met their expectations for CADE Release 1. However, we believe that to obtain more complete and detailed project execution plans that describe plan activities, and allow for meaningful tracking of plan execution, the BSMO should:
7. Ensure future project plans developed by the PRIME contractor are complete and comprehensive prior to approving them. This is particularly critical since the BSMO is in the early stages of Business Systems Modernization with numerous projects and releases scheduled for deployment over the next 10 to 15 years. A complete and detailed pilot plan could be useful in planning pilot activities for future CADE releases. This plan could also serve as a template for other BSMO projects that use PRIME contractor project plans to implement modernized products.
Management’s Response: The DCM/CIO disagreed with this recommendation. He stated that the current plan meets the
expectations agreed upon by the IRS and the PRIME contractor and that the
appropriate IRS executives thoroughly reviewed and approved this plan. Because this pilot plan was tailored specifically for the
CADE, it can be used as a reference document, but the IRS does not recommend
using it as a template or standard for other modernization projects due to its
specific scope, purpose, and uniqueness.
The ELC methodology provides pilot planning guidance for projects to use
across the program. The IRS does not
anticipate using this approach for future releases. If the need arises for the CADE to implement another Pilot Phase
in a future release, the IRS would take another course of action because it
would already have the core functionality of the CADE in place.
Office of Audit Comment: Although the IRS responded that the pilot plan met the IRS’ and the PRIME contractor’s expectations, we maintain that the plan was incomplete and does not provide the necessary guidance to effectively execute the pilot. We also believe that regardless of the methodology followed, accepting incomplete plans on this or any future modernization project increases the risk of delays in providing benefits to the IRS and taxpayers.
Taxpayer account data currently reside on numerous IRS systems. One system contains individual taxpayer accounts, one system contains business accounts, and several smaller systems contain the remaining accounts. The data conversion process involves moving taxpayer accounts from the existing IRS systems to the modernized CADE database.
The Data Conversion Plan prepared for the IRS by the PRIME contractor provides a detailed description of the taxpayer account data that will be converted. The plan identifies the sources of these data within the IRS and documents the specific destination within the modernized CADE where the data will reside. The data conversion approach is logical and uses an industry-wide process to convert the data. Additionally, the data conversion testing plan, which is part of the pilot plan, describes the data conversion test approach, the testing activities, and details testing that will occur during the CADE Release 1 Pilot.
The Data Conversion Plan and supporting documentation (see Appendix IV for a complete list of documents) adequately describe the conversion process for moving taxpayer account data from existing IRS systems to the modernized CADE database. This process is crucial since once a year the CADE database will be populated with taxpayer account data from existing IRS systems.
The Data Conversion Plan and supporting documentation also adequately describe the conversion process for returning taxpayer account data from the CADE database to the existing IRS systems when the need arises. If a situation arises in the modernized CADE for which there is an inability to properly handle the taxpayer account conditions, the taxpayer data will be moved off of the modernized CADE and returned to an existing IRS system. The taxpayer data will remain there until a future CADE release can handle the account.
The IRS’ data conversion plans adequately document the
process necessary to move taxpayer account data to and from existing IRS
systems and the modernized CADE. One
key to the pilot’s success is now to carefully track the data conversion
progress against the plans and timely address any potential barriers or risks.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine the adequacy of the Customer Account Data Engine (CADE) Release 1 Pilot and data conversion plans. To accomplish this objective, we performed the following audit tests:
I.
Determined the
adequacy of the CADE Release 1 pilot plan.
A.
Obtained the current version of the CADE
Release 1 pilot plan, including the Customer Technical Review comments, to
review the pilot plan for completeness and reasonableness and determine whether
it adheres to guidance from the Enterprise Life Cycle (ELC) and the Software
Engineering Institute’s Capability Maturity Model (CMM) criteria (see
Attachment IV for additional supporting documentation).
B.
Interviewed Business Systems Modernization
Office (BSMO) and PRIME contractor project managers regarding concerns related
to the pilot plan’s completeness, reasonableness, and adherence to guidance
from the ELC and CMM.
C.
Used Treasury Inspector General for Tax
Administration, General Accounting Office, and MITRE project and program issues
identified in prior reviews and determined whether issues with the pilot plan
could have been avoided with proper implementation of key system development
processes and management disciplines.
II.
Determined the adequacy of the CADE data
conversion plans to move taxpayer account data to and from existing
Internal Revenue Service systems and the modernized CADE.
A.
Reviewed the planning documents used by the
BSMO and PRIME contractor to develop the data conversion process (Appendix IV
lists the specific documents reviewed).
B.
Interviewed the appropriate BSMO and PRIME
contractor staff to obtain clarification related to the CADE data conversion
process.
C. Conducted interviews with the appropriate
BSMO and PRIME contractor project managers to discuss issues related to the
data conversion process, including data mapping, data analysis, data
transformation and cleansing, and data validation.
Appendix II
Major Contributors to This Report
Scott E. Wilson, Assistant Inspector General for Audit
(Information Systems Programs)
Scott A. Macfarlane, Director
Edward A. Neuwirth, Audit Manager
Allen Gray, Senior Auditor
Bruce
Polidori, Senior Auditor
Glen
J. Rhoades, Senior Auditor
Beverly Tamanaha, Senior Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Associate Commissioner, Business Systems Modernization M:B
Deputy
Associate Commissioner, Program Management
M:B:PM
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaison:
Associate
Commissioner, Business Systems Modernization
M:B
Appendix IV
Pilot Plan Documents
Updated Customer Account Data Engine (CADE) Release 1 Pilot Plan, dated
July 3, 2002.
CADE Installation Guide, dated May 17, 2002.
CADE Systems Administrators Guide, dated May 17, 2002.
CADE Operations Guide, dated May 28, 2002.
Data Conversion Documents
CADE Data Conversion Plan, dated March 29, 2002.
Appendix N “Data Conversion Test Plan” in the Pilot Plan, dated March 29, 2002.
Appendix B “Data Mapping” of the Data Conversion Plan, dated March 29,
2002.
Business System Design Report, dated March 29, 2002.
System Engineering Model View, dated March 29, 2002.
System Engineering Model View Interface Control Documents, dated March
29, 2002.
Release 1 System Integration Test Plan, dated May 3, 2002.
CADE Fiscal Year 2002 Milestone 4 Work Breakdown Structure as of July 26,
2002.
Appendix V
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.