The Customer Communications Project 2001 Release Was
Deployed, But Testing Processes Did Not Ensure All Applications Were Working As
Intended
March 2002
Reference
Number: 2002-20-056
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
March
1, 2002
MEMORANDUM FOR DEPUTY COMMISSIONER FOR MODERNIZATION & CHIEF INFORMATION OFFICER
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for
Audit
SUBJECT: Final Audit Report - The Customer
Communications Project 2001 Release Was Deployed, But Testing Processes Did Not
Ensure All Applications Were Working As Intended (Audit # 200120033)
This
report presents the results of our review of the Customer Communications Project Fiscal Year 2001
Release (CC 2001) deployment readiness and testing activities. This audit determined whether CC2001
deployment and testing activities followed approved plans to ensure intended
project capabilities were delivered.
Although
we followed Government Auditing
Standards, we encountered scope limitations by not having access to
sufficient, competent, relevant, and timely information to afford a reasonable
basis to draw conclusions and meet the audit objective. The absence of timely and complete
information impaired our ability to fully assess whether intended project
capabilities were provided.
CC
2001 became operational in August 2001.
The Business Systems Modernization Office (BSMO) reported that CC 2001
improved the Internal Revenue Service’s (IRS) ability to receive, route, and
respond to the more than 150 million taxpayer telephone calls received each
year. Major system improvements include
designs to use voice-activated programs that recognize English- or
Spanish-speaking callers, a voice-activated program that taxpayers can use to
find out the status of their refunds, and capabilities that more accurately
route taxpayer calls to the most appropriate IRS personnel.
In
summary, based on the information we were able to review, we found that the
BSMO and the Computer Sciences Corporation (CSC) did not sufficiently test all
CC 2001 project capabilities to ensure they were working as intended. Neither the BSMO nor the CSC implemented
adequate controls to ensure that testing of significant project requirements
was documented and approved.
Also,
there was a lack of documentation to show how problems or defects identified
during the tests that were conducted were resolved. Without this documentation, the BSMO had no evidence to support
how or if the problems or defects were corrected. Finally, the project deployment decision process did not consider
all significant issues and document that systems performance criteria were met
and problems resolved before the project was deployed.
Without
adequate evidence that CC 2001 met the requirements it was designed to deliver,
and that unresolved problems or defects were resolved prior to deployment,
problems arising during and after deployment could negatively affect the
services that CC 2001 was designed to deliver to taxpayers.
We recommend that, before future projects are deployed,
the Deputy Commissioner for Modernization & Chief Information Officer
strengthen processes for system testing, problem reporting and resolution, and
for making project deployment “go/no-go” decisions.
Management’s Response: Management agreed with our findings and five of the six recommendations presented in the report. They indicated that they have implemented four of the five agreed to recommendations. To ensure that systems requirements are adequately tested, the IRS Product Assurance function will conduct reviews of system requirements test plans, and will document non-compliance with the test plans. To ensure adequate configuration management procedures, the BSMO’s Office of Configuration Management was assigned responsibility to conduct configuration management audits of IRS modernization projects. To improve the defect report resolution process, the CSC developed procedures for the defect report process flow, including actions to address, close, and document a defect. This process will also provide the CSC Defect Report Coordinator with the responsibility to ensure that all defect reports entered into the control database contain complete and accurate information. Finally, to provide sound “go/no-go” decisions prior to proceeding to deployment, the CSC is drafting procedures which detail specific review items and pass/fail criteria that project teams will follow. The procedures will also include mandatory signoff from all review parties to concur that criteria have been met prior to initiating deployment.
Management did not agree
with our recommendation for the IRS to review and approve resolution and
closure of all defect reports.
Management cited that the high number of defect reports generated during
the project testing would make such reviews difficult. Management’s complete response to the draft
report is included as Appendix VI.
Office
of Audit Comment: While we agree that the extraordinarily
high number of testing defects on this project presented a challenge, we believe that because there were such a high number
of defects, management needs to have adequate assurance that all problems are
either resolved or reduced to an acceptable level. Absence of BSMO input in all defect closure decisions increases
the risk that the project will not meet performance expectations because some
defects may be closed without adequate resolution. While
we still believe our recommendation is worthwhile, we do not intend to elevate
our disagreement concerning it to the Department of Treasury for resolution. Future
Treasury Inspector General for Tax Administration reviews of modernization
project testing and deployment activities will include assessments of the
resolution of defect reports and the need for the IRS’ involvement in the
defect report closures.
Management
also indicated that three of its corrective actions were completed in July 2001. Since this was near the beginning of our
audit, and prior to the completion of the CC 2001 testing, we question whether
the completion dates provided are correct.
If these actions were in fact taken before we raised the issues, we must
conclude that the corrective actions were either incomplete or
ineffective. We are also troubled that
BSMO management did not provide us with documentation during our audit to
support the actions taken.
Copies of this
report are being sent to the IRS managers who are affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Scott E. Wilson, Assistant Inspector
General for Audit (Information Systems Programs), at (202) 622-8510.
The Customer
Communications Project 2001 Release Was Deployed to Serve Taxpayers More
Efficiently
Testing Processes
Did Not Ensure That All Capabilities Were Working As Intended
The Resolution of Problems Identified During Testing Was Not Clearly Documented
The Project Deployment Decision
Process Did Not Consider All Relevant Information
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix V – Analysis of System Requirements Testing and Test
Results
Appendix VI – Management’s Response to the Draft Report
The Customer Communications Project Fiscal Year 2001 Release (CC 2001) is intended to increase telephone and communication service levels to those of similar customer service operations in the private sector. The project’s expectations can be traced back to May 1997, when the Internal Revenue Service (IRS) issued its Modernization Blueprint to define, direct, and control investments in modernized systems and related infrastructure.
The IRS created the Business Systems Modernization Office (BSMO) to oversee the modernization efforts and contracted with experienced information technology companies to design and build the various modernization projects, including the CC 2001 project. The PRIME Alliance is a group of leading companies brought together by the Computer Sciences Corporation (CSC) to provide the IRS with access to commercial best practices, guarantee access to viable alternative solutions, and streamline the systems acquisition process. One of the processes that the CSC created for the IRS is the Enterprise Life Cycle (ELC), which guides the planning, design, development, and deployment of modernization projects.
The ELC establishes a set of repeatable processes and a system of reviews, checkpoints, and milestones that help ensure delivery of promised business results. The ELC provides specific guidelines in completing the testing phase of a project in preparation for deployment. To provide additional controls over project quality, the CSC and the BSMO developed a defect report process to resolve problems identified during testing. In addition, they developed a “go/no-go” procedure to control the decision to proceed through project deployment activities.
We conducted our audit from June through September 2001, at the BSMO facilities in New Carrollton, Maryland, and the IRS’ National Headquarters in Washington, D.C. The review was conducted in accordance with Government Auditing Standards, except that we encountered scope limitations by not having access to sufficient, competent, relevant, and timely information to afford a reasonable basis to draw conclusions and meet the audit objective. The absence of timely and complete information impaired our ability to fully assess whether intended project capabilities were provided. Detailed information on our audit objective, scope, and methodology is presented in Appendix I, and information related to the scope impairment is presented in Appendix IV. Major contributors to the report are listed in Appendix II.
The CC 2001 project became operational on August 2, 2001. The BSMO reported that the CC 2001 project has improved the IRS’ ability to receive, route, and respond to the more than 150 million taxpayer telephone calls it receives each year. The cost to design, develop, test, and deploy the CC 2001 project was approximately $65 million.
Major system improvements include designs to use:
· Voice-activated programs that recognize English- or Spanish-speaking callers.
· A voice-activated program that taxpayers can use to find out the status of their refunds.
· Capabilities that more accurately route taxpayer calls to the most appropriate IRS resource.
One of the CC 2001 features is the use of automated voice recognition to screen and route callers to a customer service representative or to an automated program. The planned system capabilities allow taxpayers with simple questions to get answers quickly, while those with issues that are more complex are guided to a customer service representative. The voice response system starts if callers do not make a selection using their telephone keypads. Callers then hear a series of options and make their selections by speaking into the telephone.
The BSMO and CSC did not adequately create, maintain, or provide documents to support the successful testing and deployment of all intended CC 2001 project capabilities.
The CC 2001 project’s pre-deployment plans included the development and execution of a series of rigorous testing processes. These plans called for application tests to ensure the individual components worked and met the business requirements, integration tests to ensure the components worked together as a whole, security tests to ensure the project was secure and met IRS security standards, and site readiness tests to ensure the project worked with existing IRS systems. The results of these various tests should have been documented and approved by the CSC and the BSMO, and copies of these documents should have been maintained by the BSMO to support that the project was working as intended when it was deployed.
We judgmentally selected a sample of 27 system requirements that we considered the most critical of the 116 total system requirements for the CC 2001 project. Our review of testing plans and results for these 27 system requirements showed that:
· Thirteen system requirements did not have evidence of any testing.
· Three system requirements passed some planned testing activities but did not undergo all planned testing.
· Eight system requirements passed planned testing activities.
· Two system requirements were deferred to later CC project releases.
· One system requirement was cancelled.
Appendix V presents an analysis of the system requirements included in our audit sample.
Without evidence that critical system requirements were successfully tested, the BSMO does not have assurance that the project will deliver all expected capabilities. The absence of testing activity documents was due, in part, to inadequate controls to trace and document system requirements to testing activities, and to maintain appropriate control over the various versions of the testing documents (known as configuration management).
Tracing and documenting system requirements to testing activities
The absence of support that testing activities addressed the CC 2001 project’s system requirements is attributable to shortcomings in tracing tests to specific requirements (traceability). The BSMO did not ensure that the CSC traced system requirements to the test cases used to control the testing processes. The CC 2001 project has over 100 individual system requirements, yet the CSC developed only 11 test cases to determine whether project capabilities met system requirements. Incomplete traceability of the system requirements to the test cases means that system requirements are difficult, if not impossible, to verify through the testing process.
The Customer Communications Project System Requirements Report states that traceability is one of the major activities of managing system requirements. Traceability provides the linkage between systems requirements and all phases of project development, and provides the ability to discover the history of each system feature. It links the lifecycle of a requirement both forward and backward, from origin to implementation. Traceability ensures that every requirement has been met because it is known which system components address each requirement.
To illustrate the difficulties we experienced in obtaining support that systems requirements were tested, the CSC stated that several system requirements could not be tested during the integration tests due to testing and system constraints. The Integration End of Test Report indicated that although the requirements could not be tested during integration, they were all thoroughly tested during the individual application tests. However, neither the BSMO nor CSC provided documents confirming these requirements were actually tested (see Appendix V for details).
Requirements traceability is an investment that increases the chances of delivering a system that satisfies all the stated customer requirements and is easier to maintain.
Maintaining appropriate
configuration management practices
The absence of support that testing activities addressed the CC 2001 project’s system requirements is also attributable to shortcomings in meeting configuration management guidance. Guidance for proper configuration management is contained in the ELC, but the CSC and BSMO had not fully implemented this guidance.
Configuration management is the process of keeping formal project documents, software code, and other key products safe from inadvertent changes. Changes to products and documents are expected, but need to be controlled. Changes are made to correct errors, provide enhancements, or simply reflect the evolutionary refinement of product definition. Proper configuration management processes keep the changes under control to eliminate the confusion and error brought about by the existence of different versions of the project’s products and documents. Configuration management also helps provide direct traceability between the projects developed by the CSC and the systems deployed to support IRS operations.
Examples of difficulties experienced from the inadequacies in configuration management included the following:
· The CSC cataloged system requirements using a number. Two systems requirement numbers were changed without any support for the change.
· Three system requirements were moved to a later release of the project without documentation indicating IRS management approval or control over the reassignment of the requirements.
Without a well-enforced configuration management process, project team members could unintentionally use different versions of products or documents, or individuals could create versions of products or documents without the proper authority. These changes could affect the validity of performance requirements and measurements, which could prevent the IRS from being assured that the systems will have the intended functionality.
To assure intended system capabilities are delivered in line with business requirements, the Deputy Commissioner for Modernization & Chief Information Officer should direct the BSMO to:
1. Ensure that
requirements management meets established ELC practices. Specifically, the BSMO should perform
reviews to ensure documentation is received from the CSC showing that project
system requirements are traced to use cases, test cases and test procedures.
Management’s Response: The IRS Product Assurance function will conduct reviews of requirements traceability, and document non-compliance for all test plans from the CSC.
Office of Audit Comment: Management’s response shows this action was completed in July
2001. We question whether that date is
accurate, since we raised this issue after that date and were not provided
information regarding corrective actions.
If this corrective action was taken before we identified the issue, then
we must conclude that the actions taken were either incomplete or ineffective.
2.
Assign
responsibility for configuration management inspections to monitor the adequacy
of the implementation and execution of the configuration management procedures.
Management’s Response: The BSMO’s
Office of Configuration Management was assigned responsibility to conduct
configuration management audits of IRS modernization projects. Also, the BSMO has contracted with the MITRE
Corporation to assist it in defining configuration management procedures.
Problems, also known as defects, may be found in software, hardware, documents, or other controlled products. Typically, defects are identified during testing or by the end user of a product. The CSC and the BSMO adopted procedures for identifying, reporting and resolving defects. The procedures provide that any person in the program may originate a defect report and submit it to the Defect Report Administrator, who reviews the defect report for completeness and assigns it a number. The information is supposed to be entered into a database (ClearQuest) that is used by the CSC to capture and manage the defect reports.
We identified 1,160 defect reports generated from the various testing phases. We found that defect report information was not always input to the ClearQuest database and the documentation supporting the resolution was not always maintained. Additionally, support was not always maintained showing BSMO approval for defect report closure or changes in defect severity ratings.
The ClearQuest database and Defect Report Record documentation were not always complete
The CSC’s draft ClearQuest
User’s Guide identifies the “Resolution,”
“Actions Taken to Resolve Defect,” and “Final Severity” as required information
for the ClearQuest database and
supporting records. The CSC did not always complete the required
fields in the database and/or document the solutions to defect reports.
·
The “Final Severity” field was not completed in 72 of
the 1,160 records on the database. This
field is important because each defect is given a severity rating to help set
the priority for resolutions (see next section for more details). Since the database is used to control the
identification and resolution of the defects, the absence of a severity rating
could allow critical defects to remain open while lesser severity defects are
being worked.
· The “Resolution” or the “Actions Taken to Resolve Defect” fields were not completed on 23 of the 34 sample defect report records we reviewed. Of the 11 records that did contain resolution information, 7 did not adequately describe how the action resolved the identified defect. The status of the defects and details on the actions taken to resolve defects are needed to ensure the actions were appropriate, and for reference in resolving future occurrences of the same or similar problems.
Support was not always maintained showing the IRS’ approval
for changes in defect severity ratings
Defect reports are given severity ratings that are used to determine the urgency in correcting the defects. There are four levels of priority to identify the severity of a defect report: Critical, High, Medium, and Low. Defect reports with a Critical or High severity are more serious and require immediate attention. The CSC and the BSMO agreed that Critical and High severity defects would be resolved before the CSC delivered the system to the IRS. They also agreed that Medium and Low severity defects would be resolved before the CSC completed its work on the project and the IRS accepted the system.
Of the 34 sampled defect reports identified above, 3 had the severity ratings changed on the
database without any indication of approval by the CSC or the BSMO. One was changed from Critical to High, one
from High to Medium, and one from Medium to High.
Significant defects that were not sufficiently resolved prior to deployment could cause problems during and after deployment that could negatively affect service to taxpayers.
Support was not always maintained showing the IRS’ approval for defect report closure
At the time of the project’s deployment phase, the CSC and the BSMO created a Defect Review Board to review and approve solutions to defect reports. The Defect Review Board is made up of CSC, BSMO, and other IRS personnel. Prior to this time, solutions to defects required the project management’s approval. Of the 1,160 defects included on the ClearQuest database, the CSC project management unilaterally closed 935 defects prior to deployment. The IRS Product Assurance Staff approved the closure of an additional 24 defects identified during the IRS’ systems acceptability testing. The remaining 201 defects had not been resolved at the time of project deployment.
The BSMO and Defect Review Board did not provide input for approving resolution of the 935 defects closed prior to deployment because the defect reporting process did not require the BSMO’s input for defect closure. The absence of the BSMO’s input in all defect closure decisions could allow defects to be closed without adequate resolution. The BSMO’s participation in the approval process would provide the IRS assurance that the system meets performance expectations.
For example, one of the defects reported closed by the CSC prior to deployment was referred to as the “Say 4” defect because the system was not recognizing a voice response of “four.” The CSC closed the defect as resolved on June 1, 2001. However, as of September 13, 2001, the IRS Product Assurance Staff was still awaiting documentation supporting adequate resolution of the defect from the CSC.
We found seven other defect reports with the same “Say 4” problem that the CSC reported and closed in the ClearQuest database. One of the defect reports for the “Say 4” problem included a notation of the resolution; however, the other six defect reports did not contain any resolution information. The first of these seven defect reports was generated during application testing in February 2001, and the last was generated during the deployment phase in June 2001. The last defect report contained the notation “Held open at Jun-26 DRB [Defect Review Board] per Business request,” yet the defect was closed the same day with no further notation.
The “Say 4” appeared as a continuing problem throughout the project’s testing. If the resolution fields had been completed, it may have assisted future developers in identifying a fix to the problem. However, based on the available ClearQuest database information, it is unclear what action was taken to resolve the issue. Since the problem continued, it is questionable if the defect was resolved.
In another example, a defect report was opened during deployment testing that indicated “Spanish calls do not route correctly.” There was no indication how, or if, this defect was actually resolved. If it was not resolved, Spanish-speaking callers may not be routed correctly and may have to call back repeatedly until they are routed to a Spanish-speaking assistor.
In a final example, a defect report was opened during integration testing that indicated “Multiple Refund Calls cannot get to CSRs (Customer Service Representatives) because the threshold in the required system is NOT allowing any calls to sit in the queue.” Only the first call would get through to a CSR and subsequent callers would receive a “Technical Difficulties” message. There was no indication how, or if, this defect was actually resolved. If it was not resolved, the system may be limited in the number of calls it can handle where a CSR’s assistance is required in providing refund information. The number of refund calls answered would only be the same as the number of assistors available at a given time.
Without adequate documentation of defect resolution actions, the CSC and the BSMO have no evidence to support whether or how a problem was corrected. Also, the solutions would not be available for the same or similar problems in the future. Taxpayers could become quite frustrated if the types of problems noted above were not resolved before the system was deployed.
To help ensure adequate control over defect reporting, resolution, and closure for future modernization projects, the Deputy Commissioner for Modernization & Chief Information Officer should direct the BSMO to ensure:
3. Details are developed for the procedures to
manage the defect identification, evaluation, reporting, and resolution
processes.
Management’s Response: The CSC developed procedures for the defect report process flow, including actions to address, close, and document a defect.
Office of Audit Comment: Management’s response shows this action was completed in July 2001, which was prior to the time we identified this issue. Management did not provide us with any documentation during our audit to support this corrective action.
4. Responsibility is assigned for ensuring
that the ClearQuest database includes
accurate and complete information to document identified defects, the defect
resolutions, and approval of closures.
Management’s Response: The CSC Defect Report Coordinator now has responsibility to ensure that all defect reports entered into the ClearQuest database contain complete and accurate information.
Office of Audit Comment: Management’s response shows this action was completed in July 2001, which was prior to the time we identified this issue. Management did not provide us with any documentation during our audit to support this corrective action.
5. Procedures
are developed for the IRS to review and approve resolution and closure of all
defect reports.
Management’s Response: Management
did not agree with this recommendation, citing that the high number of defect
reports generated during the project testing would make such reviews difficult.
Office of Audit Comment: While we agree
that the extraordinarily high number of testing defects on this one project
presented a challenge, we believe that management needs to have adequate
assurance that problems are either resolved or reduced to an acceptable level
on each project. Otherwise, glitches
could occur after projects are deployed, which could significantly impact IRS
operations and/or service to taxpayers.
Project deployment plans include instructions for performing a “go/no-go” review to assess the project’s ability to perform at full production levels. The criteria used in this review are contained in the Deployment Site Acceptance Review Plan. A Deployment Executive Board, made up of representatives from the CSC and the IRS, reviews the results of testing and recommends whether to proceed with deployment at the go/no-go review.
The Deployment Executive Board made a unanimous decision to go forward with the deployment of the CC 2001 project on July 26, 2001. However, we were not provided documentation showing any review by the Board to verify that the go/no-go criteria, detailed in the Deployment Site Acceptance Review Plan, was met.
The meeting minutes for the go/no-go review indicated that the Board relied on verbal assurance from CSC employees that the criteria for deployment were satisfied. When we requested evidence that the Board reviewed documentation to show that the criteria were met, the BSMO replied that they did not have this information and referred us to the CSC. The CSC responded that we could obtain the documents from the BSMO to show that the criteria were satisfied, not with evidence that the Board reviewed the documents. In addition, two of the documents that the CSC referred us to were deliverables that would not have been completed at the time of the go/no-go review.
We identified two additional issues that should have been considered in the go/no-go review process. Although the BSMO noted these issues, it did not require their resolution in making the go/no-go decision.
One issue involved the IRS’ executive committee responsible for overseeing the CC 2001 project. The Customer Relationship Management Sub-Executive Steering Committee approved the CC 2001 project’s deployment as long as five conditions were resolved in an expeditious and aggressive manner prior to full deployment. Two of these conditions were not satisfied prior to the Board’s go/no-go review on July 26, 2001, as follows:
· “Resolve two open systems acceptability testing defects and receive Product Assurance approval.” Neither of these defects was approved by Product Assurance prior to the go/no-go decision.
· “Resolve three Intelligent Call Management script discrepancies.” Two of the three discrepancies were not closed prior to the go/no-go decision, although both were resolved within the following week.
The second issue not considered in the go/no-go decision involved the CC 2001 project’s contingency plan. In a deployment meeting in April 2001, the Joint Operations Center management stated that it would not concur with or accept deployment of the CC 2001 release until the contingency plan was approved and implemented. There was no documentation available to show that the contingency plan was approved or implemented prior to the go/no-go decision.
The BSMO and the CSC did not develop a complete process to make the go/no-go decision. The Deployment Site Acceptance Review Plan did not include procedures for the Board to use as guidance as to what documentation it should review to ensure that the go/no-go criteria were met. Also, the process did not ensure that all issues identified during testing were included for review by the Board. Including these issues would allow the Board to make a more informed decision regarding the adequacy of the system for deployment.
Significant issues that were not sufficiently resolved prior to deployment could cause problems during and after deployment that could negatively affect service to taxpayers.
To help ensure that informed and appropriate go/no-go decisions are made on future modernization projects, the Deputy Commissioner for Modernization & Chief Information Officer should direct the BSMO to ensure:
6. Specific procedures are developed for use by
the Deployment Executive Board that include:
· Identifying issues or conditions that are raised by the various groups involved in the project deployment activities for inclusion as a criteria to be met and reviewed by the Deployment Executive Board prior to making the go/no-go decision.
Management’s Response: The CSC is
drafting procedures which detail specific review items and pass/fail criteria
that project teams will follow in all deployment activities. The procedures will also include mandatory
signoff from all review parties to concur that criteria have been met prior to
initiating deployment.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this audit was to determine whether the Customer
Communications Project Fiscal Year 2001 Release (CC 2001) deployment readiness
activities followed testing and pre-deployment plans to ensure intended
capabilities were delivered. We
performed this audit as part of our planned audit coverage of the Internal
Revenue Service’s (IRS) modernization efforts.
Although we followed Government Auditing Standards, the auditors encountered scope limitations by not having access to sufficient, competent, relevant, and timely information to afford a reasonable basis to draw conclusions and meet the audit objective. The absence of timely and complete information did not allow the auditors to fully assess whether intended project capabilities were provided. Appendix IV presents a list of the information requested by the Treasury Inspector General for Tax Administration auditors but not provided for their review.
We performed the following reviews and analyses.
I. Analyzed deployment readiness by reviewing the various testing activities to determine whether the Enterprise Life Cycle (ELC) guidelines, industry standards, and the Internal Revenue Manual were addressed.
A. To determine whether the Application Qualification Test (AQT) plan and its execution met guidelines and standards, we:
1. Reviewed ELC requirements for the AQT.
2. Compared the AQT plan to ELC requirements to identify any significant discrepancies, unusual constraints, or scope impairments.
3. Reviewed the AQT test results, completed April 3, 2001, to determine if all tests were conducted, results analyzed, and defects adequately resolved.
4. Determined whether the entire completion criteria for AQT were satisfied, including approval by the IRS to permit transition into System Integration Testing (SIT).
B. To determine whether the SIT plan and its execution met guidelines and standards, we:
1. Reviewed ELC and Software Engineering Institute requirements for the SIT.
2. Compared the SIT plan to ELC requirements to identify any significant discrepancies, unusual constraints, or scope impairments.
3. Reviewed the SIT test results, completed May 31, 2001, to determine if all tests were conducted, results analyzed, and defects adequately resolved.
4. Determined whether the entire completion criteria for SIT were satisfied, including approval by the IRS to permit transition into Deployment Site Readiness.
C. To determine whether the System Acceptability Test (SAT) plan and its execution met guidelines and standards, we:
1. Reviewed SAT requirements and standards from the Internal Revenue Manual and industry standards from the Software Engineering Institute.
2. Compared the SAT plan to SAT requirements to identify any significant discrepancies, unusual constraints, or scope impairments.
3. Reviewed the SAT test results, completed May 21, 2001, to determine if all tests were conducted, results analyzed, and defects adequately resolved.
4. Determined whether the entire completion criteria for SAT testing were satisfied, including approval by the IRS to permit transition into Deployment Site Readiness.
D. To determine whether the Deployment Site Acceptance Review Plan/Deployment Site Readiness Test Plan met guidelines and standards, we:
1. Reviewed Deployment Site Acceptance Review Plan/Deployment Site Readiness Test Plan requirements and standards from the Internal Revenue Manual and procedure documents, as well as industry standards from the Software Engineering Institute.
2. Compared the Deployment Site Acceptance Review Plan/Deployment Site Readiness Test Plan to requirements to identify any significant discrepancies, unusual constraints, or scope impairments.
Note: We were not provided Deployment Site Acceptance Review/Deployment
Site Readiness Test results to be able to assess the adequacy of the review and
test plan execution. Without this
information, our ability to determine the adequacy of the deployment of
intended CC 2001 project capabilities was impaired.
E. To determine if the Security Test and Evaluation (ST&E) Plan and its execution met guidelines and standards, we:
1. Compared the ST&E Plan to guidelines and standards to determine whether the plan adequately followed the criteria necessary to effectively test and evaluate the security controls. The security criteria included the ELC certification and accreditation process requirements and the Information Systems Security Procedural Guide (Document 9627).
2. Compared the ST&E Plan with the Customer Communications Risk Assessment Report to determine whether the plan adequately tested those risks warranting attention.
3. Interviewed the Security Oversight and Management staff to assess their responsibilities in the preparation and execution of the ST&E Plan.
4. Reviewed the ST&E test results to determine if the security was adequate to ensure the CC 2001 project met the requirements for security accreditation.
II. Analyzed the support available from deployment testing activities that demonstrated the project met planned system requirements. To accomplish this, we:
A. Judgmentally selected a sample of 27 CC 2001 project system requirements from the Systems Requirement Report dated May 8, 2000. We considered these 27 system requirements as the most critical of the Systems Requirement Report’s 116 total system requirements. These system requirements were selected to emphasize auditors’ conclusions.
B. Requested documentation and artifacts from the PRIME and the IRS presenting the test activities and results (AQT, SIT, SAT, and Deployment Site Readiness) for the 27 system requirements in our sample. The sample was subsequently reduced to 24 requirements because 2 of the requirements had been deferred to later CC releases and 1 was cancelled. The IRS approved the elimination of these three requirements from the CC 2001 release.
III. Reviewed the process to identify, control, report, and resolve project defects in all testing phases. Our reviews determined if the defects were adequately reported, tracked, and resolved, when necessary, prior to proceeding to the next phase of testing and prior to project deployment. To assess the adequacy of controls to manage the reporting and resolution of defects we:
A. Reviewed the CSC defect report procedure.
B. Obtained an extract from the CSC’s ClearQuest database of defect report information accumulated between January 25, 2001, and August 1, 2001. The 1,160 defect reports on this database were generated during AQT, SIT, SAT, and Deployment Site Readiness.
C. Reviewed the CSC’s draft ClearQuest User’s Guide to determine the method used to manage reported defects on the database.
D. Reviewed a judgmental sample of 34 defect report records from the population of 1,160 defect reports to determine the accuracy and completeness of the defect report records, the adequacy of the defect resolutions, and whether the defect resolution was approved. The defect reports were selected to find examples to emphasize auditors’ conclusions. The sample consisted of 11 AQT defect reports that were not closed prior to the end of AQT testing, 11 defect reports that related to the “Say 4” SAT issue, 9 defect reports that the auditors interpreted as significant based on the defect report titles, and 3 defect reports that did not include a severity classification.
IV. Analyzed the decision process, referred to as “go/no-go”, used by the CSC and the IRS managers to proceed with deployment of the CC 2001 project. Two go/no-go decisions were made during the CC 2001 project deployment -- the Go/No-Go Review and the Deployment Site Acceptance Review. To assess the adequacy of controls to manage and execute this process, we:
A. Reviewed criteria in the Deployment Site Acceptance Review Plan used by the Deployment Executive Board to assess the adequacy of project development to proceed with its deployment and active operation.
B. Reviewed meeting minutes in which management reviewed deployment activities to identify issues for consideration in the go/no-go decision by the Deployment Executive Board.
C. Analyzed the status of defect resolutions to determine whether any significant defects remained unresolved after a “Go” decision by the Deployment Executive Board.
D. Analyzed the documentation used by the Deployment Executive Board to make the go/no-go decision.
Appendix II
Major Contributors to This Report
Scott E. Wilson, Assistant Inspector General
for Audit (Information Systems Programs)
Scott A. Macfarlane, Director
Edward A. Neuwirth, Audit Manager
Michael Garcia, Senior Auditor
Michelle Griffin, Senior Auditor
Beverly Tamanaha, Senior Auditor
Esther Wilson, Senior Auditor
Louis V. Zullo, Senior Auditor
Suzanne Noland, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Deputy Associate Commissioner, Program
Management M:B
Deputy Associate Commissioner, Systems
Integration M:B
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaison:
Associate Commissioner, Business Systems Modernization M:B
Appendix IV
Information
Requested and Not Provided to Treasury Inspector General for Tax Administration
Auditors
Date Requested
|
Information
Not provided
|
|---|---|
|
Documentation of resolution of the Application
Qualification Testing (AQT) team-generated list of issues directly related to
the use cases (these issues were referred to in the AQT End of Test Report,
dated April 13, 2001, page 11, “Constraints” section, last bullet). |
|
|
August 9, 2001 |
Documentation showing that the contingency plan was
approved and implemented (Change Request #22). |
|
August 9, 2001 |
Documentation showing support for completion of
Milestone 4 exit conditions - Action Item #2 – Business and Product Assurance
sign-off on one of two defect reports requested. |
|
August 16, 2001 |
Complete responses from the Computer Sciences
Corporation (CSC) to auditor questions regarding additional support
documenting actual system requirements capabilities. |
|
August 16, 2001 |
Complete responses from the CSC about auditor
questions regarding defect reports, Go/No-Go decision processes and
documentation, Milestone 4 exit, and AQT constraints. |
|
August 23, 2001 |
Copies of Configuration Inspection AQT test results
procedures identified in Scenario #1 and Scenario #2 and Verification Cases
0004 through 0024 as discussed in the Customer Communication Project’s System
Verification and Validation Plan. |
|
August 29, 2001 |
Complete documentation for sampled defect reports. |
|
August 29, 2001 |
Complete answers to questions about the CSC’s defect
report summary. |
|
September 5, 2001 |
AQT sub-phase Process Flow and Work Product Dependency
Diagrams. |
|
September 5, 2001 |
Internal Revenue Service approval for requirements
moved to a later release for requirements 40.0370 and 40.0260. |
|
September 5, 2001 |
Documentation and clarification for test procedure
13.20.01.19. |
|
September 5, 2001 |
System Integration Testing documentation supporting
testing of 24 requirements in the Security Testing & Evaluation test
objectives. |
Appendix V
Analysis of System Requirements Testing
System
Requirement
|
Planned Testing* |
Test Support
|
|---|---|---|
|
40.0020: The Modernized System shall accept
taxpayers’ input from touch-tone (keypad) telephones. |
AQT SIT |
Sufficient
Support |
|
40.0031: The
Modernized System shall accept taxpayers’ input from callers using rotary
dial telephones for Front-end Screening and for Refund Status Inquirers. |
AQT SIT SAT |
Sufficient
Support |
|
40.0160: The Modernized System shall provide the capability to play ad
hoc broadcast messages to callers. |
AQT SIT SAT |
Sufficient
Support |
|
40.0260: The Modernized System shall provide the capability for an
assistor to generate a request to get a hostile caller’s Automated Number
Identification (ANI) during the telephone conversation. |
None – requirement moved to a later release without management approval. |
No
Evidence of Testing |
|
40.0310: The Modernized System shall provide the
capability to deliver and concurrently display call transfer data when the
call is delivered to an assistor. |
AQT SIT SAT |
Sufficient
Support |
|
40.0370: The Modernized System shall provide the
capability to assign and maintain priority call processing by service as
determined by business rules. |
None – requirement moved to a later release without management approval. |
No
Evidence of Testing |
|
40.0420: The Modernized System shall provide the
capability to monitor calls locally and remotely by type of service or
employee. |
Deployment |
No
Evidence of Testing |
|
40.0460: The Modernized System shall select a next
action to take if a caller makes a designated number of erroneous responses
based on business rules. |
AQT SIT SAT |
Sufficient
Support |
|
40.0490: The
Modernized System shall provide the capability to schedule and administer
closures or outages at call sites. |
Deployment |
No Evidence of
Testing |
|
40.0590: The Modernized System shall provide a call
routing and rerouting capability to and from the following resources and
services for incoming and outgoing calls: -
A
designated legacy system application. -
The
designated site for the required service. -
A
skill group with a designated skill. -
The
designated specialty service. -
The
Public Telephone Network interface. |
AQT SIT SAT Deployment |
AQT: Support only partially
satisfied requirement SIT: Sufficient Support SAT: Sufficient Support Deployment: No Evidence of Testing |
|
40.0610: The Modernized System shall provide the
capability to route and reroute calls one or more times across and within all
product lines and services: -
Callers
selecting a service from a menu or script. -
Callers
transferred from one assistor to another. -
Callers transferred from an assistor to a legacy
Telephone Routing Interactive System (TRIS) application. |
AQT SIT SAT Deployment |
AQT: Sufficient Support SIT: Sufficient Support SAT: Sufficient Support Deployment: No Evidence of Testing |
|
40.0641: The Modernized System shall provide the capability
to route calls using the following business rules: -
Caller
input/request/topic. -
ANI. -
Traffic load
balancing. -
Location/network
capacity. -
Location availability. -
Location specialty
including Business Operating
Division. -
Location use. -
Application
availability. -
Assistor availability. -
Assistor
skills/qualifications. -
Minimum expected
delay. -
Dialed Number
Identification Service. |
AQT SIT SAT Deployment |
AQT: Support only partially
satisfied requirement SIT: Support only partially satisfied
requirement SAT: Support only partially
satisfied requirement Deployment: No Evidence of Testing |
|
40.0671: The Modernized System
shall provide the capability to manage, route, and/or reroute telephone
traffic throughout the modernized Internal Revenue Service system-wide
telecommunication network to balance traffic across sites and within each
service in accordance with business rules. |
AQT Deployment |
No
Evidence of Testing |
|
40.0730: The Modernized System shall provide the
capability to route a Spanish-speaking caller to a Spanish-speaking assistor. |
SIT SAT |
No
Evidence of Testing |
|
40.0780: The
Modernized System shall adjust call routing so that the number of calls
connected to a resource will decrease to approximately zero by the scheduled
time of non-availability. |
AQT Deployment |
No Evidence of
Testing |
|
40.0810: The Modernized System shall receive calls
and data from the Public Telephone Network needed to process the call. |
AQT SIT SAT |
Sufficient
Support |
|
40.0860: The Modernized System shall provide the
capacity to service 165.5 million nationwide incoming call attempts at the
specified level of access during a fiscal year. |
AQT Deployment |
No Evidence of
Testing |
|
40.0860.01.03:
The Modernized System shall provide the capacity to service 16 million
Voice Refund Status Inquiry calls during a fiscal year, with an average call
duration of 4 minutes and with the caller selecting 1 inquiry per call. |
AQT |
No
Evidence of Testing |
|
50.0010: The interface from the front-end screening
Modernized System application via the Intelligent Call Manager (i.e., Geotel)
and via the Automated Call Distributor to the Legacy TRIS System shall
transfer data that include the following information: -
Language
Types (e.g., English, Spanish). -
Media
Types (e.g., Touch Tone, Rotary [Voice]). -
Applications Types
(e.g., Non-Account services, Accounts services, etc.). |
SIT |
No
Evidence of Testing |
|
50.0050: The interface from the Public Telephone
Network to the Modernized System shall transfer taxpayer menu selections. |
AQT SIT SAT |
Sufficient
Support |
|
50.0090: The
interface from the Modernized System to the Public Telephone Network shall
transfer Refund and Fact-of-Filing data input prompts. |
AQT SIT SAT |
Sufficient
Support |
|
50.0110: The interface from the Modernized System
to the Public Telephone Network shall transfer rerouting requests. |
AQT SIT SAT Deployment |
No Evidence of
Testing |
|
50.0110.01:
The interface from the Modernized System to the Public Telephone
Network shall transfer rerouting requests with a projected average frequency
of 81,000 times per hour. |
AQT |
No
Evidence of Testing |
|
390.0250: The Modernized System
shall include the following standard reports types: -
Americans
with Disabilities Act devices and Spanish users. -
Telephone
Infrastructure Report. -
Weekly Internet
Post-Filing Web Page Report. |
Deployment |
No
Evidence of Testing |
*
AQT = Application Qualification Testing
SIT =
System Integration Testing
SAT
= System Acceptability Testing
Deployment = Deployment Site Readiness
Appendix VI
Management’s Response to the Draft
Report
The response was removed due to its
size. To see the complete response, please
go to the Adobe PDF version of the report on the TIGTA Public Web Page.