Testing Practices for Business Systems Modernization
Projects Need Improvement
September 2003
Reference Number: 2003-20-178
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
September
23, 2003
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Assistant Inspector General
for Audit (Small Business and
Corporate Programs)
SUBJECT: Final Audit Report – Testing Practices
for Business Systems Modernization Projects Need Improvement (Audit #
200320039)
This
report presents issues and trends in
Business Systems Modernization (BSM) testing processes that need management
attention. We summarized results from
audits we recently conducted on five BSM projects, and we have highlighted our
continuing concerns with the BSM testing processes on three audits currently in
process.
As the BSM projects progress
through development and deployment, they undergo various testing processes to
ensure they meet performance specifications and can be effectively used in
their intended operational environment.
These testing processes are a key management control to ensure that
Internal Revenue Service (IRS) executives have valid, credible information upon
which to base their decisions for modernization project investments.
In summary, the BSM Office (BSMO) and the PRIME
contractor have made significant progress in establishing testing processes and
practices. Testing processes have been substantially
revised and refined based on lessons learned during the early testing efforts
for BSM projects. However, while improvements have been made in developing and defining
acceptable testing processes and practices, we have not seen the same level of
improvement with implementing the enhancements. Additional controls and further improvements are needed to ensure
that the BSM project teams incorporate and follow the defined testing processes
and practices.
In
each of the eight BSM projects we included in our analysis, we identified
concerns with the project testing practices followed by the BSMO and the PRIME
contractor. While not all of the individual testing concerns we
identified on the BSM projects are of the same significance or criticality, we
are concerned about the combined effect that these testing practices have on
the overall BSM program. These concerns
include: insufficient
test plans, incomplete testing activities, inadequate actions to resolve failed
tests, and incomplete testing results documentation.
We
believe the inadequate testing practices are the result of the BSM project
teams attempting to meet overly
optimistic project schedules. The
demands of trying to meet the project schedules have pressured the BSMO and the
PRIME contractor to reduce the emphasis placed on following defined and
approved testing processes. Several
factors contribute to the schedule pressures, such as budget constraints,
inadequate resources, and changing business requirements. While the primary intent of these testing
practice decisions was a legitimate desire to speed delivery of the project
capabilities to the IRS business functions, the desired impact has not been
achieved. All of the BSM projects we
reviewed were either delivered late, or are significantly behind their planned
delivery dates.
We realize the project
releases we reviewed are considered less risky and that the testing weaknesses
have not led directly to major known problems in the projects that have been
deployed. However, the BSMO and the
PRIME contractor should be using these less risky project releases to implement
and refine the testing processes so that these processes are well established
and integrated when the more important and significant project releases are
developed and tested. The IRS hired the
PRIME contractor to develop and implement well-established processes and best
practices, and to institutionalize the discipline of following these practices
on all BSM projects. Reducing or
eliminating some testing processes in the early stages of the BSM program does
not allow the defined and desired processes to be institutionalized into the
project teams. While the BSMO and the
PRIME contractor have deployed several BSM projects, the successes have been
more the result of the dedication and heroics of the project team members,
rather than the discipline of following established and effective
processes.
Implementing and following
defined testing processes will reduce the possibility of undetected errors,
allow identified errors to be fixed before project deployment when the fixes
are easier to make and less expensive, and increase the extent to which a
system can be relied on to provide accurate information and safeguard taxpayer
data once it is deployed. Testing
processes are part of the overall controls to help ensure that systems perform
their intended functions accurately and reliably. Focusing on the discipline of following and improving defined testing
processes is equally important since it will improve the maturity of the
project teams and will significantly improve the chances of BSM program
success.
Management’s
Response:
The Chief Information Officer (CIO) agreed with most of the comments and observations
in our report. The CIO responded that,
while significant progress has been made, testing practices have not been
uniformly implemented for all projects, and further improvement opportunities
remain. However, the CIO stated the
report does not provide a clear or correct impression of the current state of
testing activities. This disagreement
involves a difference in opinion on our observations that inadequate testing
practices are the result of attempts to meet overly optimistic project schedules,
and that the number of waivers and deferred tests is an indication of
incomplete testing activities.
Management’s complete response to the draft report is included as
Appendix V.
Office of Audit Comment: The CIO stated that he strongly disagreed with
the central thesis of the report, which he related as “inadequate testing
practices are the result of the BSM project teams attempting to meet overly
optimistic project schedules.” The
actual central thesis of our report is that testing practices for BSM projects
need improvements, which the CIO agreed with in his response.
We agree with the CIO’s
statement that testing practices should represent an appropriate balance of
technical, schedule, cost and business risks.
However, on every BSM project we reviewed, we found that some testing
activities detailed in the project test plans and defined in testing guidance
documents were reduced or eliminated, which significantly increases the
business risk that systems may not perform as intended. Based on our audit work, discussions with
BSMO and PRIME contractor personnel, and observations of project activities, we
believe the testing practices were reduced or eliminated by the project teams
in an attempt to meet specific project delivery dates.
The CIO also stated “although
the number of waived or deferred test cases may appear to be troublesome after
a superficial review, the basis of these waived and deferred test cases does
not support the assertion that testing activities are incomplete.” The CIO further stated that waivers and
deferrals usually occur because test environments cannot support specific test
cases, or requirements are modified or removed.
We agree that there are valid
reasons to waive or defer test cases, but our concern with the CIO’s statement
is that if it was known that the test environment would not support the test
cases, or that the requirements being tested were modified or removed, why were
the test cases still included in the test plans? If these issues came about after the test plans were prepared and
approved, then the plans should have been revised to reflect the actual test
conditions to be performed. The fact
that these waivers and deferrals were approved does not override the concern that
incomplete testing can lead to problems when the systems are finally deployed.
We respectfully disagree
with the CIO’s statement that this report does not provide a correct impression
of the current state of BSM testing activities. While the conditions we include in the report were identified in
past audits, we also saw many of the same conditions in current projects. While the CIO may disagree with some of our
comments and perspectives, we believe the information in this report is valid
and worthwhile based on the comprehensive audit work performed.
We commend the BSMO on the improvements that have been made to the testing processes and practices, but additional improvements need to be made to ensure that the testing activities provide the necessary confidence that systems perform their intended functions accurately and reliably. Our report points out the testing practices that require consideration to aid the efficient and effective delivery of modernized information and business systems.
Copies of this report are also being sent to the IRS managers who are affected by the comments and observations contained in the report. Please contact me at (202) 622-6510 if you have questions, or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.
While Testing
Processes Have Been Improved, Project Teams Have Not Always Followed Defined
Processes
Insufficient Testing Plans and Facilities Have Contributed to Project Delays
Testing Activities Have Not Been Completed Before
Putting Systems Into Operation
Failed Tests Are Not Being Properly Resolved During Project Development
Security Test Documentation Was Not Completed Before Systems Were Placed Into Operation
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix
IV – Description of Business Systems Modernization Projects Reviewed
Appendix V – Management’s Response to the Draft Report
The Internal Revenue Service (IRS) expects to spend more than $7 billion over a 15-year period to upgrade and modernize its information technology and business systems. Testing of new hardware and software is often the last opportunity for IRS executives and project managers to ensure that Business Systems Modernization (BSM) projects meet requirements and expectations before they become operational.
As the BSM projects progress through development and deployment, they undergo various testing processes to ensure they meet functional and performance specifications and can be effectively used in their intended operational environment. These testing processes are a key management control for ensuring that IRS executives have valid credible information upon which to base their decisions for modernization project investments.
The testing process is designed to detect errors in both software and hardware before a system is made operational. The BSM Office (BSMO) and the PRIME contractor are primarily responsible for ensuring that BSM projects have been adequately tested and that the projects perform as expected. The BSMO and the PRIME contractor developed a systems development methodology called the Enterprise Life Cycle (ELC) that provides guidance and detailed processes to be followed by project teams working on BSM projects. Included in the ELC are detailed processes for designing, developing, and testing BSM projects.
The information presented in this report is derived from audits we recently conducted on five BSM projects, as well as three audits currently in process, where we identified concerns with testing processes. We also reviewed General Accounting Office (GAO) audit reports related to the BSM program. These audits were conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II. Appendix IV presents details of the audits used in this analysis and the status of the corrective actions implemented by the BSMO.
Because the testing concerns related to individual projects and the corresponding recommendations for corrective actions have been included in previous audit reports, or will be included in upcoming reports on the three projects currently under review, we are not providing recommendations in this report. However, since we are still identifying testing concerns in our current BSM audits, we believe it is important to present the testing concerns in a program-wide perspective. By elevating the individual project issues into a program-wide concern, we expect the BSMO and the PRIME contractor to accelerate corrective actions to fully address the testing concerns identified.
The BSMO and the PRIME contractor have made significant progress in
establishing testing processes and practices.
Testing processes have been substantially revised and refined based on
lessons learned during the early testing efforts for BSM projects. However, while improvements have been made in
developing and defining acceptable testing processes, we have not seen the same
level of improvement with implementing the enhancements. Additional controls and further improvements are needed to ensure that
the BSM project teams incorporate and follow the defined testing processes and
practices.
In each of our eight BSM project audits included in our analysis, we identified concerns with the project testing practices followed by the BSMO and the PRIME contractor. While not all of the individual testing concerns we identified on the BSM projects are of the same significance or criticality, we are concerned about the combined effect that these testing practices have on the overall BSM program. The testing concerns we identified include: insufficient test plans, incomplete testing activities, inadequate actions to resolve failed tests, and incomplete testing results documentation.
We believe the inadequate testing practices are the result of the BSM project teams attempting to meet overly optimistic project schedules. The demands of trying to meet the project schedules have pressured the BSMO and the PRIME contractor to reduce the emphasis placed on following defined and approved testing processes. Several factors contribute to the schedule pressures, such as budget constraints, inadequate resources, and changing business requirements. While the primary intent of these testing practice decisions was a legitimate desire to speed delivery of the project capabilities to the IRS business functions, the desired impact has not been achieved. All of the BSM projects we reviewed were either delivered late, or are significantly behind their planned delivery dates.
We realize the project releases we reviewed are considered less risky, and that the testing weaknesses have not led to major known problems in the projects that have been deployed. However, the BSMO and the PRIME contractor should be using these less risky project releases to implement and refine the testing processes outlined in the ELC, so that these processes are well established and integrated when the more important and significant project releases are developed and tested.
The IRS hired the PRIME contractor to develop and implement well-established processes and best practices, and to institutionalize the discipline of following these practices on all BSM projects. Reducing or eliminating some testing processes in the early stages of the BSM program does not allow the defined and desired processes to be institutionalized into the project teams. While the BSMO and the PRIME contractor have deployed several BSM projects, the successes have been more the result of the dedication and heroics of the project team members, rather than the discipline of following established and effective processes.
Testing processes are part of the overall controls to help ensure that systems perform their intended functions accurately and reliably. Implementing and following defined testing processes reduce the possibility of undetected errors and increase the extent to which a system can be relied upon to provide accurate information and safeguard taxpayer data once it is deployed. Additionally, following detailed testing processes increases the chances of detecting errors earlier in the development process when they are easier to fix, which can greatly reduce the cost and time needed to resolve the errors. The National Institute of Standards and Technology estimated that, relative to resolving an error during the design phase of a project, it is 90 times more expensive to resolve during the system testing phase, but up to 880 times more expensive to resolve after the system is operational.
Focusing on the discipline of following and improving defined testing processes is equally important since it will improve the skills and maturity of the BSM project teams. The BSMO and the PRIME contractor are now developing several projects, such as the Customer Account Data Engine (CADE) and e-Services, which are very critical to the BSM program. Improving the discipline of the project teams to follow defined and established processes will significantly improve the chances of the BSM program being successful.
Our concerns regarding testing practices are not new. For example, in July 1995, the GAO expressed similar concerns with the IRS’ Tax Systems Modernization Program. The GAO reported that “systems integration is incomplete,” and “system testing and test planning are inadequate.” The report continued, “…until IRS completes its testing plans, implements effective testing processes, and establishes its Integration Test and Control Facility, it has little assurance that systems will be adequately and effectively tested.”
An effective test
plan describes the overall testing process, the test verification approach and
the test acceptance criteria. A test
plan should be developed for each testing phase (i.e., integration testing,
acceptance testing, and security testing), and cover the test environment
including all required equipment needed to perform the testing and the
availability of the test facilities.
The IRS has built a test lab to provide the ability to test system and
project capabilities in a simulated live environment.
Comprehensive testing guidance was not always provided
We reported that the CADE Release 1 project did not include
sufficient planning activities. The
PRIME contractor initiated a pilot of the CADE project to help ensure the
success of the Release 1 deployment; however, the pilot plan did not provide
comprehensive testing guidance.
Specifically, it did not include all necessary details or procedures to properly conduct
performance testing. For example, the
test case scenarios did not have complete step-by-step documentation to
run each
test, criteria for measuring test success, and test descriptions.
Insufficient guidance caused pilot team members to have difficulty in executing pilot tests, monitoring test activities, and assessing the adequacy of the tests. The CADE pilot began in July 2002 and was scheduled for completion in October 2002. However, the pilot is still ongoing and its estimated completion date is not until early 2004, about the same time Release 1 is scheduled for deployment.
The capacity of the test lab is not adequate to support the BSM program
Limitations in the test environment have resulted in project deployment
delays. Because two BSM projects, the e-Services
and the Internet Refund Fact of Filing (IRFOF), were developed with different
versions of supporting software, and the test lab did not have sufficient
equipment to provide separate environments for each software version, testing
of the projects had to occur at different times of day. A painstaking process had to be developed to
switch from one test lab configuration to another, and around-the-clock testing
had to be conducted in the test lab to support both projects.
Delays occurred, particularly in the testing of the e-Services project, because the IRFOF project remained in the lab to conduct performance testing much longer than planned. As a result, the e-Services project team could test only part of each day, rather than having full control of the testing lab.
Management Actions: The BSMO has recently taken several actions to improve the test lab environment, such as substantial procurement activities to increase the capacity of the lab and increased detail in planning and tracking lab utilization requirements.
Software and hardware testing ensures that a system meets functional and performance specifications and can be effectively used in its intended operational environment. It is often the last opportunity to make sure projects meet requirements and expectations before they become operational. Additionally, the testing process is a key management control for ensuring that IRS executives have valid, credible information upon which to base their decisions for project deployments.
Testing did not encompass all project capabilities
In March 2002, we reported that the PRIME contractor did
not sufficiently test all Customer Communications 2001 Release (CC 2001)
capabilities to ensure they were working as intended. Of the 27 system requirements that we reviewed, 13 did not have
any testing, and 3 had only partial testing.
This occurred because neither the PRIME contractor nor the BSMO implemented adequate controls to ensure that testing of significant project requirements was documented and approved. Also, the project deployment decision process did not verify that all deployment criteria were met, and that all problems were resolved before the project was put into operation.
After the 2002 Filing Season, the IRS decided to disable a portion of the CC 2001 automated call routing application because taxpayers found the telephone access menu too difficult to navigate. For the 2003 Filing Season, taxpayers with tax law questions were directed to a Customer Service Representative, who screened the calls and forwarded them to the appropriate tax law specialist. Approximately 150 additional employees were hired for the 2003 Filing Season to perform this and other filing season related tasks. A certain amount of voice script and menu tuning is expected after a system like CC 2001 is deployed. However, complete testing of all system requirements may have detected the scripts and menus concerns earlier and allowed the IRS to make adjustments before deployment, thus lessening the impact on personnel and taxpayers.
Hardware upgrades were not adequately tested
The PRIME contractor and the BSMO deployed a pilot of the IRFOF application in May 2002 with the knowledge that performance response time and user capacity needed improvement to meet contract requirements. For the 2003 Filing Season, the PRIME contractor provided upgrades to the web-server hardware to try and improve performance to meet the project requirements. However, IRFOF performance with the new hardware was not tested prior to deployment because of schedule demands and requests from the IRS to deploy the application.
Planned performance tests were not completed
We reviewed the testing performed on the Security and Technology Infrastructure Release (STIR) project and its support for the IRFOF project to determine the extent and results of performance testing. While reviewing the documentation on the STIR project testing, we found that although performance tests were planned, all of these tests were either waived or deferred.
The IRS agreed to waive or defer the performance tests because it believed the performance issues could be addressed prior to the peak tax refund issuance period. The STIR infrastructure is a critical component for the BSM projects, and deploying the first release without fully testing it is a risky practice that can affect both the IRFOF application and future projects that will use the STIR.
We currently have an audit of the e-Services Release 1 project in progress. While we have not completed our audit work or reported our results to the BSMO, our preliminary survey results indicate that a significant number of integration tests were waived during the project development phase. We also identified a significant number of CADE Release 1 tests that were waived or deferred during project development.
Some tests are performed concurrently instead of sequentially as planned
Integration testing ensures that system components are working as expected, and acceptance testing determines whether a system meets user needs. Because of this relationship, integration testing is performed before acceptance testing. However, to meet time and schedule restrictions, the PRIME contractor and the BSMO are performing these tests at the same time instead of one after the other.
We identified a trend in several projects where the BSMO is allowing acceptance testing to start before the completion of the integration testing. The STIR and the CADE Release 1 employed this testing practice, and we have also seen indications of this practice in our preliminary audit work involving the Custodial Accounting Project and the Integrated Financial System (IFS) project.
When tests that should be performed sequentially are run concurrently, such as integration and acceptance testing, the IRS runs the risks of incurring additional costs and schedule delays due to the need to re-perform some tests. For example, changes made to a system to address a problem identified in integration testing may affect the validity of previously conducted acceptance tests, so those tests may need to be performed again to ensure system changes did not impact acceptance criteria. If previously conducted tests are not re-performed, the IRS may risk accepting a system that does not work as intended or meet all contractual requirements.
System components that fail a test are known as defects. Defects are given a Severity Rating to denote the significance of the defect, with Level 1 indicating a problem that is critical to the system and Level 4 being a cosmetic or other problem that does not impact the performance of the system. The PRIME contractor and the BSMO developed procedures for identifying, reporting, and resolving test defects. The defect information is compiled into a report and entered into a database that is used by the PRIME contractor to capture and manage the resolution of the defects.
Resolution of defect reports for failed test cases was not adequately documented
We reported that the defect report information during the CC 2001 project testing was not always input to the defect database, and the documentation supporting the resolution was not always maintained. Additionally, support was not always maintained showing BSMO approval for defect report closure or changes in defect severity ratings.
The “Resolution” or the “Actions Taken to Resolve Defect” fields were not completed on 23 of the 34 sample defect report records we reviewed. Of the 11 records that contained resolution information, 7 did not adequately describe how the action resolved the identified defect. The status of the defects and details on the actions taken to resolve defects are needed to ensure the actions were appropriate, and for reference in resolving future occurrences of the same or similar problems.
Guidance was not sufficient to close defect reports for failed tests
The CADE pilot plan includes a “Problem Management” section that provides guidance to capture and track defect reports about pilot activities through the PRIME contractor’s and the IRS’ problem reporting databases. However, it does not provide sufficient direction to close defect reports and reconcile reported defects.
For example, the pilot plan does not include adequate defect report closure and reconciliation procedures to ensure accuracy of both the PRIME contractor’s and the IRS’ defect report databases. Defect report resolution and closure procedures do not include the IRS’ approval, and controls do not limit the PRIME contractor defect report administrator’s access in the IRS’ defect report database. Without adequate controls to manage the reporting and resolution of defects, the IRS does not have assurance that defects are properly resolved.
Additionally,
during our survey work of the e-Services Release 1 project, we identified
concerns regarding the capturing and resolution of defect reports. We have not completed our audit work to
determine the validity of our concerns but will include the results of our work
in an upcoming report.
A critical information system security process that all Federal Government agencies must undergo is security certification and accreditation. The main purpose of system certification and accreditation is to provide documented evidence (security test cases and results) that a system meets security standards and that the system owners accept the security risks related to its operation.
We reported that the IRS authorized the STIR project to support processing of the IRFOF application without having complete documentation of the results of security testing. This occurred because IRS executives unconditionally approved the deployment of STIR without ensuring all aspects of the security certification and accreditation processes were complete. While we did not identify any security problems with the STIR, limitations in the security testing and incomplete reporting of test findings could increase the risks to the deployed STIR system.
The potential for a similar situation to occur surfaced during our current audit work related to the IFS project. In a February 2003 memorandum, the IRS’ Security Services stated,
“The security documentation shall be updated before exit
from Milestone 4 (system deployment) IFS Release 1 is granted. Normally we would require that these changes
be accomplished as part of Milestone 3 (system development) exit, but given the
rapidly moving nature of the project and the extreme time pressures generated
as a result, we are willing to be flexible.”
*******
Testing processes are part of the overall controls to help ensure that information systems perform their intended functions accurately and reliably. The IRS hired the PRIME contractor in 1998 to bring discipline to its modernization efforts. As demonstrated by the compromises in all phases of performing project testing processes, this discipline has not been consistently realized. The discipline of following established processes needs to be achieved for the IRS to successfully modernize its information and business systems.
Management’s Response: The Chief Information Officer (CIO) agreed with most of the comments and observations in our report. The CIO responded that while significant progress has been made, testing practices have not been uniformly implemented for all projects, and further improvement opportunities remain. However, the CIO stated the report does not provide a clear or correct impression of the current state of testing activities. This disagreement involves a difference in opinion on our observations that inadequate testing practices are the result of attempts to meet overly optimistic project schedules, and that the number of waivers and deferred tests is an indication of incomplete testing activities. Management’s complete response to the draft report is included as Appendix V.
Office of Audit Comment: The CIO stated that he strongly disagreed with the central thesis of the report, which he related as “inadequate testing practices are the result of the BSM project teams attempting to meet overly optimistic project schedules.” The actual central thesis our report presents is that testing practices for BSM projects need improvements, which the CIO agreed with in his response.
We also stated our concern that the BSMO and the PRIME contractor should use the early, less risky project releases to implement and refine the testing practices. Refining the testing practices on the early projects allows them to be established and integrated when the more important and significant project releases are developed and tested. Reducing or eliminating some testing processes in the early stages of the BSM program does not allow the project teams to institutionalize defined and desired practices.
We agree with the CIO’s statement that testing practices should represent an appropriate balance of technical, schedule, cost and business risks. However, on every BSM project we reviewed, we found that some testing activities detailed in the project test plans and defined in the testing guidance documents were reduced or eliminated, which significantly increases the business risk that systems may not perform as intended. Based on our audit work, discussions with BSMO and PRIME contractor personnel, and observations of project activities, we believe the testing practices were reduced or eliminated by the project teams in an attempt to meet specific project delivery dates.
Since testing is one of the last phases of project development, reducing time spent in testing is the last opportunity for a project team to meet a schedule. In fact, the CIO stated that, while the IRS has not yet delivered a BSM project ahead of plan, the test tailoring decisions have significantly advanced final delivery dates. This statement seems to support our observation that decisions to reduce testing activities were made in an effort to meet schedules, or at least speed up delivery. While taking appropriate actions to attempt to meet a schedule commitment date is a desirable goal, we are concerned that the amount of reductions to the testing activities lessens the controls over quality that these tests are designed to evaluate.
The CIO also stated “although the number of waived or deferred test cases may appear to be troublesome after a superficial review, the basis of these waived and deferred test cases does not support the assertion that testing activities are incomplete.” The CIO further stated that waivers and deferrals usually occur because test environments cannot support specific test cases, or requirements are modified or removed.
We agree that there are valid reasons to waive or defer test cases, but our concern with the CIO’s statement is that if it was known that the test environment would not support the test cases, or that the requirements being tested were modified or removed, why were the test cases still included in the test plans? If these issues came about after the test plans were prepared and approved, then the plans should have been revised to reflect the actual test conditions to be performed. The fact that these waivers and deferrals were approved does not override the concern that incomplete testing can lead to problems when the systems are finally deployed.
We presented an example in the report where all planned performance tests on the STIR project were waived or deferred, and identified two other projects where a significant number of test cases were waived. While some of these waivers were probably valid decisions, we believe that many of the test cases were waived because of pressures to speed the delivery of the systems. As pointed out in the report, the risk of this practice is significant considering that the STIR infrastructure is a critical component for BSM projects.
We respectfully disagree with the CIO’s statement that this report does not provide a correct impression of the current state of BSM testing activities. While the conditions we include in the report were identified in past audits, we also saw many of the same conditions in current projects. This led us to conclude that the activities we report are still taking place to some degree. Further, the CIO’s response is limited to a high level assessment of the current testing practices used by the BSMO. His assessment does not provide specific examples on the steps taken to correct the issues presented in our report.
In summary, while the CIO may disagree with some of our comments and perspectives, we believe the information in this report is valid and worthwhile based on the comprehensive audit work performed. We commend the BSMO on the improvements that have been made to the testing processes and practices, but additional improvements need to be made to ensure that the testing activities provide the necessary confidence that systems perform their intended functions accurately and reliably. Our report points out the testing practices that require consideration to aid the efficient and effective delivery of modernized information and business systems.
Appendix I
Detailed Objective, Scope,
and Methodology
Our objective was to present issues and trends in Business Systems Modernization (BSM) testing processes that need management attention. To accomplish this objective, we summarized results from five audits recently conducted on BSM projects, as well as three audits currently in process, where we identified concerns with testing processes. The completed audits were:
· Customer Communications Project 2001 Release.
·
Internet
Refund Fact of Filing Release Packages 1 & 2.
· Customer Account Data Engine Release 1.
·
Security
and Technology Infrastructure Release 1.
·
Infrastructure
Shared Services.
The audits in process are:
·
e-Services
Release 1.
· Custodial Accounting Project / Enterprise Data Warehouse Release 1.
·
Integrated
Financial System Release 1.
We also reviewed General Accounting Office audit reports related to Internal Revenue Service computer modernization efforts.
Appendix II
Major Contributors to This Report
Margaret E. Begg, Assistant Inspector General for Audit
(Information Systems Programs)
Scott A. Macfarlane, Director
Edward
A. Neuwirth, Audit Manager
Bruce Polidori, Senior Auditor
George L. Franklin, Auditor
Appendix III
Commissioner C
Deputy Commissioner for Operations Support OS
Associate Commissioner, Business Systems Modernization OS:CIO:B
Chief, Information
Technology Services OS:CIO:I
Deputy Associate Commissioner,
Program Management OS:CIO:B:PM
Deputy Associate Commissioner,
Systems Integration OS:CIO:B:SI
Acting Director, Portfolio Management OS:CIO:R:PM
Chief Counsel CC
National Taxpayer Advocate
TA
Director, Office of Legislative Affairs CL:LA
Director, Office of
Program Evaluation and Risk Analysis
RAS:O
Office of Management Controls OS:CFO:AR:M
Audit Liaisons:
Associate
Commissioner, Business Systems Modernization
OS:CIO:B
Chief, Information Technology Services
OS:CIO:I
Appendix IV
This project was the first step towards achieving the objective of planning and managing the vital Customer Service telephone enterprise activity so that taxpayers get prompt and reliable access to the information they need.
The Customer Communications Project 2001 Release (CC 2001) became operational in August 2001. The Business Systems Modernization Office (BSMO) reported that the CC 2001 Project improved the Internal Revenue Service’s (IRS) ability to receive, route, and respond to the more than 150 million taxpayer telephone calls received each year. Major system improvements include designs to use voice-activated programs that recognize English or Spanish-speaking callers, a voice-activated program that taxpayers can use to find out the status of their refunds, and capabilities that more accurately route taxpayer calls to the most appropriate IRS personnel.
Audit Report
The Customer Communications Project 2001 Release Was Deployed, But
Testing Processes Did Not Ensure All Applications Were Working As Intended (Reference Number 2002-20-056, dated March 2002).
Status of Management Corrective Actions
Recommendation: The Chief Information Officer (CIO) should
direct the BSMO to ensure that requirements management meets established
Enterprise Life Cycle (ELC) practices.
Specifically, the BSMO should perform reviews to ensure it receives
documentation from the PRIME contractor showing that project system
requirements are traced to use cases, test cases, and test procedures.
Corrective Action: The PRIME contractor has created the Program
Validation and Verification Plan which requires all test plans to include a
requirement traceability matrix that maps all requirements to the test case and
test phase to verify all project requirements.
Status: Completed March 1, 2002.
Recommendation: To ensure
adequate control over defect reporting, resolution, and closure for future
modernization projects, the CIO should direct the BSMO to ensure details are
developed for the procedures to manage the defect identification, evaluation,
reporting, and resolution processes.
Corrective Action: The
PRIME contractor added the Defect Report (DR) Process flow to the Configuration
Management Plan. The DR Process
requires each project’s PRIME Coordinator to ensure that all DRs submitted for
closure contain a complete list of actions taken to address and close the
DR. The PRIME also added the DR Process
to the Program Validation and Verification Plan.
Status:
Completed July
31, 2001.
Recommendation: To help
ensure adequate control over defect reporting, resolution and closure for
future modernization projects, the CIO should direct the BSMO to ensure
responsibility is assigned for ensuring that the PRIME defect report database
includes accurate and complete information to document identified defects, the
defect resolutions, and approval of closures.
Corrective Action: The
PRIME Configuration Management Plan refined the function of the PRIME Defect
Report Coordinator to include responsibility for ensuring that all defect
reports entered into the PRIME database for his or her project contain complete
and accurate information.
Status:
Completed July
31, 2001.
Recommendation:
To ensure adequate control over defect reporting, resolution, and
closure for future modernization projects, the CIO should direct the BSMO to
ensure procedures are developed for the IRS to review and approve resolution
and closure of all defect reports.
Corrective Action: The CIO disagreed with this recommendation citing that the high number of defect reports generated during the project testing would make such reviews difficult.
Status:
Rejected.
Office of Audit Comment: While we agree that the extraordinarily high number of testing
defects on this one project presented a challenge, we believe that management needs to have
adequate assurance that problems are either resolved or reduced to an
acceptable level on each project.
Otherwise, problems could occur after projects are deployed, which could
significantly impact IRS operations and/or service to taxpayers.
2. Internet Refund Fact of Filing
The Internet Refund Fact of Filing (IRFOF) application provides secure Internet access for taxpayers to determine whether their tax returns have been received and processed by the IRS and the status of their refunds. A pilot version of the IRFOF application was deployed in May 2002, and it has handled over 1 million tax return and refund status requests. Taxpayer responses to survey questions about their use of the IRFOF application show that 78 percent of them were at least somewhat satisfied with this service.
The PRIME contractor has provided enhancements to the IRFOF application since it was initially deployed. The enhancements, entitled Release Package 1, were installed in October 2002 and provided application fixes and an improved encryption level. Further enhancements are included in Release Package 2. These enhancements include application performance changes, 2003 tax law changes, and changes to enable the IRFOF application to operate with the IRS’ upgraded Internet application.
Enhancements to the Internet
Refund Project Need to Be Completed to Ensure Planned Benefits to Taxpayers Are
Realized (Reference Number
2003-20-053, dated February 2003).
Status of Management Corrective Actions
The
Treasury Inspector General for Tax Administration (TIGTA) did not provide recommendations
since the corrective actions needed to address the issues identified had been
in previous TIGTA reports.
Finding:
IRFOF performance
with new hardware was not tested prior to deployment because of time and
schedule restrictions.
Management Actions: Following
the deployment of the IRFOF upgrades and enhancements, the PRIME contractor
conducted several benchmark performance tests.
The Customer Account Data Engine (CADE) will replace the IRS’ antiquated taxpayer account system with a modernized database. The new system, which is the centerpiece of the IRS’ modernization strategy, will facilitate faster processing of taxpayer returns and refunds, and improve customer service. The conversion of taxpayer accounts to the modernized database will happen in several stages. The first group of taxpayer accounts to be moved to the CADE includes single taxpayers who file an electronic or paper Income Tax Return for Single and Joint Filers With No Dependents. The IRS and the PRIME contractor are conducting a pilot of this first Release and started processing test data through the CADE in April 2003.
Improvements in the Customer Account Data Engine Pilot Plan Need to Be
Considered to Help Ensure the Pilot’s Success (Reference Number
2003-20-018, dated November 2002).
Status of Management Corrective Actions
Recommendation: To help ensure that the pilot test scenarios are properly executed and evaluated, the BSMO should require the PRIME contractor to complete the pilot plan scenarios with detailed description/objective, scenario steps, and success criteria.
Corrective Action: The CIO disagreed with this recommendation because the BSMO did not intend the scenario content to be detailed enough to run the scenarios.
Status: Rejected.
Office of Audit Comment: Although the CIO responded that the pilot could be accomplished with incomplete and missing scenarios, the pilot plan based the pilot execution on the ability to run the scenarios according to the related processing cycle. Without sufficient guidance provided by complete scenarios, pilot team members will have difficulty in executing pilot steps, monitoring pilot activities, and assessing the adequacy of operational activities. The completion of the CADE pilot test has been delayed, and while there are many factors contributing to the delays, we believe more detailed execution scenarios and acceptance criteria would allow the IRS and the PRIME contractor to better manage the pilot test to ensure success and minimize additional delays.
Recommendation: To ensure that the defect reporting databases provide
reliable information for the pilot defect reports, the BSMO should require the
PRIME contractor to provide detailed procedures for reconciling defect reports.
Corrective Action: The PRIME contractor drafted procedures, which
document the process the BSMO will use when interfacing defect reporting
information between the IRS’ and the PRIME contractor’s defect reporting
databases.
Status: Completed November 22, 2002.
Recommendation: To ensure that the defect reporting databases provide
reliable information for the pilot defect reports, the BSMO should require the
PRIME contractor to provide written procedures directing that the IRS approve
defect report resolution actions prior to defect report closure.
Corrective Action: The IRS now monitors IRS database usage and requires
the PRIME contractor to develop procedures to include defect report
resolution. The PRIME contractor
initiates the recommendation to close in the IRS database, and the IRS is
responsible for final closure.
Status: Completed November 30, 2002.
Recommendation: To ensure that the IRS’ defect reporting databases provide reliable information for the pilot defect reports, the BSMO should limit the PRIME defect report administrator’s privileges to “Read Only.”
Corrective Action: The CIO disagreed that the PRIME contractor should be limited to “Read Only” access to the IRS’ defect report database. The IRS will evaluate this process during the pilot and implement appropriate changes for production.
Status: Rejected
Office of Audit Comment: The CIO agreed with the need to implement appropriate internal
controls to ensure the integrity of data during production. However, by not limiting closure privileges
to the IRS’ defect report database, the IRS does not have assurance that
appropriate defect solutions took place prior to defect report closure. Without controls to limit privileges
in the IRS’ defect report database, inappropriate closure of defects without
the IRS’ knowledge and acceptance of their resolution may occur.
The Security and Technology Infrastructure Release (STIR) project provides a secure technical infrastructure used to support and enable the delivery of the IRS’ modernized business systems. For the BSMO and the PRIME contractor, the STIR is the first major project to undergo security certification testing and accreditation processes as required by the Office of Management and Budget, and the Department of the Treasury. Many challenges were encountered during this process, but the completion of the STIR Release 1.0 in May 2002 was a monumental step in providing opportunities for the development and deployment of all other modernized projects.
Security Testing and Certification of the Modernized Infrastructure
Needs to Be Strengthened (Reference Number 2003-20-127, dated June 2003).
Status of Management Corrective Actions
Recommendation: To ensure that future Business Systems
Modernization (BSM) projects meet security requirements and IRS officials
clearly understand the risks related to the projects and the impacts on their
operations, the CIO should ensure that the security certification and
accreditation process is performed, with all formal documents completed and
approved, prior to allowing any future BSM project to process sensitive
taxpayer data.
Corrective Action: The CIO
disagreed and responded that the IRS certification and accreditation process allows
for an informed management decision to be made (to deploy without formal
documents completed and approved) on a project-by-project basis that considers
the project risks at the completion of the security test and evaluation. The response also stated that the improved
certification and accreditation within the ELC process would indicate what
document is needed that communicates this authority.
Status: Rejected
Office of Audit Comment: Specific
guidance already exists within the ELC and the Department of the Treasury
Security Manual TD P-71-10 that allows for a system to temporarily operate
without full compliance to certification and accreditation. While we do not recommend this scenario, if
this situation does occur, a written exception must be obtained from the IRS
Office of Security, Privacy, and Oversight.
This process was not followed during the certification and accreditation
for the STIR.
Recommendation: To
reduce security risks for future BSM systems, the CIO should ensure that the
Certification Program Office performs security tests on all physical components
of the infrastructure located at each functional site, especially if the number
of sites is limited.
Corrective Action: The CIO
disagreed and stated that for the STIR, the IRS employed Type
accreditation. Type accreditation can
be used when the same system or configuration is being installed in multiple
locations.
Status: Rejected.
Office of Audit Comment: We
believe that applying Type accreditation for a system as critical as the infrastructure
is inappropriate. We also believe that,
while it is inappropriate to apply Type accreditation to the STIR, the IRS
relied upon the advantages of that guidance without following or performing the
recommended or suggested processes and procedures that should occur to provide
the necessary support for a Type accreditation.
Recommendation: To reduce security risks for future BSM
systems, the CIO should require the BSMO to inform the PRIME contractor that
alleviating schedule delays by executing security testing concurrently with
other critical test phases is not an acceptable practice and should be
conducted only in very rare circumstances.
Corrective Action: The CIO
disagreed and stated that a system can be tested at the same location and on the same day, but at different times.
Status: Rejected
Office of Audit Comment: We believe that
the three test phases of integration, deployment site readiness, and security
testing should occur independently and be completed prior to the start of
another test phase. Although the CIO
stated that the IRS did not allow concurrent testing to occur for the STIR, the
three test phases were performed during the same time period, which we believe
is concurrent testing. We maintain that
it is a risky practice to perform multiple testing phases on the same
system/components on the same day, especially when each test phase can require
several weeks to complete.
The success of the IRS’ modernization program depends on establishing a strong foundation from which to build business applications to support core tax processing functions. This process begins with the development of a modernized infrastructure. Currently, the modernized infrastructure is divided into three major functional areas: STIR; Enterprise Systems Management; and Development, Integration and Test.
A common operational environment, operations management, and oversight services, as well as a standardized set of hardware and software, are some of the benefits of a modernized infrastructure. Other benefits include reduced systems development time and resources, reduced operational costs, and tighter security.
Improvements to the Modernized Infrastructure Are Needed to Support the
Deployment of Business Systems Modernization Projects (Reference Number
2003-20-161, dated August 2003).
Status of Management Corrective Actions
Recommendation: To ensure that test lab capacity can
support future testing of modernized projects, the CIO should require (1)
improvements to the test lab be made a priority for future funding requests,
and (2) the test lab project team to follow the testing processes consistently
and gather test lab requirements from projects earlier.
Corrective Action: Management’s response is pending.
Status: Open.
Audits in Process
The e-Services project will provide a set of Web-based business products as incentives to third parties to increase electronic filing. The project focuses on fostering easy-to-use electronic products and services targeted at specific practitioner segments that inform, educate, and provide service to the taxpaying public. In addition, e-Services will provide electronic customer account management capabilities to all businesses, individuals, and other customers. Release 1.1 business functionality includes: Registration, Application, Indirect Channel Management, and Interactive Taxpayer Identification Number Matching.
Review of the
IRS’ e-Services Release 1 Development (Audit
Number 200320023).
2. Custodial Accounting Project/Enterprise Data
Warehouse Release 1
The Custodial Accounting Project
(CAP) will provide the IRS’ Chief Financial Officer with an automated revenue
accounting and collections allocation system that is compliant with Federal
Government requirements. To accomplish
this, the CAP will (1) use a data warehouse approach for storing, analyzing,
and reporting taxpayer accounts and collection information, and (2) design a
solution that serves as the foundation of an enterprise data warehouse. The first release of the CAP implements the
Enterprise Data Warehouse capability.
Review of the Integration and
Deployment of the Custodial Accounting Project, Release 1 (Audit Number
200320025).
The Integrated Financial System (IFS) project will provide the IRS with accurate and consistent financial data resulting in improved decision-making and management of the organization. The IFS has financial components that are being designed to improve how the IRS inputs, tracks, and reports financial data. It will help IRS employees better plan, manage, and measure performance across the Agency. The IFS will enable the IRS to integrate the majority of its financial processes, share common data and practices across the entire organization, and produce and access information in a real-time environment.
Audit Information
Review of the Development of the IRS’
Integrated Financial System Release 1 (Audit Number 200320038).
Appendix V
The response was
removed due to its size. To see the
response, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.