Customer Account Data Engine 2 Performance and Capacity Is Sufficient, but Actions Are Needed to Improve Testing
May 16, 2012
Reference
Number: 2012-20-051
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone Number /
202-622-6500
E-mail Address / TIGTACommunications@tigta.treas.gov
Website / http://www.tigta.gov
HIGHLIGHTS
CUSTOMER ACCOUNT DATA ENGINE 2 PERFORMANCE AND CAPACITY
IS SUFFICIENT, BUT ACTIONS ARE NEEDED TO IMPROVE TESTING
Highlights
Final
Report issued on May 16, 2012
Highlights of Reference Number:
2012-20-051 to the Internal Revenue Service Chief Technology Officer.
IMPACT ON TAXPAYERS
Built on the foundations of the current Customer Account Data Engine (CADE), CADE 2 will allow the IRS to convert the existing weekly individual taxpayer account processing system to daily batch processing. Such an enhancement should improve the service provided to taxpayers by allowing the IRS to more effectively and efficiently update taxpayer accounts, support account settlement and maintenance, and process refunds on a daily basis. The ability of the IRS to accurately execute, monitor, and assess performance and capacity testing directly affects whether, after implementation, CADE 2 will be capable of processing the necessary quantity and types of information within required time frames. The untimely completion of information processing could result in delayed taxpayer refunds and reduced customer service.
WHY TIGTA DID THE AUDIT
The overall objective of this review
was to determine
whether the IRS is effectively testing the performance and capacity of CADE 2. This audit was included in our Fiscal Year
2011 Annual Audit Plan and addresses the major management challenge of Modernization
of the IRS.
WHAT TIGTA FOUND
The IRS
has successfully established a testing environment for CADE 2 that is
representative of the production environment. This is allowing the IRS to obtain meaningful
data from its pre-production tests.
However, the IRS did not follow procedures to ensure that performance
requirements were completely tested during the Final Integration Test Phase I. As a result, the IRS may not have acquired all
the necessary information to make a fully informed decision on the ability of
the CADE 2 system to effectively process transactions under expected normal and
peak workload conditions, within acceptable response time thresholds. TIGTA also found that the IRS needs to develop
procedures for access to and retention and maintenance of testing artifacts for
Final Integration Testing.
WHAT TIGTA RECOMMENDED
TIGTA recommended that the Associate Chief Information
Officer, Applications Development, ensure internal controls for testing
performance and capacity requirements are formally and effectively implemented
to ensure the traceability of these requirements through the performance
testing process. This should include the use of integrated automated
tools, when warranted by program and project size, to improve the consistency
and completeness of testing performance and capacity requirements. In addition, the IRS should develop
procedures related to the access to and retention and maintenance of testing
artifacts for performance testing.
In their response to the report, IRS management agreed
with TIGTA’s recommendations. The IRS
plans to ensure internal controls for testing performance and capacity
requirements are formally and effectively implemented to ensure the
traceability of these requirements through the performance testing
process. This will include the use of
automated tools for testing, where appropriate.
The IRS also plans to review and, as warranted, develop procedures
related to the access to and retention and maintenance of testing artifacts
across all test projects.
May 16, 2012
MEMORANDUM FOR CHIEF TECHNOLOGY OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Customer Account Data Engine 2 Performance and Capacity Is Sufficient, but Actions Are Needed to Improve Testing (Audit # 201120026)
This report presents the results of our review to determine whether the Internal Revenue Service (IRS) is effectively testing the performance and capacity of the Customer Account Data Engine 2. This audit was included in our Fiscal Year 2011 Annual Audit Plan and addresses the major management challenge of Modernization of the IRS.
Management’s complete response to the draft report is included as Appendix V.
Copies of
this report are also being sent to the IRS managers affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Alan R. Duncan, Assistant Inspector
General for Audit (Security and Information Technology Services), at (202)
622-5894.
The Customer
Account Data Engine 2 Test Environment Is Representative of the Production
Environment
Procedures Need
to Be Developed for Access to and Retention and Maintenance of Testing
Artifacts
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – Glossary of Terms
Appendix
V – Management’s Response to the Draft Report
Abbreviations
|
CADE |
Customer Account Data Engine |
|
FIT |
Final Integration Testing |
|
IRS |
Internal Revenue Service |
|
RTVM |
Requirements Traceability Verification Matrix |
The Internal Revenue Service (IRS) had two primary production systems that processed individual taxpayer account data, the Individual Master File[1] and current Customer Account Data Engine (CADE). In June 2011, current CADE tax processing was taken out of production in order to implement changes necessary for its integration with CADE 2. The IRS underwent a major development effort to deploy CADE 2 functionality in January 2012.
The IRS approach for delivery of CADE 2 is a functional and technical progression through two Transition States to a Target State. Transition State 1 has two main purposes: 1) to establish a database that will house all individual taxpayer accounts and provide the ability for IRS employees to view the updated account information online (Database Implementation); and 2) to provide individual taxpayer account information to select external systems on a daily basis as opposed to the current weekly basis (Daily Processing). Affected processes include the receipt and submission of tax returns; the processing and management of accounts, examinations, collections; and criminal investigation. The key IRS customer service system, the Integrated Data Retrieval System, would also realize the benefit of more timely posted data. Such an enhancement to these business processes and systems should improve the service provided to taxpayers by allowing the IRS to more effectively and efficiently update taxpayer accounts, support account settlement and maintenance, and process refunds on a daily basis. Transition State 1 Daily Processing went live, as planned, on January 17, 2012.
Transition State 2 will address financial material weaknesses and build or modify existing applications to directly interact with the CADE 2 database. Finally, the Target State will focus on completing the transition of all applications and achieving the business benefits that are enabled by the target solution. This final step is necessary to provide for the long-term viability of the CADE 2 platform.
As part of Transition State 1, the IRS performed Final Integration Testing (FIT), which is an end-to-end integration test of multiple systems which support the high-level business requirements. The FIT is planned from the perspective that all IRS application systems are subsystems to an overall tax processing system. The tax processing system consists of hundreds of subsystems operating on many unique hardware and software platforms. The FIT is designed to ensure that IRS systems work together correctly prior to production start up. CADE 2 Daily Processing started FIT Phase I on July 7, 2011, and concluded on October 7, 2011. The testing was completed after the planned end date which allowed the IRS to perform additional testing, analyze metrics, and revalidate failed tests.
CADE 2 FIT Phase I was used to demonstrate the system’s capability to execute daily processing jobs within allocated performance windows in a near production-like environment. In order to provide testing in a simulated production environment, FIT Phase I utilized a copy of production data to prepare test cases with predetermined results. Unique test cases created based on requirements were used to verify systemic interoperability. Data were processed and passed from system to system exercising all communication links to demonstrate that information was being correctly exchanged between applications.
We focused our review on performance and capacity requirements that were tested during CADE 2 FIT Phase I. Database Implementation was removed from FIT Phase I and will be tested separately in a future test outside of FIT Phase I. As a result of Database Implementation not being tested during FIT Phase I, it was removed from the scope of this audit.
This review was performed at the Modernization and Information Technology Services organization facility in New Carrollton, Maryland, during the period November 2011 through February 2012. We conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objective. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objective. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The Customer Account Data Engine 2 Test Environment Is Representative of the Production Environment
The IRS’s Test Assurance Documentation Standards and Procedures detail the procedures and guidelines for establishing a test environment. The test environment should emulate the production environment to the greatest degree possible.
The FIT Phase I test was conducted on mainframe platforms located at the Enterprise Computing Centers in Martinsburg, West Virginia, and Memphis, Tennessee. At Martinsburg, the test environment consisted of International Business Machines Corporation z990 and z10 mainframe computers. At Memphis, the mainframe test environment consisted of a Unisys Corporation 780 Dorado mainframe computer and an International Business Machines Corporation z900 mainframe computer. The testing of tax processing that occurs on the IRS’s server environment was conducted in the IRS Enterprise Systems function’s Test Lab located in New Carrollton, Maryland.
To simulate production-like volumes for CADE 2 Daily Processing testing and test case development, the IRS used data from the submission processing sites and Master File for 2011 peak production cycles. These data represent the April 15th peak tax processing period and the weeks immediately before and after it. Data from these cycles are the closest representation of what daily processing will be like in a peak period in Processing Year 2012.
We compared the physical overview diagrams of the two environments and then validated the system specifications listed in capacity requirements. Our comparison indicated that the major infrastructure components are identical, but that there are slight differences in processing capability, storage capacity, and memory. Discussions with IRS subject matter experts and our review of IRS documentation indicated these slight differences were expected. The IRS reiterated that the CADE 2 test environment was designed to be representative of the production environment, not an exact replica of it.
Procedures Were Not Effectively
Followed to Ensure Performance Requirements Were Completely Tested During Final
Integration Test Phase I
The IRS Test Assurance Documentation Standards and Procedures detail the requirement analysis process. This analysis is the cornerstone of a systems acceptability test. A requirement, at its lowest decomposed level, provides specific information for a single function performed by a software application. A process exists whereby a person or machine can verify that the system built meets the requirement (that is, the requirement is testable). Taken collectively, requirements completely define all system capabilities.
These procedures also stress the importance of a Requirements Traceability Verification Matrix (RTVM), one of the key controls used to provide IRS management with evidence that requirements are sufficiently gathered prior to the development of test cases. An RTVM also helps to ensure all requirements are included in test cases and are tested. The absence of an RTVM increases the risk that not all requirements are tested to ensure the system works as intended.
To validate the core changes to the tax pipeline processing system as a whole, the IRS tested a subset of high-level CADE 2 requirements. The FIT Phase I End of Test Status Report stated that the requirements selected for testing were documented as scenarios, and that the IRS documented, tracked, and monitored these test scenarios in an automated tool.
We evaluated 227 performance and capacity requirements from a repository of CADE 2 requirements. All of these requirements were included in authorized and approved Database Implementation or Daily Processing internal design documents. These documents recognize the use of various requirements as the basis for the business solution design, development, integration, testing, and deployment of the CADE 2 system. We determined that 70 (31 percent) of these performance requirements were included in the RTVM and could be traced to 702 corresponding test cases. Figure 1 provides a detailed breakdown of the number of requirements that were or were not traced to test cases.
Figure 1: Performance and
Capacity Requirements Traced to Test Cases
|
Requirements
Traced to Test Cases |
Number |
|
Requirements With Test
Cases |
70 (31%) |
|
Requirements With No Test
Cases |
|
|
Environmental Design |
66 (29%) |
|
Database Implementation |
68 (30%) |
|
Daily Processing |
23 (10%) |
|
Total |
227 (100%) |
Source: Treasury
Inspector General for Tax Administration analysis of IRS performance and capacity
requirements.
The IRS indicated that 66 (29 percent) performance and capacity requirements were environmental design requirements. While these requirements were not supported with documented test cases, they were supported by Government equipment lists showing hardware purchases to meet the requirement capabilities. Eight of these 66 requirements specifically referenced hardware specifications such as number of terabytes or Millions of Instructions Per Second. For each of these eight requirements, we successfully traced the amount of storage, processing speed, or memory that was listed to the physical components shown in the FIT Plus Environment Physical Overview Diagram. We also reviewed an IRS comparison of the CADE 2 production environment to the CADE 2 FIT test environment. In addition, the IRS internally validated the addition of CADE 2 components into the existing FIT environment.
Sixty-eight (30 percent) of the requirements were categorized as Database Implementation requirements. As previously discussed, Database Implementation testing was removed from the scope of FIT Phase I. There were no test cases provided for these Database Implementation requirements. The remaining 23 (10 percent) Daily Processing requirements were not supported with test cases. One of these requirements was identified as a future test outside the scope of FIT Phase I. The remaining 22 requirements did not have documented test cases or were not traced to an alternative testing method. This lack of internal control in ensuring that all requirements are traced to an appropriate testing method increases the potential risk of future performance issues.
Test scripts establish the relationships between the requirements to be tested and the associated test cases. Each test script is run and marked as passed, failed, or incomplete/inconclusive. Figure 2 provides a detailed breakdown of the results for each category. Our review of the IRS testing results determined that 215 (31 percent) of 702 test scripts passed and 255 (36 percent) were incomplete or inconclusive due to processing issues. The IRS indicated that the processing issues included, but were not limited to, errors in programming language.
Figure 2: Performance Test Results
|
Test Script
Results |
Number |
|
Failed |
232 (33%) |
|
Incomplete/Inconclusive |
255 (36%) |
|
Passed |
215 (31%) |
|
Total |
702 (100%) |
Source: Treasury Inspector General for Tax Administration
analysis of IRS performance testing results.
The performance testing results in Figure 2 were only one of several factors considered in the decision to move forward from FIT Phase I testing and could not be considered alone in determining the production readiness of the system. An in-depth analysis of CADE 2 performance testing was conducted by IRS subject matter experts with the assistance of contractors. This analysis led to adjustments being made to several variables, including the processing time frame targets and the testing environment. These adjustments addressed the performance test failures, as well as incomplete and inconclusive test results, thereby resulting in CADE 2 performing within required time frames and providing management with confidence to move forward from FIT Phase I to further performance testing.
As a result of FIT Phase I testing and other analyses, the IRS made the determination that CADE 2 Daily Processing is capable of processing the required quantities and types of transactions within the specified predetermined time frames. However, we identified a lack of effective planning and compliance with established procedures in the development of an RTVM and ensuring that all requirements were tested.
The IRS did not effectively implement or consistently use two automated tools that could have provided improved traceability for requirements in the testing process. As a result, performance and capacity requirements could not be traced to testing results. The inconsistent use of these tools resulted in the IRS using a manual process to manage the test cases, potentially resulting in unnecessary complexity and inefficiency within the test management and reporting process.
By not ensuring that each requirement had a test case; documenting its traceability, including the results of testing; and identifying when requirements were to be tested, the IRS has increased the potential risk that the CADE 2 system will not perform in an acceptable manner. The risk exists that the IRS has not tested all requirements associated with system performance because test cases were not developed or were overlooked due to the lack of a documented schedule. Due to these risks, the potential exists that the IRS did not acquire all necessary information to make a fully informed decision on the capability of the CADE 2 system to effectively process transactions under expected normal and peak workload conditions, within acceptable response time thresholds.
Recommendation
Recommendation 1: The Associate Chief Information Officer, Applications Development, should ensure that internal controls for testing performance and capacity requirements are formally and effectively implemented to ensure the traceability of these requirements through the performance testing process. This should include the use of integrated automated tools, when warranted by program and project size, to improve the consistency and completeness of testing performance and capacity requirements.
Management’s Response: The IRS agreed with the recommendation. The Associate Chief Information Officer, Applications Development, will ensure internal controls for testing performance and capacity requirements are formally and effectively implemented to ensure the traceability of these requirements through the performance testing process. This will include the use of automated tools for testing, where appropriate.
Procedures Need to Be Developed for Access to and
Retention and Maintenance of Testing Artifacts
According to the FIT Phase I Technical Test Plan, dated
April 8, 2011, test artifacts should be saved in either hard copy or to
electronic media and retained for one year.
These artifacts include input and output data files, logs and matrices,
and test documentation (e.g., project
folders, scenarios, and test cases). The
test plan also states that once the CADE 2 FIT Phase I was completed that these
test artifacts were to be kept for an agreed-upon length of time, up to one
year. Guidelines for the file backup and
retention process can be found in IRS procedures entitled Types of Records and their Life Cycle.
Should this be CJK: No – this is the name of an IRM (procedures). “Types
of Records and their Life Cycle” Amended to correct name and edited
sentence to clarify. The term
Enterprise Life Cycle is not used in the report and will be deleted from
the glossary.
System logs are the artifacts that support test case
results for tests run during FIT Phase I performance testing. Logs are automatically created and stored for
a predetermined amount of time. The
information in these logs, specifically job start and end times, are used to
determine whether a job completed in its allotted amount of time and, therefore,
is the basis for whether a test has passed or failed.
“Enterprise Life Cycle?” If so, and
it is changed here, please disregard the comment in the glossary of terms
related to the use of Enterprise Life Cycle in the report.
During on-site observations, we presented the IRS with
multiple test cases of differing results (i.e.
pass, fail, incomplete) from the FIT Phase I performance RTVM. The process to identify test artifacts in
support of these testing results was an informal process consisting of multiple
steps. To identify the test artifacts
using this process, IRS personnel used an online test case management tool to
identify a test script within a test case, then referred to a detailed
spreadsheet used to maintain artifact details, and finally accessed a separate
application that interfaced with and pulled artifact data from the specific
mainframe computer on which the testing was performed. The process for retaining and accessing
testing artifacts for FIT Phase I performance testing was not documented or
formalized. However, the IRS indicated
that formal procedures are currently in development.
The IRS did not create formal procedures for the access to
and retention of testing artifacts prior to initiating CADE 2 FIT Phase I
testing, but relied on an informal process that was different for each system
that maintained the artifact information.
A lack of formal procedures regarding the access to and retention and
maintenance of FIT Phase I performance testing
artifacts increases the potential that artifacts may not be maintained in
accordance with data retention procedures.
The lack of standardization in maintenance of testing artifacts
increases the risk that artifacts needed for future review or for use in other
development projects may not be available.
Recommendation
Recommendation 2: The Associate Chief Information Officer, Applications Development, should develop procedures related to the access to and retention and maintenance of testing artifacts for performance testing.
Management’s Response: The IRS agreed with the recommendation. The Associate Chief Information Officer, Applications Development, will review and, as warranted, develop procedures related to the access to and retention and maintenance of testing artifacts across all test projects.
The report body states 227 performance and capacity
requirements were evaluated and Appendix II includes Dr. Katz as a major
contributor. Therefore, the reviewer
assumes a sample was taken as part of this audit. If a sample was taken, please include the
required sampling information such as type of sample and population and
sample sizes. If no sample was taken
and Dr. Katz was contacted for his advice, but his work was not actually
used as a major part of the fieldwork, then consider removing him as a
major contributor CJK: We originally consulted Dr. Katz and took a
sample. However due to the responses
during fieldwork, we expanded and reviewed the entire universe of 227
requirements. We will delete his
name from the list of contributors.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this audit was to determine whether the IRS is effectively testing the performance and capacity of CADE 2. To accomplish this objective, we:
A. For Database Implementation:
1. Determined whether all performance requirements, as stated in CADE 2 Database Implementation Transition State 1 Performance Measurement Plan - 09/02/2011, were properly approved by management by determining if source documents (e.g., Database Implementation Business System Report and Database Implementation Design Specification Report) had been properly reviewed and approved by management.
2.
Determined
whether changes to performance requirements were properly authorized and
reflected in updated test scenarios.
B.
For
Daily Processing:
1. Determined whether all performance and capacity requirements were properly approved by management by determining if source documents (e.g., Daily Processing Business System Report and Daily Processing Design Specification Report) had been properly reviewed and approved by management.
2.
Determined whether performance and capacity
requirements in our sample had been included in the RTVM.
3.
Determined
whether changes to performance and capacity requirements in our sample were
appropriately authorized and updated.
II. Determined whether the CADE 2 performance and capacity requirements were effectively tested.
A. Determined whether the environment used to conduct FIT was consistent with the production environment.
B. Determined
whether CADE 2 Program tests were conducted, results analyzed, and defects
adequately resolved.
III.
The reviewer did not find the report body where the
results of these tests were discussed. CJK: These areas were all reviewed. These items applied to the requirements
testing area of the audit and were part of our verification of
requirements, test results, and subsequent IRS actions, discussed on pages
4 through 6. If there had been
issues in any of these areas they would have been detailed in the
discussion of whether requirements were completely tested.
Determined whether CADE 2 performance and capacity
requirements that failed testing (i.e.,
defects) were properly resolved.
A. Obtained and reviewed the End-of-Test Status Report which contains the final complete test results.
1. Identified any performance/capacity issues identified during testing.
2. Verified that all performance/capacity test cases passed, were waived, or deferred.
B. Verified that the RTVM was updated to include test results as passed, waived, or deferred.
C. Determined whether the CADE 2 executive handling of risks/issues identified during testing was appropriate.
Internal controls methodology
Internal controls relate to management’s
plans, methods, and procedures used to meet their mission, goals, and
objectives. Internal controls include
the processes and procedures for planning, organizing, directing, and
controlling program operations. They include
the systems for measuring, reporting, and monitoring program performance. We determined the following internal controls
were relevant to our audit objective: the
CADE 2 Project Management office’s and the Enterprise
System Testing organization’s policies and procedures for effectively testing
performance and capacity of developing systems. We evaluated these controls by interviewing
management, reviewing the Project Management office’s
and the Enterprise System Testing organization’s policies and procedures, as
well as by reviewing test cases and other testing byproducts for tests run
during FIT Phase I.
Appendix II
Major Contributors to This Report
Alan R. Duncan, Assistant Inspector General for Audit (Security and
Information Technology Services)
Danny Verneuille, Director
Carol Taylor, Audit Manager
Myron Gulley, Acting Audit Manager
Curtis Kirschner, Senior Auditor
Elton Jewell, Information Technology
Specialist
Daniel Oakley, Information Technology Specialist
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Deputy Chief Information Officer for Operations OS:CTO
Associate Chief Information Officer, Applications Development OS:CTO:AD
Associate Chief Information Officer, Enterprise Operations OS:CTO:EO
Associate Chief Information Officer, Modernization - Program Management Office OS:CTO:MP
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis RAS:O
Office of Internal Controls OS:CFO:CPIC:IC
Audit
Liaison: Director, Risk Management Division OS:CTO:SP:RM
Appendix IV
|
Term |
Definition |
||
|
Capacity Testing |
A process of testing the program(s) under heavy load by putting a large number of inputs that are beyond the handling capacity of the application. |
||
|
When read without the text set off by commas “of” does
not seem correct. QA modified. CJK: Noted. |
Capacity is interpreted in terms of system resources required to process peak workload demand, i.e., Central Processing Unit, Input/Output, Memory, Disk space. Industry best practices indicate that modern mainframe systems are capable of running at high levels of system utilization and that it is the performance of higher priority workloads that should be managed, not the level of system utilization. |
||
|
Computer Performance |
Computer system performance relates to the response time of the system, i.e., the amount of time it takes the system to perform a given unit of work. Typical units of work are transactions. |
||
|
Enterprise Computing Center |
Supports tax processing and information management through a data processing and telecommunications infrastructure. |
||
|
Final Integration Testing |
Should this be “allocated?” CJK: Amended. |
||
|
Fiscal Year |
A 12-consecutive-month period ending on the last day of any month, except December. The Federal Government’s fiscal year begins on October 1 and ends on September 30. |
||
|
Individual Master File |
The IRS database that maintains transactions or records of individual tax accounts. |
||
|
Integrated Data Retrieval System |
IRS computer system capable of retrieving or updating
stored information. It works in
conjunction with a taxpayer’s account records. |
|
Term |
Definition |
||
|
Mainframe |
A powerful, multiuser computer capable of supporting many
hundreds of thousands of users simultaneously. |
||
|
Master File |
The IRS database that stores various types of taxpayer
account information. This database
includes individual, business, and employee plans and exempt organizations
data. |
||
|
Performance Testing |
Determines whether the system undergoing testing can
effectively process transactions under expected normal and peak workload
conditions, within acceptable response time thresholds. Performance testing will uncover any
bottlenecks and capacity constraints that may not have occurred during normal
functional testing. |
||
|
Processing Year |
The calendar year in which the tax return or document is processed by the IRS. |
||
|
Requirements Traceability Verification Matrix |
A tool showing the relationship between test requirements
and test cases. |
||
|
Suggest defining the highlighted term in the glossary. CJK: We do not believe the sentence adds anything
material to the definition (this is the only time thread is used in the
report), deleted the sentence. |
Comprised of the event (i.e., type of input data that results in an action), the entry
point into the system (e.g., Information
Data Retrieval System, Integrated Submission and Remittance Processing, etc.), and the action. |
||
|
Submission Processing Site |
The data processing arm of the
IRS. The sites process paper and
electronic submissions, correct errors, and forward data to the Computing
Centers for analysis and posting to taxpayer accounts. |
||
|
Systems Acceptability Test |
Testing performed by the Test,
Assurance, and Documentation group to independently assess the quality of the
application software by testing with controlled data to determine conformance
of the system to customer requirements and to aid the customer and developer
in determining the system’s production readiness. |
||
|
Test Case |
A document specifying the test approach for a software
feature or combination of features and the inputs, predicted results, and
execution conditions for the associated tests. |
|
Term |
Definition |
|
Test Environment |
The preliminary setup of hardware, resources, data,
utilities, etc., required to run a
successful test. |
|
Test Plan |
A document that describes the objectives, scope, approach,
and focus of a software testing effort. |
|
Test Results |
A documented summary or printed output resulting from the
execution of the test plan. |
|
Test Script |
A set of instructions that is performed on a system under test to verify that the system performs as expected. |
Appendix V
Management’s Response to the Draft Report
DEPARTMENT OF THE TREASURY
INTERNAL REVENUE SERVICE
WASHINGTON, D.C. 20224
Received Apr 18 2012
CHIEF TECHNOLOGY OFFICER
MEMORANDUM FOR DEPUTY INSPECTOR GENERAL FOR AUDIT
FROM: Terence V. Milholland
/s/ Terence V. Milholland
Chief
Technology Officer
SUBJECT: Draft
Audit Report - Customer Account Data Engine 2 Performance and Capacity Is
Sufficient, but Actions Are Needed to Improve Testing (audit #201120026)
(e-trak #2012-30810)
Thank you for the opportunity to review your draft audit
report and to discuss earlier draft report observations with the audit team.
I was pleased to read your comments and observations
acknowledging that the IRS has successfully established a testing environment
for CADE 2 representative of the production environment. Further, I appreciate your acknowledgement that establishing this testing
environment allowed the Internal Revenue Service to obtain important data
during the pre-production tests. These tests allowed us to fine tune our
processes early and contributed to the successful launch of Daily Processing.
I agree with the two recommendations provided in the
report, and the attachment details our planned actions to implement them. In the case of the first recommendation,
however, it should be noted that the use
of a specific product, such as an automated tool, does not ensure control over
the execution of a test. I therefore
appreciate TIGTA's flexibility
in requiring the use of automated tools, only where appropriate.
We are committed to continuously improving our
information technology systems and processes.
We value your continued support and the assistance and guidance your
team provides. If you have any
questions, please contact me at (202) 622-6800 or Karen Mayr at (202) 283-0015.
Attachment
RECOMMENDATION
#1: The Associate Chief Information Officer,
Applications Development, should ensure internal controls for testing
performance and capacity requirements are formally and effectively implemented
to ensure the traceability of these requirements through the performance
testing process. This should include the
use of integrated automated tools, when warranted by program and project size,
to improve the consistency and completeness of testing performance and capacity
requirements.
CORRECTIVE
ACTION #1: The IRS agrees with this recommendation and
will ensure internal controls for testing performance and capacity requirements
are formally and effectively implemented to ensure the traceability of these
requirements through the performance testing process. This will include the use of automated tools
for testing, where appropriate.
IMPLEMENTATION
DATE: December 31, 2012
RESPONSIBLE
OFFICIAL: Associate Chief Information Officer,
Applications Development
CORRECTIVE
ACTION MONITORING PLAN: We enter accepted
Corrective Actions into the Joint Audit Management Enterprise System
(JAMES). These Corrective Actions are
monitored on a monthly basis until completion.
RECOMMENDATION
# 2: The Associate Chief Information Officer,
Applications Development, should develop procedures related to the access to
and retention and maintenance of testing artifacts for performance testing.
CORRECTIVE
ACTION #2:
The IRS agrees with this recommendation and will review and, as warranted,
develop procedures related to the access, retention, and maintenance of testing
artifacts across all test projects.
IMPLEMENTATION
DATE: December 31, 2012
RESPONSIBLE
OFFICIAL: Associate Chief Information Officer,
Applications Development
CORRECTIVE ACTION MONITORING PLAN: We entered accepted Corrective Actions into the Joint Audit Management Enterprise System (JAMES). These Corrective Actions are monitored on a monthly basis until completion.