Systems Acceptability Testing for the Tax Exempt
Determination System Was Planned and Executed As Intended
July 2003
Reference
Number: 2003-10-140
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
July
1, 2003
MEMORANDUM FOR
COMMISSIONER, TAX EXEMPT AND GOVERNMENT ENTITIES DIVISION
FROM: Gordon C. Milbourn III /s/ Gordon C. Milbourn III
Assistant Inspector General
for Audit (Small Business and
Corporate Programs)
SUBJECT: Final Audit Report - Systems
Acceptability Testing for the Tax Exempt Determination System Was Planned and
Executed As Intended (Audit #
200310010)
This
report presents the results of our review to determine if
the Tax Exempt and Government Entities (TE/GE) Division management’s
process for conducting the Tax Exempt Determination
System (TEDS) Release 1 Systems Acceptability Testing (SAT) was adequate
to assure management that the TEDS will correctly
process Forms 5307.
The
TE/GE Division is developing the TEDS to replace the Employee Plans and Exempt
Organization Determination System (EDS). SAT
is one component of Internal Revenue Service (IRS) testing and consists of the
activities to assess the quality of application software to assist the business
users in determining whether the system operates as intended.
In
summary, our review determined that the planning and execution of SAT for the
TEDS Release 1 conformed to industry standards and IRS procedures and was
generally adequate to test the system
requirements. During testing, TEDS
project management identified necessary business requirements that were
overlooked during the system development process. This oversight was not significant enough to stop SAT to correct
the deficiencies. The project team
controlled the problems that were identified so they can be corrected after
completion of SAT.
Management
Response:
The Commissioner, TE/GE Division, agreed with our conclusion that the TEDS
Release 1 SAT conformed to appropriate standards and was generally adequate to
verify that the software would operate as intended. Management’s complete response to the draft report is included as
Appendix V.
Copies
of this report are also being sent to the IRS managers who are affected by the
report finding. Please contact me at
(202) 622-6510 if you have questions or Daniel R. Devlin, Assistant Inspector
General for Audit (Headquarters Operations and Exempt Organizations Programs), at
(202) 622-8500.
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Tax Exempt Determination System Release 1 Capabilities
Appendix V – Management’s Response to the Draft Report
The Tax Exempt and Government Entities (TE/GE) Division uses the Employee Plans and Exempt Organization (EP/EO) Determination System (EDS) to process and control EP/EO determination letter applications. The TE/GE Division is developing the Tax Exempt Determination System (TEDS) to replace the EDS. The TEDS is designed to provide increased systems capability and much needed improvements in overall system performance and reliability, correct existing severe shortcomings, and support new requirements of the major Internal Revenue Service (IRS) strategies and operational priorities.
Industry best practices suggest that new computer systems be tested during development to ensure they operate as the user intended. Systems Acceptability Testing (SAT) is one component of IRS testing and consists of the activities to assess the quality of application software. SAT is performed with controlled data (not actual cases) that, when processed through the system, should produce predetermined results. It provides an independent assessment of the quality of the software to assist the business users in determining whether the system meets the operational objectives of the business. The business objectives cannot be met if the software was not designed and coded to support the planned business functionality (i.e., how the system is intended to operate). SAT does not provide a critique of the system design, only whether the system is built to the design specifications and if the design meets the business requirements.
The Product Assurance Division of the IRS’ Modernization, Information Technology and Security (MITS) Services organization normally performs SAT. However, due to other priorities, it only provided guidance in developing the original SAT plan for the TEDS Release 1. Accordingly, the TEDS project team planned and performed SAT with the assistance of the Project Management/Systems Integration (PM/SI) contractor and under the direction of the TE/GE Division Information Officer (DIO).
We recently completed an audit to evaluate how well TEDS management adhered to project management techniques in the development of the TEDS Release 1. We reported that ineffective requirements gathering, performance monitoring, risk management, and configuration management did not ensure project objectives were completed on schedule or within budget, did not identify potential problems, and did not ensure that all development team members were continuously working on the most recent version of custom software or system design elements. We recommended that project management techniques be followed to effectively develop the TEDS. TE/GE Division management agreed with our findings and recommendations and has initiated corrective actions.
Based on the weaknesses identified during the development of the TEDS, we performed a review of the TEDS Release 1 SAT to determine if the system would be adequately tested prior to being placed into production. We performed audit work at the TE/GE Headquarters offices in Washington, D.C., from November 2002 through May 2003 in accordance with Government Auditing Standards.
We limited the scope of our audit work to the planning for SAT, the testing of certain key requirements that govern automatic case closure, and problem identification and resolution. We did not evaluate the TE/GE Division’s efforts to test the remaining Release 1 requirements: scanning determination applications into the system, entering data to the system, or transferring data to the Letter Information Network User System (LINUS) and the EDS. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The TE/GE Division management’s planning and execution of SAT for the TEDS Release 1 conformed to industry standards and the Internal Revenue Manual (IRM) and was generally adequate to test the system requirements. The SAT plan included the scope of tests to be performed, support needed to conduct the tests, definition of test cases, and a methodology to track and resolve problems that were identified. In addition, the planned tests were executed and data were processed through the TEDS to verify, with a few exceptions, that overall the software operated as intended. TEDS project management did identify a few problems during testing that they plan to follow up on after SAT. For example, TEDS project management identified that necessary business requirements had been overlooked during the development process and the software was not coded to evaluate one line of the determination application. This was not significant enough to stop SAT to correct the deficiencies. Instead, TEDS project management plans to correct these deficiencies after completion of SAT.
We determined that TEDS project management took the following actions during the planning and execution of SAT.
Planning for SAT
The TEDS project team developed a SAT plan that generally met IRM guidelines, Enterprise Life Cycle (ELC) standards, and industry best practices. The plan included the following:
·
SAT test information such as
test locations and dates.
·
A general description of the
system to be tested.
· The scope of the test, including the test standards and documentation.
· The support needed for conducting the tests, including personnel and computer hardware and software.
· The four phases of SAT (Initiation, Preparation, Execution, and Conclusion) and the scheduled tasks within those phases.
· The system and procedures used for reporting problems.
When we initially reviewed the SAT plan before the testing phase began, we determined that two items had not been included: the manager who had overall responsibility for SAT, and the computer hardware and software required for SAT. We advised project management of this, and they promptly revised the plan to include this information. Additionally, we reported to TEDS project management that they needed to continue their risk and configuration management processes. We later determined that they were adequately following the configuration management process for the software development. TEDS project management stated that they had identified and managed their risks; however, they were not getting everything documented. Due to the planned corrective actions regarding risk management in management’s response to our prior report, we are not making any further recommendations in this report.
We also determined that the TEDS project team developed a Requirements Traceability Matrix (RTM) as well as 200 test cases to ensure that all of the systemic requirements were tested. The RTM is a tool that shows the relationship between the requirements to be tested and the specific tests, the test cases that are designed to test the specific requirements, and the individuals who will perform the test or verify the test results.
The 200 test cases were fabricated Forms 5307 prepared by EP revenue agents with experience in working application letter cases. The test cases were not designed to test every combination of data possible, rather the more likely scenarios based on the revenue agents’ experience. The TEDS has an automatic closure feature that allows cases meeting the criteria for a favorable determination to be closed without manual intervention. Some of the test cases were developed to pass the automatic closure feature and others were designed to fail.
Additionally, these revenue agents developed a case summary matrix called the SAT Business Rule Summary. The matrix consolidated the information for each of the fabricated test cases onto one page and identified the predetermined results for the test case. Having all of the test case information on one page made comparison to the actual results easier during testing.
During test case development, the project team determined that some cases would pass the automatic case closure but should not. This problem occurred because the business rule to prevent the automatic closure was not identified during requirements development. Project management made the decision to not change the requirements until after SAT was complete. Without this business rule, cases requiring manual intervention could be given a favorable determination without meeting all of the criteria for automatic closure. This problem was controlled on the Information Technology Asset Management System (ITAMS), which is used to record and track problems identified during testing through their resolution. Proper control of this information will ensure the problems can be tracked and corrected after SAT.
Executing SAT
The TEDS project’s processes for conducting SAT were adequate to test the system requirements. We reviewed three aspects of the testing phase.
Test results verification
Verification of the test results by the project team was sufficient to identify whether errors or problems existed in the test case processing. Testing and verification included, but is not limited to, the following:
· Data entry and access control were tested at the Cincinnati Submission Processing Site.
· Processed data were tested for business rule evaluation and letter generation at the TE/GE Division’s Headquarters office.
To verify the sufficiency of the test cases, we evaluated the predetermined results of a judgmental sample of 13 test cases. The project team provided these test cases as examples of those that were (1) designed to either pass or fail the automatic closure business rules or (2) identified as those that will pass but should fail the business rules.
We found that the expected results for the 13 test cases had been accurately determined and, when compared to the processed data, would identify success or failure of the business rules. Success and failure were defined as whether the actual test results were the same as the predetermined results.
During testing, the testers used a systemically generated report that listed the actual results after the TEDS processed the data. This report was compared to the SAT Business Rule Summary, which identified the predetermined results, to determine success or failure of the business rules.
Problem identification, tracking and resolution
Overall, problem identification, tracking and resolution were adequate. When test cases are processed during SAT, problems should be identified, controlled, prioritized, and resolved. This is important because if the problems are not corrected, the system may be placed into production with defects or other uncorrected problems. Problems may range from errors in logic to keyboard behaviors (e.g., the tab key does not work).
We determined that when a problem was identified, it was entered into the ITAMS. The project team received a daily listing of the problems. They prioritized the problems and determined what course of action to take based on the severity of the error, the importance of the function tested, the affect on case processing if the error was not fixed, and the time and cost to fix the error. Courses of action included correcting the error prior to production, leaving the error until some determined time after going into production, providing either an automated or manual alternative, letting the error persist indefinitely, or changing the requirement.
After the problems were prioritized, they were given to the software development contractor for resolution. The testers, working with the development contractor programmers, identified where the problem occurred (i.e., which requirement or business rule failed) and explained the expected behavior of the requirement to the programmer so corrections could be made.
When the developers corrected a problem, they provided the corrected software to one individual who maintained the configuration control for the software. A transmittal was completed, and the DIO’s staff approved it before it and the corrected computer code were sent to the appropriate location for installation. The ITAMS was updated to record the transmittal of the corrected code. After installation, the requirement was retested to ensure that the corrective action worked, and the ITAMS was further updated to reflect the results of the retesting.
As stated earlier, we identified that problem resolution in some cases was being postponed until after completion of SAT. However, project management has properly controlled the problems so necessary actions can be taken to correct and test them after SAT but prior to going into production. For example, the menu option to update the Quality Assurance case status did not work correctly, and a print button was not always functioning correctly. Both of these problems have manual alternatives. Based on the controls the TE/GE Division has in place to track these problems, we are not making a recommendation about these postponed corrections.
Independence
Adequate independence between the test team and software development contractor was maintained to prevent test results from being manipulated. The ELC states in part that SAT should be performed independent of the organization that developed the system. Independent testing helps to ensure the validity of the testing. Because SAT was not performed by an organization totally independent of the designers and developers, an inherent risk existed that not all identified problems would be reported. Another risk factor is that it is in the development contractor’s best interest to have fewer problems identified during SAT. Without appropriate independence between the developers and the test results, it is possible that the results could be manipulated so that fewer actual problems are reported.
However, this risk was substantially reduced because the project team controlled the process for identifying and reporting problems during testing. The SAT team consisted of personnel provided by the TEDS project team and the DIO, as well as the PM/SI contractor (note: the PM/SI contractor played no role in the programming of the software). Accordingly, the development contractor did not have the ability to manipulate the test results. In addition, after completion of SAT, the DIO planned to provide TEDS project management with an End of Test Status Report identifying the status of testing. This will enable TEDS project management to ensure that problems identified during SAT are resolved prior to putting the TEDS into production.
Management’s Response: The Commissioner, TE/GE Division, agreed with our conclusion that the TEDS Release 1 SAT conformed to appropriate standards and was generally adequate to verify that the software would operate as intended.
Appendix I
Detailed Objective, Scope, and Methodology
Our overall objective was to
determine if the Tax Exempt and Government Entities
(TE/GE) Division management’s process for conducting the Tax Exempt Determination System (TEDS) Release 1
Systems Acceptability Testing (SAT) was adequate to assure management that the TEDS will correctly process Forms 5307.
To accomplish this objective, we performed the following work:
I. Reviewed the TEDS SAT plan to determine if it will fully evaluate the planned operational capabilities of the TEDS Release 1.
A.
Determined if the TEDS SAT
plan contained appropriate tests and methodology based on the Internal Revenue
Manual, the IRS Enterprise Life Cycle, and industry best practices.
B.
Determined if management had
established responsibilities for each action in the plan, along with describing
deliverables and due dates for each step.
C.
Determined if management had
developed a listing of all system requirements to be tested and if that list
included the following major functional requirements of the TEDS:
1.
Scanning of the Forms 5307.
2.
Automated determination case
closures based on the business rules.
3.
Accessibility of case
information in the TEDS by the TE/GE Division’s Customer Account Services.
4.
Generation of determination
letters for automatic case closures.
5.
Software interfaces with
external systems (e.g., software was designed to interface the Letter
Information Network User System with the TEDS to
capture fee information associated with Employee Plans and Exempt
Organization applications).
6.
Availability of the limited Quality Assurance
capabilities.
D.
Determined if the plan
included documented criteria for measuring and determining the success or
failure of the system to meet user requirements.
E.
Determined if processes and
guidelines had been included in the SAT plan to ensure independence in testing
and that test results could not be manipulated.
F.
Evaluated a judgmental sample of controlled data (13
test cases out of the total population of 200 test cases prepared by Internal
Revenue Service management) to determine if the cases would adequately test the
requirements of the system. We used a
judgmental sample because we did not intend to make any projections across the
population of test cases from the judgmental sample.
II. Determined if actual test results were compared to predetermined results and how any discrepancies were handled.
A.
Determined if the actual test
results were compared to the predetermined results.
B.
Determined if problems
identified in processing were tracked and resolved.
Appendix II
Major Contributors to This Report
Daniel R. Devlin, Assistant Inspector General
for Audit (Headquarters Operations and Exempt Organizations Programs)
Nancy A. Nakamura, Director
Gerald T. Hawkins, Audit Manager
Thomas F. Seidell, Acting Audit Manager
Barry G. Huff, Senior Auditor
Phung-Son H. Nguyen, Senior Auditor
Andrew J. Burns, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner for Services and Enforcement N:DC
Associate
Commissioner, Business Systems Modernization
M:B
Director, Business Systems Planning, Tax Exempt and Government Entities Division T:BSP
Chief, Information Technology Services M:I
Chief Counsel CC
National Taxpayer Advocate
TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk
Analysis N:ADC:R:O
Office of Management Controls N:CFO:AR:M
Liaisons:
Director, Communications and
Liaison, Tax Exempt and Government Entities Division
T:CL
Chief, Information Technology
Services M:I
Appendix IV
The Tax Exempt Determination System (TEDS) Release 1 consists of the receipt of Forms 5307 application packages and continues through the completion of the exempt determination processing for automatic closure. The system includes interfacing with the payment processing system, scanning and data storage of application packages, generating final determination letters, Quality Assurance, and error resolution.
Based on the TEDS Release 1 Physical Architecture (February 11, 2003), the following high-level functionality is to be delivered in Release 1 of the TEDS:
·
Receipt and control for
Employee Plans (EP) Revised Form 5307 application packages.
·
Automatic case closure for EP
Revised Form 5307 application packages.
·
Accessibility of case
information in the TEDS by Customer Account Services.
·
Generation of determination
letters for cases that have been automatically closed in the TEDS.
·
Interface with the Employee
Plans and Exempt Organization Determination System and
the Letter Information Network User System.
·
Limited Quality Assurance
capabilities.
Appendix V
Management’s Response to
the Draft Report
The response was removed due to
its size. To see the complete response,
please go to the Adobe PDF version of the report on the TIGTA Public Web Page.