Improvements Are Needed for Subsequent Integrated Financial
System Testing
March 2004
Reference Number:
2004-10-052
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
March
1, 2004
MEMORANDUM FOR
CHIEF FINANCIAL OFFICER
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Acting Deputy Inspector
General for Audit
SUBJECT: Final Audit Report - Improvements Are Needed for Subsequent
Integrated Financial System Testing (Audit
# 200310027)
This
report presents the limited results of our review of the Internal Revenue
Service’s (IRS) planned Integrated
Financial System (IFS). The overall
objective of this audit was to determine whether the IFS, when implemented,
would function as intended to provide administrative financial management
information that is essential for financial statement preparation as well as
useful to IRS managers and others.
Because the IRS rescheduled a material amount of the IFS testing, our
original audit scope was significantly reduced. This report presents our audit observations and evaluations
through the initial testing phase of the IFS.
The
implementation of the proposed IFS will greatly affect the way the IRS records
and reports both administrative and custodial accounting transactions. Further, this system represents a key
element of the corrective actions being taken by the IRS to ensure its
accounting system is in compliance with the Joint Financial Management
Improvement Program’s (JFMIP) Federal Financial Management System
Requirements and provides accurate and timely financial information for
management decision making.
In summary, the IFS project
has experienced delays in system testing which have caused the IRS to
reschedule the implementation of Release 1 from October 2003 to April 2004. Because of this, we were unable to fully
evaluate the functionality of the IFS-defined requirements or other
implementation activities. We were,
however, able to observe and evaluate a select number of Release 1 System
Integration Testing (SIT) Cycle 1 test cases and test scripts.
Based on our limited tests,
we determined that the IFS testing team developed test cases
that, for the most part, contained a set of conditions, data, and expected
results for a particular test objective; a corresponding test script to provide
a series of instructions to carry out the test case; and a test folder to
document the results of the testing process.
Further, the testing team prepared a Requirements Traceability
Verification Matrix (RTVM) in an effort to map accounting requirements to
the corresponding test cases. However,
we determined that some test cases and test scripts were incorrect or
incomplete and the RTVM did not always provide an accurate traceability of
requirements to be tested.
We recommended that the
Chief Financial Officer (CFO), in concert with IFS project management, ensure
that subsequent SIT test plans, cases, and scripts are complete and accurate
and that all applicable financial system requirements can be readily accounted
for during the testing process.
Management’s
Response: IRS management
agreed with the recommendation presented in the report. The
CFO has instituted a number of controls and procedures that have been applied
to subsequent SIT Cycles. The controls
and procedures include a concurrence process for each test plan that provides
multiple reviews, rewrites, and approval of test plans prior to execution; training
sessions for all SIT testers to explain general ledger postings and
requirements; and creation and maintenance of a current list of requirements
along with verification that all requirements are adequately tested. The RTVM has been reviewed and validated by
the IRS and is reviewed weekly to ensure all requirements are properly mapped
to test plans and all changes are reflected in the document. Additionally, the IRS Subject Matter Experts
will reverify that requirements are mapped properly and have been met by the
functionality tested before the test plan is approved. An approval signature is required on each
test case for it to be considered passed.
Management’s complete response to the draft report is included as Appendix IV.
Copies of this
report are also being sent to the IRS managers who are affected by the report
recommendation. Please contact me at
(202) 622-6510 if you have questions or Daniel R. Devlin, Assistant Inspector
General for Audit (Headquarters Operations and Exempt Organizations Programs),
at (202) 622-8500.
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix IV
– Management’s Response to the Draft Report
The Internal Revenue Service (IRS) is in the midst of Business Systems Modernization (BSM), a major technological and business process transformation program. BSM will radically change the IRS’ approach to satisfying its customers – the taxpayers.
To achieve legislative compliance and provide critical financial management information, the IRS selected a commercial-off-the-shelf software package, designated as the Integrated Financial System (IFS), as a cost-effective alternative to improve its financial systems. One basic, yet significant, requirement of the IFS is that the system is to be compliant with the Joint Financial Management Improvement Program’s (JFMIP) Federal Financial Management System Requirements. The IRS also expects the IFS to resolve several longstanding financial management issues identified by the General Accounting Office (GAO) during the annual financial statement audits.
The IRS plans to implement
the IFS in at least four releases over a period of several years. Release 1 will include the JFMIP core
accounting functions of Accounts Payable, Accounts Receivable, General Ledger
Management, Budget Execution, Cost Management (also referred to as Cost Accounting),
and Financial Reporting, as well as
Budget Formulation. Release 1 will also include the mandated
Health Coverage Tax Credit and Custodial Accounting functionality. Subsequent releases will provide for
additional financial management functions, including Property, Procurement, and
Performance Management, and enhancements to the cost accounting and finance
modules.
To increase the probability that the IFS will function as intended and meet defined requirements, the IRS is working with a contractor to conduct System Integration Testing (SIT) prior to deploying the IFS. System testing is an essential component of the Enterprise Life Cycle, which governs system development activities and is required by IRS procedures. The SIT test plan provides that all test cases will describe the purpose of the test, any pre-conditions or post-conditions, the requirements to be verified, and the satisfaction criteria. The satisfaction criteria will state the conditions required for a test case to pass. Before testing can begin, test cases and scenarios should be developed, reviewed, and approved, and system requirements should be traced to test cases.
This on-line audit is a follow-on review to our prior audit of the IFS requirements definition that was issued in August 2003. The audit was conducted in accordance with Government Auditing Standards at the IRS National Headquarters in the office of the Chief Financial Officer (CFO) and the contractor’s testing site located in Lanham, Maryland, from April through September 2003. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The IRS has rescheduled the implementation of the IFS Release 1 from October 2003 to April 2004, due in part to delays in system testing. Thus, we were unable to fully evaluate the functionality of IFS-defined requirements or other implementation activities. Accordingly, we limited the scope of this audit to observation and evaluation of SIT Cycle 1 test cases and scripts. We will continue to review the IRS’ subsequent test phases to fully accomplish our original objective of determining whether the IFS will provide administrative financial management information essential to financial statement preparation and management decision making.
The IFS system testing was split into two cycles. The SIT Cycle 1 testing was very basic and involved the creation of preliminary data on the system, the change and display of that data, and the recordation of simple financial transactions. The purpose of SIT Cycle 1 testing, as described to us by IRS officials and contractors, was to allow testers to become familiar with the testing process and environment to facilitate a more focused SIT Cycle 2 testing.
Conversely,
the SIT Cycle 2 testing is very comprehensive and includes test transactions
run with real data from the initial requesting entry to final payment. Further, SIT Cycle 2 testing retests most,
if not all, of the tests from Cycle 1.
The purpose of SIT Cycle 2 testing is to provide conclusions concerning
the new system’s functionality and to give assurances the system will work as
intended.
We selected nine financial requirement areas for evaluation during the IFS testing phase based on findings reported in the IRS’ financial statement audit reports, the GAO Standards for Internal Control in the Federal Government, and overall functionality risk. We observed and evaluated a selection of SIT Cycle 1 test cases and results corresponding to those areas; they are listed in Appendix I.
Based on our limited tests, we determined the IFS testing team developed test cases that, for the most part, contained a set of conditions, data, and expected results for a particular test objective; a corresponding test script to provide a series of instructions to carry out the test case; and a test folder to document the results of the testing process. Further, the testing team prepared a Requirements Traceability Verification Matrix (RTVM) in an effort to map accounting requirements to the corresponding test cases.
However, we determined some test cases and test scripts were incorrect or incomplete and the RTVM did not always provide an accurate traceability of requirements to be tested. Specifically, at least 1 of the following 9 conditions was observed in 1 or more of the 23 SIT Cycle 1 test cases reviewed:
· Test cases did not identify the post-condition or satisfaction criteria to be met.
We identified several financial management activities or transactions that could be adversely affected by these incomplete or inaccurate test processes. For example:
· The IFS test developers did not include the satisfaction criteria for the following test case scenarios: capturing Federal income tax-related data, posting user fee deposits, posting depreciation to individual cost centers, and recording audit trail information.
· The test developers included questionable or incomplete USSGL accounts for the following test scripts: posting expenses and capitalized assets in the same transaction, posting user fee deposits, and using cancelled appropriations.
Without explicit, well-defined satisfaction criteria and the identification of specific USSGL account information, the testers may be hindered from fully completing the test as intended and may not accurately evaluate test results. By following defined testing processes, the IRS can better ensure the IFS will function as intended to provide administrative financial management information that is essential for financial statement preparation and management decision making.
We recognize that the IRS and contractor are continually reassessing and, where necessary, modifying IFS tests as system development activities proceed toward final implementation. Therefore, we are providing these observations in this report to assist management in identifying conditions that should be avoided or anticipated as the IRS moves into subsequent SIT testing phases.
We have also shared these audit results with our Information Systems Program audit staff, who are responsible for evaluating systems modernization from an information technology perspective, and who have previously reported similar concerns about testing from both a project-specific and enterprise-wide viewpoint. We will continue to coordinate our audit efforts on the IFS during our subsequent reviews.
Successful implementation of
the IFS will require the collective and collaborative efforts of the CFO,
Business Systems Modernization Office, and contractor. Ultimately, the CFO, as the IRS executive
owner of the IFS, has the fiduciary responsibility to ensure that the IFS will
function as intended. Therefore,
complete and accurate testing of the IFS plays a critical role in the CFO’s
discharge of that responsibility.
1. The CFO, in concert with IFS project management, should ensure that subsequent SIT test plans, cases, and scripts are complete and accurate and that all applicable financial system requirements can be readily accounted for during the testing process.
Management’s Response: The CFO has instituted a number of controls and procedures that have been applied to subsequent SIT Cycles. The controls and procedures include a concurrence process for each test plan that provides multiple reviews, rewrites, and approval of test plans prior to execution; training sessions for all SIT testers to explain general ledger postings and requirements; and creation and maintenance of a current list of requirements along with verification that all requirements are adequately tested. The RTVM has been reviewed and validated by the IRS and is reviewed weekly to ensure all requirements are properly mapped to test plans and all changes are reflected in the document. Additionally, the IRS Subject Matter Experts will reverify that requirements are mapped properly and have been met by the functionality tested before the test plan is approved. An approval signature is required on each test case for it to be considered passed.
Appendix I
Detailed Objective, Scope,
and Methodology
We originally planned to determine whether the Integrated Financial System (IFS), when implemented, would function as intended to provide administrative financial management information that is essential for financial statement preparation as well as useful to Internal Revenue Service (IRS) managers and others. However, IFS testing efforts were delayed, and we were unable to complete the initial aspects of our original plan. The scope of this audit was limited to observation and evaluation of IFS System Integration Testing (SIT) Cycle 1 testing of Release 1 requirements for the administrative accounting activities of the IFS, as follows:
I. Selected a judgmental sample of nine key Joint Financial Management Improvement Program (JFMIP) financial requirement areas, based on findings reported in the IRS’ financial statement audit reports, the General Accounting Office (GAO) Standards for Internal Control in the Federal Government, and overall functionality risk. The areas included:
· Fixed Assets.
· Receipt and Acceptance.
· Reimbursable Work Agreements.
· Multi-Year Reporting.
· Cost Accounting.
· Travel Advances and Obligations.
· Taxable Transactions.
· Financial and Treasury Information Executive Repository Reporting.
· Audit Trail.
II. For each of the nine JFMIP financial requirement areas, we identified relevant:
· JFMIP standards.
· Statement of Federal Financial Accounting Statements standards, technical releases, or exposure drafts.
· Department of the Treasury requirements.
· Office of Management and Budget guidance.
· GAO concerns from either its financial statement audits or its High Risk series.
· Private sector Generally Accepted Accounting Principles.
· IFS System Requirements Report requirements.
· IFS financial management processes.
III. Reviewed the major SIT Cycle 1 test documents, which included the Test Plan, the Test Schedule, the file of test cases, and the Requirements Traceability Verification Matrix (RTVM) that maps accounting requirements to the corresponding test cases.
IV. Interviewed IRS Subject Matter Experts involved with the Computer Lab operation to define their role in the implementation process.
V. Reviewed 23 test cases that related to the 9 JFMIP financial requirement areas shown in Step I. above, out of the total of 168 SIT Cycle 1 test cases, to identify administrative compliance with test development criteria (e.g., requirements listed, post-condition listed, correct/complete general ledger postings, etc.). We observed the actual tests being conducted for 11 of the 23 test cases and reviewed the corresponding test folders. Also, we reviewed the RTVM for traceability to individual test cases and general completeness.
Appendix II
Major Contributors to This Report
Daniel R. Devlin, Assistant Inspector General
for Audit (Headquarters Operations and Exempt Organizations Programs)
John R. Wright, Director
Thomas J. Brunetto, Audit Manager
Bobbie M. Draudt, Senior Auditor
S. Kent Johnson, Senior Auditor
Gwenevere Bryant-Hill, Auditor
Chinita Coates, Auditor
Richard E. Louden, Auditor
Peter L. Stoughton, Auditor
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Chief, Agency-Wide Shared Services OS:A
Chief Information Officer OS:CIO
Director, Procurement OS:A:P
Chief Counsel CC
National Taxpayer Advocate
TA
Director, Office of Legislative Affairs CL:LA
Director, Office of
Program Evaluation and Risk Analysis
RAS:O
Office of Management Controls OS:CFO:AR:M
Audit Liaison: Chief Financial Officer OS:CFO
Appendix IV
Management’s
Response to the Draft Report
The response was removed due to its size. To see the response, please go to the Adobe
PDF version of the report on the TIGTA Public Web Page.