Improvements Are Needed for Subsequent Integrated Financial System Testing

 

March 2004

Reference Number:2004-10-052

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

March 1, 2004

 

 

MEMORANDUM FOR CHIEF FINANCIAL OFFICER

 

FROM:†††† Gordon C. Milbourn III /s/ Gordon C. Milbourn III

†††††††††††††††† Acting Deputy Inspector General for Audit

 

SUBJECT:†††† Final Audit Report - Improvements Are Needed for Subsequent Integrated Financial System Testing(Audit # 200310027)

 

This report presents the limited results of our review of the Internal Revenue Serviceís (IRS) planned Integrated Financial System (IFS). The overall objective of this audit was to determine whether the IFS, when implemented, would function as intended to provide administrative financial management information that is essential for financial statement preparation as well as useful to IRS managers and others.Because the IRS rescheduled a material amount of the IFS testing, our original audit scope was significantly reduced.This report presents our audit observations and evaluations through the initial testing phase of the IFS.

The implementation of the proposed IFS will greatly affect the way the IRS records and reports both administrative and custodial accounting transactions.Further, this system represents a key element of the corrective actions being taken by the IRS to ensure its accounting system is in compliance with the Joint Financial Management Improvement Programís (JFMIP) Federal Financial Management System Requirements and provides accurate and timely financial information for management decision making.

In summary, the IFS project has experienced delays in system testing which have caused the IRS to reschedule the implementation of Release 1 from October 2003 to April 2004.Because of this, we were unable to fully evaluate the functionality of the IFS-defined requirements or other implementation activities.We were, however, able to observe and evaluate a select number of Release 1 System Integration Testing (SIT) Cycle 1 test cases and test scripts.

Based on our limited tests, we determined that the IFS testing team developed test cases that, for the most part, contained a set of conditions, data, and expected results for a particular test objective; a corresponding test script to provide a series of instructions to carry out the test case; and a test folder to document the results of the testing process.Further, the testing team prepared a Requirements Traceability Verification Matrix (RTVM) in an effort to map accounting requirements to the corresponding test cases.However, we determined that some test cases and test scripts were incorrect or incomplete and the RTVM did not always provide an accurate traceability of requirements to be tested.

We recommended that the Chief Financial Officer (CFO), in concert with IFS project management, ensure that subsequent SIT test plans, cases, and scripts are complete and accurate and that all applicable financial system requirements can be readily accounted for during the testing process.

Managementís Response:IRS management agreed with the recommendation presented in the report.The CFO has instituted a number of controls and procedures that have been applied to subsequent SIT Cycles.The controls and procedures include a concurrence process for each test plan that provides multiple reviews, rewrites, and approval of test plans prior to execution; training sessions for all SIT testers to explain general ledger postings and requirements; and creation and maintenance of a current list of requirements along with verification that all requirements are adequately tested.The RTVM has been reviewed and validated by the IRS and is reviewed weekly to ensure all requirements are properly mapped to test plans and all changes are reflected in the document.Additionally, the IRS Subject Matter Experts will reverify that requirements are mapped properly and have been met by the functionality tested before the test plan is approved.An approval signature is required on each test case for it to be considered passed.Managementís complete response to the draft report is included as Appendix IV.

Copies of this report are also being sent to the IRS managers who are affected by the report recommendation.Please contact me at (202) 622-6510 if you have questions or Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs), at (202) 622-8500.

 

Table of Contents

Background

Integrated Financial System Implementation Has Been Rescheduled to Complete System Testing and Development

Some Test Cases and Scripts Were Incomplete or Inaccurate, and Some System Requirements Could Not Be Easily Traced

Recommendation 1:

Appendix I Ė Detailed Objective, Scope, and Methodology

Appendix II Ė Major Contributors to This Report

Appendix III Ė Report Distribution List

Appendix IV Ė Managementís Response to the Draft Report

 

Background

The Internal Revenue Service (IRS) is in the midst of Business Systems Modernization (BSM), a major technological and business process transformation program.BSM will radically change the IRSí approach to satisfying its customers Ė the taxpayers.

To achieve legislative compliance and provide critical financial management information, the IRS selected a commercial-off-the-shelf software package, designated as the Integrated Financial System (IFS), as a cost-effective alternative to improve its financial systems.One basic, yet significant, requirement of the IFS is that the system is to be compliant with the Joint Financial Management Improvement Programís (JFMIP) Federal Financial Management System Requirements.The IRS also expects the IFS to resolve several longstanding financial management issues identified by the General Accounting Office (GAO) during the annual financial statement audits.

The IRS plans to implement the IFS in at least four releases over a period of several years.Release 1 will include the JFMIP core accounting functions of Accounts Payable, Accounts Receivable, General Ledger Management, Budget Execution, Cost Management (also referred to as Cost Accounting), and Financial Reporting, as well as Budget Formulation.Release 1 will also include the mandated Health Coverage Tax Credit and Custodial Accounting functionality.Subsequent releases will provide for additional financial management functions, including Property, Procurement, and Performance Management, and enhancements to the cost accounting and finance modules.

To increase the probability that the IFS will function as intended and meet defined requirements, the IRS is working with a contractor to conduct System Integration Testing (SIT) prior to deploying the IFS.System testing is an essential component of the Enterprise Life Cycle, which governs system development activities and is required by IRS procedures.The SIT test plan provides that all test cases will describe the purpose of the test, any pre-conditions or post-conditions, the requirements to be verified, and the satisfaction criteria.The satisfaction criteria will state the conditions required for a test case to pass.Before testing can begin, test cases and scenarios should be developed, reviewed, and approved, and system requirements should be traced to test cases.

This on-line audit is a follow-on review to our prior audit of the IFS requirements definition that was issued in August 2003.The audit was conducted in accordance with Government Auditing Standards at the IRS National Headquarters in the office of the Chief Financial Officer (CFO) and the contractorís testing site located in Lanham, Maryland, from April through September 2003.Detailed information on our audit objective, scope, and methodology is presented in Appendix I.Major contributors to the report are listed in Appendix II.

Integrated Financial System Implementation Has Been Rescheduled to Complete System Testing and Development

The IRS has rescheduled the implementation of the IFS Release 1 from October 2003 to April 2004, due in part to delays in system testing.Thus, we were unable to fully evaluate the functionality of IFS-defined requirements or other implementation activities.Accordingly, we limited the scope of this audit to observation and evaluation of SIT Cycle 1 test cases and scripts.We will continue to review the IRSí subsequent test phases to fully accomplish our original objective of determining whether the IFS will provide administrative financial management information essential to financial statement preparation and management decision making.

The IFS system testing was split into two cycles.The SIT Cycle 1 testing was very basic and involved the creation of preliminary data on the system, the change and display of that data, and the recordation of simple financial transactions.The purpose of SIT Cycle 1 testing, as described to us by IRS officials and contractors, was to allow testers to become familiar with the testing process and environment to facilitate a more focused SIT Cycle 2 testing.

Conversely, the SIT Cycle 2 testing is very comprehensive and includes test transactions run with real data from the initial requesting entry to final payment.Further, SIT Cycle 2 testing retests most, if not all, of the tests from Cycle 1.The purpose of SIT Cycle 2 testing is to provide conclusions concerning the new systemís functionality and to give assurances the system will work as intended.

Some Test Cases and Scripts Were Incomplete or Inaccurate, and Some System Requirements Could Not Be Easily Traced

We selected nine financial requirement areas for evaluation during the IFS testing phase based on findings reported in the IRSí financial statement audit reports, the GAO Standards for Internal Control in the Federal Government, and overall functionality risk.We observed and evaluated a selection of SIT Cycle 1 test cases and results corresponding to those areas; they are listed in Appendix I.

Based on our limited tests, we determined the IFS testing team developed test cases that, for the most part, contained a set of conditions, data, and expected results for a particular test objective; a corresponding test script to provide a series of instructions to carry out the test case; and a test folder to document the results of the testing process.Further, the testing team prepared a Requirements Traceability Verification Matrix (RTVM) in an effort to map accounting requirements to the corresponding test cases.

However, we determined some test cases and test scripts were incorrect or incomplete and the RTVM did not always provide an accurate traceability of requirements to be tested.Specifically, at least 1 of the following 9 conditions was observed in 1 or more of the 23 SIT Cycle 1 test cases reviewed:

        Test cases did not identify the post-condition or satisfaction criteria to be met.

We identified several financial management activities or transactions that could be adversely affected by these incomplete or inaccurate test processes.For example:

        The IFS test developers did not include the satisfaction criteria for the following test case scenarios:capturing Federal income tax-related data, posting user fee deposits, posting depreciation to individual cost centers, and recording audit trail information.

        The test developers included questionable or incomplete USSGL accounts for the following test scripts:posting expenses and capitalized assets in the same transaction, posting user fee deposits, and using cancelled appropriations.

Without explicit, well-defined satisfaction criteria and the identification of specific USSGL account information, the testers may be hindered from fully completing the test as intended and may not accurately evaluate test results.By following defined testing processes, the IRS can better ensure the IFS will function as intended to provide administrative financial management information that is essential for financial statement preparation and management decision making.

We recognize that the IRS and contractor are continually reassessing and, where necessary, modifying IFS tests as system development activities proceed toward final implementation.Therefore, we are providing these observations in this report to assist management in identifying conditions that should be avoided or anticipated as the IRS moves into subsequent SIT testing phases.

We have also shared these audit results with our Information Systems Program audit staff, who are responsible for evaluating systems modernization from an information technology perspective, and who have previously reported similar concerns about testing from both a project-specific and enterprise-wide viewpoint.We will continue to coordinate our audit efforts on the IFS during our subsequent reviews.

Successful implementation of the IFS will require the collective and collaborative efforts of the CFO, Business Systems Modernization Office, and contractor.Ultimately, the CFO, as the IRS executive owner of the IFS, has the fiduciary responsibility to ensure that the IFS will function as intended.Therefore, complete and accurate testing of the IFS plays a critical role in the CFOís discharge of that responsibility.

Recommendation

1.      The CFO, in concert with IFS project management, should ensure that subsequent SIT test plans, cases, and scripts are complete and accurate and that all applicable financial system requirements can be readily accounted for during the testing process.

Managementís Response:The CFO has instituted a number of controls and procedures that have been applied to subsequent SIT Cycles.The controls and procedures include a concurrence process for each test plan that provides multiple reviews, rewrites, and approval of test plans prior to execution; training sessions for all SIT testers to explain general ledger postings and requirements; and creation and maintenance of a current list of requirements along with verification that all requirements are adequately tested.The RTVM has been reviewed and validated by the IRS and is reviewed weekly to ensure all requirements are properly mapped to test plans and all changes are reflected in the document.Additionally, the IRS Subject Matter Experts will reverify that requirements are mapped properly and have been met by the functionality tested before the test plan is approved.An approval signature is required on each test case for it to be considered passed.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

We originally planned to determine whether the Integrated Financial System (IFS), when implemented, would function as intended to provide administrative financial management information that is essential for financial statement preparation as well as useful to Internal Revenue Service (IRS) managers and others.However, IFS testing efforts were delayed, and we were unable to complete the initial aspects of our original plan.The scope of this audit was limited to observation and evaluation of IFS System Integration Testing (SIT) Cycle 1 testing of Release 1 requirements for the administrative accounting activities of the IFS, as follows:

I.                    Selected a judgmental sample of nine key Joint Financial Management Improvement Program (JFMIP) financial requirement areas, based on findings reported in the IRSí financial statement audit reports, the General Accounting Office (GAO) Standards for Internal Control in the Federal Government, and overall functionality risk.The areas included:

        Fixed Assets.

        Receipt and Acceptance.

        Reimbursable Work Agreements.

        Multi-Year Reporting.

        Cost Accounting.

        Travel Advances and Obligations.

        Taxable Transactions.

        Financial and Treasury Information Executive Repository Reporting.

        Audit Trail.

II.                 For each of the nine JFMIP financial requirement areas, we identified relevant:

        JFMIP standards.

        Statement of Federal Financial Accounting Statements standards, technical releases, or exposure drafts.

        Department of the Treasury requirements.

        Office of Management and Budget guidance.

        GAO concerns from either its financial statement audits or its High Risk series.

        Private sector Generally Accepted Accounting Principles.

        IFS System Requirements Report requirements.

        IFS financial management processes.

III.               Reviewed the major SIT Cycle 1 test documents, which included the Test Plan, the Test Schedule, the file of test cases, and the Requirements Traceability Verification Matrix (RTVM) that maps accounting requirements to the corresponding test cases.

IV.              Interviewed IRS Subject Matter Experts involved with the Computer Lab operation to define their role in the implementation process.

V.                 Reviewed 23 test cases that related to the 9 JFMIP financial requirement areas shown in Step I. above, out of the total of 168 SIT Cycle 1 test cases, to identify administrative compliance with test development criteria (e.g., requirements listed, post-condition listed, correct/complete general ledger postings, etc.).We observed the actual tests being conducted for 11 of the 23 test cases and reviewed the corresponding test folders.Also, we reviewed the RTVM for traceability to individual test cases and general completeness.

 

Appendix II

 

Major Contributors to This Report

 

Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

John R. Wright, Director

Thomas J. Brunetto, Audit Manager

Bobbie M. Draudt, Senior Auditor

S. Kent Johnson, Senior Auditor

Gwenevere Bryant-Hill, Auditor

Chinita Coates, Auditor

Richard E. Louden, Auditor

Peter L. Stoughton, Auditor

 

Appendix III

 

Report Distribution List

 

CommissionerC

Office of the Commissioner Ė Attn:Chief of StaffC

Deputy Commissioner for Operations SupportOS

Chief, Agency-Wide Shared ServicesOS:A

Chief Information OfficerOS:CIO

Director, ProcurementOS:A:P

Chief CounselCC

National Taxpayer AdvocateTA

Director, Office of Legislative AffairsCL:LA

Director, Office of Program Evaluation and Risk AnalysisRAS:O

Office of Management ControlsOS:CFO:AR:M

Audit Liaison:Chief Financial OfficerOS:CFO

 

Appendix IV

 

Managementís Response to the Draft Report

 

The response was removed due to its size.To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.