REVIEW OF THE INTEGRATED SUBMISSION
AND REMITTANcE PROcESSING (ISRP)
SYSTEM SOFTWARE DEVELOPMENT AND
PILOT AcTIVITIES
Reference No. 090903 Date: November 6, 1998
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Detailed Objectives and Scope of Review
ISRP System Pre-Pilot Development Milestones
Analysis of the ISRP Pilot Production Results
Management's Response to Internal Audit Memorandum #1
Management's Response to Internal Audit Memorandum #2
Management's Response to Internal Audit Memorandum #3
On August 22, 1996, the Service’s Investment Review Board (IRB) approved the initial Business case to replace the existing Distributed Input System (DIS) and the Remittance Processing System (RPS). The project was later named the Integrated Submission and Remittance Processing (ISRP) system. Replacement of legacy DIS and RPS, the primary data input systems for processing paper submissions, with ISRP is critical. The legacy systems are 13 and 20 years old respectively, and neither is capable of processing dates beyond the year 1999.
On January 30, 1998, we issued a report on the Initial System Development Activities of the ISRP system (Report #082204), identifying the aggressive rollout schedule, the absence of contingency plans, and increased development risk for the Residual Remittance Processing System (RRPS) functionality. In this review, we assessed the decisions and activities regarding the design, development, and installation of the ISRP pilot system to determine if they were complete and reliable.
The ISRP systems development project is one of four critical Information Systems projects monitored monthly by the commissioner’s Year 2000 Executive Steering committee. We initiated our review based on the Service’s mission critical need for a year 2000 compliant system to process the taxpayer's paper submissions. The audit was performed in accordance with generally accepted government auditing standards. We plan to continue to evaluate the system rollout in future audits.
Results
The pilot produced mixed results and risks remain for successful implementation of a nationwide production system. From the start-up of the pilot on February 9, 1998, through June 30, 1998, the pilot processed 4.1 million tax returns and 2.3 million payments. Although not all functionality was delivered, tested, or working at the inception of the pilot, management reported that the Austin Service center (AUSc) did meet both the April deposit program completion date (PcD) and May other-than-full-paid individual tax return PcD. However, the RRPS did not demonstrate consistent reliability as a production system, and neither ISRP subsystem conclusively demonstrated anticipated productivity gains.
Management appropriately elected to mitigate the risk associated with a nationwide rollout by limiting the RRPS rollout for the 1999 processing year to five additional service centers. We concur with management’s actions to control risks associated with the nationwide rollout. However, there are additional system design, project scheduling and resource allocation risks, which management must also continue to monitor and address to ensure successful nationwide implementation before January 1, 2000.
System Design Risks
Software development and hardware configuration issues affect the stability of the RRPS and its ability to meet all contractual and operational production requirements. (See page 5 for details.)
As of the Preliminary Pilot Review, the Service had not yet accepted the ISRP pilot system because it had not meet the contract's 99 percent effectiveness requirement and RRPS remittance processing requirements.
Work process changes to accommodate incompatible enhancements increase the project's non-technical system design risk. For example:
Once programming changes for all non-RPS processed remittances are made, we will follow up to quantify the number of taxpayers who would have received erroneous notices and /or refunds. Management plans to implement a partial corrective action for the 1999 processing season. We plan to follow-up and review the effectiveness of this corrective action.
Project Scheduling Risks
The project office's strong configuration management controls have proactively identified and/or mitigated many of the project's scheduling risks, such as the effect of contractor delivery delays on system testing activities. However, continued delays in software delivery by the contractor require management to continue its strong executive oversight to ensure that the system is adequately developed and tested prior to the final nationwide implementation. (See page 13 for details.)
Resource Allocation Risks
Productivity gains established in the project's initial business case were the basis for projected labor savings and comprise approximately 34 percent of the overall benefit of the Service's investment in the ISRP system. Since the projected benefits of major information technology investments are closely integrated into the Service's financial planning and budgeting activities, it is important that the Service evaluate the productivity gains of ISRP operations closely during the 1999 filing season. (See page 16 for details.)
To provide management with assessments of the project's system development risks and recommendations to mitigate those risks, we issued Internal Audit Memorandums on the interim results of our review. While management did not always agree with our assessments and conclusions, they took or initiated appropriate corrective actions to most of our recommendations. As a result, we have not restated all of our recommendations in this report. Instead, we have highlighted the risks which management must continue to mitigate to ensure a successful implementation. Because this was an on-line audit, the recommendations, which were not immediately addressed, were subsequently resolved or acknowledged by management during pilot operations. copies of management’s responses to our memorandums are included as Attachments IV, V, and VI. We will continue to provide audit coverage as the ISRP implementation progresses.
The Integrated Submission and Remittance Processing (ISRP) systems development project is one of four critical Information Systems projects monitored monthly by the commissioner’s Year 2000 Executive Steering committee. We initiated the review based on the Service’s mission critical need for a year 2000 compliant system to process the taxpayer's paper submissions. The audit was performed in accordance with generally accepted government auditing standards.
The overall objective of this audit was to determine if the decisions and activities regarding the design, development, and installation of the ISRP systems were complete and reliable. We concentrated our audit tests on:
We conducted audit work from November 4, 1997, through June 30, 1998, at the National Office and the Austin Service center (AUSc), which hosted the ISRP pilot. Details of our audit objectives and scope of review are presented in Attachment I.
On August 22, 1996, the Service’s Investment Review Board (IRB) approved the initial Business case to replace the existing Distributed Input System (DIS) and the Remittance Processing System (RPS). These systems are the Service’s core components for paper return and payment input processing. The systems handle up to 170 million paper returns and 50 million paper remittances per year.
On December 4, 1996, the IRB approved the project’s acquisition strategy, established the DIS/RPS Replacement Project Office, and authorized the project’s immediate funding requirements. The combined replacement system was later named the Integrated Submission and Remittance Processing (ISRP) system. See Attachment II for a chronology of key pre-pilot development milestones.
Replacement of legacy DIS and RPS with ISRP is critical to the mission of the Service since they are the primary data input systems for processing paper submissions. The legacy DIS and RPS are 13 and 20 years old respectively, and neither is capable of processing dates beyond the year 1999. Both have outlived their originally designed system life. For these reasons it is incumbent on the Service to fully test and ensure the reliability of the ISRP system prior to nationwide installation and year 2000 operations.
The replacement plan began on December 20, 1996, and scheduled the development of the system in two increments with the completion of the nationwide rollout by January 1, 1999. Increment 1 technical requirements include all RPS functionality and the DIS applications for Forms 1040 family, 940, and 1120S. All other form specific DIS applications were scheduled for delivery in Increment 2.
To accommodate the aggressive schedule in a very condensed development period, the design and build stages of the project have run concurrently and the prototype stage for the RPS replacement was omitted. As a result, the project’s original statement of work has required numerous revisions because functional requirements were omitted.
To better manage these configuration changes, minimize cost increases, and facilitate formal communications with the contractor, the Service established an ISRP configuration control Board (ccB) as a single point of authority for system design changes. The ccB is comprised of representatives from Information Systems, Operations, Acquisition and Procurement, as well as, the contractor. Through the configuration management of the ccB, the Service and the contractor have bilaterally agreed to and definitized the statement of work.
On January 30, 1998, we released a report on the Initial System Development Activities of the ISRP system (Report #082204), identifying the aggressive rollout schedule, absence of contingency plans, and increased development risk for the Residual Remittance Processing System (RRPS) functionality. The report's recommendations included the re-assessment of RPS enhancements (e.g. digital imaging, image retrieval, and image archiving) and the development of contingency plans should the system not be ready for nationwide rollout. Management agreed with these recommendations and implemented contingency planning. They also tasked an independent contractor to study the ISRP Image Storage and Retrieval Subsystem, but the results of this study were not available prior to the start-up of pilot operations and are not yet complete.
The pilot produced mixed results and risks remain for successful implementation of a nationwide production system. From start-up on February 9, 1998, through June 30, 1998, the pilot processed 4.1 million tax returns and 2.3 million payments. Although not all functionality was delivered, tested, or working at the inception of the pilot, management reported that AUSc met both the April deposit program completion date (PcD) and May other-than-full-paid (OTFP) individual tax return processing PcD. However, the Business case projected that the DIS and RRPS portions of ISRP would produce a minimum of 10 percent and 25 percent productivity gains over legacy operations, respectively. The RRPS has not yet demonstrated consistent reliability as a production system, and neither ISRP subsystem conclusively demonstrated these gains during AUSc pilot operations. (See Attachment III for an analysis of the pilot production results.)
Management appropriately elected to mitigate risk by implementing contingency plans.
Because the ISRP RRPS functionality did not fully meet expectations, management appropriately elected to mitigate the risk associated with a nationwide rollout by limiting the RRPS rollout for the 1999 processing year to five additional service centers. The ISRP DIS will roll out to all sites for the 1999 processing year.
We concur with management’s actions to control risks associated with the nationwide rollout. However, there are additional risks which management must also continue to monitor and address to ensure successful implementation before January 1, 2000. Many of these risks were discussed in Internal Audit memorandums issued throughout the review.
System Design Risks
Project Scheduling Risks
Resource Allocation Risks
The current ISRP system configuration has not yet demonstrated overall systemic stability.
On May 18, 1998, we reported our concerns that the RRPS functionality had not yet demonstrated processing reliability. Specifically, the RRPS pilot at AUSc:
Management comment: Management agreed with our assessment of the image archive system and acknowledged that the RRPS had experienced several systemic problems during Systems Acceptability Tests (SAT) and AUSc pilot operation. They responded that since the last week of April 1998, RRPS remained stabilized and functioned well throughout the period of peak processing. Also, AUSc was provided latitude in the area of deposit requirements since the primary purpose of the pilot was to test the system as much as possible.
Throughout pilot operations, the ISRP project office has implemented strong configuration management controls. These controls include conducting incremental development reviews and requiring the resolution of the most critical risks before continuing into the next system development phase. While continuing to control system development, management must also ensure the following problems/risks are addressed before the ISRP system is fully implemented nationwide:
Account for and resolve all problem reports.
RRPS experienced problems posting transaction data to the Master File and numerous production interruptions throughout pilot operations. AUSc continues to generate trouble tickets regarding the operations of the RRPS.
Ensure that the ISRP system meets all contractual and operational production requirements.
On May 6, 1998, the ISRP project office reported that the Service had not yet accepted the ISRP system because:
While RRPS was the service center's primary deposit system, AUSc was unable to consistently meet the Service’s minimum deposit standards. Specifically:
Ensure the reliability of the image archive sub-system.
To assess the overall stability of RRPS, the performance of all of its sub-systems must be considered. On May 18, 1998, we reported that:
Since completing the IMF peak deposit season, AUSc has generated additional trouble tickets regarding the operations of the RRPS image archive sub-system.
At the conclusion of the May 6, 1998, Preliminary Pilot Review, management identified 10 critical issues impacting the rollout decision. These ten issues were subsequently reported to the Service’s Executive Steering committee (ESc). At least five of these issues relate directly to RRPS operations; and on July 14, 1998, management reported to the Executive Steering committee that the vendor had acknowledged reliability and stability problems regarding the RRPS image archive sub-system.
RRPS transaction processing changes circumvent legacy RPS Unpostable controls.
On February 4, 1998, we reported to management that work process changes required to accommodate the RRPS architecture would circumvent existing Unpostable controls designed to prevent the issuance of erroneous balance due notices and non-rebate refunds. Outstanding risks for erroneous notices and refunds are discussed below.
Existing legacy controls ensure that remittance transactions are associated with their payment documents.
When legacy RPS processes remittances received with a tax return, the corresponding transactions are coded with RPS indicators.
The Individual Master File (IMF) Unpostable condition (UPc) 140 and Business Master File (BMF) UPc 399 are based upon the matching of the document locator numbers of returns and payments. These Unpostable conditions prohibit the posting of RPS coded returns unless a matching payment transaction has posted to the Master File. This control prevents:
ISRP Residual Remittance Processing System (RRPS) transactions are similar to the remittance transactions processed through the Service’s Lockbox program. Neither the Lockbox program nor ISRP RRPS supports the legacy RPS transaction coding process. The legacy DIS does not set the RPS indicator or assign a matching DLN to the returns with remittances processed through Lockbox banks. Similarly, the original procedures for transcribing returns with remittances processed through ISRP RRPS did not instruct data transcribers to set the RPS indicator.
To determine the potential effect of this condition, we reviewed over 90,000 reversed payment transactions not processed through legacy RPS (i.e. Lockbox transactions) and determined that UPc 140/399 could have prevented over 19,340 erroneous refunds and 53,377 erroneous notices from generating.
The analysis revealed that 60 percent of the IMF cases reviewed (39,146 IMF tax modules) and 75 percent of the BMF cases reviewed (20,251 BMF tax modules) allowed a tax return to post along with a remittance transaction that was later reversed. Further analysis revealed that:
We also determined that approximately 66 percent of the cross-reference tax modules associated with reversed remittance transactions (affected tax modules) generated erroneous settlement notices during the time their payment was misapplied. Further analysis revealed that:
Management comment: Management agreed with our finding and issued a request for information services (RIS Number TSF-8-0067) on February 25, 1998, to improve the controls over IMF Lockbox and ISRP transaction processing. Due to established work processes, the RIS does not include BMF transactions. Management has agreed that the BMF control weakness is significant and committed resources to develop the appropriate corrective actions.
In May 1998, part of the RIS was cancelled because of its effect on partially paid returns. As written, the RIS incorrectly rejected partially paid tax return transactions because the balance due amount did not match the remittance amount exactly and the transactions could not be associated by matching DLNs.
In a separate study, the Service’s Non-Rebate Erroneous Refund Task Force determined that 78 percent of the non-rebate erroneous refund cases reviewed were caused by misapplied payments. In their draft report, the Task Force defined non-rebate erroneous refunds as ones resulting from a clerical or ministerial mistake, not from a redetermination of a taxpayer’s liability.
The study's first recommendation was to expand UPc 140 to unpost a tax return transaction when the amount is less than a posted remittance and creates a credit balance of ten dollars or more. Operations has submitted a request for these programming changes (RIS number TFS-8-0153); but, due to resource limitations, the implementation of this RIS has been delayed until January 2000. Once implemented, this RIS will resolve the previous RIS’s affect on partially paid returns but will still not address the weaknesses of the BMF controls (UPc 399).
Preliminary data indicates that re-engineered RRPS work processes have affected the productivity of some downstream functions.
In a document titled "Roadmap for ISRP Implementation," AUSc has prepared a clear and comprehensive synopsis of the ISRP pilot system performance, which warns other Service centers not to underestimate the impact of the ISRP system on downstream functions. According to the Roadmap, the RRPS sub-system's lack of audit trails had the most significant impact on downstream functions.
Problems associated with the RRPS image archive sub-system and the disassociation of return and remittance DLN's have affected the productivity of various downstream functions. AUSc downstream units, such as ERS, Rejects, Unpostables and Unidentified Remittances, experienced lower than expected production rates, increases in their overage inventories, and/or increases in their overall inventories. For example:
ERS Unit: AUSc BMF ERS inventory, which is normally processed within two days, reached as high as nine days old. Remittance images of 58 RRPS cases in this inventory were not available on the RRPS archive system.
Rejects Units: At various times during the filing season, the Rejects inventory was 2,000 cases above prior year levels, and the number of aged cases (those in excess of 60 days) increased by 200.
Unpostable Units: Problems with RRPS printers and missing images from the RRPS image archive contributed to a backlog of cases. On June 30, 1998, the unpostable inventory was up 60 percent over prior year levels.
Unidentified Remittance Section: cases received from April 14 to June 12, 1998, were up 53 percent over the same period in 1997. The majority of the unresolved cases are 1040 payments for which source documents are not available. AUSc reported that an inability to quickly resolve these types of cases increased the need for taxpayer correspondence.
For other units, such as Erroneous Refunds and Excess collections, RRPS related cases may not be received for six months to a year after the transaction is processed. It is too early to assess the RRPS impact on the productivity of these downstream functions.
On July 14, 1998, management reported to the Service's Executive Steering committee that these performance problems were not associated with identified ISRP requirements, and the issue was tabled until after the year 2000. In future audits, we will continue to evaluate the effects of ISRP re-engineered work process on downstream functions.
Delays in contractor Software Deliveries Impact Testing.
The ISRP Pilot began operations with unresolved system development problems. In preparation for the start-up of pilot operations, the ISRP project (including System Acceptability Testing) conducted over 1,000 tests to evaluate 714 allocated requirements. One week prior to start-up, 267 problem reports were outstanding. The contractor had unilaterally modified the number and content of software delivery drops. Delays in the contractor's delivery of software were responsible for Increment I problem reports.
At the January 30, 1998, Operational Readiness Review, management identified 22 of these open problem reports as high priority reports, requiring resolution prior to pilot start-up and directed the projects office's activities to resolve these issues. On February 4, 1998, Internal Audit advised management of additional risks associated with pilot start–up and made recommendations for testing activities to:
Although management did delay the posting of ISRP DIS transactions until test data was confirmed as accurately posted to Master File, they rejected our recommendations for "extended operation" tests and RRPS Master File posting accuracy tests.
contractor software deliveries delayed both Increment I and II testing activities.
Five days prior to the scheduled receipt of nationwide rollout software (Increment III), the ISRP SAT team reported a total of 10 Priority I, 139 Priority II and 7 Priority III overage Increment II SAT Problem Reports (Forms 5534). They were also continuing to test ISRP Remittance Processing Increment I SAT problem reports.
On June 4, 1998, the contractor unilaterally extended Increment II software deliveries into 5 drops (Drops I - V), and a sixth drop (DROP VI) was added in July. In July, the SAT team reported that they would not be able to complete testing of Drops V and VI prior to the implementation of Increment II functionality on the AUSc pilot system. The contractor's proposed solution was to install Drop IV software functionality at AUSc so the pilot system could begin Increment II operations on August 3, 1998, as scheduled. According to SAT status reports, Drop IV software does not include Run control Record (RcR) functionality, which will seriously impact the Service's ability to process General Purpose Programs (GPP) during the Increment II pilot. The contractor proposed the installation of the balance of Increment II functionality on August 15 and 16, 1998.
Based upon the number of open SAT Problem Definitions and the amount of regression testing that remained to be completed, the SAT team reported concerns regarding the ISRP interface-testing schedule. While the SAT team actively continued interface testing, the time available to process submission data through both the ISRP and the Generalized Mainline Framework (GMF) was severely constrained by the contractor's delivery of Increment II, Drops V and VI functionality and the need for further regression testing. SAT was unable to complete ISRP Increment II testing prior to the receipt of the software for the ISRP nationwide rollout; and as of August 29, 1998, they had not yet verified the GMF interface for Employee Plan Master File (EPMF) documents and had processed a limited amount of GPP documents.
continued delays in software deliveries will impact the nationwide rollout.
The SAT team is also concerned about the software delivery schedule for the nationwide rollout. The contractor has begun referring to the ISRP nationwide rollout as Increment III, to include Tax Year 1998 changes, security functionality, and image archive balance and audit functionality. According to the project's Master Schedule, SAT is to receive and begin testing the initial Tax Year 1998 changes prior to the installation of ISRP application software at the nationwide rollout sites.
On July 18, 1998, SAT reported that the contractor's schedule to deliver Increment III in three drops would hamper their ability to conduct interface testing with the other tax processing systems in sufficient time to detect and correct errors prior to production. On August 8, 1998, SAT reported that the contractor added a fourth drop of Increment III software for October 10, 1998, which would delay the completion of SAT testing activities until late November 1998.
Management implemented contingency plans to mitigate the risks confronting the nationwide rollout.
Because the ISRP RRPS functionality did not fully meet expectations, management appropriately elected to limit the RRPS rollout for the 1999 processing year to five additional service centers. The ISRP DIS will roll out to all sites during 1998.
While mitigating much of the risk associated with a nationwide rollout, this action created additional risks associated with funding sources for the contingency options and project schedule modifications to accommodate a delayed installation of RRPS at the remaining four service centers. Through the proactive development of detailed risk mitigation work sheets required by the project's contingency plans, the Service is prepared to implement these activities.
Estimated savings in FY 1999 and FY 2000 Submission Processing budgets were affected by optimistic ISRP pilot production results.
In a memorandum to the commissioner dated June 11, 1998, the chief Financial Officer’s Financial Analysis Division presented estimated cost savings related to the effects of productivity gains from ISRP on the Submission Processing budgets. The estimate was based upon the 10 percent DIS and 25 percent RPS productivity gains documented in the ISRP project’s business case.
In our prior report on the Initial System Development Activities of the ISRP System (Report #082204), we determined that the project’s business case did not have quantitative support for the projected productivity gains. We recommended that management design the pilot’s evaluation plan to capture data to support the productivity gains for any individual segment of the system. Management agreed with our recommendations and responded accordingly.
In an Internal Audit Memorandum dated July 2, 1998, we advised the chief Financial Officer that pilot productivity information presented to the Service’s Executive Steering committee (ESc) appeared very optimistic and incomplete. As a result, we recommended that the Service not include estimated ISRP productivity savings in the FY 1999 or FY 2000 budget projections until the AUSc pilot productivity information is complete and actual savings are determined. (See Attachment III for our complete analysis of the pilot results.) These findings were based upon our conclusions that:
Management comments: The cFO’s office agreed that there is uncertainty about the actual ISRP pilot productivity results and revised the most recent projection to assume no productivity savings in FY 2000 from the RRPS component of ISRP. Because the pilot results did not produce any credible data to challenge the DIS productivity gains established in the project’s original Business case, they were obliged to maintain the 10 percent DIS productivity gains in their estimate.
Since the projected benefits of major information technology investments are closely integrated into the Service’s financial planning and budgeting activities, it is important that the Service evaluate the productivity gains of ISRP operations closely during the 1999 filing season. In a response to our memorandum, the Assistant commissioner (Forms and Submission Processing) announced that during FY 1999, his office would retain and analyze comparative data between the pilot RRPS center and a center still processing on the Legacy RPS system to determine the productivity improvement for FY 2000.
Management implemented strong system development and configuration management controls and is proceeding cautiously with the nationwide rollout. Despite these accomplishments, the Service needs to continue its strong executive oversight of the project's development and take additional actions to mitigate the remaining project development risks.
/s/ Gary S. Swilley
Audit Manager
|
Detailed Objectives and Scope of Review |
|
ISRP System Pre-Pilot Development Milestones |
|
Date: |
Development Milestone: |
Status/comments: |
|
August 22, 1996 |
The Investment Review Board (IRB) approved the initial Business case to replace the existing Distributed Input System (DIS) and Remittance Processing System (RPS). |
The project was latter named the Integrated Submission and Remittance Processing (ISRP) system. |
|
November 14, 1996 |
The chief Information Officer’s (cIO) System Engineering Office released a DIS/RPS engineering design study. |
Recommended replacement solutions for both DIS and RPS, with an aggressive development schedule to achieve rollout by January 1, 1999. The study concluded that prompt executive decisions regarding the project acquisition strategy would reduce schedule risk. |
|
December 4, 1996 |
The IRB approved the project’s acquisition strategy and established the DIS/RPS Replacement Project Office. |
The acquisition strategy was to develop the system within an existing Information Technology contract. |
|
December 20, 1996 |
The Service selects Lockheed Martin Federal Systems (LMFS) as the contractor. |
The project’s statement of work will be submitted as a modification to the existing Document Processing System (DPS) contract, TIRNO-94-D-0028. |
|
January 7, 1997 |
The Service established Integrated Product Teams (IPTs) to define system requirements. |
IPTs are technical workgroups comprised of both contractor and IRS personnel. |
|
February 12, 1997 |
Inspection issued an Internal Audit Memorandum (IAM) to the: Associate commissioner for Modernization/chief Information Officer IS |
Reported concerns:
|
|
February 18, 1997 |
The Service conducts its initial field site survey at the Memphis Service center (MSc) |
Site surveys define local site preparation activities and establish local ISRP contacts at each Service center. |
|
February 26, 1997 |
Management submits the project's initial Statement of Work to the contractor as contract Modification 164. |
The system's technical and business requirements were presented to National Office and Field Representatives on February 13 and 20, 1997. |
|
March 4, 1997 |
Austin Service center (AUSc) is selected as the ISRP pilot site. |
Rationale for selection includes building capacity, established infrastructure, and the existence of a remote processing facility. |
|
March 18, 1997 |
Inspection issued an Internal Audit Memorandum (IAM) to the: Associate commissioner for Modernization/chief Information Officer IS chief Taxpayer Service/chief compliance Officer T |
Reported concerns:
|
|
Date: |
Development Milestone: |
Status/comments: |
|
July 18, 1997 |
The contractor begins incremental deliveries of ISRP system software to the Program Development Site (PDS). |
Throughout the development of the pilot system, the project office conducted various testing activities, such as:
|
|
August 27, 1997 |
Internal Audit issued its Draft Report on the "Initial Systems Development Activities of the ISRP System" The final report, including management's response, was issued on January 30, 1998 (Reference No. 082204). |
Findings:
|
|
October 21, 1997 |
The contractor certified that the AUSc and the San Antonio Remote Site met all requirements for the installation of Increment 1 of the pilot. |
Hardware and incremental software deliveries to AUSc began on October 3, 1997. |
|
Date: |
Development Milestone: |
Status/comments: |
|
October 29, 1997 |
Inspection issued an IAM to the: Associate commissioner for Modernization/chief Information Officer IS chief Taxpayer Service/chief compliance Officer cP chief Management and Administration M |
Reported concerns:
|
|
January 7, 1998 |
The Service and contractor sign contract Modification (cM) 184. |
cM 184 superceded previous contract modifications and definitized the project’s statement of work. |
|
January 30, 1998 |
The Service conducted an Operational Readiness Review (ORR) of the AUSc pilot system. |
The ISRP project office conducted over 1000 tests to evaluate 714 requirements. Management identified 22 of the 267 open problem reports as high priority reports requiring resolution prior to start-up. |
|
Date: |
Development Milestone: |
Status/comments: |
|
February 5, 1998 |
Inspection issued a draft IAM on the "operational readiness prior to processing live taxpayer data on the ISRP pilot system" On March 17, 1998, the final version of this IAM was issued to the: Associate commissioner for Modernization/chief Information Officer IS chief Taxpayer Service/chief compliance Officer cP |
Reported concerns:
|
|
February 9, 1998 |
ISRP DIS Pilot start-up. |
AUSc begins processing tax returns through the pilot system. |
|
February 17, 1998 |
ISRP RRPS Pilot start-up. |
AUSc begins processing remittances through the pilot system. |
|
Analysis of the ISRP Pilot Production Results |
On May 6, 1998, the Service conducted a Preliminary Pilot Review (PPR) of the Austin Service center (AUSc) ISRP pilot operations. According to the Service’s Systems Development Life cycle, pilot performance should be carefully measured against the benefits established in the project’s business case. The latest version of the ISRP business case, dated July 1, 1997, established a 10 percent productivity gain for the Distributed Input System (DIS) and a 25 percent productivity gain for the Residual Remittance Processing System (RRPS) as the basis for projected labor savings.
Although the labor savings from ISRP productivity gains comprise approximately 34 percent of the total estimated benefits from the ISRP investment, the production data presented at the PPR requires additional analysis to determine if pilot operations met the 10 percent and 25 percent productivity gains. In addition, the project office did not control the production volumes, return complexity, or operator experience level of AUSc legacy operations to ensure that they were comparable with ISRP pilot production results. Without a legacy control group, ISRP productivity gains based upon current year AUSc legacy operations are less reliable than other data sources (i.e. current year national average production rates or prior year production rates).
The Service’s Executive Steering committee was presented the most optimistic DIS production results without contrasting information.
The May 6, 1998 PPR presented ISRP DIS production rates in a matrix along with AUSc current year legacy rates; current year scheduled rates; prior year legacy rates; and the current year national averages. The PPR matrix indicated that pilot operations achieved acceptable production rates when processing Forms 1040, 1040A, 1120S, and 1040EZ. The reported results were based upon current year AUSc legacy production and additional analysis indicates that they represented the most optimistic presentation of productivity gains (see Table 1).
|
TABLE 1: Distributed Input System (DIS) productivity analysis based upon data presented in the May 6, 1998, Preliminary Pilot Review (PPR) handouts |
|||||
|
ISRP DIS Productivity |
1998 AUSc Legacy Act |
1998 AUSc Legacy Sch |
1997 AUSc Legacy Act |
1998 National Average |
|
|
Form 1040 OTFP |
17.8% |
(6.6%) |
(1.0%) |
2.2% |
|
|
Form 1040A OTFP |
17.1% |
(8.0%) |
(2.9%) |
12.9% |
|
|
Form 1040Pc OTFP |
(3.1%) |
(5.4%) |
(0.8%) |
7.5% |
|
|
Form 1120S |
6.2% |
5.4% |
5.4% |
23.1% |
|
|
Form 940 |
(32.1%) |
(27.9%) |
(22.7%) |
(21.4%) |
|
|
Form 1040EZ |
85.2% |
52.7% |
6.7% |
(1.0%) |
|
|
Form 1040X |
(17.4%) |
4.5% |
4.9% |
8.3% |
|
DIS productivity gains of 16.8 percent for Forms 1040 and 17.1 percent for Forms 1040A were presented to the Service’s Executive Steering committee at their May 12, 1998 meeting. Documents for the June 11, 1998 Executive Steering committee reported the completion of the Preliminary Pilot Review with an 18 percent peak performance improvement over Legacy DIS operations. Although these gains are similar to the AUSc legacy productivity gains displayed above (see Table 1), the executives were not presented alternative results.
The pilot’s RRPS productivity gain over prior year Legacy RPS operations is approximately 80 percent lower than the business case estimate.
The PPR RRPS production data was exhibited in a matrix comparing AUSc’s current year RRPS production rate to its prior year legacy RPS production rate. The analysis was based on comparable time periods spanning the entire pilot-operating period (i.e., February 17, 1998 through April 25, 1998). Additional analysis indicates that the RRPS system achieved a 15.2 percent cumulative productivity gain on April 25, 1998.
As the following graph illustrates, this production rate was significantly higher than any time period before or after this date (see Table 2). When augmented with current production data, the analysis shows that the cumulative productivity gain dropped to 6.1 percent the following week and steadily declined to 2.8 percent as of the week ended June 27, 1998. As a result of management's conclusion that the RRPS has remained stable since the end of April 1998, we calculated the average cumulative productivity gain for the ten weekly periods from April 25 to June 27, 1998. During this period, RRPS averaged a 5.5 percent productivity gain, approximately 80 percent lower than the business case estimate.
|
TABLE 2: Graph of the Residual Remittance Processing System's (RRPS) productivity gains based upon prior year legacy RPS production. |
Graph has been removed due to its size. To see the complete graph, please go to the Adobe PDF version of this report.
Response has been removed due to its size. To see the complete Response, please go to the Adobe PDF version of this report.
|
Management's Response to Internal Audit Memorandum (IAM) #2 |
Response has been removed due to its size. To see the complete Response, please go to the Adobe PDF version of this report.
|
Management's Response to Internal Audit Memorandum (IAM) #3 |
Response has been removed due to its size. To see the complete Response, please go to the Adobe PDF version of this report.