TREASURY
INSPECTOR GENERAL FOR TAX ADMINISTRATION
Complete Actions Were Not Taken to Validate the Best Software Solution Was Chosen for the Private Debt Collection Program
April
10, 2007
Reference
Number: 2007-20-065
This
report has cleared the Treasury Inspector General for Tax Administration
disclosure review process and information determined to be restricted from
public release has been redacted from this document.
Phone Number |
202-927-7037
Email Address | Bonnie.Heald@tigta.treas.gov
Web Site |
http://www.tigta.gov
April 10, 2007
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Complete Actions Were Not Taken to Validate the Best Software Solution Was Chosen for the Private Debt Collection Program (Audit # 200620006)
This report presents the results of our review of the Filing and Payment Compliance Project (hereafter referred to as the Project). The overall objective of this review was to determine whether the Internal Revenue Service (IRS) revalidated its decision to purchase commercially available software for the Project.
The IRS does not actively attempt to collect billions of dollars in delinquent taxes because it does not have sufficient staff, collection processes, or systems. To address this issue, in 2001, the IRS initiated the Project, which was initially designed to route collection cases to the appropriate IRS collection unit. The IRS later added plans for the Private Debt Collection program to the Project. Under the Private Debt Collection program, the IRS has the authority to hire private collection agencies to assist in the collection of delinquent taxes. In July 2004, the Department of the Treasury estimated the IRS will collect $1.4 billion through the program over the next 10 years (Fiscal Years 2006-2015).[1]
Impact on the Taxpayer
The IRS compared several alternatives to identify the best
solution for the Project. Based on the
comparison, the IRS decided to purchase commercially available software. In 2005, we determined the documentation
prepared to support the Alternatives Analysis[2] was not accurate,
complete, and timely.[3] The IRS agreed with our conclusion and
planned corrective actions to revalidate its decision to purchase commercially
available software for the Project.
However, the IRS did not adequately complete those corrective actions
and thus cannot prove it spent taxpayer funds wisely.
Synopsis
The IRS has implemented the first phase of the Private Debt Collection program. As of December 2006, the IRS reported the private collection agencies had collected about $11 million.[4] In addition, based on our December 2005 report recommendation, the IRS hired a contractor to estimate the acquisition cost for one alternative.
Corrective actions planned to address our previous recommendation were not fully implemented.
While the IRS did hire a contractor to provide an estimate for part of one alternative, the IRS’ efforts to revalidate its decision to purchase commercially available software for the Project were incomplete. For example, neither the IRS nor its contractor estimated maintenance costs or benefits for one alternative. Additionally, the IRS did not develop estimates and supporting documentation for the acquisition and maintenance costs for another alternative.
As we were completing our audit, the IRS implemented the last phase of the Private Debt Collection program and cancelled the remaining phase. Because the IRS significantly changed the scope of the Project, Department of the Treasury regulations require the next Capital Asset Plan and Business Case (Exhibit 300) for the Project to include a new Alternatives Analysis. The Project will be approaching “steady state”[5] as it submits its next Exhibit 300. Once the Project is considered steady state, it will no longer be required to prepare an Alternatives Analysis. Therefore, it may not be cost effective to expend significant resources developing estimates and supporting documentation for a new Alternatives Analysis.
Recommendation
We recommended the Chief Information Officer work with officials from the Department of the Treasury and the Office of Management and Budget to determine the level of effort required to develop estimates and supporting documentation for a new Alternatives Analysis for submission to the Department of the Treasury in June 2007.
Response
The Chief Information Officer agreed with our recommendation and stated the IRS would use our observations as lessons learned for preparing future Alternatives Analysis. The Chief Information Officer also stated the IRS will commence working with the Department of the Treasury and the Office of Management and Budget to determine an appropriate solution for fulfilling the Exhibit 300 requirements. Management’s complete response to the draft report is included as Appendix V.
Copies of this report are also
being sent to the IRS managers affected by the report recommendation. Please contact me at (202) 622-6510 if you
have questions or Margaret E. Begg, Assistant Inspector General for Audit
(Information Systems Programs), at (202) 622-8510.
Efforts to Revalidate
the Alternatives Analysis Were Not Sufficient
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – Enterprise Life Cycle Overview
Appendix V
– Management’s Response to the Draft Report
Abbreviations
|
ELC |
|
|
IRS |
Internal Revenue Service |
As of September 30, 2006, the gross accounts receivable to the Internal Revenue Service (IRS) totaled $271 billion.[6] However, the IRS does not actively attempt to collect billions of dollars in delinquent taxes because it does not have sufficient staff, collection processes, or systems. In 2001, the IRS initiated the Filing and Payment Compliance Project (hereafter referred to as the Project) to address these shortcomings and increase collections.
The initial design for the Project included plans to implement commercially available software that would identify the most appropriate collection method for each collection case and route the case to the appropriate collection unit within the IRS. The IRS later developed plans to identify and route simple collection cases to private collection agencies and incorporated these plans into the Project.
The revised scope of the Project consisted of three releases.[7] The first release, the Private Debt Collection program, is designed to route simple collection cases to private collection agencies. The IRS received authority in October 2004 to hire private collection agencies when the President signed the American Jobs Creation Act of 2004.[8] In July 2004, the Department of the Treasury estimated the IRS will collect $1.4 billion through the program over the next 10 years (Fiscal Years 2006-2015).[9] The second release would implement plans to route collection cases to the appropriate collection units within the IRS Automated Call Sites[10] and campuses.[11] The third release would implement plans to route collection cases to the IRS Collection Field function.[12]
After reactivating the Project, the IRS identified and
evaluated three alternatives to select the best solution for the Project. Based on the estimated costs, benefits, and
other financial indicators for each alternative, the IRS decided the best
solution was to purchase commercially available software. During
a prior audit of the Project,[13] we could not verify whether the IRS had selected
the best solution for the Project because supporting documentation was not
always accurate, complete, and timely.
We recommended the Chief Information Officer complete corrective actions
to validate the IRS’ decision to purchase commercially available software for
the Project. In August 2006, the IRS
reported all corrective actions were complete.
In August 2006, the IRS also significantly reduced the scope of the Project by canceling plans to route collection cases to units within the IRS Automated Call Sites and campuses (Release 2) and the IRS Collection Field function (Release 3). As we were completing our audit work, the IRS also cancelled the last phase of the Private Debt Collection program and implemented the final phase of the software.
Additional audits have been, or are being, conducted of the Private Debt Collection program by the Treasury Inspector General for Tax Administration.[14] This audit was performed while changes were being made at the project level, including a change in the scope and schedule for the Project. Any changes that have occurred since we concluded our analyses in December 2006 are not reflected in this report. As a result, this report may not reflect the most current status of the Project.
This review was performed at the Modernization and
Information Technology Services organization office in New Carrollton,
Maryland, during the period August through December 2006. The audit was conducted in accordance with Government Auditing Standards. This review was part of our Fiscal Year 2007
audit plan for reviews of the Modernization and Information Technology Services
organization and contributes to our efforts to address the Fiscal Year 2007
Major Management Challenge to assess the progress of the “Modernization of the
Internal Revenue Service.” Detailed
information on our audit objective, scope, and methodology is presented in
Appendix I. Major contributors to the
report are listed in Appendix II.
Efforts to Revalidate the Alternatives Analysis Were Not Sufficient
The IRS acquired commercially available software for the Private Debt Collection program to identify and route specific collection cases to private collection agencies. The software acquired is a major investment. Therefore, the IRS must prepare and submit a Capital Asset Plan and Business Case (Exhibit 300) annually to the Office of Management and Budget, which uses the Exhibit 300 to allocate funds for information technology investments. In addition, agencies use the Exhibit 300 to request funds, monitor the progress of projects, and improve management decision making related to information technology investments.
A key component of
the Exhibit 300 is the Alternatives Analysis, which is required for projects in
the planning and acquisition phase and is used to identify, compare, and assess
viable alternatives for a project and document the rationale for selecting the
chosen alternative. The Alternatives
Analysis must include at least three viable alternatives and document the
criteria used to compare and assess the alternatives. The criteria used to compare and assess the
alternatives include estimated costs, benefits, and other financial indicators,
such as return on investment[15] and net present value.[16] Department
of the Treasury guidelines require the Alternatives Analysis to be as accurate
as possible and documentation to be maintained to support the estimated costs,
benefits, and financial indicators.
The IRS prepared an Exhibit 300 and Alternatives
Analysis for the Project to support its decision to purchase commercially
available software. The Alternatives
Analysis included the following alternatives:
· Buy Alternative: Implement commercially available software to improve collection processes, and contract with private collection agencies to pursue delinquent taxes.
· Build Alternative: Modify the current processing environment.
· Full-Time Equivalent[17] Alternative: Obtain an additional 1,600 full-time equivalents to work collection cases.
The IRS completed some steps to revalidate
the Alternatives Analysis and implemented the first phase of the Private Debt
Collection program
During our December
2005 audit, we could not verify the IRS’ decision to purchase commercially
available software because the supporting documentation for the Exhibit 300 and
Alternatives Analysis was not always accurate, complete, and timely. As a result, we recommended the Chief
Information Officer revalidate the Alternatives Analysis, develop and maintain
adequate documentation to support the IRS’ decision to purchase commercially
available software, and revise the Exhibit 300 if warranted. As part of the revalidation process, we
recommended the IRS perform a quality review of all supporting documentation
for the Exhibit
300 to ensure the reliability of the documentation.
The Chief Information Officer agreed with our recommendation and planned the following corrective actions:
· Hire a contractor to revalidate the Alternatives Analysis by developing new estimates (re-costing) for the Buy Alternative and the Build Alternative.
· Hire a contractor to help review the quality of the documentation developed to support the revalidation.
Based on our recommendation, the IRS hired a contractor, PRICE Systems, to estimate the acquisition cost for the Build Alternative. The contractor used the PRICE TruePlanning® applications and specific information provided by IRS employees to develop the estimate. We believe this was a solid first step for the IRS, as the PRICE TruePlanning® applications consist of industry-specific cost models, benchmarks, and experience obtained from previous projects and research. The IRS reported that another contractor, MITRE Corporation, reviewed the quality of the documentation developed to support the estimates. The IRS reported all corrective actions related to our recommendation were complete in August 2006.
During Fiscal Year 2006, the IRS also implemented the first phase of the Private Debt Collection program. In September 2006, the IRS assigned the initial inventory of collection cases to three private collection agencies. As of December 2006, the IRS reported the 3 private collection agencies had collected about $11 million.[18]
Efforts to revalidate the Alternatives Analysis were narrowly
focused
Corrective actions planned to address our previous recommendation were not fully implemented.
While the IRS did hire a contractor to provide an estimate for part of the Build Alternative, the IRS’ efforts to revalidate its decision to purchase commercially available software for the Project were incomplete. For example, neither the IRS nor its contractor estimated maintenance costs or benefits for the Build Alternative. Additionally, the IRS did not develop estimates and supporting documentation for the acquisition and maintenance costs for the Buy Alternative as part of the revalidation.
IRS employees stated they never intended to develop supporting documentation for the Buy Alternative. However, this statement is not consistent with the corrective actions planned by the IRS to address our recommendation, which included plans to develop estimates for both the Build and Buy Alternatives. Without these estimates and supporting documentation, it is not possible to compare the Build and Buy Alternatives to identify the best solution for the Project. Thus, the IRS cannot prove it spent taxpayer funds wisely.
While we believe the preparation of estimates for the Build Alternative using industry cost models and benchmarks was a positive first step to responding to our recommendation, we identified two issues that could cause the estimated acquisition costs for the Build Alternative to be overstated. The first issue relates to the growth of software during the acquisition phase. During software acquisition, the planned size of the software being developed can increase because of several factors, such as changes in customer needs. The IRS and the contractor assumed the size of the software could grow by 50 percent during the acquisition phase. This assumption appears to be high considering the IRS has a Business Rules and Requirements Management Office[19] and an Enterprise Life Cycle (ELC)[20] in place to help mitigate excessive software growth.
The second issue relates to the experience of the team that would be developing the software. The IRS and the contractor assumed the system would be developed by a team with limited experience. Therefore, Development Team Capability was assessed as “slightly experienced.” Because the IRS intended to hire a contractor to complete 75 percent of the effort, we believe the IRS would have hired an experienced contractor. Therefore, we believe the Development Team Capability should have been assessed as having more experience.
IRS officials stated the assessments discussed above were developed by the contractor, and the IRS should not revise conclusions made by an independent contractor. However, the assessments were developed based on information obtained from IRS officials. Additionally, the IRS should challenge all contractor assessments that appear to be unreasonable.
The scope of the Project was significantly
reduced
As we were completing our audit, the IRS implemented the last phase of the Private Debt Collection program and cancelled the remaining phase. Due to earlier changes in scope, the Project will be considered “steady state”[21] by September 30, 2007. This will complicate matters for the IRS as it prepares its next Exhibit 300, which is due to the Department of the Treasury in June 2007, because Department regulations require a new Alternatives Analysis when there is a significant change to the scope of a project. As a result, the IRS will need to provide updates for all three viable alternatives. As mentioned previously, we determined the IRS does not have complete support for the estimated costs and benefits for the Buy or Build Alternatives. Lastly, the Project will be approaching steady state as it submits its next Exhibit 300. Once the Project is considered steady state, it will no longer be required to prepare an Alternatives Analysis. Therefore, it may not be cost effective to expend significant resources developing estimates and supporting documentation for a new Alternatives Analysis.
Recommendation
Recommendation 1: The Chief Information Officer should work with officials from the Department of the Treasury and the Office of Management and Budget to determine the level of effort required to develop estimates and supporting documentation for a new Alternatives Analysis for submission to the Department of the Treasury in June 2007.
Management’s Response: The Chief Information Officer agreed with our recommendation and stated the IRS would use our observations as lessons learned for preparing future Alternatives Analysis. The Chief Information Officer also stated the IRS will commence working with the Department of the Treasury and the Office of Management and Budget to determine an appropriate solution for fulfilling the Exhibit 300 requirements.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine whether the IRS revalidated its decision to purchase commercially available software for the Filing and Payment Compliance Project. To accomplish this objective, we:
I. Determined whether the IRS had adequate (timely and reliable) supporting documentation for the estimated costs, benefits, and related financial indicators for each viable alternative[22] for the Filing and Payment Compliance Project.
II. Determined whether the revalidation performed by PRICE Systems provided assurance the estimates for each viable alternative are reliable.
Appendix II
Major Contributors to This Report
Margaret
E. Begg, Assistant Inspector General for Audit (Information Systems Programs)
Gary
V. Hinkle, Director
Troy
D. Paterson, Audit Manager
James
A. Douglas, Lead Auditor
Tina
Wong, Senior Auditor
Olivia
DeBerry, Auditor
Appendix III
Commissioner C
Office of the Commissioner – Attn:
Chief of Staff C
Deputy Commissioner for Operations Support OS
Deputy Chief Information Officer
OS:CIO
Associate Chief Information
Officer, Applications Development
OS:CIO:AD
Associate Chief Information
Officer,
Director, Stakeholder Management
OS:CIO:SM
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk
Analysis RAS:O
Office of Internal Control
OS:CFO:CPIC:IC
Audit Liaisons:
Associate Chief Information Officer, Applications Development OS:CIO:AD
Manager,
Program Oversight Office OS:CIO:SM:
Appendix IV
Enterprise Life
Cycle Overview
The ELC is the IRS standard approach to business change and information systems initiatives. It is a collection of program and project management best practices designed to manage business change in a successful and repeatable manner. The ELC addresses large and small projects developed internally and by contractors.
The ELC includes such requirements as:
·
Development of
and conformance to an enterprise architecture.
·
Improving
business processes prior to automation.
·
Use of
prototyping and commercial software, where possible.
·
Obtaining early
benefit by implementing solutions in multiple releases.
·
Financial
justification, budgeting, and reporting of project status.
In addition, the ELC improves the IRS’ ability to manage changes to the enterprise; estimate the cost of changes; and engineer, develop, and maintain systems effectively. Figure 1 provides an overview of the layers, paths, phases, and milestones (shown as “MS” in Figure 1) within the ELC Framework.
Figure
1: ELC Framework
Figure 1 was removed due to its size. To see Figure 1, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
ELC Layers
The ELC is a
framework for organizing and using IRS directives, processes, procedures, templates,
and standards to accomplish business change.
It is organized as a set of six interacting layers.
·
The Management Layer specifies how to plan
and control business change programs, projects, acquisitions, and solutions
throughout the ELC.
·
The Governance Layer specifies additional
controls imposed from outside the project or program.
·
The Solution Life Cycle Layer specifies
what should be done but not how to do it.
·
The Solution Layer manages the solution as
it is produced, including providing standards for consistent solution
specification and formal review of solution content. This Layer provides control over work
products that may be produced by multiple internal and external developers
using differing methodologies.
·
The Methodology Layer details how to do the
work and specifies a unique set of work products to be produced. Specific methodologies are not part of the
ELC Framework.
·
The Specialty Areas Layer provides
additional guidance for areas of particular importance within the IRS. These areas include Enterprise Integration,
Test, and Evaluation;[23] Business Rules Harvesting[24] and Management; Transition Management;[25] Enterprise Architecture; Capital Planning and
Investment Control;[26] Security and Privacy; and Requirements Development
and Management.
A path specifies a
unique “philosophy” or orientation for performing the work. Although the ELC specifies a standard for the
work required to produce and operate business change solutions, there are
multiple ways to approach and accomplish the required work. Paths are like alternate roads, each of which
crosses different terrain, but all of which lead to the same destination. The ELC provides five distinct paths or approaches
to developing systems:
·
The Large Custom Path is for large
projects.
·
The Small Custom Path is for small
projects.
·
The Commercial-Off-the-Shelf Path is a
commercial software-based approach.
·
The Joint Application Development/Rapid
Application Development Path is a highly accelerated, prototyping-based
approach for very small, standalone solutions or solution components.
·
The Iterative Custom Path is a hybrid
approach that combines elements of the other approaches.
A phase is a broad segment of
work encompassing activities of similar scope, nature, and detail and providing
a natural breakpoint in the life cycle.
Each phase begins with a kickoff meeting and ends with an executive
management decision point (called a milestone) at which IRS executives make
“go/no-go” decisions for continuation of a project. Project funding decisions are often
associated with milestones.
Figure
2: ELC Phases and Milestones
|
Phase |
General Nature of Work |
Concluding Milestone |
|
Vision and
Strategy/Enterprise Architecture Phase |
High-level direction setting. This is the only phase for enterprise
planning projects. |
0 |
|
Project Initiation
Phase |
Startup of development projects. |
1 |
|
Domain Architecture
Phase |
Specification of the operating concept,
requirements, and structure of the solution. |
2 |
|
Preliminary
Design Phase |
Preliminary design of all solution
components. |
3 |
|
Detailed
Design Phase |
Detailed design of solution components. |
4A |
|
System Development
Phase |
Coding, integration, testing, and
certification of solutions. |
4B |
|
System Deployment
Phase |
Expanding availability of the solution
to all target users. This is usually
the last phase for development projects. |
5 |
|
Operations
and Maintenance Phase |
Ongoing management of operational
systems. |
System Retirement |
Source: The ELC Guide.
Appendix V
Management’s Response to the Draft Report
The
response was removed due to its size. To
see the response, please go to the Adobe PDF version of the report on the TIGTA
Public Web Page.
[1] Information obtained from the Department of the Treasury. We did not verify the accuracy of this information.
[2] The Alternatives Analysis is a critical component of the Capital Asset Plan and Business Case (Exhibit 300), which must be submitted to the Office of Management and Budget annually for major information technology investments. The Exhibit 300 is used to request funds, monitor the progress of projects, and improve management decision making related to information technology investments.
[3] The Alternatives for Designing and Developing the Filing and Payment Compliance Project Should Be Revalidated (Reference Number 2006-20-026, dated December 2005).
[4] Information obtained from IRS briefings related to the Private Debt Collection program. We did not verify the accuracy of this information.
[5] Steady state refers to ongoing operations and support
of the system, which begin after the business processes and system have been
installed and have begun performing business functions.
[6] We did not verify the accuracy of this information.
[7] A release is a specific edition of software.
[8] Pub. L. No. 108-357, 118 Stat. 1418 (2004).
[9] Information obtained from the Department of the Treasury. We did not verify the accuracy of this information.
[10] A telephone contact system through which telephone assistors collect unpaid taxes and secure tax returns from delinquent taxpayers who have not complied with previous notices.
[11] The data processing arm of the IRS. The campuses process paper and electronic submissions, correct errors, and forward data to the Computing Centers for analysis and posting to taxpayer accounts.
[12] A unit consisting of revenue officers who handle personal contacts with taxpayers to collect delinquent accounts or secure unfiled returns.
[13] The Alternatives for Designing and Developing the Filing and Payment Compliance Project Should Be Revalidated (Reference Number 2006-20-026, dated December 2005).
[14] The Private Debt Collection Request for Quotation Outlines Adequate Procedures and Controls (Reference Number 2005-10-156, dated September 2005); Management Needs to Continue Monitoring Some Case Selection Issues As the Private Debt Collection Program Is Implemented (Reference Number 2006-30-064, dated April 2006); The Revised Private Debt Collection Request for Quotation Adequately Addressed Prior Deficiencies in the Solicitation Methodology (Reference Number 2006-10-078, dated April 2006); and The Private Debt Collection Program Was Effectively Developed and Implemented, but Some Follow-up Actions Are Still Necessary (Reference Number 2007-30-066, dated March 27, 2007).
[15]
The net profit or loss in an accounting period
divided by the capital investment used during the period, usually expressed as
an annual percentage return.
[16]
Used to analyze the profitability of an
investment or project. Net present value
is the present value of an investment’s future net cash flows minus the initial
investment.
[17] A full-time equivalent is a measure of labor hours in which 1 full-time equivalent is equal to 8 hours multiplied by the number of compensable days in a particular fiscal year. For Fiscal Year 2005, 1 full-time equivalent was equal to 2,088 staff hours.
[18] Information obtained from IRS briefings related to the Private Debt Collection program. We did not verify the accuracy of this information.
[19] The Business Rules and Requirements Management Office is designed to ensure projects have fully developed, understood, agreed to, and documented rules and requirements, which should reduce the risk of significant software growth.
[20] See Appendix IV for an overview of the ELC.
[21] Steady state refers to ongoing operations and support
of the system, which begin after the business processes and system have been
installed and have begun performing business functions.
[22] This step
was performed only for the Buy Alternative and Build Alternative. It was not performed for the alternative
related to hiring additional staff because that alternative was not included in the President’s budget.
[23] Enterprise Integration, Test, and Evaluation includes processes for integrating multiple components of a solution and conducting various types and levels of testing on the solution.
[24] A business rule is a statement that defines or
constrains some aspect of the business. Harvesting
is a general term used to broadly describe the entire set of activities
involved in gathering, formalizing, analyzing, and validating business rules
for a particular scope.
[25] Transition Management helps ensure personnel and organizations are prepared to receive, use, operate, and maintain the business processes and technology provided by business change solutions.
[26] The Capital Planning Investment and Control process
manages a central portfolio of information technology investments across the
IRS.