The New Risk-Based Collection Initiative Has the Potential to Increase Revenue and Improve Future Collection Design Enhancements

 

September 2004

 

Reference Number: 2004-30-165

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

September 1, 2004

 

 

MEMORANDUM FOR COMMISSIONER, SMALL BUSINESS/SELF-EMPLOYED DIVISION

 

FROM:     Gordon C. Milbourn III /s/ Gordon C. Milbourn III

                 Acting Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - The New Risk-Based Collection Initiative Has the Potential to Increase Revenue and Improve Future Collection Design Enhancements  (Audit # 200330042)

 

This report presents the results of our review of the Internal Revenue Service’s (IRS) Risk-Based Collection (RBC) initiative. The overall objectives of the review were to determine whether the RBC initiative in the Small Business/Self-Employed (SB/SE) Division was developed using sound methodology and whether its effectiveness can be adequately measured.

Between Fiscal Years 1996 and 2001, the IRS Collection function program results declined sharply, due in part to a large decrease in Collection Field function (CFf) staffing.  Although some Collection function results have improved since then, the IRS still faces a tax gap estimated at over $300 billion and an increasing belief among taxpayers that cheating on their taxes is acceptable.  To help reduce this tax gap, the IRS Commissioner is emphasizing stronger enforcement.  Part of this emphasis involves an IRS collection project, called Filing and Payment Compliance (F&PC), which includes risk-based scoring to predict collectibility.  However, it is a long-term project dependent on the IRS’ historically problematic modernization program, leaving the IRS operating divisions to develop their own short-term improvements.

Limited to making minimal system changes and implementing them within 3 years, in May 2001, the SB/SE Division Collection Strategy (CS) team began devising a short-term solution as part of the Collection function reengineering efforts.  The CS team’s RBC initiative used predictive models to characterize accounts according to the probability of productive or unproductive closure, for routing to the most effective treatment area.  The CS team’s most recent calculations estimate additional revenue from the initiative, which began operating in January 2003, at over $1.9 billion annually when fully implemented.

In summary, the SB/SE Division should be recognized for taking significant steps to develop risk-based criteria for improving casework selection in the Automated Collection System (ACS) and the CFf. The CS team implemented the initiative within the targeted time periods and met programming constraints.  Their comprehensive methodology considered risks and used sophisticated software modeling employing hundreds of variables from appropriate available data sources.

The SB/SE Division Research function is testing the predictive accuracy of the models, and statistical data to date show mixed results.  The CFf’s average monthly dollars collected from January through May 2004 exceeded preinitiative results by nearly $120 million, and the ACS’ monthly collections increased by over $20 million.  On the other hand, decreases in the ACS’ productive closure rates and an increase in its unproductive closure rate contrast with the RBC initiative’s predicted results.  However, it may be premature to draw any conclusions from the limited initial data. It is also difficult to determine whether the statistics are indications of the success of the RBC initiative, or whether there are factors from other initiatives contributing to the changes.

SB/SE Division management did not have an adequate method to measure the results of the models on revenue, productive and unproductive closings, and compliance, or the effects of potential incorrect predictions. Ideally, the measurement should differentiate the effects on revenue from those of several other initiatives that have taken place.  Additionally, because the SB/SE Division had to implement the RBC initiative within tight time periods, the CS team had to make decisions and accept limitations in the design of the models that inherently make them less ideal than a longer development period using more complete data, such as that envisioned in the F&PC project, would have allowed.  For example, the modeling does not include information on nondelinquent taxpayers, complete information on a taxpayer’s entire account history, or third-party data. In some cases, it does not account for all types of returns for a business taxpayer and related compliance work that Collection function employees must consider.  The SB/SE Division has the opportunity to apply its experiences from model development to future risk-based efforts.

We recommended the Director, Payment Compliance, develop an effective measurement system which quantifies and reports on the results of the RBC initiative.  The Director should share the CS team’s experiences in developing a risk-based collection system with the developers of the F&PC project, or use those experiences to improve the SB/SE Division’s system if the F&PC project becomes significantly delayed.

Management’s Response:  The Commissioner, SB/SE Division, disagreed with one of our two recommendations.  The Commissioner, SB/SE Division, disagreed with the recommendation to develop a measurement system to quantify and report on the results of the initiative, saying it would require retroactively developing a methodology to quantify and evaluate the change.  Instead, the SB/SE Division will evaluate progress by looking at overall results of its incremental reengineering improvements at various points in time and will work on ways to measure and report on benefits of future recommendations.  SB/SE Division management agreed to modify existing procedures to include a representative from the F&PC project in the periodic status discussions on SB/SE Division research projects and annual progress reviews.  Management’s complete response to the draft report is included as Appendix IV.

Office of Audit Comment:  Although we understand the difficulties of isolating and attributing improvements to various reengineering efforts, the RBC models projected very specific results, and we believe that a measurement process is needed to determine whether the results are being achieved.  While we still believe our recommendation is worthwhile, we do not intend to elevate our disagreement concerning it to the Department of the Treasury for resolution.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Philip Shropshire, Acting Assistant Inspector General for Audit (Small Business and Corporate Programs), at (215) 516-2341.

 

Table of Contents

Background

A Comprehensive Methodology Was Used to Implement a Risk-Based Modeling Approach to Select Case Work

Testing Is Designed to Gauge the Models’ Accuracy but Will Not Fully Quantify Their Success

Recommendation 1:

Opportunities Exist to Apply Lessons Learned From Model Development to Future Risk-Based Efforts

Recommendation 2:

Appendix I – Detailed Objectives, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Management’s Response to the Draft Report

 

Background

The mission of the Internal Revenue Service (IRS) is to provide America’s taxpayers top-quality service by helping them understand and meet their tax responsibilities and by applying the tax law with integrity and fairness to all.  In its 1999 publication, Modernizing America’s Tax Agency, the IRS recognized that, regardless of how successful it is in preventing taxpayer errors, intervention would always be necessary when taxpayers did not file their returns or pay their tax liabilities. The publication further noted the need to promptly identify customers who may present a risk of nonpayment and to intervene in the most effective way to work out a payment program that addresses a particular customer’s payment problem.  Risk-based compliance techniques are dependent upon centralized management of compliance resources; accurate, up-to-date data; and modern technology.

The Government Accountability Office (GAO) reported a serious decline in the IRS’ Collection function program between Fiscal Years (FY) 1996 and 2001.  This was due in part to a large decrease in Collection Field function (CFf) staffing.  The GAO estimated that, by the end of FY 2001, the IRS had deferred collecting taxes from about 1.3 million taxpayers who collectively owed about $16.1 billion.  The Taxpayer Delinquent Account (TDA) inventory increased 22 percent from FYs 1997 to 2000, while use of enforcement tools such as liens and levies dropped significantly.  More recent data show that the IRS has improved some of its collection results.  For example, there was a substantial increase in the use of liens and levies, TDA closings increased 26 percent, and dollars collected in FY 2003 increased 15 percent over FY 2002.  Nevertheless, the IRS Commissioner has emphasized stronger enforcement, citing an increase from 11 to 17 percent (55 percent) in the number of taxpayers who believe tax cheating is acceptable and a tax gap estimated by the IRS at over $300 billion.

To address many collection enforcement challenges, the IRS launched a series of programs to modernize its technologies and processes, one of which is the Filing and Payment Compliance (F&PC) project.  Projected capabilities of the F&PC project include increased and timely use of third-party data in case detection and case resolution; decision analytics for risk-based scoring, customer segmentation, and treatment assignments; and improved treatment streams (a series of tailored consecutive collection actions).  Decision analytics is a software application that builds scoring models to identify segments or groupings of receivables and predicts collectibility; it designs treatment streams aimed at efficiently collecting the receivables the models identify.

Although the F&PC project is estimated to result in $27 billion in additional tax revenues through FY 2016, it is a long-term solution dependent on the IRS’ historically problematic modernization efforts. Funding has not been consistently provided, and the F&PC project is not likely to enhance the collection process for some time, leaving the IRS operating divisions to develop their own short-term improvements.  The Risk-Based Collection (RBC) initiative was the Small Business/Self-Employed (SB/SE) Division’s solution.

This review was performed at the SB/SE Division’s National Headquarters in the Office of Payment Compliance and in the Research function in the Office of Strategy, Research, and Performance Management, during the period September 2003 through May 2004.  We discussed the RBC initiative’s development with officials in these functions and reviewed the documentation pertaining to the development and testing of the RBC initiative.  Analyzing the mechanisms of the developed models themselves was beyond the scope of this review.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objectives, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

A Comprehensive Methodology Was Used to Implement a Risk-Based Modeling Approach to Select Case Work

As part of its Collection function reengineering efforts, in May 2001 the SB/SE Division chartered a Collection Strategy (CS) team with the goal of achieving short-term collection improvements.  Guiding principles for its work included implementing projects within 3 years and selecting solutions that would require only minimal system changes.

In developing the RBC initiative, the CS team used predictive models to characterize aspects of balance due modules that would indicate a higher probability of a productive closure.  Financial institutions, underwriters, and credit card companies use predictive models to assess credit risk and collectibility of accounts.

The models screen new modules to determine whether incoming cases are likely to be unproductive (Currently Not Collectible (CNC)) or productive (Full Pay (FP)).  The screening device, called a predictive filter, bases predictions on a taxpayer’s prior payment history in collection and tax return information.  The CS team chose the Collection Inventory Delivery System (IDS) as the implementation point for these models.  The IDS has the capacity to accommodate short-term changes and is the key to routing and prioritizing Collection function work.  After filtering, the modules can be routed to the most effective treatment area.  The IRS expects that eliminating unproductive cases from the Automated Collection System (ACS) and the CFf  will ensure less time will be spent on unproductive work.  Consequently, a greater volume of productive cases and a greater total volume of cases can be worked, resulting in more dollars collected.

The filters were brought on line in January 2003, although related programming for the ACS and CFf was not completed until April and August 2003, respectively.  The SB/SE Division Research function established projects to test the predictive accuracy of the models and profile or to describe the characteristics of tax modules that have been identified by the models.  Interim reports issued in February 2004 followed model case results from January through September 2003.

Although it may be too early to tell whether the models will deliver the anticipated increased productive closings and revenue, as discussed in the next section of this report, the SB/SE Division should be recognized for its efforts to implement a risk-based methodology for selecting case work.  The CS team implemented the initiative within the targeted time periods and met programming constraints.  Their comprehensive methodology considered risks and used sophisticated software modeling employing hundreds of variables from appropriate available data sources.  While the IRS awaits the delayed F&PC project, the SB/SE Division has taken the first significant steps to apply decision analytics to develop risk-based criteria for improving resource allocation in the ACS and CFf.

Testing Is Designed to Gauge the Models’ Accuracy but Will Not Fully Quantify Their Success

As the CS team worked on the development of the models, it was concerned that there was no way to test the results of the RBC initiative.  Although research testing is underway and discussed below, it is not designed to fully quantify the initiative’s success.

A prior Treasury Inspector General for Tax Administration report on Collection function process improvements identified a concern about measuring results of the improvements. The report noted that it would be difficult to measure the impact of the process improvements because other initiatives that should affect measures were being implemented almost simultaneously. These additional initiatives included revising the priority system for assigning cases, hiring additional staff, implementing a management consultation initiative, and developing the risk-based modeling discussed in this report.  Our prior audit report concluded that it would be difficult to correlate any productivity gains to specific initiatives and that, while overall results might improve, one or more of the initiatives’ recommendations could actually have a negative impact without this being clearly identified.

In discussing this concern with us, SB/SE Division management explained that the Research function’s current research on the models addresses measurement.  Although they acknowledged it is an ongoing challenge, they believe they are better measuring the RBC initiative.  However, neither our discussion nor a review of an interim report on the research test project clearly revealed how the impact of the initiative would be measured. 

The objectives of that research project were to measure the predictive accuracy of models and to provide some descriptive statistics on variables from cases whose dispositions (closures) were inconsistent with the models’ predictions.  The research plan’s methodology, sample design, and selection appear to be sound and designed to deliver an appropriate measurement of the models’ predictive accuracy.  While that measurement is a valuable and necessary research objective, a measurement system should determine whether the desired business outcomes are being realized or, in the case of the RBC initiative, quantify the models’ effects on revenue and differentiate their effects from those of any other initiatives. The test plan and interim results do not address these issues.

Statistical information may provide some indication of the initiative’s outcomes.  For example, the rate of FP or Installment Agreement (IA) module closings may indicate how well the filters identify productive work.  Conversely, the rate of CNC module closings may indicate how well the filters identify unproductive work.  The number of dollars collected may also indicate effectiveness.

Figure 1 presents mixed results, with dollar collections showing significant increases but some productive and unproductive closing rates showing no improvement.  It compares key indicator averages from January through May 2004, the most recent data available, to averages for the same period for 2002, the year before the initiative was implemented.

Figure 1:  Key SB/SE Division Measures

Source:  IRS Collection 5000-1, 5000-5, and 5000-139 Reports.

In January 2002, the CS team estimated implementing the RBC initiative would generate an 11 percent increase in productive dispositions, an 11 percent decrease in unproductive dispositions, and about $1 billion annually in additional revenue.  The statistics above indicate an increase in average monthly dollars collected in the CFf of nearly $120 million and in the ACS of over $20 million.  However, decreases in the ACS FP and IA rates, and increases in the ACS CNC rates, contrast with the RBC initiative’s predicted results.

Because many modules have not had sufficient time to close, it may be premature to draw any conclusions from these data.  As we stated in our May 2002 report, it is also difficult to determine whether the statistics are indications of the success of the RBC initiative or whether there are factors from other initiatives contributing to the changes.

Throughout the development process, the CS team arrived at various estimates of improved results from the initiative.  Calculations of additional yearly revenue, when the initiative is fully implemented, ranged from an original assumption of about $1 billion to over $1.9 billion in the most recent estimate.  SB/SE Division management advised us the Research function would measure the results of the models on revenue, productive and unproductive closings, and compliance, and the effects of potential incorrect predictions.  The research project documentation did not specify how these results would be tracked or in what report or series of reports they would be documented.  Therefore, there appears to be no process or method in place that can effectively measure whether the modeling will achieve the improvements envisioned.

Recommendation

1.      The Director, Payment Compliance, should implement a process to report on attainment of expected benefits of the RBC initiative.  This should include all proposed benefits whether specifically quantified or not.  Quantified benefits included increased productive closures, decreased unproductive closures, and increased revenue.

Management’s Response:  The Commissioner, SB/SE Division, disagreed with this recommendation, saying it would require retroactively developing a methodology to quantify and evaluate the change.  Instead, the Division will evaluate progress by looking at overall results of its incremental reengineering improvements at various points in time and will work on ways to measure and report on benefits of future recommendations.

Office of Audit Comment:  Although we understand the difficulties of isolating and attributing improvements to various reengineering efforts, the RBC models projected very specific results, and we believe that a measurement process is needed to determine whether the results are being achieved. 

Opportunities Exist to Apply Lessons Learned From Model Development to Future Risk-Based Efforts

SB/SE Division executives tasked the CS team with identifying options for rapid implementation of the initiative.  The CS team’s short-term time periods prevented developing a solution requiring substantial reprogramming of the IRS’ main computer systems, the Master File (MF) and the Integrated Data Retrieval System (IDRS).  Because of the limited time periods, the CS team had to make decisions and accept limitations in the design of the models that inherently make them less ideal than a longer development period using more complete data, such as that envisioned in the F&PC project, would have allowed. The CS team fully considered the limitations, which are presented in its documentation and discussed below, and its experience should be helpful in refining the SB/SE Division’s risk-based systems, if needed, or in assisting with the IRS’ future overall risk-based collection efforts.

Placement of filters after the notice stream limits treatment opportunities

A risk analysis performed around the time of the initiative’s inception detailed several important points in identifying risk-based characteristics. These include determining the amount of risk at the time of assessment, routing accounts with different risk to different types of treatment, and identifying taxpayers’ payment and nonpayment characteristics from assessment to resolution.

Since the MF and IDRS could not be reprogrammed, programming filters had to be implemented at the IDS level, which routes cases that have completed the notice stage.  The filters identify the overall likelihood of payment, but their placement provides no mechanism to treat a taxpayer’s delinquency from the assessment’s inception by using different notice treatments for low- and high-risk cases.

In addition to the programming constraints, SB/SE Division management believed the limitation was acceptable because the notice process effectively resolves a high percentage of cases and collects substantial money.  While the notice process does result in large collection of dollars, filtering out those unproductive cases could allow IRS resources to be used more effectively to address the large amount of dollars deferred each year.  Tailoring complete treatment streams, including different types and numbers of notices, will not be fully addressed until the F&PC project is in place.

Models are based on less than complete data

The selection of data sources for modeling was one important element the CS team had to consider.  During risk-based research predating the SB/SE Division’s RBC initiative, researchers wanted to use a database extracted from MF data containing a taxpayer’s entire history, as well as data for nondelinquent taxpayers.  Comparing the attributes of delinquent and nondelinquent taxpayers was considered necessary for the development of any prediction model and useful for identifying possible misleading model data.  However, modeling was based on only those cases in various Collection functions.  The models do not have information on the entire population of taxpayers who have had no collection activity or complete information on the taxpayers’ entire MF account histories.

All of a taxpayer’s accounts and third-party information were not considered in the models

Modeling is also performed by module, which, while eliminating the necessity of dealing with multiple disposition types for a particular entity, means giving up some critical information about the entity.  For example, in the analysis of a business taxpayer’s employment tax returns, there is no information about the income of the business or the business owner. Small businesses file several different tax forms containing substantially different information. 

There are often multiple modules for a single taxpayer arising from balances due on different forms that may have been resolved in a variety of ways.  A business taxpayer may have fully paid some modules while other modules may have been abated or reported as uncollectible, making modeling taxpayer payment characteristics very complex. 

Modeling was further limited because third-party data, such as credit history or state income tax information, were not available for analysis.  These types of data are available from various electronic sources and could be integrated into the modeling process.  This could significantly enhance the models’ ability to predict productive versus unproductive cases.

The models do not account for some collection policies

If a balance due module is issued on a self-employed individual’s return, Collection function employees must determine whether the related business entity is in filing and payment compliance and, if not, must secure unfiled returns and/or balance due amounts and make a collection determination on both entities simultaneously.  The models do not account for such cross-compliance and may filter the account as unproductive, allowing no chance for the investigation of the related entity.

Additionally, Collection function employees must consider assessing a Trust Fund Recovery Penalty (TFRP) against responsible persons involved with corporate failure to pay trust fund monies.  Similarly, the models do not fully address this issue and may filter the account as unproductive with no TFRP investigation.  SB/SE Division management explained that these investigation results have not traditionally been good and they are willing to forego such cases for more productive cases the model predicts.

Models have inaccuracy rates when predicting case resolutions

An inherent risk in modeling is the possibility that some predictions will be inaccurate.  For example, according to standards identified in testing research, for 1 tax form, the model accurately predicts productive cases 80 percent of the time but misclassifies unproductive cases as productive 35 percent of the time.  For another tax form, these percentages were 78 percent and 42 percent, respectively. This type of misclassification means that resources may be expended on cases that eventually prove to be uncollectible.

There is also the opposite risk that modeling will misclassify productive cases as unproductive, meaning that cases that could be collectible will not be worked.  Although the standards showed that the relative percentages of misclassifications to accurate predictions were smaller for the latter type of risk, both types exist in every tax form modeled.  The percentages evolved as the models were refined, and SB/SE Division management accepted the risks as business policy decisions, reasoning that using any model results which better allocate finite resources is preferable to leaving case assignment practices unchanged. Additionally, as testing continues the models will be adjusted and inaccuracy rates may be reduced.

Recommendation

2.      The Director, Payment Compliance, should ensure the SB/SE Division’s experiences in developing a risk-based collection system are fully shared with the developers of the IRS’ overall F&PC project, including how the limitations necessary during the shorter development period of the RBC initiative can be addressed through the more comprehensive F&PC project.  As test results and other data become available, these can be shared with the F&PC team through distribution lists.  In the event that the F&PC project becomes significantly delayed, the Director should determine the feasibility of eliminating limitations and enhancing the SB/SE Division’s system, especially with regard to including external data in modeling.

Management’s Response:  The SB/SE Division will modify existing procedures to include a representative from the F&PC project in the periodic status discussions on SB/SE Division research projects and annual progress reviews.

 

Appendix I

 

Detailed Objectives, Scope, and Methodology

 

The overall objectives of the review were to determine whether the Risk-Based Collection (RBC) initiative in the Small Business/Self-Employed (SB/SE) Division was developed using sound methodology and whether its effectiveness can be adequately measured.  To accomplish our objectives, we:

I.                   Determined whether the methodology used in working toward the models was appropriate and comprehensive.

A.    Obtained documentation for the preliminary and subsequent models used during the development phase of the initiative and other documentation from the Collection Strategy (CS) team and the SB/SE Division Research function staff.

B.     Reviewed all documentation to determine the data obtained, data limitations, analyses performed, assumptions made, assessment of risks, and logical steps employed in developing recommendations.

C.     Determined how the additional dollar collections projected in developing the models were derived.

D.    Discussed the rationale behind the assumptions with SB/SE Division management, the CS team, and Research function personnel and reviewed documentation showing explanations for decisions made.

II.                Determined whether the case-filtering models that were developed incorporated a logical and comprehensive set of data and variables to enable dependable identification of probable collectible and not collectible cases.

A.    Obtained plans, prospectuses, and any other documentation for the models from the Research function and CS team.

B.     Reviewed the documentation to determine what data were used, whether limitations or restrictions of data existed, and what variables were identified as possible predictors of collectibility.

C.     Discussed with SB/SE Division management, the CS team, and Research function personnel the reasons behind decisions made in developing the models that were eventually adopted.

III.             Determined whether the RBC initiative’s costs and benefits are properly accounted for, measured, and managed.

A.    Reviewed the research plans and prospectuses obtained in Step II.A.

1.      Determined whether all costs of the initiative have been identified.

2.      Determined how the results and success of the RBC initiative are being measured and how the measurement systems will differentiate the results of the RBC initiative from other Collection function reengineering initiatives.

3.      Compared the measurements against available reports to gauge their accuracy.

4.      Compared the results against the additional collections projected in developing the models.

B.     Determined whether a program manager and staff have been selected and how they plan to measure and monitor the RBC initiative.

IV.             Determined whether testing of the models is designed to deliver an accurate forecast of the models’ effectiveness.

A.    Determined how the selected models are being tested.

B.     Reviewed database documentation and discussed with Collection function management to determine whether there are any identifiers or characteristics that would distinguish the cases as being filtered.

 

Appendix II

 

Major Contributors to This Report

 

Philip Shropshire, Acting Assistant Inspector General for Audit (Small Business and Corporate Programs)

Parker Pearson, Director

Preston Benoit, Audit Manager

Richard Hayes, Lead Auditor

Erlinda Foye, Auditor

Janis Zuika, Auditor

 

Appendix III

 

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Services and Enforcement  SE

Deputy Commissioner, Small Business/Self-Employed Division  SE:S

Acting Director, Compliance, Small Business/Self-Employed Division  SE:S:C

Acting Deputy Director, Compliance Policy, Small Business/Self-Employed Division SE:S:C:CP

Director, Payment Compliance, Small Business/Self-Employed Division  SE:S:C:CP:PC

Director, Centralized Workload Selection and Delivery, Small Business/Self-Employed Division  SE:S:C:CP:CW

Director, Research, Small Business/Self-Employed Division  SE:S:SF:SR:R

Chief Counsel  CC

National Taxpayer Advocate TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaison:  Commissioner, Small Business/Self-Employed Division  SE:S

 

Appendix IV

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.