The Statistical Sampling Method Used in the Earned Income Tax Credit Proof of Concept Test Appears Valid

 

May 2004

 

Reference Number:  2004-40-100

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

May 12, 2004

 

 

MEMORANDUM FOR COMMISSIONER, WAGE AND INVESTMENT DIVISION

 

FROM:     Gordon C. Milbourn III /s/ Gordon C. Milbourn III

                 Acting Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - The Statistical Sampling Method Used in the Earned Income Tax Credit Proof of Concept Test Appears Valid  (Audit # 200440016)

 

This report presents the results of our review of the Earned Income Tax Credit (EITC) Proof of Concept Test (the Test).  The overall objective of this review was to determine the usefulness of the Test in enabling the Internal Revenue Service (IRS) to make decisions regarding the future of its EITC Program.  The EITC is a refundable credit available to taxpayers that file returns with certain earned income.  While the EITC has provided significant benefits to taxpayers, it has also resulted in a significant loss of revenue to the Federal Government.  An IRS compliance study of Tax Year (TY) 1999 returns estimated that between $8.5 and $9.9 billion (27 to 32 percent) of the $31 billion in EITC claimed for TY 1999 should not have been paid.

This audit focused on the statistical sampling methodologies used to select the various samples for the Test.  However, we did not assess the selection criteria used for each sample.  In addition, because the IRS had an independent third party validate the design of the Certification of Qualifying Child Residency Requirements (Certification) portion of the Test, our review of the Certification sample was limited to asking the Treasury Inspector General for Tax Administration’s contracted statistician to review and provide an overall assessment of the third party report.

Overall, the statistical sampling method used to select the samples for the Test appears adequate and should provide reliable information on which to base future decisions.  The Test consists of three portions—Certification, Verification of Filing Status (Filing Status), and Verification of Income (Income).  The IRS selected statistical samples for both the Certification and Filing Status portions of the Test.  The IRS actually selected larger samples than necessary, so the Test results should be very precise.  The IRS did not use a statistical sample for the Income portion of the Test.  It developed a new case selection methodology and selected the Income inventory based on a set of information filters and available resources.  The new case selection methodology supports the intent of the Income portion of the Test and should also provide reliable results.

We do, however, have some concerns with the sizes of the subsamples in the Certification portion of the Test and the impact IRS contact with taxpayers included in this portion could have on the Test results.  While these concerns do not affect the reliability of the samples, they could affect the interpretation of the Test results.  We suggested the IRS use caution when interpreting and relying upon these results.

Management’s Response:  IRS management is pleased that the Test should enable them to make well-informed decisions regarding the future of the EITC Program.  The IRS recognizes the limitation of the small sample sizes for the subsamples related to the Certification portion of the Test but believes the subsamples will provide meaningful insights that can be used to develop more quantifiable tests.  In addition, management believes the design of the Certification portion of the Test includes procedures to address our concerns about the impact IRS contact with taxpayers included in this portion could have on the Test results.  Management’s complete response to the draft report is included as Appendix VI.

Copies of this report are also being sent to the IRS managers affected by the report results.  Please contact me at (202) 622-6510 if you have questions or Michael R. Phillips, Assistant Inspector General for Audit (Wage and Investment Income Programs), at (202) 927-0597.

 

Table of Contents

Background

The Samples Used in the Test Should Generally Provide Reliable Results

Data Collected During the Test Should Provide the Information Needed to Evaluate the Test Results

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Prior Treasury Inspector General for Tax Administration and General Accounting Office Reviews of the Earned Income Tax Credit Proof of Concept Test

Appendix V – Income Verification Program Earned Income Tax Credit Inventory Categories

Appendix VI – Management’s Response to the Draft Report

 

Background

The Earned Income Tax Credit (EITC) is a refundable credit available to taxpayers that file returns with certain earned income.  While the EITC has provided significant benefits to taxpayers, it has also resulted in a significant loss of revenue to the Federal Government.  An Internal Revenue Service (IRS) compliance study of Tax Year (TY) 1999 returns estimated between $8.5 and $9.9 billion (27 to 32 percent) of the $31 billion in EITC claimed for TY 1999 should not have been paid.

On February 28, 2002, the Department of the Treasury and the IRS announced that a Task Force would examine the administration and complexity of the EITC.  The Treasury Assistant Secretary for Tax Policy and the IRS Commissioner headed the Task Force.  The Task Force used the TY 1999 compliance study as the basis for its recommendations.  The IRS used these recommendations to develop a future vision for administering the EITC.  This vision is outlined in the IRS’ EITC Concept of Operations (CONOPS).

The first step the IRS is taking to implement its future vision is to test the basic concepts outlined in the CONOPs.  This test is referred to as a Proof of Concept Test (the Test) and is intended to provide the IRS with information necessary to determine if these basic concepts are feasible for improving compliance with the EITC without harming participation in the EITC Program. 

The Test consists of three portions designed to test concepts for improving EITC compliance:

·        Certification of Qualifying Child Residency Requirements (Certification) – The IRS will ask a sample of EITC claimants to verify, when they file their TY 2003 returns, that the qualifying child claimed for EITC purposes resided with them in the United States for more than one-half of the tax year, as required by law.  These taxpayers are being treated differently from other taxpayers that claim the EITC, and the IRS is trying to capture the impact this treatment has on improving compliance with the EITC.

·        Verification of Filing Status (Filing Status) – The IRS will review the TY 2003 returns for a sample of EITC claimants filing as “Head of Household” or “Single” to verify whether the filing status they claimed is accurate.  The IRS hopes to find a better way to identify filing status errors by evaluating the usefulness of adding additional information to the Schedule Earned Income Credit.  The IRS will also assess the value of comparing the information to other available sources in the verification process.

·        Verification of Income (Income) – The IRS will review the TY 2002 returns for taxpayers that claimed the EITC but failed in the past to report all their income.  These taxpayers may not be eligible because their income, when corrected, is too high to qualify for the credit.  The IRS is testing whether cases selected for its income verification program based on the potential change in the EITC are as productive as its regular inventory.

This audit focused on the statistical sampling methodologies used to select the various samples for the Test.  However, we did not assess the sample selection criteria used for each sample.  In addition, because the IRS had an independent third party validate the design of the Certification portion of the Test with a focus on the sample selection, our review of the Certification sample was limited to asking the Treasury Inspector General for Tax Administration’s contracted statistician to review the third party report and provide an overall assessment of the results.

This audit is the third in a series of audits we are conducting on the Test.  We have completed one review and the second is in process.  Information on our previous audit report, along with the results of a review of the Test by the General Accounting Office, can be found in Appendix IV.

This review was performed in the Office of Research, Analysis, and Statistics in the National Headquarters in Washington, D.C., and in the Wage and Investment Division Office of Compliance and EITC Office in Atlanta, Georgia, during the period November 2003 through February 2004.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

The Samples Used in the Test Should Generally Provide Reliable Results

The samples the IRS selected for the Certification and Filing Status portions of the Test are statistically valid and should provide reliable results.  The IRS actually selected larger samples than necessary, so the Test results should be very precise.  The Income portion of the Test did not use a statistical sample.  Rather, the IRS is testing a new method for selecting EITC inventory for income verification.  This new case selection methodology supports the intent of the Income portion of the Test and should also provide reliable results.

The sampling method used for the Certification portion of the Test is statistically valid and should provide reliable results

Overall, the method used to select the various samples for the Certification portion of the Test is statistically valid.  The method should also result in reliable results that the IRS can use to make future EITC Program decisions. 

In the Certification portion of the Test, the IRS is asking certain taxpayers to validate the residency requirements for their qualifying children.  The IRS started with a representative sample of approximately 1.6 million taxpayers that claimed the EITC with qualifying child(ren) and filed timely tax returns for TY 2002.  The IRS systemically certified as many taxpayers as possible using available computer databases.  It also applied various exclusions (e.g., those already selected for examination) to ensure the IRS would contact the taxpayers only once on the same return.  It then selected a statistical sample of 25,000 taxpayers from approximately 402,000 taxpayers that could not be systemically certified.  The IRS used a systematic sampling method to select the sample. 

These 25,000 taxpayers are being asked to provide affidavits, letters, or documents from a landlord or property manager, police officer, teacher, or other official in the taxpayer’s community that verify the residency requirement of the qualifying child claimed.  The IRS subsequently selected 2 subsamples out of the 25,000 to test alternative documentation requirements.  One subsample of 1,000 taxpayers will be able to ask friends and relatives to sign an affidavit attesting to a qualifying child’s residence.  A second subsample of 1,000 taxpayers will receive forms printed in both English and Spanish to gauge whether participation in the Test increases due to the distribution of materials in both languages. 

The IRS contracted with an independent third party to validate the design of the Certification portion of the Test.  To evaluate alternative forms, the third party assessed the sample design by reviewing how the IRS identified the population eligible for certification, the designation of a control group, the sample selection, and the subsamples.  The third party assessment also reviewed the Certification process by looking at IRS contact with the participants and verification of residency.  The third party looked at the IRS’ plans to evaluate the results of the Test by assessing the strengths and limitations of the overall Certification design, reviewing how the IRS can estimate the net effect of Certification, and discussing how the IRS can further develop the analysis plan.

The third party endorsed the method the IRS used to identify the taxpayers eligible for the Certification portion of the Test.  It determined that a valid approach was used to select the sample and the control group and that the sample size is more than sufficient to address the objectives of the Certification test.  The third party also determined that a subsample of 1,000 taxpayers is sufficient to test the effects of the alternative affidavit.  However, the third party indicated the second subsample of 1,000 taxpayers is not sufficient to reliably detect whether participation in the Test increases among those families that can read Spanish but not English. 

We reviewed the third party’s assessment and generally agree with it.  We are, however, concerned that the sizes of the subsamples within the Certification portion of the Test are not large enough to provide substantial and significant results.  This issue was also raised in the IRS’ third party assessment but only in reference to the Spanish notice subsample.  The IRS acknowledged that the Spanish notice subsample would probably not provide reliable results.  At best, it will provide anecdotal information.  While neither the third party’s nor our concerns affect the statistical validity of the subsamples, they do affect how the information gained from these subsamples should be used.  As a result, we suggest the IRS use the results of these two subsamples with caution.

We also have a concern about how IRS contact with the taxpayers in the Certification portion of the Test could affect the test results.  The IRS sent notices in December 2003 informing the 25,000 taxpayers in the sample that they were part of this Test.  We are concerned that some of these taxpayers may have moved and would not have received the materials that explain how the certification process can be accomplished successfully.  Also, some tax return preparers may be unaware of the Test and the importance of certification for this small group of taxpayers.  While the effect of the IRS contact does not affect the reliability of the sample, it could affect the interpretation of the Test results.  Again, we suggest the IRS use caution when interpreting and relying upon these results.

The sampling method used for the Filing Status portion of the Test is statistically valid and should provide reliable results

The samples selected for the Filing Status portion of the Test are statistically valid.  In addition, the samples should provide reliable results for the IRS to use to make future EITC Program decisions. 

In the Filing Status portion of the Test, the IRS plans to verify the filing status of two different samples of taxpayers.  The IRS used the same representative sample of approximately 1.6 million taxpayers to determine the Filing Status sampling population.  The Certification sampling population was removed from the 1.6 million before the Filing Status samples were selected.  The taxpayers identified for the Income portion of the Test were also removed so taxpayers would be affected by only one portion of the Test.  The IRS used the same systematic sampling approach as that used in the Certification portion of the Test to select both of the Filing Status samples.  We reviewed the sampling methodology and determined the samples are appropriate to address the purpose of the Test.

The IRS selected a statistical sample of 36,000 taxpayers from approximately 69,000 taxpayers whose filing status history indicated they had been previously married and then filed as “Single” or “Head of Household” on their TY 2002 tax returns.  A second statistical sample of 5,000 taxpayers was selected from approximately 600,000 taxpayers that have filed “Head of Household” for the last 4 years but for whom there is not enough information available in the IRS’ databases to determine the taxpayers’ marital status. 

The taxpayers in both samples will be asked to validate that they have chosen the correct filing status, which could affect their eligibility for the credit.  They will be asked to provide divorce decrees, legal separation documents, or other evidence to substantiate their filing status through an inquiry or the examination process.

The inventory selection model used for the Income portion of the Test supports the intent of the Test and should provide reliable results

The new case selection model, used to select income verification program EITC inventory, supports the purpose of the Income portion of the Test.  In this portion of the Test, the IRS is using a new EITC inventory selection model to select TY 2002 returns for verification.  Generally, the taxpayers selected for review have claimed the EITC but failed in the past to report all their income.  The new selection model was designed to isolate EITC returns and allow inventory selection based on the potential EITC change.

The IRS has always worked the EITC as part of its income verification program, but it has not selected returns based specifically on the potential EITC change.  In the past, the EITC was worked as a secondary issue.  The IRS will determine if the new EITC selection model is more productive than the general selection model. 

The IRS identified 1.2 million TY 2002 returns on which the taxpayers claimed the EITC and had underreported income.  Using the taxpayers’ history in its income verification program, the IRS created new EITC inventory categories.  Factors such as whether a taxpayer’s return was identified or worked in the past, whether additional tax was assessed, and whether the return fell out in the same income category were used to determine in which EITC inventory category the return belonged.  The cases were then filtered by:

·        The location in which the case would be worked.

·        EITC Inventory Category.  (An explanation of the EITC inventory categories can be found in Appendix V.)

·        Filing Status.

·        The number of EITC-qualifying children claimed. 

·        Position on the EITC bell curve. 

As illustrated in Chart 1, the EITC bell curve has five phases that cover the EITC tables.  As a taxpayer’s earned income amount increases, the EITC amount increases to the point at which the maximum EITC is reached.  Then as the earned income amount continues to increase, the EITC amount begins to decrease, which creates the EITC bell curve.

Chart 1:  The EITC Bell Curve

 

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

The filters allow the IRS to analyze and focus on specific sections of the income verification program EITC population.  Once the cases were filtered, the potential EITC changes were calculated and used to select the most productive cases.  The IRS used a cut-off amount applicable to the potential EITC change amount as a guide in selecting most of the cases.  This should also ensure a broad cross-section of cases is selected to develop a comprehensive income verification program EITC taxpayer profile.

Using the process described above, the IRS selected 245,717 returns using the new EITC case selection method.  An additional 54,283 returns were selected using the general selection model.  The volume of returns was selected based on resources available to work cases at each of the three locations where the IRS’ income verification program is worked and the amount of potential EITC change. 

In addition to using sound sampling methodologies for the Test, the IRS took additional steps to validate that the Certification and Filing Status samples were similar to their sampling populations.  The IRS validated the percentage of male/female taxpayers, the filing status claimed, and the state in which the taxpayer filed the return.  The IRS stated it validated that the data pulled from source files contained valid values for filing status; however, it was unable to provide documentation to support this effort.  The IRS also informed us that it performed manual calculations to test the accuracy of the new case selection model used in the Income portion of the Test.  However, it was also unable to provide documentation to support this effort.

Despite the IRS’ efforts to validate the various Test samples, it later found an error in the process it used to identify the Filing Status sampling population.  The IRS did, however, identify this problem early enough to be able to reselect the individual samples so the reliability of the samples was not affected. 

Data Collected During the Test Should Provide the Information Needed to Evaluate the Test Results

The data being gathered by the IRS during the Test should enable it to make well-informed decisions regarding the future of the EITC Program.  The IRS has developed a plan to collect data that will be used to evaluate the results of the Test.  The purpose for collecting these data is to provide the IRS with the information necessary to answer three key questions.

·        Does certifying the residency requirement for EITC-qualifying children reduce erroneous EITC claims without harming participation in the EITC Program?

·        Does verifying the filing status reduce erroneous EITC claims by married taxpayers incorrectly using the  “Single” or “Head of Household” filing status without harming participation in the EITC Program?

·        Does selecting EITC income verification program cases based on potential change in the EITC result in cases that are as productive as those selected through other income verification program methods? 

The IRS plans to collect data that will provide the information necessary to enable it to answer these key questions.  In addition to collecting the data, the IRS plans to validate the data through its quality review process.  This process should help ensure that the data are input into the various tracking systems correctly and that the data are reasonable.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to determine the usefulness of the Earned Income Tax Credit (EITC) Proof of Concept Test (the Test) in enabling the Internal Revenue Service (IRS) to make decisions regarding the future of its EITC Program.  To accomplish our objective, we:

I.                    Determined whether the Test samples were statistically valid and would provide reliable results to determine the success of the Test.  We obtained the sampling methodology documents for the Certification of Qualifying Child Residency Requirements (Certification) and Verification of Filing Status (Filing Status) samples and the case selection documents for the Verification of Income portion of the Test.  We also interviewed IRS personnel.

A.                 Determined whether the samples were implemented as designed and whether the case selection methodology for the EITC income verification program matched the intent of the Test.  We also determined the validation steps taken by the IRS to ensure the Test included the taxpayers intended.

B.                 Reviewed the IRS’ independent third party evaluation of the Certification portion of the Test.  We requested the opinion of the Treasury Inspector General for Tax Administration’s contracted statistician on the third party evaluation and the sampling methodology for the Filing Status portion of the Test.

II.                 Determined whether the IRS had plans to gather from the Test data elements that would provide information useful in making decisions regarding the future of the EITC Program. 

A.                 Interviewed IRS personnel and reviewed the initiative evaluation plan, the Wage and Investment Division’s Research Prospectus for Qualifying Child, and the independent third party evaluation of the Certification sample to determine the IRS’ plans for evaluating the results of the Certification subsamples.  We interviewed IRS personnel and reviewed the above documents to identify the data elements to be used in evaluating the Test and determined the reasons behind selecting those data elements.  We also compared the above documents to determine if there were data the IRS should collect that it was not planning to collect.

B.                 Interviewed IRS personnel and reviewed the Audit Information Management System Data Dictionary, information available from the Joint Operations Center Enterprise Telephone Data Dictionary, Automated Underreporter (AUR) Glossary, and AUR reports to determine whether the data the IRS plans to collect for the Test are already available from current Management Information Systems.

C.                 Interviewed IRS personnel and reviewed the Quality Job Aid for EITC Certification paper cases and AUR paper cases to determine whether the IRS will perform data validations on the data elements captured and on reports to be used for evaluating the Test.

 

Appendix II

 

Major Contributors to This Report

 

Michael R. Phillips, Assistant Inspector General for Audit (Wage and Investment Income Programs)

Scott A. Macfarlane, Director

Deann L. Baiza, Audit Manager

Karen C. Fulte, Senior Auditor

Kathleen A. Hughes, Senior Auditor

Steven Stephens, Senior Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Services and Enforcement  SE

Deputy Commissioner, Wage and Investment Division  SE:W

Director, Office of Research, Analysis, and Statistics  RAS

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Director, Compliance, Wage and Investment Division  SE:W:CP

Director, Earned Income Tax Credit, Wage and Investment Division  SE:W:EITC

Director, Reporting Compliance, Wage and Investment Division  SE:W:CP

Earned Income Tax Credit Program Manager  SE:W:EITC

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Office of Management Controls  OS:CFO:AR:M

Audit Liaison:  GAO/TIGTA Liaison, Wage and Investment Division  SE:W:S:PA

 

Appendix IV

 

Prior Treasury Inspector General for Tax Administration and General Accounting Office Reviews of the Earned Income Tax Credit Proof of Concept Test

 

Prior reviews of the Internal Revenue Service’s (IRS) Earned Income Tax Credit (EITC) Proof of Concept Test by the Treasury Inspector General for Tax Administration (TIGTA) and General Accounting Office (GAO) were completed and reports were issued in December and September 2003, respectively.

Management Controls Over the Proof of Concept Test of Earned Income Tax Credit Certification Need to Be Improved (Reference Number 2004-40-032, dated December 2003)

The TIGTA reported that, although the IRS accurately identified taxpayers for the Proof of Concept Test of EITC certification of qualifying children and used an appropriate sampling approach, management controls needed improvement.  Noting that continued change could jeopardize the usefulness of the Proof of Concept Test of EITC certification, the TIGTA advised that additional controls should be implemented, such as quantifying goals, establishing measures, and improving oversight of management information systems.

Earned Income Credit:  Qualifying Child Certification Test Appears Justified, but Evaluation Plan Is Incomplete (GAO-03-794, dated September 2003)

The GAO reported the Qualifying Child Certification Test appeared justified, but the evaluation plan was incomplete.  The report noted that the IRS’ plan for evaluating this test presented some information on how the IRS would evaluate whether certification would reduce the EITC overclaim rate, minimize burden, and maintain a relatively high participation rate.  However, it did not provide details on when decisions would be made or on the specific data that would be collected.  The GAO recommended the IRS Commissioner accelerate the development of the plan to evaluate the Certification test and have the plan demonstrate how the program’s objectives would be evaluated, including milestones for conducting the evaluation.

 

Appendix V

 

Income Verification Program Earned Income Tax Credit Inventory Categories

 

The Internal Revenue Service (IRS) created new Earned Income Tax Credit (EITC) inventory categories for its income verification program to test the new case selection methodology for the Proof of Concept Test.  The new categories and their criteria are as follows:

Repeater Freeze Criteria

·        Taxpayer’s returns were worked by the IRS’ income verification program 2 years in a row.

Repeater Non-Freeze Criteria

·        Taxpayer’s return had to be worked by the IRS’ income verification program at least once during the last 3 years.

·        Return does not meet the Repeater Freeze criteria.

Repeater Non-Worked Criteria

·        Taxpayer’s return showed up at least once in the income verification program inventory during the last 3 years.

Other

·        Return does not meet Repeater Freeze criteria.

 

Appendix VI

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.