The Statistical Sampling Method Used in the Earned Income
Tax Credit Proof of Concept Test Appears Valid
May 2004
Reference
Number: 2004-40-100
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
May
12, 2004
MEMORANDUM FOR
COMMISSIONER, WAGE AND INVESTMENT DIVISION
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Acting Deputy Inspector
General for Audit
SUBJECT: Final Audit Report - The Statistical
Sampling Method Used in the Earned Income Tax Credit Proof of Concept Test
Appears Valid (Audit # 200440016)
This
report presents the results of our review of the Earned Income Tax Credit
(EITC) Proof of Concept Test (the Test).
The overall objective of this review was to determine the usefulness of
the Test in enabling the Internal Revenue Service (IRS) to make decisions
regarding the future of its EITC Program.
The EITC is a refundable credit available to taxpayers that file returns
with certain earned income. While the
EITC has provided significant benefits to taxpayers, it has also resulted in a
significant loss of revenue to the Federal Government. An IRS compliance study of Tax Year (TY) 1999
returns estimated that between $8.5 and $9.9 billion (27 to 32 percent) of the
$31 billion in EITC claimed for TY 1999 should not have been paid.
This
audit focused on the statistical sampling methodologies used to select the
various samples for the Test. However,
we did not assess the selection criteria used for each sample. In addition, because the IRS had an
independent third party validate the design of the Certification of Qualifying
Child Residency Requirements (Certification) portion of the Test, our review of
the Certification sample was limited to asking the Treasury Inspector General
for Tax Administration’s contracted statistician to review and provide an
overall assessment of the third party report.
Overall,
the statistical sampling method used to select the samples for the Test appears
adequate and should provide reliable information on which to base future
decisions. The Test consists of three
portions—Certification, Verification of Filing Status (Filing Status), and
Verification of Income (Income). The
IRS selected statistical samples for both the Certification and Filing Status
portions of the Test. The IRS actually
selected larger samples than necessary, so the Test results should be very
precise. The IRS did not use a
statistical sample for the Income portion of the Test. It developed a new case selection
methodology and selected the Income inventory based on a set of information
filters and available resources. The
new case selection methodology supports the intent of the Income portion of the
Test and should also provide reliable results.
We do, however, have some concerns with the sizes of the
subsamples in the Certification portion of the Test and the impact IRS contact
with taxpayers included in this portion could have on the Test results. While these concerns do not affect the
reliability of the samples, they could affect the interpretation of the Test
results. We suggested the IRS use
caution when interpreting and relying upon these results.
Management’s Response: IRS management is pleased that the Test should enable them to
make well-informed decisions regarding the future of the EITC Program. The IRS recognizes the limitation of the
small sample sizes for the subsamples related to the Certification portion of
the Test but believes the subsamples will provide meaningful insights that can
be used to develop more quantifiable tests.
In addition, management believes the design of the Certification portion
of the Test includes procedures to address our concerns about the impact IRS
contact with taxpayers included in this portion could have on the Test
results. Management’s complete response
to the draft report is included as Appendix VI.
Copies of this
report are also being sent to the IRS managers affected by the report results. Please contact me at (202) 622-6510 if you
have questions or Michael R. Phillips, Assistant Inspector General for Audit
(Wage and Investment Income Programs), at (202) 927-0597.
The Samples Used in the Test Should Generally
Provide Reliable Results
Data Collected During the Test Should Provide the Information Needed to Evaluate the Test Results
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix V – Income
Verification Program Earned Income Tax Credit Inventory Categories
Appendix VI
– Management’s Response to the Draft Report
The Earned Income Tax Credit (EITC) is a refundable credit available to taxpayers that file returns with certain earned income. While the EITC has provided significant benefits to taxpayers, it has also resulted in a significant loss of revenue to the Federal Government. An Internal Revenue Service (IRS) compliance study of Tax Year (TY) 1999 returns estimated between $8.5 and $9.9 billion (27 to 32 percent) of the $31 billion in EITC claimed for TY 1999 should not have been paid.
On February 28, 2002, the Department of the Treasury and the IRS announced that a Task Force would examine the administration and complexity of the EITC. The Treasury Assistant Secretary for Tax Policy and the IRS Commissioner headed the Task Force. The Task Force used the TY 1999 compliance study as the basis for its recommendations. The IRS used these recommendations to develop a future vision for administering the EITC. This vision is outlined in the IRS’ EITC Concept of Operations (CONOPS).
The first step the IRS is taking to implement its future vision is to test the basic concepts outlined in the CONOPs. This test is referred to as a Proof of Concept Test (the Test) and is intended to provide the IRS with information necessary to determine if these basic concepts are feasible for improving compliance with the EITC without harming participation in the EITC Program.
The Test consists of three portions designed to test concepts for improving EITC compliance:
· Certification of Qualifying Child Residency Requirements (Certification) – The IRS will ask a sample of EITC claimants to verify, when they file their TY 2003 returns, that the qualifying child claimed for EITC purposes resided with them in the United States for more than one-half of the tax year, as required by law. These taxpayers are being treated differently from other taxpayers that claim the EITC, and the IRS is trying to capture the impact this treatment has on improving compliance with the EITC.
· Verification of Filing Status (Filing Status) – The IRS will review the TY 2003 returns for a sample of EITC claimants filing as “Head of Household” or “Single” to verify whether the filing status they claimed is accurate. The IRS hopes to find a better way to identify filing status errors by evaluating the usefulness of adding additional information to the Schedule Earned Income Credit. The IRS will also assess the value of comparing the information to other available sources in the verification process.
· Verification of Income (Income) – The IRS will review the TY 2002 returns for taxpayers that claimed the EITC but failed in the past to report all their income. These taxpayers may not be eligible because their income, when corrected, is too high to qualify for the credit. The IRS is testing whether cases selected for its income verification program based on the potential change in the EITC are as productive as its regular inventory.
This audit focused on the statistical sampling methodologies used to select the various samples for the Test. However, we did not assess the sample selection criteria used for each sample. In addition, because the IRS had an independent third party validate the design of the Certification portion of the Test with a focus on the sample selection, our review of the Certification sample was limited to asking the Treasury Inspector General for Tax Administration’s contracted statistician to review the third party report and provide an overall assessment of the results.
This audit is the third in a series of audits we are conducting on the Test. We have completed one review and the second is in process. Information on our previous audit report, along with the results of a review of the Test by the General Accounting Office, can be found in Appendix IV.
This review was performed in the Office of Research, Analysis, and Statistics in the National Headquarters in Washington, D.C., and in the Wage and Investment Division Office of Compliance and EITC Office in Atlanta, Georgia, during the period November 2003 through February 2004. The audit was conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The samples the IRS selected for the Certification and Filing Status portions of the Test are statistically valid and should provide reliable results. The IRS actually selected larger samples than necessary, so the Test results should be very precise. The Income portion of the Test did not use a statistical sample. Rather, the IRS is testing a new method for selecting EITC inventory for income verification. This new case selection methodology supports the intent of the Income portion of the Test and should also provide reliable results.
Overall, the method used to select the various samples for the Certification portion of the Test is statistically valid. The method should also result in reliable results that the IRS can use to make future EITC Program decisions.
In the Certification portion of the Test, the IRS is asking certain taxpayers to validate the residency requirements for their qualifying children. The IRS started with a representative sample of approximately 1.6 million taxpayers that claimed the EITC with qualifying child(ren) and filed timely tax returns for TY 2002. The IRS systemically certified as many taxpayers as possible using available computer databases. It also applied various exclusions (e.g., those already selected for examination) to ensure the IRS would contact the taxpayers only once on the same return. It then selected a statistical sample of 25,000 taxpayers from approximately 402,000 taxpayers that could not be systemically certified. The IRS used a systematic sampling method to select the sample.
These 25,000 taxpayers are being asked to provide affidavits, letters, or documents from a landlord or property manager, police officer, teacher, or other official in the taxpayer’s community that verify the residency requirement of the qualifying child claimed. The IRS subsequently selected 2 subsamples out of the 25,000 to test alternative documentation requirements. One subsample of 1,000 taxpayers will be able to ask friends and relatives to sign an affidavit attesting to a qualifying child’s residence. A second subsample of 1,000 taxpayers will receive forms printed in both English and Spanish to gauge whether participation in the Test increases due to the distribution of materials in both languages.
The IRS contracted with an independent third party to validate the design of the Certification portion of the Test. To evaluate alternative forms, the third party assessed the sample design by reviewing how the IRS identified the population eligible for certification, the designation of a control group, the sample selection, and the subsamples. The third party assessment also reviewed the Certification process by looking at IRS contact with the participants and verification of residency. The third party looked at the IRS’ plans to evaluate the results of the Test by assessing the strengths and limitations of the overall Certification design, reviewing how the IRS can estimate the net effect of Certification, and discussing how the IRS can further develop the analysis plan.
The third party endorsed the method the IRS used to identify the taxpayers eligible for the Certification portion of the Test. It determined that a valid approach was used to select the sample and the control group and that the sample size is more than sufficient to address the objectives of the Certification test. The third party also determined that a subsample of 1,000 taxpayers is sufficient to test the effects of the alternative affidavit. However, the third party indicated the second subsample of 1,000 taxpayers is not sufficient to reliably detect whether participation in the Test increases among those families that can read Spanish but not English.
We reviewed the third party’s assessment and generally agree with it. We are, however, concerned that the sizes of the subsamples within the Certification portion of the Test are not large enough to provide substantial and significant results. This issue was also raised in the IRS’ third party assessment but only in reference to the Spanish notice subsample. The IRS acknowledged that the Spanish notice subsample would probably not provide reliable results. At best, it will provide anecdotal information. While neither the third party’s nor our concerns affect the statistical validity of the subsamples, they do affect how the information gained from these subsamples should be used. As a result, we suggest the IRS use the results of these two subsamples with caution.
We also have a concern about how IRS contact with the taxpayers in the Certification portion of the Test could affect the test results. The IRS sent notices in December 2003 informing the 25,000 taxpayers in the sample that they were part of this Test. We are concerned that some of these taxpayers may have moved and would not have received the materials that explain how the certification process can be accomplished successfully. Also, some tax return preparers may be unaware of the Test and the importance of certification for this small group of taxpayers. While the effect of the IRS contact does not affect the reliability of the sample, it could affect the interpretation of the Test results. Again, we suggest the IRS use caution when interpreting and relying upon these results.
The samples selected for the Filing Status portion of the Test are statistically valid. In addition, the samples should provide reliable results for the IRS to use to make future EITC Program decisions.
In the Filing Status portion of the Test, the IRS plans to verify the filing status of two different samples of taxpayers. The IRS used the same representative sample of approximately 1.6 million taxpayers to determine the Filing Status sampling population. The Certification sampling population was removed from the 1.6 million before the Filing Status samples were selected. The taxpayers identified for the Income portion of the Test were also removed so taxpayers would be affected by only one portion of the Test. The IRS used the same systematic sampling approach as that used in the Certification portion of the Test to select both of the Filing Status samples. We reviewed the sampling methodology and determined the samples are appropriate to address the purpose of the Test.
The IRS selected a statistical sample of 36,000 taxpayers from approximately 69,000 taxpayers whose filing status history indicated they had been previously married and then filed as “Single” or “Head of Household” on their TY 2002 tax returns. A second statistical sample of 5,000 taxpayers was selected from approximately 600,000 taxpayers that have filed “Head of Household” for the last 4 years but for whom there is not enough information available in the IRS’ databases to determine the taxpayers’ marital status.
The taxpayers in both samples will be asked to validate that they have chosen the correct filing status, which could affect their eligibility for the credit. They will be asked to provide divorce decrees, legal separation documents, or other evidence to substantiate their filing status through an inquiry or the examination process.
The new case selection model, used to select income verification program EITC inventory, supports the purpose of the Income portion of the Test. In this portion of the Test, the IRS is using a new EITC inventory selection model to select TY 2002 returns for verification. Generally, the taxpayers selected for review have claimed the EITC but failed in the past to report all their income. The new selection model was designed to isolate EITC returns and allow inventory selection based on the potential EITC change.
The IRS has always worked the EITC as part of its income verification program, but it has not selected returns based specifically on the potential EITC change. In the past, the EITC was worked as a secondary issue. The IRS will determine if the new EITC selection model is more productive than the general selection model.
The IRS identified 1.2 million TY 2002 returns on which the taxpayers claimed the EITC and had underreported income. Using the taxpayers’ history in its income verification program, the IRS created new EITC inventory categories. Factors such as whether a taxpayer’s return was identified or worked in the past, whether additional tax was assessed, and whether the return fell out in the same income category were used to determine in which EITC inventory category the return belonged. The cases were then filtered by:
· The location in which the case would be worked.
· EITC Inventory Category. (An explanation of the EITC inventory categories can be found in Appendix V.)
· Filing Status.
· The number of EITC-qualifying children claimed.
· Position on the EITC bell curve.
As illustrated in Chart 1, the EITC bell curve has five phases that cover the EITC tables. As a taxpayer’s earned income amount increases, the EITC amount increases to the point at which the maximum EITC is reached. Then as the earned income amount continues to increase, the EITC amount begins to decrease, which creates the EITC bell curve.
Chart 1: The EITC Bell Curve
The chart was removed due to its size. To see the chart, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
The filters allow the IRS to analyze and focus on specific sections of the income verification program EITC population. Once the cases were filtered, the potential EITC changes were calculated and used to select the most productive cases. The IRS used a cut-off amount applicable to the potential EITC change amount as a guide in selecting most of the cases. This should also ensure a broad cross-section of cases is selected to develop a comprehensive income verification program EITC taxpayer profile.
Using the process described above, the IRS selected 245,717 returns using the new EITC case selection method. An additional 54,283 returns were selected using the general selection model. The volume of returns was selected based on resources available to work cases at each of the three locations where the IRS’ income verification program is worked and the amount of potential EITC change.
In addition to using sound sampling methodologies for the Test, the IRS took additional steps to validate that the Certification and Filing Status samples were similar to their sampling populations. The IRS validated the percentage of male/female taxpayers, the filing status claimed, and the state in which the taxpayer filed the return. The IRS stated it validated that the data pulled from source files contained valid values for filing status; however, it was unable to provide documentation to support this effort. The IRS also informed us that it performed manual calculations to test the accuracy of the new case selection model used in the Income portion of the Test. However, it was also unable to provide documentation to support this effort.
Despite the IRS’ efforts to
validate the various Test samples, it later found an error in the process it
used to identify the Filing Status sampling population. The IRS did, however, identify this problem
early enough to be able to reselect the individual samples so the reliability
of the samples was not affected.
The data being gathered by the IRS during the Test should
enable it to make well-informed decisions regarding the future of the EITC
Program. The IRS has developed a plan
to collect data that will be used to evaluate the results of the Test. The purpose for collecting these data is to
provide the IRS with the information necessary to answer three key questions.
· Does certifying the residency requirement for EITC-qualifying children reduce erroneous EITC claims without harming participation in the EITC Program?
· Does verifying the filing status reduce erroneous EITC claims by married taxpayers incorrectly using the “Single” or “Head of Household” filing status without harming participation in the EITC Program?
· Does selecting EITC income verification program cases based on potential change in the EITC result in cases that are as productive as those selected through other income verification program methods?
The IRS plans to collect data that will provide the information necessary to enable it to answer these key questions. In addition to collecting the data, the IRS plans to validate the data through its quality review process. This process should help ensure that the data are input into the various tracking systems correctly and that the data are reasonable.
Appendix I
Detailed Objective, Scope,
and Methodology
The
overall objective of this review was to determine the usefulness of the Earned
Income Tax Credit (EITC) Proof of Concept Test (the Test) in enabling the
Internal Revenue Service (IRS) to make decisions regarding the future of its
EITC Program. To accomplish our objective,
we:
I.
Determined whether the Test
samples were statistically valid and would provide reliable results to
determine the success of the Test. We
obtained the sampling methodology documents for the Certification of Qualifying
Child Residency Requirements (Certification) and Verification of Filing Status
(Filing Status) samples and the case selection documents for the Verification
of Income portion of the Test. We also
interviewed IRS personnel.
A.
Determined whether the
samples were implemented as designed and whether the case selection methodology
for the EITC income verification program matched the intent of the Test. We also determined the validation steps
taken by the IRS to ensure the Test included the taxpayers intended.
B.
Reviewed the IRS’ independent
third party evaluation of the Certification portion of the Test. We requested the opinion of the Treasury
Inspector General for Tax Administration’s contracted statistician on the third
party evaluation and the sampling methodology for the Filing Status portion of
the Test.
II.
Determined whether the IRS
had plans to gather from the Test data elements that would provide information
useful in making decisions regarding the future of the EITC Program.
A.
Interviewed IRS personnel and
reviewed the initiative evaluation plan, the Wage and Investment Division’s
Research Prospectus for Qualifying Child, and the independent third party
evaluation of the Certification sample to determine the IRS’ plans for
evaluating the results of the Certification subsamples. We interviewed IRS personnel and reviewed
the above documents to identify the data elements to be used in evaluating the
Test and determined the reasons behind selecting those data elements. We also compared the above documents to
determine if there were data the IRS should collect that it was not planning to
collect.
B.
Interviewed IRS personnel and
reviewed the Audit Information Management System Data Dictionary, information
available from the Joint Operations Center Enterprise Telephone Data
Dictionary, Automated Underreporter (AUR) Glossary, and AUR reports to
determine whether the data the IRS plans to collect for the Test are already
available from current Management Information Systems.
C.
Interviewed IRS personnel and
reviewed the Quality Job Aid for EITC Certification paper cases and AUR paper cases
to determine whether the IRS will perform data validations on the data elements
captured and on reports to be used for evaluating the Test.
Appendix II
Major Contributors to This Report
Michael R. Phillips, Assistant Inspector General for Audit
(Wage and Investment Income Programs)
Scott
A. Macfarlane, Director
Deann L. Baiza, Audit Manager
Karen C. Fulte, Senior Auditor
Kathleen
A. Hughes, Senior Auditor
Steven Stephens, Senior Auditor
Appendix III
Commissioner C
Office of the
Commissioner – Attn: Chief of
Staff C
Deputy Commissioner
for Services and Enforcement SE
Deputy
Commissioner, Wage and Investment Division
SE:W
Director, Office of Research, Analysis, and Statistics RAS
Director, Office of Program Evaluation and Risk
Analysis RAS:O
Director,
Compliance, Wage and Investment Division
SE:W:CP
Director, Earned
Income Tax Credit, Wage and Investment Division SE:W:EITC
Director, Reporting Compliance, Wage and Investment
Division SE:W:CP
Earned Income Tax Credit Program Manager SE:W:EITC
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Office of
Management Controls OS:CFO:AR:M
Audit Liaison: GAO/TIGTA Liaison, Wage and Investment
Division SE:W:S:PA
Appendix IV
Prior reviews of the Internal Revenue Service’s (IRS)
Earned Income Tax Credit (EITC) Proof of Concept Test by the Treasury Inspector
General for Tax Administration (TIGTA) and General Accounting Office (GAO) were
completed and reports were issued in December and September 2003, respectively.
Management Controls Over the Proof of Concept Test
of Earned Income Tax Credit Certification Need to Be Improved (Reference
Number 2004-40-032, dated December 2003)
The TIGTA reported that, although the IRS accurately identified taxpayers for the Proof of Concept Test of EITC certification of qualifying children and used an appropriate sampling approach, management controls needed improvement. Noting that continued change could jeopardize the usefulness of the Proof of Concept Test of EITC certification, the TIGTA advised that additional controls should be implemented, such as quantifying goals, establishing measures, and improving oversight of management information systems.
Earned Income Credit: Qualifying Child Certification Test Appears Justified, but
Evaluation Plan Is Incomplete (GAO-03-794, dated September 2003)
The GAO reported the Qualifying Child Certification Test
appeared justified, but the evaluation plan was incomplete. The report noted that the IRS’ plan for
evaluating this test presented some information on how the IRS would evaluate
whether certification would reduce the EITC overclaim rate, minimize burden,
and maintain a relatively high participation rate. However, it did not provide details on when decisions would be
made or on the specific data that would be collected. The GAO recommended the IRS Commissioner accelerate the
development of the plan to evaluate the Certification test and have the plan
demonstrate how the program’s objectives would be evaluated, including
milestones for conducting the evaluation.
Appendix V
Income Verification Program
Earned Income Tax Credit Inventory Categories
The Internal Revenue Service (IRS) created new Earned Income Tax Credit
(EITC) inventory categories for its income verification program to test the new
case selection methodology for the Proof of Concept Test. The new categories and their criteria are as
follows:
Repeater Freeze Criteria
· Taxpayer’s returns were worked by the IRS’ income verification program 2 years in a row.
Repeater Non-Freeze Criteria
· Taxpayer’s return had to be worked by the IRS’ income verification program at least once during the last 3 years.
· Return does not meet the Repeater Freeze criteria.
Repeater Non-Worked Criteria
· Taxpayer’s return showed up at least once in the income verification program inventory during the last 3 years.
Other
· Return does not meet Repeater Freeze criteria.
Appendix VI
Management’s Response to
the Draft Report
The response was removed due to its size. To see the response, please go to the Adobe
PDF version of the report on the TIGTA Public Web Page.