The Examination Quality Measurement System Program Can Be Enhanced
January 2003
Reference Number: 2003-30-041
This report has cleared the Treasury Inspector General
for Tax Administration disclosure review process and information determined to
be restricted from public release has been redacted from this document.
January 15, 2003
MEMORANDUM
FOR COMMISSIONER, SMALL BUSINESS/SELF-EMPLOYED DIVISION
FROM: Gordon
C. Milbourn III /s/ Gordon C. Milbourn III
Acting Deputy Inspector General for Audit
SUBJECT: Final Audit Report - The Examination Quality Measurement System Program Can Be Enhanced (Audit # 200230021)
This report presents the results of our review of the Internal Revenue Service’s (IRS) Examination Quality Measurement System (EQMS) Program. The overall objective of this review was to determine whether the EQMS Program is effective to measure the quality of the Examination function’s work and if the results are used to improve overall effectiveness.
The EQMS Program is the Small Business/Self-Employed (SB/SE) Division’s process to measure examination quality and assess long-term trends of performance, in keeping with the IRS’ balanced measures. The EQMS Program’s results are used to identify national trends, system changes, and training needs; to establish baselines; and to provide an understanding of the quality of examinations. The EQMS Program is responsible for conducting reviews of a statistically valid sample of completed examination cases, based on certain standard criteria. During Fiscal Year (FY) 2001, the EQMS Program reviewed 14,075 cases.
The SB/SE Division’s Examination function is effectively using the EQMS Program results to help improve overall examination quality. For example, a group manager in one area office contributed articles to a territory office newsletter informing revenue agents which standards/issues they need to focus on to improve their case closures, based on the EQMS Program’s results. Also, some area directors and territory managers prepared action plans to identify areas to improve the quality of examinations. Based on the most recent EQMS Program reports, the Examination function’s overall quality improved by 5 percent for field examination cases and 1 percent for office examination cases from the end of FY 2001 through the first 3 quarters of FY 2002.
However, we identified four areas in which the EQMS Program could be improved. First, the EQMS Program did not identify the complete population of qualified, completed examinations that should have been subject to the sampling process. Only 38 percent of the EQMS Program-qualified field examination cases, and only 51 percent of the EQMS Program-qualified office examination cases, were actually selected for review from the 3 area offices we visited. As a result, the EQMS Program results were not statistically valid, since each case did not have an equal chance of being selected for review. In addition, EQMS Program management did not adequately document or monitor the sampling process or make periodic adjustments to the sample size when needed.
Second, the number
of cases actually reviewed nationwide by the EQMS Program in FY 2001 was
approximately 3,400 more than required to produce statistically valid
results. The additional hours needed to
review these returns could have been put to better use in examining tax
returns. Based on an average time of 5
and 2.5 hours, respectively, to review a field case and an office case, and
based on Examination function statistics for FY 2001, we estimate an additional
800 tax returns could have been examined, producing additional assessments of
about $9.8 million in tax liabilities in FY 2001. The decline in the number of examinations conducted has been a
concern of the Congress and the IRS Oversight Board in the past several
years. Taking the staff hours needed to
conduct these excess EQMS Program case reviews and using them, instead, to
conduct actual examinations of tax returns would help alleviate this concern. The IRS has recently changed its approach
regarding the type of tax returns it will be examining. It has aligned its strategic priorities to
address six specific areas of non-compliance.
Therefore, the results from shifting the resources from quality reviews
to conducting examinations may differ from FY 2001 statistics.
Third, the SB/SE Division’s plan to make the
EQMS Program results statistically valid at the territory office level, rather
than at the current area office level, will greatly increase the resources
needed to conduct the case reviews. The
total statistically valid area office sample size would need to be increased by
about 400 percent to make territory results statistically valid, requiring
approximately 110 additional EQMS Program reviewers.
Finally,
neither the examiners nor their group managers receive individual feedback from
the EQMS Program reviewers concerning problems identified with the quality of
their cases. Twelve of 14 area
directors, territory managers, group managers, and Technical Support managers
we interviewed believe that individual feedback is crucial for individual
examiners to improve examination case quality.
We recommended the
Director, Compliance, SB/SE Division, resubmit the previously rejected Request
for Information Services (RIS) to automate the sampling process to help ensure
all qualified cases are subject to review, thus helping promote statistical
validity. Until such time when
electronic sampling is implemented, the Director, Compliance, SB/SE Division,
should re-emphasize the EQMS Program criteria and the need for reviews so that
all cases meeting the criteria are being properly identified. In addition, the Director, Compliance, SB/SE
Division, should establish and document an approved sampling plan that provides
for statistical validity, as well as monitor and periodically evaluate the plan
to make any appropriate revisions. The
Director, Compliance, SB/SE Division, should ensure that
only the number of cases that constitute a statistically valid sample are
reviewed and evaluate the impact that valid territory office statistical
sampling would have. Finally, while we
understand past agreements with the National Treasury Employees Union (NTEU)
prevent individual feedback to examiners, it could be a very effective training
tool. Therefore, the Director,
Compliance, SB/SE Division, should consider providing this type of feedback
after resolving this issue with the NTEU.
Management’s Response: The
Commissioner, SB/SE Division, agreed that they must improve the EQMS
Program. SB/SE Division management recognized
the need for an automated system and submitted a RIS on October 10, 2002, to
request automation support. The
Compliance Support Design Team is also considering automating the EQMS sampling
process. SB/SE Division management will
re-issue the sampling procedures and expectations to all Case Processing
sites. They will provide training and
assistance visits to the Case Processing sites to ensure consistent and correct
application of the sampling procedures.
SB/SE Division management
will maintain a documented sampling plan at all times, monitor the sampling
process, and make any necessary periodic adjustments. Additionally, they will re-evaluate the impact statistically
valid territory level sampling will have on staffing levels. This evaluation will include a discussion of
the impact on balanced measures.
Finally, they agreed with
the need to share as much feedback with the examiners as possible and to pursue
resolution of this issue with the NTEU.
In the interim, all levels of SB/SE Division management will review the
results of the EQMS Program and share them generally with the examiners to
improve performance.
While agreeing with our
recommendations, SB/SE Division management disagreed with our conclusion about
the revenue impact of reviewing more cases than the sample required. Management’s complete response to the draft
report is included as Appendix VI.
Office of Audit Comment: Although
SB/SE Division management agreed with the report’s recommendations and plans to
take appropriate corrective action, they do not agree with the revenue impact
of using EQMS Program resources to examine tax returns. SB/SE Division management stated that they
assign reviewers in the EQMS Program for a specified period of time; they do
not shift reviewers between reviewing and examining returns. Their reviewers are also responsible for
providing support to the field in understanding the EQMS Program standards and
applications through making presentations, attending group meetings, and
conducting various other activities.
The Compliance initiatives in the FY 2003 Business Plan require
examination of more complex returns, which would require even more in-depth
reviews than those conducted during the period the audit report covered.
We agree that shifting reviewers
throughout the year between reviewing and examining returns would not be
prudent. At the beginning of each year,
however, management should be able to use their sampling plan to determine the
number of returns that need to be reviewed based upon the number of returns
planned for examination in the Annual Examination Plan. If all existing EQMS Program resources are
not needed to review those returns and provide support to the field, this is an
opportunity for the IRS to conduct additional examinations of tax returns.
Copies of this report are
also being sent to IRS officials who are affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Parker F. Pearson, Acting Assistant
Inspector General for Audit (Small Business and Corporate Programs), at (410)
962-9637.
Examination Quality Measurement System Results Were Being Used Effectively
Examination Quality Measurement System Results Were Not Statistically Valid
More Cases Were Being Reviewed Than Required for Statistical Validity
Sufficient Feedback Was Not Given to Examiners
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Outcome Measures
Appendix V – Comparison of Estimated Statistically Valid Territory and Area Office Samples
Appendix VI – Management’s Response to the Draft Report
The Internal
Revenue Service (IRS) Examination function is a nationwide program that selects
and examines Federal tax returns to determine taxpayers’ correct
liabilities. Examinations are conducted
at the “field” and “office” levels. Field examinations involve individuals, partnerships, and
corporations and generally occur at the taxpayer’s place of business. Office examinations usually involve
individuals and are conducted through interviews at an IRS office.
The Examination Quality
Measurement System (EQMS) Program is the Small Business/Self-Employed (SB/SE)
Division’s process to measure examination quality and assess long-term trends
of performance. The EQMS Program is
part of the IRS’ balanced measures process that includes customer satisfaction,
employee satisfaction, and business results.
The EQMS Program’s results are not used to evaluate individual employee
performance. Instead, they are to be used
to identify national trends, system changes, and training needs; to establish
baselines; and to provide an understanding of the quality of examinations.
Approximately
59 EQMS Program reviewers are located in 9 offices nationwide. These employees conduct reviews of completed
examination cases and provide data to the Office of Performance Analysis, which
in turn provides results data to the area offices. During Fiscal Year (FY) 2001, the EQMS Program reviewed 14,075
cases, including 6,879 field examination and 7,196 office examination
cases. A case is defined as either an
examination of one return for a taxpayer or any group of multiple year returns
and/or related returns closed as a related package. Random sample selection is used to help ensure the reliability of
the results.
Cases are reviewed to assess adherence to established quality
standards. Quality standards are
concise statements of expectations for quality examinations and are guidelines
to assist examiners in fulfilling their professional responsibilities. The eight standards essentially define
“quality” for examinations and provide objective criteria against which case
quality is evaluated. They address
planning, scope of examination, income probes, techniques, workpapers and
reports, tax law application, time management, and customer relations. Each quality standard is defined by elements
representing components that are present in a quality examination.
At the time of our review, the EQMS Program was organizationally placed
in the Office of Performance Measurement (Compliance) and reported to the
Office of Strategy, Research, and Performance Measurement (SRPM). However, as of June 1, 2002, Performance
Measurement (Compliance) was moved to the SB/SE Division’s Compliance function.
We visited the Chicago,
Los Angeles, and Detroit Area Offices where we interviewed Examination and Case
Processing management. We also
interviewed EQMS Program management in Buffalo, New York, and Los Angeles,
California, and at the SB/SE Division’s Headquarters office. We conducted the audit between February and
July 2002 in accordance with Government Auditing Standards. Detailed
information on our audit objective, scope, and methodology is presented in Appendix I.
Major contributors to the report are listed in Appendix II.
The SB/SE Division’s Examination function is effectively using the EQMS
Program’s results to help improve overall examination quality. Based on the most recent EQMS Program
reports, the Examination function’s overall quality score improved by 5 percent
for field examinations and 1 percent for office examinations from the end of FY
2001 through the first 3 quarters of FY 2002.
We compared the
EQMS Program standards to the SB/SE Division’s strategic goals for FYs 2002 and
2003 and the Critical Job Elements (CJE) for revenue agents (field) and tax
compliance officers (office). The EQMS
Program standards adequately supported the IRS business results measures and
the CJEs.
We visited two EQMS Program sites and determined they were timely
reviewing and providing results to Examination function management. The EQMS Program goal is to assign the cases
to reviewers within 60 days of receipt and to complete the reviews within 30
days of the assignment. Our analyses in
2 EQMS Program offices showed that they were generally assigning cases to
reviewers within a 60-day period. Our
review of inventory reports in 1 EQMS Program office showed that 81 percent of
the cases were reviewed within 30 days and 98 percent were reviewed within 45
days. In addition, the EQMS Program
submits completed review results to the SRPM office once a month and makes them
available to SB/SE Division management within 5 to 10 days of receiving them.
The EQMS Program has several internal methods for ensuring the quality of
its own reviews. The EQMS Program
managers and reviewers periodically meet and conduct consistency checks to help
ensure reviewers are uniformly applying the quality standards. Reviewers and managers discuss specific
elements and case scenarios and independently evaluate the same case, discuss
the ratings, and critique the review process.
In the area offices we visited, EQMS Program management had also
provided on-site guidance/training on proper sample selection procedures for
case processing employees.
The SB/SE
Division’s Examination function managers in the three area offices we visited
were actively involved with the improvement of examination quality. Some of the actions taken
as a result of the EQMS Program reviews included:
·
Some area directors and territory managers prepared
action plans to identify areas to improve the quality of examinations. Some also periodically conducted workload and operational reviews.
·
A group manager in one area office contributed
articles to her territory’s newsletter to inform the revenue agents which
standards/issues they need to focus on to improve their case closures.
·
One area
director conducted a town hall meeting and discussed the EQMS Program results
and plans to improve quality in examinations.
The SRPM office
also plays a role in improving examination quality. For example, SRPM office staff reviewed the narratives pertaining
to all the cases that did not meet the standard involving report preparation in
FY 2001. To help improve on this
standard, they issued a number of quick fixes/job aids for the examiners to
follow prior to closing cases. The SRPM
office also developed a website where the EQMS Program results are accessible
to more IRS employees and can be obtained at more frequent intervals (monthly)
than before. In addition, it prepares a
quarterly report that gives an analysis of the quality standards and includes
recommendations to improve examination quality. The SRPM office also produces a national EQMS Program newsletter
that gives procedures and methods for improvements when quality standards are
not being met.
Finally,
Technical Support liaisons analyze the EQMS Program results and issue quarterly
reports to Examination function management for their specific areas. A National Task Force was convened and
developed a standardized report format for the EQMS Program results to provide
consistency of reporting to Examination managers. The standardized report includes: a table of area office results by standards and elements, an
illustrative area office chart, a table of territory office results by
standards and elements for each territory manager, and a report of reviewer
narratives for each territory manager.
Managers can also request optional reports containing more detailed
analysis of the EQMS Program results.
The EQMS
Program did not identify the complete population of qualified, completed
examinations that should have been subject to the sampling process. Therefore, each case did not have an equal
chance of being selected for review. As
a result, the EQMS Program results were not statistically valid. In addition, EQMS Program management did not
adequately document or monitor the sampling process or make periodic
adjustments to the sample size when needed.
Not all the
EQMS Program-qualified cases were identified for sampling
The EQMS
Program reviews randomly selected cases to collect information from completed
examinations. Most types of examined
income tax return cases qualify for the EQMS Program review. However, unexamined (surveyed) income tax
returns and examined estate, gift, excise, and some employment tax returns do
not qualify for EQMS Program review.
The EQMS
Program set out to select and review a statistically valid random sample of
cases because it was not physically and economically possible to review 100 percent
of the population. Examining a
statistically valid sample of a universe allows one to make conclusions about
the universe based on information from the sample. A random sample is one that seeks to represent, as closely as
possible, the population from which it was drawn. Random sample selection means that every item in the population
has an equal chance of being selected.
The sample size
originally established for the EQMS Program called for selecting 1 in every 25
completed field examination cases and 1 in every 30 completed office
examination cases. Management advised
us that due to a decrease in the number of closed examinations and problems
with implementation of the sample selection procedures, the number of cases
being sampled and submitted for EQMS Program review was not adequate. In an effort to increase the number of cases
sampled and submitted for review, revised procedures went into effect October 1,
2000, increasing the sample size to 1 in 3 for field examination cases and 1 in
5 for office examination cases.
To determine
whether sampling procedures were adequate, we analyzed the Examination
function’s file of closed examinations generated from the Audit Information
Management System (AIMS) for the three offices we visited. We also reviewed the EQMS Program reports
for FY 2001 to identify the number of cases the EQMS Program actually reviewed.
We estimate
that, in FY 2001, the SB/SE Division in these 3 area offices conducted 8,200
field and 15,500 office examination cases that met the EQMS Program review
criteria. If the 1 of 3 and 1 of 5
selection plan was used, the number of cases selected should have been about 2,700
(field) and 3,100 (office). However,
only 1,016 field cases and 1,594 office cases were actually selected for
review, 38 and 51 percent of what should have been selected, respectively.
To further evaluate the adequacy of the case selection process, we
requested the Case Processing function in 2 area offices to set aside all the
EQMS Program cases for a 1-day period that they determined did not meet EQMS
Program review criteria. Case
Processing units in the area offices are responsible for identifying and
processing cases for the EQMS Program sample.
The Case Processing staff identified 19 cases not meeting EQMS
Program criteria.
Our review of these 19 cases determined that 6 (32 percent) should have
been included in the population, since they met the EQMS Program criteria. Case Processing management agreed that these
six cases should have been included in the population and informed us that some
of their personnel were new to the position.
Also, one office was using an obsolete listing of the sample selection
criteria.
Technical Support liaisons in area offices are responsible for monitoring
the sampling process to ensure that qualified cases are properly selected in
accordance with prescribed EQMS Program procedures, and that the sample is
selected timely, continuously, and at the proper selection rate. However, the Technical Support liaison in
one area office did not conduct any reviews to ensure all EQMS Program-qualified
cases were identified during the sampling process.
Management recognized the inherent risks associated with a manual sample
selection process, including human error, which could make their results not
statistically valid. They submitted a
Request for Information Services (RIS) in November 2000 to automate the sample
selection process. However, the RIS was
rejected due to the priority of other RISs received and limited resources.
The EQMS
Program sampling process was not adequately documented
The EQMS Program did not
adequately document the steps followed in establishing a sampling plan. In addition, the staff did not adequately
monitor the sampling process or make appropriate periodic adjustments to the
sample size, such as when the expected population of cases decreased. Management advised us they did not have a
documented plan. Not having a documented
plan, and not making the necessary and timely adjustments to it, can lead to
invalid results.
According to the IRS’ Statistical Sampling Handbook, sampling guidelines
require documenting the sequence of steps that are taken to arrive at desired
sample results. The process should
begin by establishing and defining, as precisely as possible, the objectives of
the program to be sampled. Then the
targeted population from which the sample will be drawn needs to be identified.
The next step in the sampling process is to develop a sampling plan,
which includes determining:
·
The sampling scheme to be used (judgmental,
statistical, etc.).
·
How the data are to be gathered (manually,
electronically, etc.).
·
The sample size that is needed.
·
How the
data are going to be evaluated.
After the
sampling plan has been prepared, the final steps in the sampling process are
the actual selection of the sample, review of sample items, and evaluation of
the results. The sample process should
be monitored periodically to determine if it is producing the required results
and to make any necessary adjustments that are required.
The Director, Compliance, SB/SE Division, should:
1.
Resubmit the
RIS to automate the sampling process to help ensure all qualified cases are subject
to review, thus helping to promote statistical validity. The RIS should contain a business case
explaining that such a selection process is necessary to ensure that the SB/SE
Division can meet its goal of adequately measuring its business results under
the balanced measures concept.
Management’s Response: EQMS Program management recognized the need for an automated
system and submitted a RIS on October 10, 2002, to request automation
support. In addition, the Compliance
Support Design Team is considering automating the EQMS Program sampling
process.
2. Until such time when electronic sampling is implemented, re-emphasize the criteria for cases qualifying for EQMS Program sampling and the need for reviews so that all cases meeting the criteria are being properly identified.
Management’s Response: The Director, Case Management, will re-issue the sampling
procedures and expectations to all Case Processing sites. The Director, Case Management, assisted by
the Director, Strategy-Research Performance Management, will provide training
and assistance visits to the Case Processing sites to ensure consistent and
correct application of the sampling procedures.
3.
Establish
and document an approved sampling plan that ensures statistical validity,
periodically evaluate the plan, and make appropriate revisions.
Management’s
Response: The EQMS
Program Manager will maintain a documented sampling plan at all times. The Director, Case Management, SB/SE
Division, will monitor the sampling process with assistance from the Office of
Centralized Workload Selection & Delivery and Performance Excellence and
make any necessary periodic adjustments.
As stated
previously, the EQMS Program revised the sample size in October 2000 to select
1 in 3 field and 1 in 5 office examination cases. The EQMS Program management did not adequately monitor the
sampling process and revised the sampling procedures without using a statistically
valid plan to determine the appropriate sample size needed, thereby resulting
in the over-sampling of cases.
We analyzed the SB/SE
Division Examination function’s closed case populations for 15 area offices
(the International Office was not included) for both field and office
examinations. Even though all EQMS
Program-qualified cases were not being selected for quality review, we
determined that the actual sample taken for FY 2001 was about 3,400 cases
larger than necessary to produce statistically valid results. Had
the EQMS Program actually reviewed one in three field and one in five office
examination EQMS Program-qualified cases as planned, the impact would have been
much greater.
Figure 1 shows the number
of cases that the EQMS Program reviewed for the 15 area offices versus the number
that would be required based on a statistically valid sampling plan. Statistically valid samples were determined
for each of the 15 area offices and then added together.
Figure 1 –
Comparison of Actual Sample Taken to the Estimated Statistically Valid Sample
|
Field Examination |
Number of Cases |
Review Hours3 |
|---|---|---|
|
EQMS Program-Qualified Cases1 |
65,500 |
N/A |
|
Actual Sample Taken |
6,879 |
34,395 |
|
Statistically Valid Sample2 |
5, 300 |
26,500 |
|
Overage |
1,579 |
7,895 |
|
Office Examination |
Number of Cases |
Review Hours |
|
EQMS Program-Qualified Cases |
103,500 |
N/A |
|
Actual Sample Taken |
7,196 |
17,990 |
|
Statistically Valid Sample |
5,400 |
13,500 |
|
Overage |
1,796 |
4,490 |
Source: Examination Closed Data File/FYs 2000 &
2001.
1
Since FY 2000 closed examination data could not be broken down into area
populations, we used FY 2001 closed examination data and increased field area
populations by 35 percent and office area populations by 30 percent to simulate
the FY 2000 population size.
2
Statistically valid samples were based on the estimated FY 2000 population and
a 95 percent confidence level, a plus or minus 5 percent precision rate, and an
expected error rate of 50 percent.
3A responsible management official indicated
that the average times required to review an EQMS Program field case and an
office case were 5 hours and
2.5 hours, respectively.
Selecting a statistically valid sample would allow the EQMS Program to be
more economical by reviewing a minimum number of cases in the minimum amount of
time necessary. In addition, the extra
hours used to review the over-sampled cases could be put to better use by
having examiners conduct additional examinations of tax returns. Based on an average time of 5 and 2.5 hours,
respectively, for an EQMS Program reviewer to review a field and an office
case, an estimated 12,400 hours could have been saved and used to examine tax
returns. Using Examination statistics
for FY 2001, we estimate that an additional 800 tax returns could have been
examined, producing potential tax assessments of about $9.8 million in tax
liabilities in FY 2001. The IRS has
recently changed its approach regarding the type of returns it will be
examining. It has aligned its strategic
priorities to address six specific areas of non-compliance. Therefore, the results from shifting the
resources from quality reviews to conducting examinations may differ from FY
2001 statistics.
The number of returns examined has been declining in recent years, which
has been a concern of the Congress and the IRS Oversight Board. The Commissioner of the IRS has said the
decline is due in part to the decline in available staffing. Using these additional hours for examining
tax returns would help alleviate this concern.
4.
The Director, Compliance, SB/SE Division,
should ensure that the EQMS Program reviews only the number of cases needed to
constitute a statistically valid sample.
Management’s
Response:
The EQMS Program Manager will maintain a documented
sampling plan at all times. The
Director, Case Management, SB/SE Division, will monitor the sampling process
with assistance from the Office of Centralized Workload Selection &
Delivery and Performance Excellence and make any necessary periodic
adjustments. Note: This is the
same corrective action as that provided for Recommendation 3.
Office of Audit Comment: Although SB/SE Division management agreed with the report’s
recommendations and plans to take appropriate corrective action, they do not
agree with the revenue impact of using EQMS Program resources to examine tax
returns. SB/SE Division management
stated that they assign reviewers in the EQMS Program for a specified period of
time; they do not shift reviewers between reviewing and examining returns. Their reviewers are also responsible for
providing support to the field in understanding the EQMS Program standards and
applications through making presentations, attending group meetings, and
conducting various other activities.
The Compliance initiatives in the FY 2003 Business Plan require
examination of more complex returns, which would require even more in-depth
reviews than those conducted during the period the audit report covered.
We agree that shifting reviewers throughout the year
between reviewing and examining returns would not be prudent. At the beginning of each year, however,
management should be able to use their sampling plan to determine the number of
returns that need to be reviewed based upon the number of returns planned for
examination in the Annual Examination Plan.
If all EQMS Program resources are not needed to review those returns and
provide support to the field, this is an opportunity for the IRS to conduct
additional examinations of tax returns.
The current EQMS Program sampling plan is designed to produce results
that are valid at the area office level.
However, SB/SE Division management is considering making the EQMS
Program results statistically valid at the territory office level.
A statistically valid sample at the territory office level will greatly increase the resources needed to conduct the case reviews. In fact, the sample size would need to be increased by about 400 percent to make territory office results statistically valid; the sample size would go from about 10,400 (for total area office samples) to approximately 53,500 (for total territory office samples), based on a 95 percent confidence level. Consequently, it would take approximately 110 additional EQMS Program reviewers to make the sampling statistically valid at the territory office level, compared to what would be needed at the area office level. Specific details of our analysis are presented in Appendix V.
These additional reviewers would likely come from the field and office examination programs, resulting in a significant reduction in examination staff and a decrease in the number of tax returns that could be examined. Increasing the number of cases to be quality reviewed would demand significant additional resources, which need to be balanced with the benefits to be gained.
5.
The
Director, Compliance, SB/SE Division, should evaluate the impact of having
valid territory office statistical sampling, to ensure the expected benefits
justify the additional resources that would be needed.
Management’s Response: The SB/SE Division will re-evaluate the
impact statistically valid territory level sampling will have on staffing
levels. This evaluation will also
include a discussion of the impact on balanced measures.
The EQMS Program returns completed examinations to individual examiners
under certain circumstances, such as when there is a substantive tax effect
($10,000 or more) or when a tax effect exists in the taxpayer’s favor. However, of the
eight standards that the EQMS Program reviews, there are various quality issues
such as timeliness, customer relations, examination techniques, and workpapers
on which individual feedback is not provided when these are the
only exceptions noted.
The EQMS Program reviewers prepare narratives on
those cases that do not meet any one of the eight quality standards. These narratives are available to territory
level managers; however, neither the examiners nor their group managers are
receiving individual feedback from the EQMS Program reviews.
Management informed us that results are not provided
to employees based upon a past agreement with the National Treasury Employees
Union (NTEU). However, 12 of
14 area directors, territory managers, group managers, and Technical
Support managers we interviewed believe that individual feedback is crucial for
examiners to improve examination case quality.
The Internal Revenue Manual states that the EQMS Program’s results are to
be used by Examination function management to identify system changes,
training, and educational needs and to improve work processes. Area office responsibilities include
identifying opportunities for improvement and then conducting quality
improvement initiatives to affect the EQMS Program results.
While the EQMS Program reviews
can identify problem trends that need to be addressed on a national or local
level, individual examiners may be making errors that are unique to them. Consequently,
they may continue to repeat them if not notified of the problem. Addressing the performance of individual
examiners can help improve the quality of that individual’s performance, which
will result in an overall improvement in the quality of examination work.
6.
The Director,
Compliance, SB/SE Division, should consider providing individual feedback to
examiners from the EQMS Program case reviews, after resolving this issue with
the NTEU.
Management’s
Response: The Director, Compliance, SB/SE Division,
agreed with the need to share as much feedback with the examiners as possible
and will pursue resolution of this issue with the NTEU. In the interim, all levels of management
will review the results of the EQMS Program and share them generally with the
examiners to improve performance.
Detailed Objective, Scope, and Methodology
The
overall objective of this review was to determine whether the Examination
Quality Measurement System (EQMS) Program is effective to measure the quality
of the Examination function’s work and if the results are used to improve
overall effectiveness. To accomplish
our objective, we:
I.
Determined whether the EQMS Program standards
adequately addressed examination quality issues such as taxpayer rights,
technical tax law application, timeliness of work, and customer
relations/professionalism.
A.
Reviewed the Small Business/Self-Employed Strategy
and Program Plan and compared the strategic goals to the EQMS Program
standards.
B.
Compared revenue agents’ and tax compliance
officers’ “critical job elements” to the EQMS Program standards to determine if
they matched appropriately.
II.
Determined the effectiveness of the EQMS Program
procedures, controls, and the sampling process in the quality assessment of
examination cases.
A.
Interviewed EQMS Program management and analyzed all
available EQMS Program documentation to determine the sampling methodology and
the monitoring system used for the sampling procedures.
B.
Determined how the numbers of cases to be selected
for sampling for both field and office examinations were identified and if they
were statistically valid.
1.
Reviewed all 19 cases in 2 (Detroit and Los Angeles)
of the 3 area offices visited that were excluded from the EQMS Program sampling
process in a 1-day period to determine if they met EQMS Program exclusion
criteria.
2.
Analyzed the closed case examination files for
Fiscal Year (FY) 2001 to determine the EQMS Program-qualified population by
area office.
3.
For the three areas visited, compared the number of
cases that the current EQMS Program sample size should have been with the
number of cases actually sampled to determine if the correct numbers were
sampled.
4.
Determined the EQMS Program-qualified case
population for 15 area offices (not including the International Office) by
analyzing the closed case examination files for FY 2000 and comparing the total
population to the total FY 2001 population.
Since the FY 2000 population could not be broken down into areas, we
adjusted the FY 2001 area populations to account for the percentage decrease in
the population that occurred from FY 2000 to FY 2001.
5.
Calculated a statistically valid sample for each
area office by using a 95 percent confidence level, a 50 percent expected error
rate, and a plus or minus 5 percent precision.
6.
Compared the statistically valid sample size to the
number of cases actually reviewed to determine if over- or under-sampling
occurred.
7.
Calculated the number of review hours the EQMS
Program used to review the over-sampled cases.
Analyzed Table 37, Examination Program Monitor Report, and calculated
the average hours per return and the average tax dollars per hour for both
field and office examinations.
Calculated the potential increased revenue (number of tax dollars
assessed) that could have resulted by using the unnecessary overage of EQMS
Program review hours to examine tax returns.
C.
Determined whether the deployment of EQMS Program results to the
territory level would affect the current sampling process and its statistical validity.
1.
Using the area office populations of EQMS
Program-qualified cases, determined the qualified populations for 164 territory
offices.
2.
Determined a statistically valid sample for each of
the 164 territories by using a 95 percent confidence level, a 50 percent error
rate, and a plus or minus 5 percent precision.
3.
Determined the percentage increase in sample size
needed to bring statistically valid EQMS Program results to the territory
level. Based on the increase in sample
size needed, determined the number of additional reviewers required.
III.
Determined how effectively the results of the EQMS
Program reviews were used to improve overall Examination effectiveness
(training, new procedures, etc.).
A.
Compared the EQMS Program scores from the last 3 quarters of FY 2001 to
the first 3 quarters of FY 2002 to determine if results improved. (Excluded the first quarter of FY 2001
[September 30 through December 31, 2000] since the EQMS Program standards and
scoring system changed beginning January 2001.)
B.
Interviewed three area directors, five territory managers, five group
managers, and one Technical Support manager to determine when, what type, and
how much feedback was given to revenue agents and tax compliance officers
regarding EQMS Program results and reviewed all relevant documentation
regarding this issue.
Appendix
II
Major
Contributors to This Report
Parker
F. Pearson, Acting Assistant Inspector General for Audit (Small Business and
Corporate Programs)
James D.
Dorrell, Acting Audit Manager
Joseph P.
Snyder, Senior Auditor
Phyllis E. Heald, Auditor
Cristina
Johnson, Auditor
Julian E.
O’Neal, Auditor
Appendix III
Acting Commissioner
N:C
Deputy Commissioner, Small Business/Self-Employed Division S
Director, Compliance, Small Business/Self-Employed Division S:C
Director, Strategy, Research, and Performance Management S:S:SR
Director, Case Management S:C:CS:CM
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation
and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaison:
Commissioner,
Small Business/Self-Employed Division
S:C:CP:I
Appendix IV
This appendix
presents detailed information on the measurable impact that our recommended
corrective actions will have on tax administration. These benefits will be incorporated into our Semiannual Report to
the Congress.
Type and Value of Outcome
Measure:
·
Increased Revenue –
Potential; $9.8 million in assessed tax liabilities by using overage in staff
hours for the Examination Quality Management System (EQMS) Program reviews to
examine tax returns (see page 9.)
Methodology Used to
Measure the Reported Benefit:
We conducted an analysis of Small Business/Self-Employed (SB/SE) Division
closed examination files for Fiscal Years (FY) 2000 and 2001 to determine if
the EQMS Program was reviewing the proper number of cases. These data were obtained through the
Treasury Inspector General for Tax Administration (TIGTA) Data Central
Warehouse from the Examination Closed Case Files and were originally provided
to the TIGTA by SB/SE Division Examination Return Selection from the Audit
Information Management System database.
The EQMS Program determined its current sample size (1 in 3 field and 1
in 5 office cases) from the previous year’s (FY 2000) closed examination case
population. Since the FY 2000 closed
examination data are not broken down by area office, we had to simulate the FY
2000 populations by using FY 2001 closed examination data.
To accomplish this, we applied the same EQMS sampling criteria (type of
returns reviewed) to the FY 2001 closed examination data, which we then were
able to query by the 15 area office populations (the International Office was
not included). We had determined that
the number of EQMS Program-qualified returns examined nationwide decreased by
35 percent for field and 30 percent for office examinations from FY 2000 to FY
2001. Therefore, after we determined
the number of EQMS Program-qualified cases by area office for FY 2001, we
increased each area office population by these percentages to arrive at the
simulated FY 2000 area office populations.
To further qualify our populations, we had to account for the fact that
the Examination function closes cases by tax period and the EQMS Program
reviews cases by key case (with all related returns attached). We determined the average number of tax
periods (cases) per taxpayer by querying the related Taxpayer Identification
Numbers from our total estimated FY 2000 EQMS Program-qualified population and
arrived at ratios of 1.79 for field and 1.28 for office examination. We divided these ratios into each area
population to arrive at our final estimated EQMS Program-qualified sample
populations.
We then computed what a statistically valid sample should have been for
FY 2001 for each of the 15 area offices by using the simulated FY 2000 data and
a confidence level of 95 percent, an expected error rate of 50 percent, and a
precision rate of plus or minus 5 percent.
We added the 15 sample sizes together and compared these results to the
actual number of tax returns reviewed by the EQMS Program in FY 2001 and found
that there were about 3,375 cases reviewed that were unnecessary for
statistically valid area office results.
A responsible management official indicated that, on average, 5 and 2.5
hours, respectively, were required to review a field and an office case, which
resulted in approximately 12,385 extra hours used to review the cases.
We estimated that, by using these hours to examine more tax returns, the
Internal Revenue Service could have assessed approximately $9.8 million in tax
liabilities in FY 2001. To arrive at
this number, we used Table 37, Examination Program Monitoring report, from FY
2001 to compute the average number of hours spent to examine a tax return (31
for field and 8 for office examination) and the average dollar amount of
assessed tax liabilities per tax return ($30,795 for field and $3,416 for
office examination). Next, we divided
the extra hours the EQMS Program unnecessarily spent reviewing tax returns
(7,895 for field and 4,490 for office examination) by the average hours spent
to examine a tax return. This resulted
in 255 additional field and 561 additional office examination tax returns that
could have been reviewed. Multiplying
these additional tax returns by the average dollar per tax return resulted in
potential assessments of about $7.9 million for field and about $1.9 million
for office examination.
Appendix V
Comparison of Estimated Statistically Valid
Territory and Area Office
Samples
|
Field
Examination |
Closed Cases3 |
Number of ReviewersRequired4 |
|---|---|---|
|
EQMS Program-Qualified Cases |
46,100 |
N/A |
|
Valid Area Sample1 |
5,100 |
20 |
|
Valid Territory Sample2 |
25,300 |
90 |
|
Percent Increase |
396% |
N/A |
|
Office
Examination |
Closed Cases |
Number of
Reviewers Required |
|
EQMS Program-Qualified Cases |
78,500 |
N/A |
|
Valid Area Sample |
5,300 |
10 |
|
Valid Territory Sample |
28,200 |
50 |
|
Percent Increase |
432% |
N/A |
Source: Examination Closed Data Files/FY 2001.
1Valid area sample based on Fiscal Year (FY) 2001 closed cases and a 95
percent
confidence level (CL), a
50 percent expected error rate, and a plus or minus 5 percent
precision rate.
2Valid territory sample based on a 95 percent CL, an expected error rate
of 50 percent,
and a plus or minus 5
percent precision rate.
3Based on all Small Business/Self-Employed (SB/SE) Division area offices
(excluding the
International Office) and 164 SB/SE Division territory offices for FY
2001. Numbers in
this column are rounded to
the nearest one hundredth for estimation purposes.
4Based on 2,080 hours per year times 70 percent direct time (per EQMS Program
management) and 5 hours or
2.5 hours per case, respectively, for field and office
examinations – 1 staff
person can review approximately 291 field cases or 582 office
cases per year. Numbers in this column are rounded to the nearest
tenth for estimation
purposes.
Appendix VI
The
response was removed due to its size.
To see the complete response, please go to the Adobe PDF version of the
report on the TIGTA Public Web Page.