Management Advisory Report: Appeals Can Enhance the Usefulness of
Appeals Quality Measurement System Results
August 2002
Reference Number:
2002-10-158
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
August 30, 2002
MEMORANDUM FOR
CHIEF, APPEALS
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Acting Inspector General
SUBJECT: Final Management Advisory Report – Appeals Can
Enhance the Usefulness of Appeals Quality Measurement System Results (Audit #
200210024)
This
report presents the results of our review to assess whether the Appeals Quality Measurement System (AQMS)
provides results that are statistically valid, accurate, and useful for making
program improvements. Appeals
implemented the AQMS in 2001 to provide for a quality measure as part of the
Internal Revenue Service’s (IRS) efforts to adopt a balanced performance
measurement system. The IRS’ balanced
measures consist of three elements: customer satisfaction, employee
satisfaction, and business results.
Business results include measures of quality and quantity.
In summary, Appeals AQMS Team Managers took steps to assure
accuracy of results by holding reviewers meetings to discuss study cases,
distributing newsletters, conducting post-reviews of reviewers’ work, and
reviewing data input. However, Appeals did not
determine the statistical validity of its AQMS results. Consequently,
it cannot be sure whether it has achieved results that are as reliable as it
needs. Finally, although Appeals has taken steps to
provide AQMS data, it needs to establish responsibility for follow-up analysis
and corrective actions.
Management’s
Response: The Chief, Appeals agreed with our findings
and recommendations, and Appeals management has initiated corrective
actions. Appeals has contacted
technical assistance groups within the IRS to help calculate the actual
confidence level and precision rate for Fiscal Year 2001 and 2002 results, and
to provide continued support in selecting samples and other statistical
issues. Appeals also agreed to
establish procedures to use the AQMS results to identify national training
needs and emphasis areas, require area offices to report quarterly on actions
taken based on the AQMS results, and assist in analyzing and producing area
office reports on the AQMS results.
Management’s complete response to the draft report is included as
Appendix IV.
Copies
of this report are also being sent to the IRS managers who are affected by the
report recommendations. Please contact
me at (202) 622-6510 if you have questions, or Daniel R. Devlin, Assistant
Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs),
at (202) 622-8500.
Appeals
Took Steps to Assure Accuracy of Quality Results
Appeals Should Calculate the Confidence Level and Precision Rate
for Quality Results
Appeals Should Assure Actions Are Taken on Quality Results
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Management’s Response to the Draft Report
Appeals provides taxpayers with an independent, impartial review of their cases after an audit is completed or collection action is proposed. The Appeals Quality Measurement System (AQMS) was developed to provide statistically valid data on the quality of Appeals cases.
Appeals implemented the AQMS in 2001 to provide for a quality measure as part of the Internal Revenue Service’s (IRS) efforts to adopt a balanced performance measurement system. Regulations require the IRS to develop balanced measures of performance throughout the organization. The IRS’ balanced measures consist of three elements: customer satisfaction, employee satisfaction, and business results. Business results include measures of quality and quantity. The goals of balanced measures are to provide information on organizational performance and to highlight areas for performance improvement.
The AQMS is expected to:
· Serve as the quality measure for Appeals decisions and settlements.
· Ensure customer rights are protected.
· Increase consistency in Appeals decisions and settlements.
· Allow identification of training needs for Appeals Officers.
· Provide feedback to management on areas requiring improvement in the Appeals process.
Appeals put together an AQMS design team to make recommendations on how to implement the AQMS. The design team recommended the methodology, reporting standards, staffing and responsibilities, and implementation plans for the AQMS process in a report dated October 30, 1999.
The design team recommended that the quality review cover six standards: fair and equitable treatment of taxpayers, quality of Appeals decision, accuracy of liability computation, consideration of case memoranda, timeliness, and procedural compliance.
The design team laid out an AQMS process which consists of selecting the sample cases, reviewing the cases, providing for accuracy and consistency of review results, and informing Appeals users about results.
In selecting sample cases for review, the design team recommended that the General Appeals Programs business unit plan for statistically valid results with a 95 percent confidence level and a 5 percent precision rate. It recommended that the Large Business and Specialty Programs (LBSP) business unit plan for statistically valid results with an 80 percent confidence level and a 5 percent precision rate. The AQMS design team chose the target confidence level and precision for the General Appeals program because other quality review programs in the IRS used them. The AQMS design team chose a confidence level that was somewhat lower for LBSP due to staffing constraints.
To review the cases, the design team proposed that there should be from 23 to 28 reviewers. These reviewers should be located nationwide, and report to the AQMS Team Manager for the business unit. The design team also specified that the reviewers should have procedural guidance on reviewing cases and use a standardized check sheet to record review results.
To provide for accuracy and consistency of review results, the design team stated that AQMS Team Managers should randomly select reviewed cases for a post-review process. In addition, the AQMS Team Managers should hold periodic reviewer meetings to promote consistency. All participants at the reviewer meetings should complete study cases and then discuss them as a group to reconcile differences.
To inform Appeals users about results, the design team anticipated that reports with AQMS results would be provided monthly to Appeals business unit Directors and quarterly to other national managers and the 12 area level field managers (area managers). Also, the Chief, Appeals and business unit Directors, along with their staff, should use AQMS results to identify national and business unit training needs.
By March 2001, Appeals had trained AQMS staff and started reviewing cases. The AQMS Team Managers provided the first AQMS reports to national and area managers in February 2002, containing data for the period January 2001 through September 2001.
The review was performed from March to June 2002 in the
Appeals National Headquarters and in 11 of the 12 Appeals area offices. The review was conducted in accordance with
the President’s Council on Integrity and Efficiency’s Quality Standards for
Inspections. Detailed information on our objective,
scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
To help promote consistency and accuracy of reviews, the General Appeals AQMS Team Manager performed post-reviews of selected cases and the LBSP AQMS Team Manager post-reviewed all cases. Both AQMS Team Managers conducted AQMS meetings in which reviewers discussed study cases. Both also reviewed data input for accuracy, and both used a standardized checklist that they developed to document their post-reviews.
The design team recommended that AQMS Team Managers post-review the work of each reviewer. The LBSP AQMS Team Manager reviewed all 235 cases worked by the LBSP reviewers. The General Appeals AQMS Team Manager reviewed 28 of the 1,853 General Appeals AQMS cases, involving 10 of the 12 General Appeals reviewers prior to issuing AQMS reports. Treasury Inspector General for Tax Administration (TIGTA) auditors discussed with General Appeals management the importance of conducting post-reviews of a sample of cases for all AQMS reviewers. General Appeals AQMS management immediately took corrective action by implementing a systematic post-review of each reviewer’s work on a quarterly basis.
Appeals worked with an IRS statistician to develop a sampling process that would provide statistically valid results. The process was designed to provide General Appeals results that could be projected to the area office level for an annual period with a 95 percent confidence level and a 5 percent precision rate. General Appeals used an interval sampling process that, according to the IRS statistician, should yield results that can be statistically projected. However, due to the start-up nature of the program and staff turnover, General Appeals did not calculate the actual confidence level and precision rate for its AQMS results. Consequently, it cannot be sure whether it has achieved results that are as reliable as it needs.
Appeals planned for an 80 percent confidence level and 5 percent precision rate for LBSP results. Although LBSP had originally planned to use statistical sampling, during implementation the LBSP reviewers worked 100 percent of two inventory categories comprising a large part of the LBSP inventory. In addition, they worked a sample of field examination cases. LBSP did not perform statistical sampling because they thought that the method they used provided even better assurance. However, since LBSP could not quantify the number of cases in each inventory category, it could not be sure whether it had achieved the desired confidence level and precision rate.
Appeals plans to use the AQMS to achieve a number of benefits, such as protecting customer rights, treating customers consistently, and identifying areas for training. The more precise the results, the more reliance can be placed on the data for decision-making purposes. TIGTA auditors discussed the importance of this issue with Appeals management. Management informed us that they would immediately start working with a statistician to calculate actual confidence levels and precision rates for AQMS results.
1.
The
Chief, Appeals should assure that actual confidence levels and precision rates
are calculated for AQMS results and make adjustments as appropriate.
Management’s Response: The Chief, Appeals agreed with this recommendation and has initiated corrective action by contacting the IRS Reports and Analysis office and the Office of Program Evaluation and Risk Analysis to obtain their assistance to:
· Determine the actual confidence level and precision rate for Fiscal Year (FY) 2001 and 2002 AQMS results.
· Determine the sample size to use for FY 2003.
· Select a statistician for continued involvement in the AQMS program.
The AQMS Team Managers informed Appeals managers and employees about the AQMS process and results through an AQMS website and through newsletters. While some Appeals managers took corrective actions in response to AQMS results, there were no procedures describing how Appeals managers should follow up.
The AQMS website contained copies of the AQMS newsletters, and reviewer check sheets and guides. The newsletters contained observations on cases reviewed, information on the AQMS review process, and trends or concerns. The Appeals employees we spoke with, although not necessarily representative of all Appeals employees, told us that they were familiar with and generally found the AQMS newsletter and website useful.
Although national and area managers were provided reports, Appeals had not assigned responsibility for initiating national or local actions in response to quality results. Specifically, Appeals had not assigned anyone to analyze national quality results to identify training needs or areas for improvement, and local area managers were not required to initiate actions based on their quality results.
Although not required, some Appeals managers had initiated actions in response to the quality results. The LBSP Director informed us that he had discussed AQMS standards needing improvement with his area managers during operational reviews. LBSP also used the quality results to identify areas for training in its most recent training seminars. Four of the 11 area managers we interviewed in the 2 business units had emphasized areas for improvement with their staffs. Although General Appeals has not yet established a process for responding to quality results data they told us that they planned to use the first year’s data as a baseline to measure future performance.
While Appeals has taken the first steps to achieve better performance by developing quality measures and collecting data, good management practice suggests that the next steps are to analyze results and take corrective actions. The AQMS was to provide the quality measure as part of the balanced performance measurement system. The overall goals of balanced measures are to provide information on organizational performance and to highlight areas for performance improvement. Appeals had expected AQMS to provide a quality measure and to ensure that customer rights were protected, decisions were consistent, and training areas were identified.
AQMS results indicate that there may be areas for improvement that are national in scope. For example, cases met the timeliness standard about 60 percent of the time on average, and ranged from 42 percent to 78 percent. Providing for national level analysis and oversight could help Appeals to better realize the intended benefits of the AQMS.
The AQMS Team Managers and the General Appeals and LBSP Directors of Appeals Measures, Analysis, and Planning informed us that they did not develop guidance requiring analysis and corrective action on quality results because their role was to deliver accurate AQMS results, not to determine which areas need improvement or to develop strategies for improvement. After discussing this issue with TIGTA auditors, LBSP AQMS analyzed LBSP results and issued a report to LBSP management identifying areas of concern, possible causes, and suggested actions.
2. The Chief, Appeals should establish
procedures to require that next FY, quality review results be used to identify
national training needs and emphasis areas.
The Chief, Appeals should also require area offices to then report
quarterly on the corrective actions taken to address quality standards that
warrant improvement.
Management’s Response: The Chief, Appeals agreed with this
recommendation and will:
·
Establish procedures to use
AQMS results to identify national training needs and emphasis areas.
·
Require area offices to
report quarterly on their performance that addresses any corrective actions
taken on AQMS results.
·
Assist in analyzing and
producing area reports as needed.
Appendix I
Detailed Objective, Scope, and Methodology
Our overall objective was to determine whether the Appeals Quality Measurement System (AQMS) provides results that are statistically valid, accurate, and useful for making program improvements. We established the following sub-objectives and audit steps to accomplish them.
I. To determine whether the General Appeals Programs (General Appeals) AQMS sampling methodology provided statistically valid results, we:
A. Obtained and reviewed the AQMS design plan to identify the sampling goals.
B. Interviewed the General Appeals and Appeals Large Business and Specialty Programs (LBSP) AQMS Team Managers and the Internal Revenue Service (IRS) statistician to identify the actual sampling procedures and sample sizes selected, and whether confidence levels and precision rates were calculated for the AQMS results.
C. Interviewed the IRS statistician who assisted in developing the AQMS sampling methodology to determine whether he was consulted on the impact of any deviations from the sampling plan.
II. To determine whether AQMS provided for consistency in case reviews and validity of the case review data, we interviewed the General Appeals and LBSP AQMS Team Managers to identify their post-review procedures.
III. To determine whether AQMS results were useful to Area Directors, Appeals managers, and Appeals Officers, we:
Appendix II
Major Contributors to This Report
Daniel R. Devlin, Assistant Inspector General for Audit
(Headquarters Operations and Exempt Organizations Programs)
Mary V. Baker, Director
Mary L. Jankowski, Audit Manager
William D. Lessa, Senior Auditor
Abraham B. Millado, Senior Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Director, Appeals Large Business and Specialty
Programs AP:L:HQ
Director, General Appeals Programs AP:G:HQ
Chief
Counsel CC
National
Taxpayer Advocate TA
Director,
Legislative Affairs CL:LA
Director,
Office of Program Evaluation and Risk Analysis
N:ADC:R:O
Office of
Management Controls N:CFO:F:M
Audit
Liaison: Chief, Appeals N:AP
Appendix IV
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.