Management Advisory Report - Appeals Should Further Evaluate Team Concept and Settlement Authority Before Implementing Nationwide

 

 

September 2002

Reference Number:  2002-10-164

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

September 4, 2002

 

 

MEMORANDUM FOR CHIEF, APPEALS

 

FROM:     Pamela J. Gardiner /s/ Pamela J. Gardiner

                Acting Inspector General

 

SUBJECT:     Final Management Advisory Report – Appeals Should Further Evaluate Team Concept and Settlement Authority Before Implementing Nationwide (Audit #200210007)

 

This report presents the results of our review of Appeals’ pilot program to implement Team Concept and Settlement Authority.  At the request of the Chief, Appeals, we conducted a limited scope review to assess the two recommendations in the   December 19, 2001, Team Concept/Settlement Authority Oversight Committee Interim Report.  The report recommended that Appeals explore the possibility that the team concept be rolled out to specialty groups, and that the settlement authority pilot be continued in the three pilot sites.  In addition, the Chief, Appeals requested we make additional recommendations, if appropriate, and identify the critical indicators for judging the success of the pilot as it continues. 

In an effort to reduce the length of the appeals process, Appeals decided to test a new business process involving both a team concept and settlement authority.  The team concept and settlement authority were intended to expedite case closure by eliminating post-settlement approval by a manager for a majority of cases and creating a supportive work group climate of mutual assistance and information sharing.

In October 1999, Appeals charged a Design Team with developing a pilot for empowering work teams with a team concept and delegated settlement authority.  An Appeals Oversight Committee issued an interim report in December 2001 stating that the team concept and settlement authority did not achieve the benefits envisioned by the Design Team.  The Oversight Committee recommended that Appeals consider rolling out the team concept to specialty groups and to delay any decision on implementing settlement authority nationwide until an additional 6 months of pilot testing was conducted.  The Oversight Committee reported the number of cases eligible for settlement authority was far less than projected. The Design Team expected settlement authority would allow employees to close 80 percent of cases.  However, in the initial pilot only about 35 percent (1,292 out of 3,697) of the cases received were eligible for settlement authority and only 10 percent (396 out of 4,043) of the cases were closed using settlement authority. 

In summary, we believe that based on the low volume of cases, Appeals should reconsider the type and dollar amount of cases excluded from settlement authority if it wants to have an impact on the appeals process.  Also, although the Oversight Committee tried to use the Internal Revenue Service’s (IRS) balanced measures of customer satisfaction, employee satisfaction, and business results to evaluate the success of the team concept and settlement authority pilot, the tools developed for measuring customer satisfaction and business results were not effective.  In addition, the Oversight Committee did not address the employee concerns raised when it attempted to measure employee satisfaction. 

Management Response

Appeals management agreed with our recommendations and will evaluate the criteria for determining the types of collection cases that may qualify for settlement authority.  In addition, Appeals management will take actions to address employee concerns, refine the existing customer satisfaction survey, request and analyze decision quality and lapse time data from the Appeals Quality Measurement System, and work to integrate cost benefit measures into future programs. Management’s complete response to the management advisory report is included as Appendix V.

Please contact me at (202) 622-6510 if you have questions or Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs) at (202) 622-8500.

 

Table of Contents

Background

The Team Concept and Settlement Authority Pilot Did Not Achieve the Desired Results

Recommendation 1:

Pilot Measurement Tools Were Not Appropriate

Recommendation 2:

Recommendations 3, 4 and 5:

Appendix I – Detailed Objectives, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Pilot Evaluation Measures

Appendix V – Management’s Response to the Draft Report

 

Background

If taxpayers disagree with a tax liability or certain collection actions proposed by an Internal Revenue Service (IRS) Compliance employee, they have the right to ask for an administrative review by the office of the Chief, Appeals.  The Compliance employee will then forward the case to an Appeals employee who will conduct an independent review of the merits of the proposed compliance actions. 

The Appeals employee will review the issues in the case and give them a fresh look before making a determination.  An Appeals manager will then conduct a post-settlement review and approve the Appeals employee’s determination.

One of Appeals’ goals is to reduce the length of the appeals process.  Taxpayers continue to express their dissatisfaction with the length of the appeals process, which averages 293 days.  As a way to reduce this time, Appeals decided to test a new business process involving a team concept and settlement authority.  The team concept and settlement authority were intended to expedite case closure by eliminating the Appeals manager post-settlement approval for a majority of cases and creating a supportive work group climate of mutual assistance and information sharing.  This concept is consistent with the IRS goal to provide quality service to each customer, Appeals’ goal to improve customer satisfaction, and the Congressional desire to improve program efficiency and service delivery.

In October 1999, Appeals charged a Design Team with developing a pilot to test the team concept and settlement authority process.  The Design Team recommended testing at three pilot sites and defined which Appeals Employees would have settlement authority and what cases would meet the criteria.  Cases excluded from settlement authority included cases with emerging or developing issues, cases having sensitive issues, or cases with unusually high dollar values.  In addition, the pilot excluded complex cases and cases that might have precedent-setting issues affecting many taxpayers or future years.

An objective of the pilot was to identify any process and training modifications needed, and opportunities for improvement prior to full implementation.  The success of the team concept and settlement authority pilot would be evaluated based on the IRS’ balanced measures of customer satisfaction, employee satisfaction, and business results.  See Appendix IV for details of the planned measures and measurement tools. The IRS did not perform a cost benefit analysis to evaluate the results of the pilot in terms of efficiency (i.e. costs of running the program versus the benefits received).

By April 2001, Appeals implemented settlement authority in the Newark, South Florida, and St. Paul offices.  An Appeals Oversight Committee monitored the pilot and issued an interim report in December 2001, stating that the team concept and settlement authority did not achieve the benefits envisioned by the Design Team.  There was not enough support from the pilot groups to initiate a nationwide rollout of the team concept, therefore they recommended that Appeals consider only rolling it out to specialty groups with common issues and goals. The Oversight Committee also deferred any decision on implementing settlement authority until an additional 6 months of pilot testing was conducted.

The Chief, Appeals asked the Treasury Inspector General for Tax Administration (TIGTA) to review the recommendations in the Team Concept/Settlement Authority Oversight Committee Interim Report, determine if we agree with them, and provide any additional recommendations if appropriate.  Also, the Chief, Appeals requested we identify the critical indicators for judging the success of the settlement authority pilot as it continues.

The audit was performed in accordance with the President’s Council on Integrity and Efficiency’s Quality Standards for Inspections.  The scope of this review was limited to reviewing documents and conducting interviews with select Appeals personnel, responsible for conducting and overseeing the pilot.  The review was performed from February to June 2002, in Fort Lauderdale, Miami, and the National Headquarters.  Detailed information on our objectives, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

The Team Concept and Settlement Authority Pilot Did Not Achieve the Desired Results

The TIGTA agrees with the Oversight Committee’s observations and recommendations to not rollout the team concept and to delay settlement authority until further analysis is conducted.  Factors that impacted the Committee’s decision included: the original benefits of the concepts were not realized, the number of cases eligible for settlement authority was far less than projected, organizational workload and geographic changes inhibited teams, and the relationship to balanced measures was not conclusive.

The Design Team intended for the team concept to promote knowledge and skills among Appeals employees and their managers.  They expected this concept to empower individuals to engage in “process focused problem solving.”  However, the Oversight Committee concluded that it was difficult to determine how the implementation of the team concept affected operational results due to organizational and workload changes that occurred during the pilot.  The Committee did report that the team concept was successful where specialty teams had been developed with common issues and goals.  As a result, they recommended that the team concept be rolled out to specialty groups.

We also concur with the Oversight Committee recommendation to delay the implementation of the settlement authority nationwide until additional pilot testing is completed.  The Oversight Committee reported the number of cases eligible for the settlement authority was far less than projected.  The Design Team expected settlement authority would allow employees to close 80 percent of its cases (approximately 3,234 cases). However, in the initial pilot only about 35 percent of the cases received (1,292 out of 3,697) were eligible for the settlement authority and only 10 percent of all cases closed during the pilot (396 out of 4,043) were closed using settlement authority. 

 

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

During the 1999 planning, the Design Team excluded cases from the settlement authority with emerging issues, including Collection Due Process (CDP) cases.  This significantly impacted the number of cases eligible for the settlement authority as the CDP receipts grew from 13 percent of total inventory in September 1999, to almost 35 percent in February 2002.

Based on the low volume of cases, we believe that the type and dollar amount of cases excluded from the settlement authority should be reconsidered if Appeals wants to have an impact on the appeals process.  For example, Appeals could consider including CDP cases for the settlement authority since the CDP process has now been in place over 3 years.  Though issues are still emerging, as with any new law, there have been numerous cases presented to the court with precedent setting issues.  Both the law and procedures have been significantly developed and clarified.  In addition, since FY 1999, Appeals has provided extensive training to Appeals Employees to help ensure employees have the skills and knowledge needed to understand and work CDP cases.

The TIGTA believes that if the majority of cases are not eligible for the settlement authority, Appeals cannot achieve its goal to create cultural changes to expedite case closures by eliminating post-settlement approval for the majority of cases.  If the case criteria for the settlement authority are not changed, there may not be a sufficient number of cases meeting the criteria to warrant implementing the settlement authority, which will require Appeals to revisit its strategy to reduce the length of the appeals process.

Recommendation

The Chief, Appeals, should

1.      Re-evaluate the type and dollar amount criteria of cases excluded from the Settlement Authority to determine if the criteria can be changed to increase the percentage of cases eligible for settlement authority.

Management’s Response:  Appeals management will analyze the types of collection cases that may qualify for settlement authority, re-evaluate existing dollar limitations, and evaluate the “lifecycle” of other types of cases previously excluded.

Pilot Measurement Tools Were Not Appropriate

The Oversight Committee tried to use the IRS’ balanced measures of customer satisfaction, employee satisfaction, and business results to evaluate the success of the team concept and settlement authority pilot.  Unfortunately, the tools developed for measuring customer satisfaction and business results were not effective.  Also, although the pilot attempted to measure employee satisfaction, the Oversight Committee did not address the employee concerns raised.  Until Appeals develops effective measurement tools, it will not be able to assess the success of the team concept and settlement authority or make any needed modifications before a nationwide rollout.

The Oversight Committee stated in their interim report the link to the IRS’ balanced measures was not conclusive and acknowledged the need to address measurement tools in their required next steps.  We believe that to fully evaluate the program Appeals should consider implementing additional measures to evaluate the success of the pilot.

An additional measure that Appeals should consider is a cost benefit analysis.  The success of the pilot needs to be reflected in terms of the costs associated with the program and the benefits achieved.  The President’s Management Agenda outlines an initiative where over time agencies will be expected to identify high quality outcome measures, accurately monitor the performance of programs, and begin integrating this presentation with associated costs.  We believe that capturing the cost of the pilot in relationship with the benefits achieved will provide another assessment of the effectiveness of the program.  Also, to effectively evaluate whether to rollout the program nationwide, management needs to assess whether the additional expenses incurred to deploy the team concept and the settlement authority significantly reduced the length of the appeals process.

Customer Satisfaction

Appeals planned to measure customer satisfaction with the settlement authority using the existing IRS customer satisfaction survey results.  The Oversight Committee requested copies of the survey results for the three pilot sites but did not receive the results prior to concluding their report. However, the Oversight Committee concluded that the IRS customer satisfaction survey might not have provided conclusive evidence on taxpayers’ satisfaction.  The IRS customer survey canvassed a sample of all taxpayers using Appeals services, not specifically taxpayers that had participated in the settlement authority.  The Oversight Committee did not recommend an alternative tool to measure customer satisfaction.

We agree that the surveys were not a valuable tool to measure the effect and success of the settlement authority pilot since the number of cases closed using the settlement authority would be insignificant to the IRS’ customer satisfaction survey.  An alternate approach to evaluating customer satisfaction would be for Appeals to canvas taxpayers that had participated in the settlement authority using the existing IRS customer satisfaction survey and also determine the feasibility and need of supplementing the survey with additional questions.

Employee Satisfaction

The Oversight Committee ensured that Appeals employees participating in the pilot completed a survey and took part in focus groups.  The Oversight Committee reported an overall satisfaction rate of 2.4 out of 5.

We believe that the survey and focus groups were good methods to gauge employee satisfaction. The results of the survey illustrate program problems to management.  When properly executed, the surveys build a sense of trust and support between employees and managers.  Appeals employees believed that training on the team concept was good and setting goals and creating a common approach improved the work environment.  Further, they believed each employee benefited from the problem solving activities and has improved the level of customer service.

Employees also raised concerns regarding the program during the employee satisfaction feedback process. For example, two pilot sites reported they had not regularly conducted internal reviews or discussed performance and results with employees.  Appeals employees can use internal reviews to informally discuss settlement strategies or new and emerging issues with their respective team.  In addition, two of the sites believed their team manager did not have time to be an effective leader of a team as envisioned in the training.  The Oversight Committee, in the interim report, did not state how it would address the concerns raised by employees.  If employees’ concerns are solicited and not addressed, employee satisfaction with both the team concept and settlement authority might be adversely affected.

Business Results

The Design Team proposed evaluating business results by measuring lapse time and the quality of case decisions using the Appeals Quality Measurement System (AQMS).  The AQMS is a case review process developed to provide statistically valid data on case quality for Appeals.  Case quality is defined by standards such as: fair and equitable treatment of taxpayers, quality of decision, accuracy of liability computation, timeliness, compliance with procedures, etc. 

At the time that the Design Team developed the business measures for the pilot, they thought that AQMS would be operational. However, implementation delays prevented AQMS from being used to evaluate the business results of the pilot. As a result of AQMS not being operational, the Oversight Committee could not obtain data to assess whether the team concept and settlement authority was having an impact on the appeals process. The Oversight Committee did not recommend an alternative tool to measure business results.

Since the Oversight Committee’s review, AQMS has become operational.  We believe that AQMS should provide sufficient data to evaluate the business results of the pilot.  Therefore, Appeals should request the necessary reports from AQMS to evaluate the lapse time and the quality of the case decisions. 

Recommendations

The Chief, Appeals, should do the following before proceeding with the settlement authority pilot and rolling out the settlement authority nationwide:

2.      Determine the actions needed to address employee concerns identified in the survey and focus groups.  Specifically, employee concerns on peer reviews, performance and results discussions, and effective leadership.

3.      Evaluate customer service by having the Oversight Committee canvas taxpayers that had participated in the settlement authority using existing IRS customer satisfaction surveys and determine the feasibility and need to supplement the survey with additional questions. 

4.      Evaluate business results by having the Oversight Committee request reports from AQMS on lapse time and the quality of the case decisions. 

5.      Establish measures and collect data on the cost savings of the settlement authority, and consider the reduction in time on cases by both team leaders and Appeals employees.

Management’s Response:  Appeals management will re-evaluate the usefulness of peer review and determine the best approach to accomplish its benefits, appropriately integrate the demands of settlement authority into the team leaders workload, re-evaluate and refine the existing customer satisfaction survey to better capture appropriate information, request and analyze decision quality and lapse time data from AQMS, and work to integrate cost benefit measures into future programs.

 

Appendix I

 

Detailed Objectives, Scope, and Methodology

 

At the request of the Chief, Appeals, we conducted a limited scope review to assess two recommendations in the December 19, 2001, Team Concept/Settlement Authority Oversight Committee Interim Report.  The report recommended that Appeals, (1) explore the possibility that team concept be rolled out to specialty groups, and (2) that the settlement authority pilot be continued in the three pilot sites.  In addition, the Chief, Appeals requested we make additional recommendations, if appropriate, and identify the critical indicators for judging the success of the settlement authority pilot as it continues. 

To accomplish this we evaluated the Oversight Committee’s use of the measurement tools in addressing the success of the team concept and the settlement authority using the IRS’ three balanced measures: employee satisfaction, customer satisfaction and business results.  Specifically, we:

A.                Interviewed Appeals personnel responsible for conducting the initial pilot and analyzing the pilot’s results. 

B.                 Obtained copies of data compiled and documents used to evaluate the pilot and prepare the interim report.

C.                Analyzed data to determine if the interim report addressed the balanced measures and the measurement tools, included all appropriate issues, and determined if the data was sufficient to support the information and/or recommendations in the interim report. 

D.                Identified any barriers not addressed in the interim report.

E.                 Contacted the Treasury Inspector General for Tax Administration’s Counsel to determine if Appeals Employees concerns about the settlement authority are valid. Determined what could be done to resolve the concerns and if there are legal barriers to authorizing the settlement authority for collection due process cases.

Appendix II

 

Major Contributors to This Report

 

Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

Mary V. Baker, Director

Kenneth L. Carlson, Jr., Acting Audit Manager

Nelva U. Blassingame, Auditor

Andrew J. Burns, Auditor

Tracy K. Harper, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  N:C

Director, Appeals SB/SE-TEGE Operating Unit  AP

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O

Office of Management Controls  N:CFO:F:M

Audit Liaison:                        Chief, Appeals  AP

                        Chief Counsel  CC

 

Appendix IV

 

Pilot Evaluation Measures

 

The table provides the measures used in the pilot plan to identify any process and training modifications needed before full-scale rollout.

 

Source:  Appeals Design Team Settlement Authority & Team Concept Pilot Plan, New Carrollton, MD, November 30, 1999, Phase IIB Design Team, page I-14.

 

Appendix V

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.