Letter Report: Opportunities Exist to Improve the Performance Indicators Used to Convey Toll-Free Telephone Accuracy Accomplishments
August 2001
Reference Number: 2001-40-130
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
August 9, 2001
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Letter Report - Opportunities Exist to Improve the Performance Indicators Used to Convey Toll-Free Telephone Accuracy Accomplishments
This report presents the results of our review to determine whether the Internal Revenue Service’s (IRS) quality performance indicators for toll-free telephone assistance for tax law questions and account issues provide stakeholders with a clear picture of the quality provided to taxpayers. In summary, we found that the two measures the Internal Revenue Service (IRS) used to communicate its quality of toll-free telephone calls did not provide a complete assessment of its achievements. We also found that the IRS has recently begun an effort to identify and implement an improved approach to quality.
We recommended that the Commissioner, Wage and Investment Division: (1) rename the indicators used to report the accuracy of responses to taxpayers’ tax law questions and account issues, (2) incorporate the results of the quality review of Spanish-speaking and r-mail calls into the measure of toll-free accuracy for assisted calls when reporting to external stakeholders, (3) create an additional measure to satisfy the Government Performance and Results Act of 1993 reporting requirements to convey the quality of automated services, and (4) establish a long-term goal based on taxpayer needs and expectations for each of its toll-free "quality" indicators.
Management substantially agreed with the recommendations we presented. While management chose not to fully implement our second recommendation at this time, they have planned to implement a significant portion in the future.
Management’s comments have been incorporated into the report where appropriate, and the full text of their comments is included as Appendix III. Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or a member of your staff may contact Susan Boehmer, Acting Assistant Inspector General for Audit (Wage and Investment Income Programs) at (770) 936-4591.
Objective and Scope
This review supports our Fiscal Year (FY) 2001 emphasis area focusing on customer service activities. The overall objective of the audit was to determine whether the Internal Revenue Service’s (IRS) quality performance indicators for toll-free telephone assistance for tax law questions and account issues provide stakeholders with a clear picture of the quality provided to taxpayers. Our purpose in reviewing this area was to determine whether the indicators that the IRS uses to communicate the quality of answers taxpayers receive from toll-free telephone assistance provide an adequate assessment of the quality of service.
To accomplish our objective, we reviewed toll-free telephone quality information disseminated to stakeholders, reviewed documentation of the IRS’ strategic planning efforts, and interviewed IRS management officials responsible for the quality reviews of toll-free telephone calls. We did not review the accuracy of the IRS’ reported data nor did we review any procedures that the IRS may have in place to verify its data. Our work was limited to the toll-free telephone quality performance indicators; there was a concurrent Treasury Inspector General for Tax Administration audit conducted on the toll-free telephone quantity performance measures.
We performed audit work at the National Headquarters for Wage and Investment, the Customer Account Services Office in Atlanta, Georgia, the IRS offices in New Carrollton, Maryland, and the Centralized Quality Review System site in Philadelphia, Pennsylvania. Our review was conducted from January to March 2001 and was performed in accordance with Government Auditing Standards. Major contributors to this report are listed in Appendix I. Appendix II contains the Report Distribution List.
Background
The Government Performance and Results Act of 1993 (GPRA) requires federal agencies to establish standards for measuring performance and improving program effectiveness. The law requires executive agencies to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments. Among other things, the strategic plan should contain general goals and objectives with the goals defined in a method that allows for a future assessment to be made on whether they were achieved. The GPRA performance results are intended to help Federal managers improve service and help the Congress evaluate an agency’s budget requests and make future appropriations.
The IRS has established three strategic goals to support its mission:
The IRS provides various ways for taxpayers to obtain service. One of the most widely used by taxpayers is the IRS’ toll-free telephone assistance program. The IRS has numerous toll-free telephone numbers for taxpayers to use; however, there are three main toll-free numbers for taxpayers with questions about their accounts or tax law topics (1-800-829-1040, 1-800-829-8815, and 1-800-808-4262). These toll-free numbers provide both "live assistance" and automated services. "Live assistance" calls to these three toll-free numbers are the basis for the IRS’ toll-free quality indicators.
During FY 2000, the IRS reported that it received almost 80 million calls to its 3 main toll-free numbers and that its Customer Service Representatives (known as "assistors") answered 32.3 million of those calls. Requested FY 2001 funding for the function that includes the toll-free telephone systems represents approximately $1 billion of the IRS’ $8.8 billion total budget.
To fulfill GPRA performance reporting requirements, the IRS has selected many indicators to include in its annual performance report to the Congress. Among those indicators, two have been included to convey its FY 2001 toll-free accuracy accomplishments—Toll-free Tax Law Quality and Toll-free Account Quality. The IRS reported a Toll-Free Tax Law Quality rate of 76.2 percent and a Toll-Free Account Quality rate of 60 percent for FY 2000.
Results
The IRS has stated in its Modernizing America’s Tax Agency publication and its Strategic Plan for FYs 2000-2005, "Whenever the IRS deals with a taxpayer, the taxpayer should receive first quality service and treatment that is helpful based on the particular situation and need."
The IRS has recognized the need to improve customer service quality, as well as the need to develop realistic and appropriate indicators of its performance. To that end, the IRS has begun an effort with a consulting firm to identify and implement an improved, "embedded" approach to quality. The project is in the initial stages of development. If the objectives of the initiative are met, it should provide the IRS with a vision to guide the content, process, and organizational changes necessary to improve performance.
Although the IRS has long been concerned with a taxpayer’s experience during telephone calls and has pledged to provide "world class" and "first quality" service, the two indicators the IRS uses to gauge the accuracy of responses taxpayers receive when calling the IRS with tax law questions or account issues did not provide a complete assessment of the agency’s actual achievements.
Internal Revenue Service Quality Indicators Implied More Than Was Actually Measured
The titles of the IRS toll-free quality indicators were not descriptive enough to communicate that only a portion of the toll-free telephone service was assessed. The IRS also did not explain that its quality indicators did not represent all aspects of its toll-free service when reporting to outside stakeholders.
The indicators for toll-free quality did not include all elements of the IRS’ toll-free system, such as the questions answered by a return telephone call or electronic mail (known as r-mail), calls from Spanish-speaking taxpayers, and calls to the IRS’ automated telephone systems. The two indicators the IRS reported included only the accuracy of responses from telephone calls that were handled live by an IRS assistor. The IRS did have separate internal indicators for the quality of r-mail and Spanish-speaking calls. However, the IRS did not have accuracy or quality indicators for its automated systems.
As detailed in the following table, the methods of telephone assistance missing from the indicators make up a significant portion of all toll-free telephone calls to the IRS.
Types of Assistance Provided for FY 2000
|
Method of Assistance |
Quantity of Calls Answered |
Percentage of Total 3 |
|
Assistor-answered calls |
32.3 |
39.3 |
Questions answered by return telephone calls or electronic mail (r-mail)1 |
.9 |
1.1 |
Spanish-speaking calls1 |
N/A2 |
|
|
Tele-Tax calls3 |
35.9 |
43.7 |
|
Automated answered calls (other than Tele-Tax) |
14.0 |
17.0 |
|
Total4 |
82.2 |
|
1
These calls are a sub-set of assistor-answered calls.2
The IRS did not conduct a national quality review of Spanish-speaking calls during FY 2000.3
Tele-Tax (1-800-829-4477) provides taxpayers with pre-recorded tax law information.4
Total includes calls to the 1-800-829-1040, 1-800-829-8815, 1-800-808-4262, andSource: The IRS’ Weekly Customer Service Report as of September 30, 2000, and the IRS’ Business Performance Review Report dated November 2000.
Current quality indicators measure only the accuracy of the answer a taxpayer receives. The IRS has separate indicators for the taxpayer’s experience accessing the toll-free system and for customer satisfaction.
According to the GPRA and the Office of Management and Budget’s (OMB) Circular No. A-11, which details the preparation and submission of strategic plans and performance reports, agencies should establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes of each program activity and provide a basis for comparing actual performance with the established performance goals. The General Accounting Office (GAO) provides further interpretation in its publication, The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans, by adding that goals and measures should be precise and allow for the assessment of performance. The GAO states that indicators must clearly represent or be related to the performance they are trying to assess.
Indicators that imply more than actual performance jeopardize the IRS’ ability to make sound management decisions and the Congress’ ability to make accurate budget decisions.
Recommendations
The Commissioner, Wage and Investment Division, should consider the following recommendations during IRS’ effort to identify and implement an improved approach to quality:
Management’s Response: The IRS responded, "We will clarify our supporting documentation and data dictionaries to more precisely reflect the specific elements of activity we are measuring. This will further ensure that readers understand we are measuring the quality of live assistance."
Management’s Response: The IRS responded, "We will continue our current practice of reporting these indicators separately. In FY 2001, we established enterprise-wide expanded Spanish-speaking service. After we gain additional experience with this new service and make the necessary adjustments to ensure comparable data, we will combine the reporting of the indicators as suggested.
We review calls that result in r-mail responses when initially received. We include the results of this review in our quality indicator for Toll-Free Tax Law Quality. We conduct a quality review of r-mail responses whether e-mail or answered by return call and report the results under our quality indicator for r-mail. To include them with our Toll-Free Quality would be inconsistent with the purpose of the indicator, which is on-line "live" phone assistance received on the three major Toll-Free incoming lines with the exception of Spanish assistance."
Management’s Response: The IRS responded, "We have added the Toll-Free Automated Completion Rate to our suite of measures for FYs 2002 and 2003. This rate is a calculation based on the customer completing a specific application in our automated system. It will capture customer experience in the automated system. The Completion Rate, when combined with other management information from our automated systems, will be an indicator of the quality of the service being provided. We will identify, define, and maintain these measures separately and will not include them in any measure designed to assess accuracy of answers provided to taxpayers through "live" assistance."
The Internal Revenue Service Needs To Set Long-term Goals for the Quality of Toll-Free Service
The IRS has not established long-term goals for indicators related to the assessment of quality. Although the IRS set interim yearly improvement goals through FY 2003 of 3 percentage points increase per year for its toll-free quality indicators, it has not established overall goals to strive toward.
In the IRS’ Strategic Plan for FYs 2000-2005, there are no specific indicators listed nor any quantitative or qualitative goals. Only the general strategic goals and objectives of "Service to Each Taxpayer," "Service to All Taxpayers," and "Productivity Through a Quality Work Environment" are highlighted in the strategic plan. No specific information is discussed for assessing achievement of the strategic goals.
When describing the term "general goals and objectives" as used in the GPRA, OMB Circular No. A-11 states that, "strategic plans set out long-term programmatic, policy, and management goals…outlining planned accomplishments…" and that "the goals(s) must be defined in a manner that allows a future assessment to be made on whether or not the goal was or is being achieved." The Circular also states that a strategic plan must describe the relationship between general goals in the strategic plan and the performance goals in the annual performance plan. Performance goals that represent milestones in achieving the general goals of the strategic plan may be appropriate.
Because the IRS has not established long-term goals for the indicators it uses to measure accuracy, it will not be able to determine when it has achieved a reasonable level or a stakeholder’s expectation of accuracy. Consequently, the IRS does not have adequate information to use as a basis for long-term funding requests and training decisions. The Congress also does not have a sound basis for making assessments of whether the IRS is making sufficient progress in achieving its strategic goal of providing service to each taxpayer.
Recommendation
The Commissioner, Wage and Investment Division, should:
Management’s Response: The IRS responded, "Based on testing in August 2001 of a revised quality measure for Toll-Free designed by the Embedded Quality project, a long-term goal for "live assistance" quality will be determined and added to the strategic plan in FY 2002."
Conclusion
The IRS has recognized the need to improve customer service quality and develop appropriate indicators of its performance and has begun an effort to identify and implement an improved approach to quality. Until that study is completed and the IRS has developed appropriate indicators, it will continue to report its "quality" accomplishments with indicators that do not provide an adequate assessment of the agency’s achievements.
Appendix I
Major Contributors to This Report
Walter E. Arrison, Assistant Inspector General for Audit (Wage and Investment Income Programs)
M. Susan Boehmer, Director
Deann L. Baiza, Audit Manager
Gregory Dix, Senior Auditor
Doris J. Hynes, Senior Auditor
Bonnie G. Shanks, Auditor
Appendix II
Report Distribution List
Commissioner, Wage and Investment Division W
Chief Counsel CC
National Taxpayer Advocate TA
Office of Management Controls N:CFO:F:M
Director, Accounts Management W:CAS:AM
Director, Customer Account Services W:CAS
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Director, Strategy and Finance W:S
Director, Legislative Affairs CL:LA
Audit Liaisons:
Commissioner, Wage and Investment Division W
Director, Accounts Management W:CAS:AM
Director, Customer Account Services W:CAS
Appendix III
Management’s Response to the Draft Report
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.