Toll-Free Customer Satisfaction Survey Results Should Be Qualified if Used for the Government Performance and Results Act
September 2000
Reference Number: 2000-10-137
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
September 20, 2000
MEMORANDUM FOR COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Toll-Free Customer Satisfaction Survey Results Should Be Qualified if Used for the Government Performance and Results Act
This report presents the results of our review of the Toll-Free Customer Satisfaction Survey Results as they relate to the Government Performance and Results Act of 1993 (GPRA). The overall objective of this review was to evaluate the accuracy, validity, and reliability of the information used by the Internal Revenue Service (IRS) to measure customer satisfaction with the Toll-Free Program. The Toll-Free Program is a significant part of the IRS’ activities, providing customers free telephone tax assistance 24 hours a day, 7 days a week.
In summary, we found that the Toll-Free Customer Satisfaction Survey results are not statistically valid and should not be used to satisfy requirements of the GPRA unless they are appropriately qualified. We recommended that the Deputy Chief Financial Officer for Strategic Planning and Budgeting, in conjunction with the Assistant Commissioner (Customer Service); the Chief, Customer Service Field Operations; and the Director, Office of Program Evaluation and Risk Analysis, consider establishing a process to ensure the Toll-Free Customer Satisfaction Survey is administered properly and that the data used to report Survey results are reliable, valid, and verifiable. We also recommended that the Survey selection technique be revised and surveyors’ work hours expanded to correspond with site operation hours to ensure all customers’ interactions have an opportunity of being selected. In addition, the GPRA reports should be appropriately qualified with any limitations of the Survey’s design and reliability of its data.
IRS management believes that their management control processes are sufficient and additional controls are not needed to ensure the Survey results are reliable, valid, and verifiable. They also disagree that the current sample selection methodology should be altered. They cannot realistically expand the Survey coverage at this time to correspond with site operation hours. Where appropriate, we have included additional comments in the body of the report to further explain our position on these issues.
IRS management agreed to disclose in the reports the hours included in the Survey coverage and will ensure that all future Toll-Free Customer Satisfaction Survey Reports and other GPRA reports include information on which groups of customers are included in the Survey population. The IRS will also reassess, with new vendors, the Survey program and controls over Survey administration. Management’s comments have been incorporated into the report where appropriate, and the full text of their comments is included as an appendix.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions, or your staff may call Maurice S. Moody, Associate Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs), at (202) 622-8500.
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Management’s Response to the Draft Report
The audit was performed as part of the Treasury Inspector General for Tax Administration’s overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer (taxpayer) satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The GPRA is intended to improve agency performance and provide objective information to congressional and executive branch decision-makers to assist them in appropriating and allocating federal funds. The law requires federal agencies to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments.
The IRS established balanced performance measures to support achievement of its strategic goals: provide quality service to each taxpayer, serve all taxpayers, and be productive through a quality work environment. Achievement of these goals is measured through customer satisfaction, employee satisfaction, and business results. Taxpayers who receive specific kinds of services from the IRS might be asked to rate the service. These survey results are being summarized by a vendor and used by the IRS to evaluate the overall satisfaction with IRS service.
The IRS originally designed the surveys as part of its balanced performance management system and did not anticipate using the surveys for GPRA reporting purposes. Subsequently, the IRS used the survey results to fulfill the GPRA reporting requirements. The intent of the GPRA is that the Congress will use the performance measurement results to help evaluate IRS budget appropriations. Therefore, it is essential that the IRS accurately measure its success in meeting the performance goals.
This review evaluated the accuracy, validity, and reliability of the information used by the IRS to measure customer satisfaction with the Toll-Free Program. The Toll-Free Program is a significant part of the IRS’ activities, providing customers free telephone tax assistance 24 hours a day, 7 days a week. Toll-Free telephone lines provide various types of customer service such as ordering forms, reporting fraud, answering tax-related questions, providing special assistance for long-standing tax problems, etc.
Results
The present results of the Toll-Free Customer Satisfaction Survey should not be used to satisfy all requirements of the GPRA, unless appropriately qualified. The original purpose of the Toll-Free Customer Satisfaction Survey was not for GPRA reporting but to measure the overall trend of taxpayer satisfaction with Toll-Free service. While the Survey may be an effective marketing tool to gauge taxpayers’ satisfaction with the services provided by the Toll-Free Program, a higher standard is required when these results are used for GPRA reporting purposes. Internal controls should be in place to ensure the Survey results are reliable and that the data used for GPRA reporting purposes have been verified and validated.
IRS executives have not established management controls to continuously monitor and improve the administration of the Toll-Free Customer Satisfaction Survey and the means to verify and validate the Survey results to measure the level of satisfaction all customers receive from interactions with Toll-Free Program employees. Without reliable information, the IRS cannot provide a basis for comparing program results with the established performance goals and will not achieve the benefits of the GPRA intended by the Congress.
Toll-Free Customer Satisfaction Survey Results Should Be Qualified if Used to Report Performance Measures for the Government Performance and Results Act of 1993
The current management control process is not adequate to ensure the Toll-Free Customer Satisfaction Survey is administered properly and the results are reliable and can be verified and validated for GPRA purposes. Specifically:
We believe that without reliable Survey results, the IRS cannot meet the GPRA requirements of establishing measurable performance goals and reporting of accurate results. Without improvements in the above conditions, the Toll-Free Customer Satisfaction Survey results should be qualified if used to meet the requirements of the GPRA.
Summary of Recommendations
We recommended that the Deputy Chief Financial Officer for Strategic Planning and Budgeting, in conjunction with the Assistant Commissioner (Customer Service); the Chief, Customer Service Field Operations; and the Director, Office of Program Evaluation and Risk Analysis, consider establishing a process to ensure the Toll-Free Customer Satisfaction Survey is administered properly and that the data used to report the Survey results are reliable, valid, and verifiable. We also recommended that the Survey selection technique be revised and surveyors’ work hours expanded to correspond with site operation hours to ensure all customers’ interactions have an opportunity of being selected. In addition, the GPRA reports should be appropriately qualified with any limitations of the Survey’s design and reliability of its data.
Management’s Response: IRS management believes that their management control processes are sufficient and additional controls are not needed to ensure the Survey results are reliable, valid, and verifiable. They also disagree that the current sample selection methodology should be altered. And though they cannot realistically expand the Survey coverage at this time to correspond with site operation hours, IRS management agreed to disclose in the reports the hours included in the Survey coverage and the related population of taxpayers whose calls are subject to survey.
IRS management will ensure that all future Toll-Free Customer Satisfaction Survey Reports and other GPRA reports include information on which groups of customers are included in the Survey population. In addition the IRS will reassess, with new vendors, the Survey program and controls over Survey administration.
Management’s complete response to the draft report is included as Appendix IV.
Office of Audit Comment: We believe that the control processes established by IRS management have not been effective in verifying and validating the Survey results. On-site reviews would have identified that the surveyors do not always follow the sample selection method and that the selection methods vary among sites.
In addition, we believe the current sample selection methodology may not produce a sample representative of the entire population and may introduce bias. A basic requirement of any sample is that it should be representative of the entire population from which it was selected.
The IRS’ unit of selection is a taxpayer and each taxpayer call is randomly routed to the next available Customer Service Representative (CSR). However, since the IRS does not have the technology to randomly select calls received by a taxpayer, it must select calls received by the CSRs. Because taxpayers are randomly routed to the next available CSR, each taxpayer has an equal opportunity of being selected to participate in the Survey, with the exception of taxpayers who call after hours. However, since all CSRs do not answer the same number of calls, the IRS must also select CSRs in a random manner that is representative of the entire population.
By working down the list of active/eligible CSRs one by one (the skip-interval (k) is equal to one), the IRS’ systematic selection does not account for the differences in the number of calls answered by each CSR during each sample period. Use of a skip-interval (k) that is equal to one does not produce a systematic CSR selection that ensures the number of CSRs in the sample is proportionate to the number of calls answered by each CSR. For example, if CSR "1" answers twice as many calls as CSR "2," then CSR "1" should be selected twice as many times as CSR "2."
Therefore, we believe the IRS should revise the sample selection technique or weight the sample results by CSR to improve the reliability of the Survey results and to decrease potential bias.
This audit is part of the Treasury Inspector General for Tax Administration’s (TIGTA) overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The IRS is implementing a balanced performance measurement system to balance customer (taxpayer) satisfaction, employee satisfaction, and business results. These quantitative measures will support and reinforce the IRS’ achievement of its overall strategic goals. The TIGTA is conducting several reviews to address separate elements of the customer satisfaction measurement system.
The overall objective of this audit was to evaluate the accuracy, validity, and reliability of the information used by the IRS to measure customer satisfaction with the Toll-Free Program services. We performed this audit from February to June 2000 in accordance with Government Auditing Standards. In addition, we relied upon the prior audit work conducted from October 1999 to February 2000 and reported in the TIGTA audit report Automated Collection System Customer Satisfaction Survey Results Should Be Qualified if Used for the GPRA (Reference Number 2000-10-078, dated May 2000). Audit work performed in the National Headquarters and the Buffalo call site during that review will be included in this report.
We performed on-site testing in the National Headquarters and the Atlanta Customer Service Center. In addition, we made telephone contacts with surveyors in the Dallas, Fresno, Jacksonville, Nashville, and Pittsburgh Customer Service call sites. These sites were selected based on their 24 hours a day, 7 days a week operations; past audit coverage; multi-lingual capabilities; and proximity to the TIGTA staff. During the audit, we:
Our scope of work was limited to evaluating the process the IRS uses to gather the information for measuring customer satisfaction with the Toll-Free Program. Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.
The GPRA requires federal agencies to establish standards for measuring their performance and effectiveness. Federal agencies are required to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments.
The overall goal of the GPRA is to improve agency performance and provide objective information to congressional and executive branch decision-makers to assist them in appropriating and allocating federal funds. Therefore, it is essential that the data used for the performance measures are reliable and the results are valid and verifiable to ensure proper conclusions are made by both the Congress and the IRS.
The IRS prepared a multi-year strategic plan and annually prepares a performance plan. It also established three strategic goals: provide quality service to each taxpayer, serve all taxpayers, and be productive through a quality work environment. Providing quality service to each taxpayer is a key part of customer satisfaction and includes:
The IRS measures its success in achieving this goal by using a customer satisfaction survey to measure its programs. The IRS contracted with a vendor to conduct 11 customer satisfaction surveys. The vendor designed and prepared the surveys with IRS executives and staff. Customers who receive specific kinds of services might be asked to complete a survey to rate the service they received. The vendor summarizes the responses to measure the overall trend in satisfaction with IRS service.
One of the 11 programs using surveys to measure customer satisfaction is the Toll-Free Program. The Toll-Free Program is a significant part of the IRS’ customer service activities. Toll-free telephone lines provide various types of customer service such as ordering forms, reporting fraud, answering tax-related questions, providing special assistance for long-standing tax problems, etc. There are numerous toll-free telephone numbers in the Program. However, only three main toll-free telephone lines are included in the Toll-Free Customer Satisfaction Survey, the lines designated as: (1) Refund Inquiries; (2) General Tax Information, Tax Law, and Procedural Inquires; and (3) Account Notice Inquiries. These lines provide free telephone tax assistance to customers 24 hours a day, 7 days a week.
Taxpayers who call any of the three main toll-free telephone lines either obtain automated assistance or reach a Customer Service Representative (CSR). Only those taxpayers who reach a CSR are asked to participate in the Survey.
The Customer Service organization is responsible for the Toll-Free Program. IRS telephone reports for Fiscal Year (FY) 1999 show over 53.5 million calls to the IRS’ 3 Toll-Free Program telephone lines were answered. CSRs answered over 37.8 million (71 percent) Toll-Free Program calls from customers.
The IRS is developing a new planning process that will provide support for its efforts to comply with the GPRA. Until this process is completed, the IRS Commissioner designated the Deputy Chief Financial Officer for Strategic Planning and Budgeting as responsible for overseeing and coordinating the implementation of all the GPRA-related activity for the IRS. However, other offices are responsible for conducting the Survey.
For GPRA reporting, the IRS should clearly disclose the limitations of the Survey’s design and the reliability of its data if the results are used to report the level of satisfaction customers have with the Toll-Free Program’s services.
The original purpose of the Survey was not for GPRA reporting but to measure the overall trend of taxpayer satisfaction with the Toll-Free Program’s services. Surveyors and CSRs took steps to administer specific aspects of the Survey. For example:
However, the Survey results are being reported to the Congress for GPRA purposes, which requires that the data obtained from the Surveys be reliable, verified, and validated. While the Survey may be an effective marketing tool to gauge taxpayers’ satisfaction with the services provided by the Toll-Free Program, a higher standard is required when these results are used for GPRA reporting purposes. A management process of internal controls should be in place to ensure the Survey results are reliable and that the data used for GPRA reporting purposes have been verified and validated.
Without reliable information, the IRS cannot provide a basis for comparing program results with the established performance goals, as intended by the Congress when it established the GPRA.
Toll-Free Customer Satisfaction Survey Results Should Be Qualified if Used to Report Performance Measures for the Government Performance and Results Act of 1993
IRS executives have not established an adequate management process to ensure the Survey is administered properly and the results are reliable and can be verified and validated for GPRA purposes. As a result, the Survey results for the Toll-Free Program include only the opinions of those taxpayers who interact with IRS employees and only those who call during regular work hours Monday through Friday. In addition, the selection methods used do not ensure that each item in the population has an equal chance of being selected.
Without a reliable survey process and a system of internal controls, there is a significant risk the IRS is not collecting data essential for accurate and reliable Survey results that represent all customers who interact with Toll-Free Program CSRs.
If improvements are not made to the above conditions, the Survey results should be qualified if used to meet the requirements of the GPRA. Without reliable Survey results, the IRS cannot provide a basis for comparing program results with the established performance goals and will not fully achieve the benefits of the GPRA intended by the Congress.
More accountability is needed for the Toll-Free Customer Satisfaction Survey results
Three IRS offices currently provide oversight for the Toll-Free Customer Satisfaction Survey.
The OPERA and the Office of the Assistant Commissioner (Customer Service) held quality conferences and provided the Toll-Free Program sites with memoranda, training materials, and guidelines. However, there were no on-site reviews to ensure that surveyors were properly selecting CSRs or that surveyors and CSRs were properly administering the Survey. None of the three offices appears to be accountable for the reliability of the Survey results. Nor has any office taken responsibility for ensuring that procedures have been established to verify and validate the Survey results.
OMB Circular No.A-123, Management Accountability and Control, states that management accountability is the expectation that managers are responsible for program performance. As managers develop and execute strategies for implementing agency programs, they should design management structures to help ensure accountability for results.
Including customers who telephone the toll-free lines on the weekends and after the surveyors’ regular work hours Monday through Friday will increase the reliability of the results
The IRS reported to the Congress in its FY 2000 Congressional Justification (submitted in FY 1999) that it would determine the level of satisfaction Toll-Free Program customers receive from interactions with the Toll-Free Program services. However, the Survey is provided only to those customers who reach a CSR. This represents about 71 percent of the calls made to the 3 main toll-free telephone lines. In addition, not all customers who reached CSRs had the opportunity to participate in the Survey. Customers telephoning the toll-free lines during evening hours and on weekends are usually not included in the Survey process.
Generally, call sites conduct the Survey only during surveyors’ regular work hours (i.e., from 7:30 a.m. to 5:00 p.m. Monday through Friday). However, approximately 20 percent of the calls to CSRs are made after 5:00 p.m. Not including in the Survey process those taxpayers who call the IRS in the evening hours and weekends could distort the sample results because the types of customers and issues may differ between daytime and evening or weekdays and weekends.
A better random sampling technique would improve the reliability of the Survey results and decrease potential bias
The basic criterion in selecting a random sample is that every item (e.g., every CSR) must be given an equal opportunity of being selected. Otherwise, bias is introduced, and there is no way of assessing how accurately the sample reflects the characteristics of the population. We discussed the sampling techniques with surveyors at six call sites. Our results follow.
Instructions in the Internal Revenue Manual (IRM) do not produce a representative sample.
CSRs are assigned unique identification (ID) numbers in a non-sequential manner. In the instructions on the selection method illustrated in the January 1, 2000, IRM, CSRs with a wide separation between their ID number and those of other CSRs are more likely to be included in the sample. For the selection of CSRs to be a completely random process, every CSR must have an equal chance of being selected.
Each working day (Monday through Friday), six CSRs are selected at three or four random times throughout the day. The customers who the CSRs are talking to at the time of the selections are asked to participate in the Survey.
To identify the first CSR to monitor for each assigned time, surveyors should select a random number within the range of all CSR ID numbers, as well as a direction indicator of either higher (+) or lower (-). For example, assume the random number selected is 3335 with a direction indicator of higher (+). According to the IRM, the surveyor should identify the first eligible CSR whose ID number is 3335 or higher and monitor that CSR’s call.
Using the table below as an example and assuming all CSRs are answering telephone calls at the assigned time, the surveyor would select the CSR whose employee number is 3339.
Customer Service Representative
Identification Numbers
|
3329 |
3331 |
|
3332 |
3334 |
|
3339 |
3344 |
|
3346 |
3349 |
|
3350 |
3351 |
|
3352 |
3353 |
This CSR has at least nine chances of being selected as the first CSR selected at any time, no matter who is answering telephone calls:
In contrast, the CSR whose ID number is 3351 has only one chance of being selected when the CSRs whose ID numbers are 3350 and 3352 are also available to answer telephone calls.
In the instructions on the selection method illustrated in the May 8, 2000, IRM, the surveyors are instructed to randomly select the first CSR during the first round of monitoring and simply proceed down a list of CSRs. All CSRs are monitored once before any CSR is monitored a second time. Similarly, all CSRs should be monitored twice before the third rotation begins, and so on.
The surveyors do not always follow sample selection procedures, and selection methods vary among call sites.
Surveyors do not always understand IRM procedures for sample selection. This results in improper use of the random number tables to select CSRs. We identified the following examples:
The sampling methods used to date increase the risk that results will not be representative of all Toll-Free Program customers. Weighting of sample results is one technique that can be used to improve the efficiency of the Survey results. However, we did not identify this or other techniques being used to correct for sampling deficiencies.
Recommendations
We recommend that the Deputy Chief Financial Officer for Strategic Planning and Budgeting, in conjunction with the Assistant Commissioner (Customer Service); the Chief, Customer Service Field Operations; and the Director, OPERA, consider the following:
Management’s Response: IRS management believes that their management control processes are sufficient and additional controls are not needed to ensure the Survey results are reliable, valid, and verifiable. They also disagree that the current sample selection methodology should be altered. And though they cannot realistically expand the Survey coverage at this time to correspond with site operation hours, IRS management agreed to disclose in reports the hours included in the Survey coverage and the related population of taxpayers whose calls are subject to survey.
IRS management will ensure that all future Toll-Free Customer Satisfaction Survey Reports and other GPRA reports include information on which groups of customers are included in the Survey population. In addition the IRS will reassess, with new vendors, the Survey program and controls over Survey administration.
Management’s complete response to the draft report is included as Appendix IV.
Office of Audit Comment: We believe that the control processes established by IRS management have not been effective in verifying and validating the Survey results. On-site reviews would have identified that the surveyors do not always follow the sample selection method and that the selection methods vary among sites.
In addition, the current sample selection methodology may not produce a sample representative of the entire population and may introduce bias. A basic requirement of any sample is that it should be representative of the entire population from which it was selected.
The IRS’ unit of selection is a taxpayer and each taxpayer call is randomly routed to the next available CSR. However, since the IRS does not have the technology to randomly select calls received by a taxpayer, it must select calls received by the CSRs. Because taxpayers are randomly routed to the next available CSR, each taxpayer has an equal opportunity of being selected to participate in the Survey, with the exception of taxpayers who call after hours. However, since all CSRs do not answer the same number of calls, the IRS must also select CSRs in a random manner that is representative of the entire population.
By working down the list of active/eligible CSRs one by one (the skip-interval (k) is equal to one), the IRS systematic selection does not account for the differences in the number of calls answered by each CSR during each sample period. Use of a skip-interval (k) that is equal to one does not produce a systematic CSR selection that ensures the number of CSRs in the sample is proportionate to the number of calls answered by each CSR. For example, if CSR "1" answers twice as many calls as CSR "2," then CSR "1" should be selected twice as many times as CSR "2."
Therefore, we believe the IRS should revise the sample selection technique or weight the sample results by CSR to improve the reliability of the Survey results and to decrease potential bias.
Presently, the Toll-Free Customer Satisfaction Survey results are not valid for establishing a statistical basis for comparing program results with established performance goals as required by the GPRA. There are limitations with the Survey process that result in an increased risk that the data are not reliable to report measures for the GPRA. Until the IRS revises the Survey selection method, the IRS should clearly disclose the limitations of the Survey’s data and the reliability of the results.
Appendix I
Detailed Objective, Scope, and MethodologyThe overall objective of this review was to evaluate the accuracy, validity, and reliability of the information used to measure customer satisfaction with the Toll-Free Program services. We performed the following work:
Appendix II
Major Contributors to This ReportMaurice S. Moody, Associate Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)
Stanley C. Rinehart, Director
John R. Wright, Director
Augusta R. Cook, Audit Manager
Kenneth L. Carlson, Jr., Senior Auditor
Mary Lynn Faulkner, Senior Auditor
Cindy Harris, Auditor
David Lowe, Auditor
Cindy Wright, Auditor
Appendix III
Report Distribution ListDeputy Commissioner Operations C:DO
Assistant to the Deputy Commissioner Operations C:DO
Chief Financial Officer C:CFO
Chief Operations Officer OP
Director, National Headquarters Management and Finance M
Assistant Commissioner (Customer Service) OP:C
Chief, Customer Service Field Operations OP:CSF
Controller/Director for Financial Management CFO:F
Deputy Chief Financial Officer for Strategic Planning and Budgeting CFO:SPB
Director, Office of Program Evaluation and Risk Analysis M:O
Director, Organizational Performance Management Division C:DO:OPME
Office of the Chief Counsel CC
Office of Management Controls CFO:A:M
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Atlanta Customer Service Center
Director, Austin Customer Service Center
Director, Brookhaven Customer Service Center
Director, Fresno Customer Service Center
Director, Memphis Customer Service Center
Director, Philadelphia Customer Service Center
Audit Liaisons:
Director, National Headquarters Management and Finance M
Assistant Commissioner (Customer Service) OP:C
Deputy Chief Financial Officer for Strategic Planning and Budgeting CFO:SPB
Director, Office of Program Evaluation and Risk Analysis M:O
Appendix IV
Management’s Response to the Draft ReportThe response has been removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.