The audit was performed as part of the Treasury Inspector General for Tax Administration’s overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer (taxpayer) satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The GPRA is intended to improve agency performance and provide objective information to congressional and executive branch decision-makers to assist them in appropriating and allocating federal funds. The law requires federal agencies to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments.

The IRS established balanced performance measures to support achievement of its strategic goals: provide quality service to each taxpayer, serve all taxpayers, and be productive through a quality work environment. Achievement of these goals is measured through customer satisfaction, employee satisfaction, and business results. Taxpayers who receive specific kinds of services from the IRS might be asked to rate the service. These survey results are being summarized by a vendor and used by the IRS to evaluate the overall satisfaction with IRS service.

The IRS originally designed the surveys as part of its balanced performance management system and did not anticipate using the surveys for GPRA reporting purposes. Subsequently, the IRS used the survey results to fulfill the GPRA reporting requirements. The intent of the GPRA is that the Congress will use the performance measurement results to help evaluate IRS budget appropriations. Therefore, it is essential that the IRS accurately measure its success in meeting the performance goals.

This review evaluated the accuracy, validity, and reliability of the information used by the IRS to measure customer satisfaction with the Toll-Free Program. The Toll-Free Program is a significant part of the IRS’ activities, providing customers free telephone tax assistance 24 hours a day, 7 days a week. Toll-Free telephone lines provide various types of customer service such as ordering forms, reporting fraud, answering tax-related questions, providing special assistance for long-standing tax problems, etc.

The present results of the Toll-Free Customer Satisfaction Survey should not be used to satisfy all requirements of the GPRA, unless appropriately qualified. The original purpose of the Toll-Free Customer Satisfaction Survey was not for GPRA reporting but to measure the overall trend of taxpayer satisfaction with Toll-Free service. While the Survey may be an effective marketing tool to gauge taxpayers’ satisfaction with the services provided by the Toll-Free Program, a higher standard is required when these results are used for GPRA reporting purposes. Internal controls should be in place to ensure the Survey results are reliable and that the data used for GPRA reporting purposes have been verified and validated.

IRS executives have not established management controls to continuously monitor and improve the administration of the Toll-Free Customer Satisfaction Survey and the means to verify and validate the Survey results to measure the level of satisfaction all customers receive from interactions with Toll-Free Program employees. Without reliable information, the IRS cannot provide a basis for comparing program results with the established performance goals and will not achieve the benefits of the GPRA intended by the Congress.

The current management control process is not adequate to ensure the Toll-Free Customer Satisfaction Survey is administered properly and the results are reliable and can be verified and validated for GPRA purposes. Specifically:

• More accountability is needed for the Toll-Free Customer Satisfaction Survey results. Three different IRS offices share responsibility for all the facets of the Toll-Free Customer Satisfaction Survey, but no one is responsible for ensuring procedures established by one office are properly executed by another office. No office has taken responsibility for ensuring that procedures have been established to verify and validate the Survey results.
• Including customers who telephone the toll-free lines on the weekends and after the surveyors’ regular work hours Monday through Friday will increase the reliability of the results. The Toll-Free Customer Satisfaction Survey sample selection methodology excludes specific customers who call on Saturday and Sunday or after surveyors’ work hours (i.e., generally after 5:00 p.m.).
• A better random sampling technique would improve the reliability of the Survey results and decrease potential bias. The selection methods used do not ensure that each item in the population has an equal chance of being selected. There is no way of assessing how accurately the sample reflects the characteristics of the entire population (i.e., taxpayers using Toll-Free services). In addition, the instructions for the sample selection are not clear and are not always followed.

We believe that without reliable Survey results, the IRS cannot meet the GPRA requirements of establishing measurable performance goals and reporting of accurate results. Without improvements in the above conditions, the Toll-Free Customer Satisfaction Survey results should be qualified if used to meet the requirements of the GPRA.

We recommended that the Deputy Chief Financial Officer for Strategic Planning and Budgeting, in conjunction with the Assistant Commissioner (Customer Service); the Chief, Customer Service Field Operations; and the Director, Office of Program Evaluation and Risk Analysis, consider establishing a process to ensure the Toll-Free Customer Satisfaction Survey is administered properly and that the data used to report the Survey results are reliable, valid, and verifiable. We also recommended that the Survey selection technique be revised and surveyors’ work hours expanded to correspond with site operation hours to ensure all customers’ interactions have an opportunity of being selected. In addition, the GPRA reports should be appropriately qualified with any limitations of the Survey’s design and reliability of its data.

Management’s Response: IRS management believes that their management control processes are sufficient and additional controls are not needed to ensure the Survey results are reliable, valid, and verifiable. They also disagree that the current sample selection methodology should be altered. And though they cannot realistically expand the Survey coverage at this time to correspond with site operation hours, IRS management agreed to disclose in the reports the hours included in the Survey coverage and the related population of taxpayers whose calls are subject to survey.

IRS management will ensure that all future Toll-Free Customer Satisfaction Survey Reports and other GPRA reports include information on which groups of customers are included in the Survey population. In addition the IRS will reassess, with new vendors, the Survey program and controls over Survey administration.

Management’s complete response to the draft report is included as Appendix IV.

Office of Audit Comment: We believe that the control processes established by IRS management have not been effective in verifying and validating the Survey results. On-site reviews would have identified that the surveyors do not always follow the sample selection method and that the selection methods vary among sites.

In addition, we believe the current sample selection methodology may not produce a sample representative of the entire population and may introduce bias. A basic requirement of any sample is that it should be representative of the entire population from which it was selected.

The IRS’ unit of selection is a taxpayer and each taxpayer call is randomly routed to the next available Customer Service Representative (CSR). However, since the IRS does not have the technology to randomly select calls received by a taxpayer, it must select calls received by the CSRs. Because taxpayers are randomly routed to the next available CSR, each taxpayer has an equal opportunity of being selected to participate in the Survey, with the exception of taxpayers who call after hours. However, since all CSRs do not answer the same number of calls, the IRS must also select CSRs in a random manner that is representative of the entire population.

By working down the list of active/eligible CSRs one by one (the skip-interval (k) is equal to one), the IRS’ systematic selection does not account for the differences in the number of calls answered by each CSR during each sample period. Use of a skip-interval (k) that is equal to one does not produce a systematic CSR selection that ensures the number of CSRs in the sample is proportionate to the number of calls answered by each CSR. For example, if CSR "1" answers twice as many calls as CSR "2," then CSR "1" should be selected twice as many times as CSR "2."

Therefore, we believe the IRS should revise the sample selection technique or weight the sample results by CSR to improve the reliability of the Survey results and to decrease potential bias.