Executive Summary

Objective and Scope

Background

Results

Conclusion

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Management’s Response

Executive Summary

This audit was performed as part of the Treasury Inspector General for Tax Administration’s overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The GPRA is intended to improve agency performance and to provide objective information to congressional and executive branch decision-makers to assist them in appropriating and allocating federal funds. The law requires federal agencies to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments.

The IRS established balanced performance measures to support achievement of the IRS’ strategic goals: provide quality service to each taxpayer, serve all taxpayers, and be productive through a quality work environment. Achievement of these goals is measured through customer satisfaction, employee satisfaction, and business results. Taxpayers who receive specific kinds of services from the IRS might be asked to rate the service. These survey results are summarized by a vendor and will be used by the IRS to evaluate the overall satisfaction with the IRS’ service.

The survey results will also be used to set goals and expectations for the GPRA. The intent of the GPRA is that the Congress will use the performance measurement results to help evaluate the IRS budget appropriation. Therefore, it is essential that the IRS accurately measure its success in meeting the performance goals.

The overall objective of this review was to evaluate the reliability of the information used to measure customer satisfaction with the IRS Appeals process. The Appeals function has the authority to consider and negotiate settlements of internal revenue controversies. Appeals’ responsibility includes, but is not limited to, the determination of liability for taxes, penalties, and interest.

The present results of the Appeals Customer Satisfaction Survey should not be used to satisfy all requirements of the GPRA, unless appropriately qualified. The original purpose of the Appeals Customer Satisfaction Survey was not for GPRA reporting but to measure the overall trend of taxpayer satisfaction with the Appeals function’s service. While the Survey may be an effective marketing tool to gauge taxpayers’ satisfaction with the services provided by Appeals offices, a higher standard is required when these results are used for GPRA reporting purposes.

IRS executives have not established an adequate process to ensure the Survey is conducted appropriately and that the data are reliable, valid, and verifiable. Without reliable information, the IRS cannot provide a basis for comparing program results with the established performance goals as intended by the Congress when it established the GPRA.

The controls over the automated case control system used to identify the Survey population for the Appeals Customer Satisfaction Survey were largely in place. The controls were sufficient to preclude the existence of a significant number of uncontrolled Appeals cases, which would have had an adverse effect on the validity of the Survey population.

However, there are limitations with the Appeals Customer Satisfaction Survey process that result in an increased risk that the data are not reliable to report measures for the GPRA.

• Controls over the transmission of Appeals closed case listings to the vendor are not adequate to ensure the data are complete and protected. The vendor contracted by the IRS to conduct the Survey uses closed case listings to select samples of taxpayers to participate in the Appeals Survey. The listings are transmitted from the IRS to the vendor monthly on an unsecured telecommunications network. This could result in lost, incomplete, or inaccurate data. It could also result in unauthorized access to the data.
• The current response rate to the Appeals Customer Satisfaction Survey is low; only 44 percent of the taxpayers who were sent a survey form in 1998 responded to the Appeals Survey. A low response rate increases the risk that the Survey results are not reliable and may not be representative of the Survey population.

Until the data transmitted to the vendor are adequately protected and a higher response rate is achieved, the IRS should clearly disclose the limitations of the Appeals Survey’s data and the reliability of the results if used for GPRA purposes in reporting the level of satisfaction customers have with the Appeals function’s service.

To ensure that the IRS accurately measures the level of satisfaction customers receive from the Appeals function’s services, additional actions are needed. We recommend that the Deputy Chief Financial Officer for Strategic Planning and Budgeting, in conjunction with the National Chief Appeals and the Director, Office of Program Evaluation and Risk Analysis, improve the process to oversee the Appeals Customer Satisfaction Survey. This will help ensure that the data used to report the Survey results are reliable, valid, and verifiable.

Management’s Response: Appeals management agreed with our recommendation that the data transmitted to the vendor be encrypted and that the limitations of the Survey be disclosed for purposes of the GPRA. However, Appeals management stated that the National Chief Appeals, working in conjunction with the Director, Office of Program Evaluation and Risk Analysis, is responsible for the Appeals Customer Satisfaction Survey, not the Deputy Chief Financial Officer, Strategic Planning and Budgeting.

Objective and Scope

This audit is part of the Treasury Inspector General for Tax Administration’s (TIGTA) overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The IRS is implementing a performance measurement system to balance customer (taxpayer) satisfaction, employee satisfaction, and business results. These quantitative measures will support and reinforce the IRS’ achievement of its overall strategic goals. The TIGTA is conducting several reviews that will address separate elements of the customer satisfaction measurement system.

The objective of this audit was to evaluate the reliability of the information used to measure customer satisfaction with the Appeals process. We performed this audit from January to June 2000 in accordance with Government Auditing Standards.

We conducted our audit tests in the National Headquarters and in the Georgia and Rocky Mountain District Appeals offices. We selected these offices after considering the historical data of closed cases and survey responses per office and the locations of IRS offices and audit staff.

During the audit, we:

• Discussed with National Headquarters IRS management how they established the Survey for Appeals offices and how it would be used for the GPRA.
• Interviewed Appeals managers and employees on current survey procedures and controls over the Survey process and data.
• Obtained a listing of Appeals cases opened the week of February 22, 1999, and compared a sample of those cases to IRS records and the vendor’s lists of closed cases used to select customers for the Appeals Customer Satisfaction Survey.
• Evaluated controls over the Appeals automated case control system and the transmittal of data to the vendor.

Our scope of work for the current audit was limited to evaluating the process the IRS uses to gather information for measuring customer satisfaction with the Appeals function’s services. Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

Background

The GPRA requires federal agencies to establish standards for measuring their performance and effectiveness. Federal agencies are required to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments.

The overall goal of the GPRA is to improve agency performance and provide objective information to congressional and executive branch decision-makers to assist them in appropriating and allocating federal funds. Therefore, it is essential that the data used for the performance measures are reliable and the results are verifiable and valid to ensure the proper conclusions are made by both the Congress and the IRS.

The IRS prepared a multi-year strategic plan and annually prepares a performance plan. It also established three strategic goals: provide quality service to each taxpayer, serve all taxpayers, and be productive through a quality work environment. Providing quality service to each taxpayer is a key part to customer satisfaction and includes:

• Making filing easier.
• Providing first quality service to each taxpayer needing help with a return or account.
• Providing prompt, professional, and helpful treatment to taxpayers in cases where additional taxes may be due.

The IRS measures its success in achieving this goal by using a customer satisfaction survey to measure its programs. The IRS contracted with a vendor to conduct 11 customer satisfaction surveys. The vendor designed and prepared the surveys with IRS executives and staff. Taxpayers receiving specific kinds of services might be asked to complete a survey to rate the service they received. The responses are summarized by the vendor and used by the IRS to measure the overall trend in satisfaction with the IRS’ service.

One of the 11 programs using the surveys to measure customer satisfaction is the Appeals function. The Appeals function is the IRS’ dispute resolution program. The Commissioner has granted the Appeals function authority to consider and negotiate settlements of internal revenue controversies. Appeals responsibility includes, but is not limited to, the determination of liability for taxes, penalties, and interest.

Appeals gained increased jurisdiction under the IRS Restructuring and Reform Act of 1998 (RRA 98). RRA 98 § 3401 states that taxpayers are now entitled to a Due Process Appeal for most Collection enforcement actions (e.g., lien, levy, and seizure). These changes should dramatically increase the number and degree of direct taxpayer contacts with the Appeals function.

Each month, Appeals supplies a vendor with data for all cases closed during the month. The vendor selects a random sample of cases, stratified by district, and adds key variables for tracking. The vendor then forwards the sample file to a sub-contractor who mails questionnaires to the customers and processes the responses. A toll-free number has been established to answer any questions and to provide questionnaires in Spanish when requested.

Appeals has recently undergone a reorganization which places Appeals outside the IRS’ four new business units. However, in preparation for the transition, Appeals has already had some preliminary discussions with the vendor regarding using the current Survey results to compute baselines for the new Appeals organization.

Appeals has grouped its customers into four main market segments to align itself with the new IRS organizational structure: (1) Large and Mid-size Business, (2) Small Business/Self-Employed, (3) Tax Exempt and Government Entities, and (4) Wage and Investment.

In addition, the IRS is developing a new planning process that will provide support for its efforts to comply with the GPRA. Until this process is completed, the Commissioner designated the Deputy Chief Financial Officer for Strategic Planning and Budgeting as responsible for overseeing and coordinating the implementation of all GPRA-related activity for the IRS. This became effective December 1, 1999.

Results

The present results of the Appeals Customer Satisfaction Survey should not be used to satisfy the requirements of the GPRA, unless appropriately qualified. IRS executives have not established an adequate management process to ensure the Survey is conducted appropriately and that the data are reliable, valid, and verifiable as required for GPRA reporting.

The original purpose of the Appeals Customer Satisfaction Survey was not for GPRA reporting but to measure the overall trend of taxpayer satisfaction with the Appeals function’s service. While the Survey may be an effective marketing tool to gauge taxpayers’ satisfaction with the services provided by Appeals offices, a higher standard is required when these results are used for GPRA reporting purposes. The Survey results are being reported to the Congress for GPRA purposes, requiring that the data obtained from the Surveys be reliable, valid, and verifiable.

The Appeals Customer Satisfaction Survey Results Should Be Qualified if Used to Report Performance Measures for the Government Performance and Results Act of 1993

Controls over the automated case control system used to identify the Survey population for the Appeals Customer Satisfaction Survey were largely in place and did not contain material internal control weaknesses that would significantly invalidate the Survey population. However, there are limitations with the Appeals Customer Satisfaction Survey process, which increase the risk that the measure is not reliable.

Transmission of closed case listings is not protected and secured. This could result in lost, incomplete, or inaccurate data. It could also result in unauthorized access to the data.

In addition, only 44 percent of the taxpayers who were sent a survey form in 1998 responded to the Appeals Survey. Non-response is a cause of non-sampling error and increases the risk that the Survey results are not reliable and may not be representative of the Survey population.

Without improvements in the above conditions, the Appeals Customer Satisfaction Survey results may not be reliable and should be qualified if used to meet the requirements of the GPRA. Without reliable information, the IRS cannot provide a basis for comparing program results with the established performance goals and will not fully achieve the benefits of the GPRA intended by the Congress.

Cases are referred to Appeals from various IRS functions: Collection, Examination (including Office Audit), and IRS service centers. The cases are controlled on the Appeals Centralized Database System (ACDS), an inventory management system used to control and track cases, monitor statuses, and record important actions taken as cases move through the Appeals process. Internal controls over the ACDS are adequate to preclude significant instances of uncontrolled cases.

Internal controls are a series of actions and activities that occur throughout a function’s operations on an ongoing basis. They should be an integral part of each system that management uses to regulate and guide its operations. Management is responsible for establishing effective controls to provide reasonable assurance that the following objectives are being achieved: (1) effectiveness and efficiency of operations, (2) reliability of financial reporting, (3) compliance with applicable laws and regulations, and (4) safeguarding of assets. Proper controls over access to the inventory systems help ensure the accuracy of the data within the systems.

Access to the ACDS should be limited to those obtaining the proper approval to access the system. We obtained copies of the approval forms and compared them to ACDS users. There were no significant discrepancies.

Level of access to the ACDS is based on "Permissions" that control what actions a user can perform. ACDS Permissions are assigned to employees based on their job functions. Permissions prevent unauthorized changes to Appeals case inventory information. We obtained organization charts for the two offices visited and determined whether each employee’s ACDS Permissions were commensurate with their duties and assignments. There were no significant discrepancies.

We obtained from Appeals a download of the 1,158 cases opened on the ACDS the week of February 22, 1999. We also obtained a list of cases that were sent to the vendor to be included in the survey. This list contained cases sent to the vendor for a 13-month period, February 1999 through February 2000.

We selected a statistical sample of 317 cases from the 1,158 cases opened during that week and determined whether they were still open or were closed. We then determined if the cases had been included in the Survey by tracing the closed cases to the vendor list.

Of the 317 cases sampled, 57 were verified as open and 260 as closed. Of the 260, all but 10 (less than 4 percent) were included on the listings sent to the vendor. According to Appeals analysts, the 10 cases closed on the ACDS but not on the vendor list resulted from a timing issue. Some case closures are input into the ACDS after the closed cases are downloaded for the vendor. The number is currently not significant, and we do not believe the number of closed cases excluded from the vendor list for this reason would significantly affect the Survey results. Continued monitoring of this timing issue should ensure that Appeals does not exclude a significant number of taxpayers from the Survey, which could affect future Survey results.

Annual validations and monthly system validation reports are effective.

Two key controls over the ACDS are annual inventory validations and system validation reports. We reviewed the following reports in the Georgia and Rocky Mountain District Appeals offices:

• Nineteen Inventory Validations for Fiscal Years 1998, 1999, and 2000.
• Thirty-two ACDS reports for various months.

Inventory validations are used by Appeals to verify the data on the ACDS with each appeals officer’s inventory. Inventory validations should identify inaccurate case data and any uncontrolled cases.

We judgmentally selected a sample of inventory validations conducted in the two Appeals offices. The results did not reveal any instances of a significant number of Appeals cases not controlled on the ACDS. The two local offices followed proper procedures for the inventory validation and any discrepancies or problems identified were appropriately resolved.

The ACDS also includes numerous system-generated reports for validating data with the Examination function. These reports are an effective method of ensuring all appealed Examination cases (audited tax returns) are controlled, assigned, and tracked on the ACDS. In the two offices visited, the monthly validation reports were generated and worked effectively.

Unlike for Examination cases, there is no automated system to ensure all Collection and service center cases (non-Examination cases) are controlled on the ACDS. Neither of the offices visited did monthly validations to ensure all non-Examination cases were accounted for. However, the annual inventory validations in the two offices visited did not identify a significant number of uncontrolled cases.

In 1998, non-Examination cases made up 50 percent of the Appeals function’s total case inventory. Although our limited testing did not identify uncontrolled cases, the lack of a monthly inventory validation process for non-Examination cases increases the risk of lost or misplaced cases.

Cases closed on the ACDS each month are downloaded from the ACDS and transmitted to the vendor. The data are transferred from the IRS to the vendor using a file transfer protocol. An Appeals computer specialist transmits the data and telephones the vendor to advise them of the transmittal. The vendor picks up the data off the file site immediately.

However, the data are not encrypted. File transfers incur risks, such as lost or incomplete data, inaccurate transmission, misdirected transmission, or inappropriate use of the data. To mitigate these risks, data should be encrypted. IRS requires that encryption be used for transferring sensitive but unclassified information between IRS facilities when not using the Treasury network.

The response rate for the Appeals Customer Satisfaction Survey for 1998 was 44 percent. A low response rate results in a greater number of non-respondents for whom the IRS will be projecting results. The 44 percent response rate to the Appeals Survey means that the IRS is projecting the results of these respondents to the 56 percent of the taxpayers who did not respond.

The IRS Manual requires that the response rate for all surveys conducted by the IRS should be at least 70 percent. Further, because non-response is a cause of non-sampling error, all personnel conducting surveys should use follow-up letters to try to achieve at least a 70 percent response rate.

To help increase the response rate, the vendor is sending a postcard reminder and a "second-wave" letter to non-respondents. In addition, a toll-free number has been established for the purposes of answering respondents’ questions and sending questionnaires in Spanish to those who request them.

Until Appeals achieves a higher response rate, the IRS should qualify the Survey results if they are used to report performance measures for the GPRA.

We recommend that the Deputy Chief Financial Officer for Strategic Planning and Budgeting, in conjunction with the National Chief Appeals and the Director, Office of Program Evaluation and Risk Analysis, improve the process to oversee the Appeals Customer Satisfaction Survey by:

1. Ensuring the data transmitted to the vendor are protected through encryption.
2. Clearly disclosing the limitations of the Survey results until the Survey response rate reaches a level acceptable for GPRA purposes.

Management’s Response: Appeals management agreed with our recommendation that the data transmitted to the vendor be encrypted and that the limitations of the Survey be disclosed for purposes of the GPRA. However, Appeals management stated that the National Chief Appeals, working in conjunction with the Director, Office of Program Evaluation and Risk Analysis, is responsible for the Appeals Customer Satisfaction Survey, not the Deputy Chief Financial Officer, Strategic Planning and Budgeting.

Conclusion

Presently, the Appeals Customer Satisfaction Survey should be qualified if used to report results of performance measures for the GPRA. There are limitations with the Appeals Customer Satisfaction Survey process that result in an increased risk that the data are not reliable to report measures for the GPRA. Until the data transmitted to the vendor are adequately protected and a higher response rate is achieved, the IRS should clearly disclose the limitations of the Survey’s data and the reliability of the results if used for GPRA purposes in reporting the level of satisfaction customers have with the Appeals function’s services.

Detailed Objective, Scope, and Methodology

The overall objective of this review was to evaluate the reliability of the information used to measure customer satisfaction with the Internal Revenue Service (IRS) Appeals process. We performed the following work:

1. Determined if the Appeals Customer Satisfaction Survey was designed to satisfy Government Performance and Results Act of 1993 (GPRA) requirements.
1. Identified the primary purpose and objectives of the Survey to determine if the results will meet the needs of reporting GPRA performance measures to the Congress.
2. Determined how the sampling criteria were established and whether the sampling components ensure valid and reliable results.
2. Determined whether all Appeals closed cases have an opportunity to be included in the Survey and are sent to the vendor.
1. Determined if taxpayer information for all closed Appeals cases was transmitted to the vendor during the month the cases were closed.
1. From Appeals, obtained a listing of taxpayers whose cases Appeals opened during the week of February 22, 1999. Note: We judgmentally selected a week without holidays to ensure we obtained a full week’s worth of data (closed cases) and selected a date that should result in a sufficient quantity of closed cases.
1. Selected a statistically valid sample of 317 from a total population of 1,158 Appeals cases opened the week of February 22, 1999. We used a sampling application designed in Microsoft Access to calculate the sample size and randomly select the cases for review. We based our sample size on a confidence level of 95 percent, an expected error rate of 5 percent, and a precision level of 2 percent.
2. Researched the Appeals Centralized Database System (ACDS) to determine the status and disposition of the sample of the cases opened during the week of February 22, 1999. Compared the open cases to the Integrated Data Retrieval System to evaluate the accuracy of the ACDS data. Compared closed cases to the lists of closed cases provided the vendor for selection for the Appeals Customer Satisfaction Survey.
2. Reviewed the lists of cases furnished to the vendor for representation from all Appeals offices.
2. Determined if the data had been sent to the vendor every month.
3. In the selected Appeals offices (Georgia District and Rocky Mountain District Appeals Offices), evaluated controls over the ACDS.
1. Obtained a list of Appeals employees and evaluated their permissions and access to the ACDS.
2. Obtained and reviewed a judgmental sample of ACDS monthly and annual reports used to verify the system’s data and evaluated the nature of problems identified to determine if they could affect the data given to the vendor.
4. Determined whether results of the Survey will be transportable to the new organizational structure.

Major Contributors to This Report

Maurice S. Moody, Associate Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

Stanley C. Rinehart, Director

John R. Wright, Director

Augusta R. Cook, Audit Manager

Kenneth Carlson, Senior Auditor

Sharon Shepherd, Senior Auditor

Catherine Cloudt, Auditor

Tracy Harper, Auditor

David Lowe, Auditor

Report Distribution List

Deputy Commissioner Operations C:DO

Assistant Deputy Commissioner Operations C:DO

Chief, Appeals C:AP

National Taxpayer Advocate C:TA

Chief, Management and Finance M

Chief Operations Officer OP

Chief Financial Officer C:DO:CFO

Deputy Chief Financial Officer for Strategic Planning and Budgeting CFO:SPB

Director, Office of Program Evaluation and Risk Analysis M:O

Director, Legislative Affairs CL:LA

Office of Chief Counsel CC

Organizational Performance Management Executive C:DO:OPME

Chief, Office of Management Controls M:CFO:A:M

Audit Liaisons:

Management’s Response

The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.