Executive Summary

Objective and Scope

Background

Results

Conclusion

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Executive Summary

This audit was performed as part of the Treasury Inspector General for Tax Administration’s overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The overall objective of our review was to assess the validity of the information used to measure customer satisfaction for the Examination Division.

The GPRA requires federal agencies to establish standards for measuring their performance and effectiveness. The law requires executive agencies to prepare multi-year strategic plans, annual performance plans, and performance reports on prior year accomplishments. The first annual performance reports must be provided to the Congress by March 31, 2000. The Congress will use the GPRA measurement results to help evaluate the IRS’ budget appropriation. Therefore, it is essential that the IRS accurately measure its success in meeting the performance goals.

The IRS has prepared a strategic plan and an annual plan establishing goals for the agency. One of the IRS’ three strategic goals is to provide top quality service to each taxpayer. The IRS is measuring its success in achieving this goal through surveys conducted by an outside vendor. Taxpayers who receive specific kinds of services are being asked to complete a survey to rate the service they received. These survey results are summarized and used to evaluate the overall satisfaction with the IRS’ service.

IRS management has not established an effective process to ensure that the survey is conducted appropriately to measure the level of satisfaction customers receive from interactions with all Examination Division program areas. As a result, the IRS needs to qualify any of the data from the Examination Division Customer Satisfaction Surveys that will be presented in the Fiscal Year (FY) 1999 Annual Performance Report.

We identified limitations with the customer satisfaction survey process that result in an increased risk that the measure is not reliable. We determined that the data are not fully representative of the universe of examination cases. The survey population is based solely on audit closures of individual taxpayers. Audit closures involving corporate, estate, excise, and gift tax returns are not included in the survey population.

Another limitation is that the customer satisfaction measure includes only those contacts with taxpayers that occur as part of an audit. For example, taxpayers contacted as part of working innocent spouse determinations will not be surveyed unless the case involves an audit. The volume of innocent spouse cases increased to the point where the Examination Division had approximately 46,000 cases in its inventory as of December 31, 1999.

In addition, only 34 percent of the taxpayers to whom a survey was sent responded. The IRS requires at least a 70 percent response rate to the surveys. The low response rate of 34 percent means the IRS is using the opinion of those who responded to the survey and is assuming that the non-respondents have the same opinion.

We did find that the controls over the automated case control system, used to identify the survey population for the Examination Division’s field and office functions Customer Satisfaction Survey, largely were in place. The controls were sufficient to preclude the existence of a significant number of uncontrolled examinations, which would have an adverse effect on the validity of the survey population.

The Assistant Commissioner (Examination) should clearly state the limitations of the sampling process and the resultant effect on the Examination Division’s ability to report customer satisfaction in the FY 1999 Annual Performance Report. The Assistant Commissioner should also consider actions to increase the response rate to the surveys in order to decrease the risk associated with projecting to those taxpayers who do not respond to the survey. One possible approach would be to conduct a telephone survey of the taxpayers not responding to the mail survey.

Management’s Response: Management’s response was due on May 5, 2000. IRS management requested, and was given an extension of 5 workdays; however, as of May 12, 2000, IRS management had not responded to this draft report.

Objective and Scope

This audit was performed as part of the Treasury Inspector General for Tax Administration’s overall strategy to assess the reliability of the Internal Revenue Service’s (IRS) customer satisfaction performance measures as they relate to the Government Performance and Results Act of 1993 (GPRA). The overall objective of the review was to assess the validity of the customer satisfaction measure for the Examination Division.

The IRS measures customer satisfaction by asking customers to participate in customer satisfaction surveys. The IRS uses information from the surveys to make goals, measure progress, and report to the Congress as required by the GPRA. We performed this audit from October 1999 to January 2000 in accordance with Government Auditing Standards.

We conducted this review in the National Office, the Kansas-Missouri and North Texas District Offices, and the Austin and Ogden Service Centers. We held discussions with the IRS’ Office of Performance Evaluation and Risk Analysis (OPERA) in the National Office to address issues that may affect the validity of the customer satisfaction measure. We also assessed the controls over the Examination Division’s inventory systems in the two district offices to determine if the existing controls prevented uncontrolled examinations. Uncontrolled examinations would not be included in the survey population and could, therefore, impact the validity of the customer satisfaction measure. We also conducted tests in the two service centers to evaluate the controls over the computer program that selects the survey population and the shipment of the resulting data to the vendor.

The IRS uses an automated case control system to identify the survey population for the Examination Division’s Customer Satisfaction Survey. As such, the procedures for the survey process do not require the employees in the Examination Division to either identify the survey population or conduct the survey. Because of this, our review in the districts was limited to a review of the controls over the automated case control system.

Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

Background

The GPRA was enacted to improve the quality and delivery of government services. The Act holds federal agencies accountable for program results by emphasizing goal setting, customer satisfaction, and results measurement. Agencies were required to submit, by September 30, 1997, strategic plans covering a period of not less than five years forward from the fiscal year in which it was submitted. Strategic plans are to be updated at least every three years. The Act also requires each agency to prepare an annual performance plan covering each program activity. Finally, federal agencies are to submit a report on program performance for the previous fiscal year to the Office of Management and Budget and the Congress no later than March 31, 2000, and no later than March 31, of each succeeding year.

The IRS has established three strategic goals that will be measured and the results reported to the Congress. One of these goals is service to each taxpayer (Customer Satisfaction). The IRS contracted with a vendor to help conduct 11 customer satisfaction surveys, covering areas such as Collection, Examination, Taxpayer Service, and Appeals.

One of the customer satisfaction surveys involves the Examination Division’s field and office audit functions (Examination). All individuals whose income tax returns were examined through an office interview or a field examination with the IRS and whose cases were then closed make up the survey population. The Information Systems Division has designed a computer program that is run against the Audit Information Management System (AIMS) at each of the 10 service centers to select the survey population. The AIMS is a computerized system used to secure returns, maintain inventory control of examinations, record examination results, and provide IRS management with the statistical reports required under examination and compliance programs. A computer tape with the AIMS information is created monthly and shipped to the vendor.

The vendor forwards the information to a sub-contractor who administers the survey process. The sub-contractor mails the questionnaires, tabulates the results, and follows up with a second questionnaire to taxpayers who did not respond initially. Summary results are furnished to the OPERA. The IRS uses the summary information to both develop the customer satisfaction measure, which it will report to the Congress as part of the IRS’ budget submission, and to find ways to improve service to its customers.

Results

The Examination Division’s Customer Satisfaction Survey process contains certain limitations that should be disclosed when reporting the customer satisfaction measure. The survey population includes only individual taxpayers contacted as part of an examination. Business taxpayers and taxpayers contacted outside of the examination process are excluded from the survey population.

In addition, the IRS should also disclose the current response rates to the customer satisfaction surveys. A low response rate increases the risk that the survey results may not be representative of the survey population because the IRS is using the opinion of those who responded to the survey and is assuming that the non-respondents have the same opinion.

IRS management has not established an effective process to ensure that the survey is conducted appropriately to measure the level of satisfaction customers receive from interactions with all Examination Division program areas. The IRS could be reporting misleading results to the Congress if it does not fully disclose the limitations of the Examination Division customer satisfaction measure.

The Examination Division’s Customer Satisfaction Survey Process Contains Limitations That Could Affect the Validity of the Measure

The Department of the Treasury requires that information used to measure customer satisfaction for a particular work unit be gathered from a statistically valid sample of customers served by that operating unit. The Examination Division’s Customer Satisfaction Survey population represents only 80 percent of the audits conducted by the Division. The criterion used to select the survey population includes only individual taxpayers. It does not include audit closures involving areas such as corporate, estate, excise, and gift tax returns in the survey population. The only corporate closures included in the survey population are audits of shareholders that occur as a result of a corporate audit.

In addition, the Examination Division’s customer satisfaction measure measures only those contacts with taxpayers that occur as part of an audit. The measure does not include contacts made with taxpayers outside of the audit process. An example of contacts made outside of the audit process are those involving the Examination Division’s processing of innocent spouse claims for relief. The volume of innocent spouse claims has recently increased due to provisions in the Internal Revenue Service Restructuring and Reform Act of 1998. As a result of the increase in volume, many of the cases were sent to the Examination Division to be worked. As of December 31, 1999, the Examination Division had approximately 46,000 innocent spouse cases in its inventory. These cases will be included in the Customer Satisfaction Survey only if they involved an audit.

We also determined that the IRS should disclose the current response rate to the customer satisfaction surveys. A direct correlation exists between the reliability of the survey results and the survey response rate. A low response rate results in a greater number of non-respondents that the IRS will need to project to. The 34 percent response rate to the Examination Survey for 1998 means that the IRS is projecting the results of these respondents to the 66 percent of the taxpayers who did not respond. The vendor conducting the surveys agreed that the non-respondents’ attitudes are often different from those of respondents and the low response rate should be considered when reporting the survey results. The Examination Division should consider additional attempts to contact the non-respondents to attain the 70 percent response rate required by the IRS.

The Examination Returns Control System (ERCS) is an inventory management system in addition to the AIMS that is used by the Examination Division. Our analysis of the controls around the ERCS did not identify any instances of a large number of uncontrolled cases.

Two key controls over the AIMS are inventory validations and operational reviews. While not required by the Internal Revenue Manual, operational reviews can be used to ensure the validity of the AIMS.

We reviewed the inventory validations conducted at the district level. Our analysis of the results of a sample of 32 inventory validations conducted in the Kansas-Missouri and North Texas Districts did not reveal any instances of significant numbers of examinations not controlled on the AIMS inventory system. Inventory validations are used by the Examination Division to verify the data on the AIMS with each examiner’s inventory. Inventory validations should identify any uncontrolled examinations.

We reviewed a sample of 27 operational reviews conducted by the Examination Division in the 2 districts. The operational reviews that we analyzed did not identify a large number of uncontrolled examinations. Operational reviews are used by the Examination Division to ensure that the Examination groups use the inventory reports produced by the AIMS. These reports help the Examination Division ensure that the data on the system are accurate and the returns are properly accounted for and controlled.

We also interviewed the Planning and Technical Support Branch (PTSB) Chiefs in the Kansas-Missouri and North Texas Districts to determine if, as a result of normal operations, they conduct examinations or contact taxpayers without controlling the cases on the AIMS. In both districts, any time a taxpayer’s tax return is requested for the branch, it is controlled on the AIMS. In addition, PTSB staffs in both districts do not contact taxpayers as part of their normal duties. Both Branch Chiefs told us that there was almost no chance of a significant number of uncontrolled examinations existing in their branches.

Proper controls over access to the inventory systems help ensure the accuracy of the data within the systems. We selected a sample of employees with access to the AIMS or ERCS inventory systems in the Kansas-Missouri and North Texas Districts to determine if the employees’ levels of access were commensurate with their job assignments. We also selected a sample of employees with access to the systems to determine whether their access was properly approved. We found that the controls over the AIMS and ERCS inventory systems adequately limit access to the systems.

During our review, we also obtained from the Austin and Ogden Service Centers, copies of the data that are sent to the vendor. We verified that the data provided to the vendor matched the selection criteria. In addition, in order to verify that data from closed cases were included on the AIMS and forwarded to the vendor, we selected a sample of 44 closed examination files in the North Texas District. We were able to verify that the data from each of the cases were sent to the vendor and that the data provided to the vendor matched the case files.

We also performed an analysis of the Functional Specification Package (FSP) for the computer program that is run at each of the service centers. The FSP communicates the functional requirements of the system. We determined that the logic used in the programming matches the Customer Satisfaction Survey selection criteria.

Based on our discussions with computer personnel at the Austin and Ogden Service Centers and review of documentation furnished by them, we concluded that the service centers’ controls over the production and shipment of the customer satisfaction tapes to the vendor are adequate. From inspection of Magnetic Media Shipment Control Documents (Form 9255-AWMS), we determined that a tape was sent to and acknowledged by the vendor each month for the 12-month period ending November 1999.

Although the tapes are created, shipped, and acknowledged as received by the vendor each month, we found that 12 out of 250 (4.8 percent) workdays a year are not included in the tapes sent to the vendor. We conducted an analysis of when the tapes were created to determine if all workdays in the year were captured on a tape. We found that 3 consecutive days were excluded in 4 separate monthly tapes. However, we believe that these missing days would not have an adverse effect on the validity of the survey population.

The current Examination Division Customer Satisfaction Survey process gives the IRS the ability to use the survey results to report on one of its four future business units. The IRS can use the Examination Division Customer Satisfaction Survey to measure taxpayers’ satisfaction in the Wage and Investment Business Unit. The Customer Satisfaction Survey population represents the majority of Small Business/Self-employed Business Unit taxpayers; however, it does not include corporate filers. This exclusion prevents the IRS from using the current survey to report on the customer satisfaction for this business unit.

The Large and Mid-size Business Unit also cannot measure taxpayers’ satisfaction through the current Examination Division Customer Satisfaction Survey because the survey population includes only individual taxpayers. The IRS initially plans to use the existing Employee Plans and Exempt Organization Customer Satisfaction Survey for the Tax Exempt and Government Entities business unit. We have not yet reviewed this survey but expect to at a later date.

The Assistant Commissioner (Examination) should establish an effective process to ensure that the survey is conducted appropriately to measure the level of satisfaction customers receive from interactions with all Examination Division program areas. The Assistant Commissioner (Examination) should also oversee the efforts of the Examination Division and coordinate with the OPERA to implement the following recommendations. These actions should improve the accuracy of the Examination Division’s Customer Satisfaction Survey results.

1. Include, in detail, in the Fiscal Year (FY) 1999 Annual Performance Report the limitations of the sampling procedures and the plans and timelines to correct the issues. The data from the current reports should not be used without a very clear explanation of its limitations, to include the following:

• Report that the survey population is based solely on the audit closures of individual taxpayers. Audit closures involving corporate, estate, excise, and gift tax returns are not included in the survey population.
• Report that the measure does not include contacts that the Examination Division had with individuals that did not result in an audit closure.

1. Develop methods to increase the survey response rate to the IRS’ mandated 70 percent. Disclose in the FY 1999 Annual Performance Report that the response rate is only 34 percent and the efforts being made to meet the IRS requirements. One possible approach to accomplishing this would be to contract with the vendor to conduct a telephone survey of taxpayers not responding to the mail survey.

Management’s Response: Management’s response was due on May 5, 2000. IRS management requested, and was given an extension of 5 workdays; however, as of May 12, 2000, IRS management had not responded to this draft report.

Conclusion

The IRS needs to make full disclosure of its efforts to date to establish a valid measurement of the Examination Division’s customer satisfaction, the shortcomings of the current process, and the dates by which it will be able to accurately report on customer satisfaction. This qualification is necessary because the data do not include all customer contacts and the response rate is less than the 70 percent required by the IRS.

Detailed Objective, Scope, and Methodology

The overall objective of the review was to assess the validity and accuracy of the information used to measure customer satisfaction for the Examination Division in the Internal Revenue Service (IRS). To accomplish this objective, we conducted the following tests:

1. Determined if the Audit Information Management System (AIMS) is an accurate and valid source of information from which to select a sample of taxpayers to be surveyed.

1. Obtained a copy of the file containing the AIMS information that was sent to the vendor.

1. Selected 44 closed examination cases from the Examination Support and Processing Branch (ESPB) in the North Texas District and compared the information on the returns with the information on the AIMS file that the IRS sent to the vendor.
2. Reviewed the AIMS file and determined whether the data in the fields were within logical boundaries and agreed with the case selection criteria.
3. Analyzed the Functional Specification Package for the AIMS file and determined whether the programming logic matches the survey extract criteria.

1. Determined how representative the survey population is of the Examination Division’s workload (both the numbers of examinations and Full Time Equivalents [FTEs] used). Only the cases of examined individual taxpayers were included in the sample population. We analyzed an AIMS report to identify the percentage of examination closures with a Masterfile Tax code of 30 and the corresponding investment of FTEs by the Examination Division.

1. Evaluated the internal controls over the Examination Division’s inventory systems. Our review of the Examination Division’s inventory controls was limited to determining if the controls identified a significant number of uncontrolled examinations. As such, we reviewed the controls over the AIMS and the Examination Returns Control System (ERCS) to determine if a significant number of examinations were conducted and not controlled on either of the inventory systems.

1. Determined whether inventory controls are sufficient to ensure the accuracy of the AIMS.

1. Reviewed a judgmental sample of 32 inventory validations out of the 127 validations that were completed in the North Texas and Kansas-Missouri Districts.
2. Reviewed a judgmental sample of 27 operational reviews out of a total of 38 reviews conducted by the AIMS coordinators in the North Texas and Kansas-Missouri Districts.
3. Evaluated the controls over the ERCS in the North Texas and Kansas-Missouri Districts.

1. Evaluated how the ERCS is used in the Examination groups to both control examinations and validate the AIMS.
2. Evaluated the process that the ESPBs used to ensure that they receive all of the cases from the Examination groups.

1. Interviewed the Chief Planning and Technical Support Branch in the North Texas and Kansas-Missouri Districts regarding the controls over projects within their branches.

1. Determined if the input of information to the AIMS and ERCS is properly controlled. Evaluated the controls over access to the AIMS and ERCS in the North Texas and Kansas-Missouri Districts.

1. Obtained a judgmental sample of 60 of 451 employees with access to the ERCS and all Examination employees with access to the AIMS for both districts and determined if their capabilities are commensurate with their job descriptions.
2. Reviewed a judgmental sample of 30 Automated Information System (AIS) User Registration/Change Requests (Form 5081) for both the AIMS and ERCS for both districts out of the 1,098 employees with access to the systems to determine whether they have been properly approved.

1. Evaluated the IRS’ plan to migrate the data to the appropriate business unit.

1. Interviewed Office of Program Evaluation and Risk Analysis personnel and determined if the vendor can readily produce survey results along the new business units. Determined if there has been any consideration to restructuring along those lines.
2. Determined if the procedures for conducting the survey using the AIMS as the source for case selection will allow identification by business unit.

1. Evaluated the controls in place at the Austin and Ogden Service Centers for the processing of the AIMS tapes sent to the vendor.

1. Interviewed Service Center personnel and identified the process used to schedule, create, and ship AIMS tapes to the vendor.
2. Analyzed Magnetic Media Shipment Control Documents (Forms 9255-AWMS) for the period December 1998 through November 1999 to determine if the tapes were shipped by the IRS and receipt acknowledged by the vendor each month.
3. Analyzed the number of workdays that were included in the AIMS tapes sent to the vendor that were created during the 12-month period ending November 1999.

Major Contributors to This Report

Maurice S. Moody, Associate Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

Stanley Rinehart, Director

Kevin Riley, Audit Manager

David Robben, Senior Auditor

David Cox, Senior Auditor

Steve Holmes, Auditor

Gene Luevano, Auditor

William Thompson, Auditor

Report Distribution List

Deputy Commissioner Operations C:DO

Chief Operations Officer OP

Assistant Commissioner (Examination) OP:EX

Director, Strategic Planning and Budgeting M:SPB

National Director for Legislative Affairs CL:LA

Chief, Office of Management Controls M:CFO:A:M

Director, Office of Program Evaluation and Risk Analysis M:O

Office of the National Taxpayer Advocate C:TA

Office of Chief Counsel CC

Audit Liaisons: