TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

Improved Internal Controls and Contact Recording Are Needed to Ensure the Accuracy and Reliability of the Taxpayer Assistance Centers Quality Measurement System

 

 

 

December 2005

 

Reference Number:  2006-40-022

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

Phone Number  |  202-927-7037

Email Address  |  Bonnie.Heald@tigta.treas.gov

Web Site           |  http://www.tigta.gov

 

 

December 6, 2005

 

 

MEMORANDUM FOR COMMISSIONER, WAGE AND INVESTMENT DIVISION

                                        

FROM:                            Michael R. Phillips /s/ Michael R. Phillips

                                         Deputy Inspector General for Audit

 

SUBJECT:                    Final Audit Report – Improved Internal Controls and Contact Recording Are Needed to Ensure the Accuracy and Reliability of the Taxpayer Assistance Centers Quality Measurement System (Audit # 200440045)

 

This report presents the results of our review to determine whether the Embedded Quality measurement system is efficiently and effectively improving business results for the Internal Revenue Service (IRS) Wage and Investment Division Field Assistance Office.

Synopsis

Each year, millions of taxpayers visit IRS Taxpayer Assistance Centers (TAC) seeking assistance in understanding the tax law and meeting their tax obligations.  According to the IRS, the TACs served approximately 7.3 million taxpayers during Fiscal Year 2004.  The Wage and Investment Division Field Assistance Office has overall responsibility for over 400 TACs and 2,200 IRS employees (called assistors) that provide taxpayers with face-to-face assistance.

To measure its customer service, the IRS uses a quality measurement system called the Embedded Quality, which links employee performance to organizational results related to the quality of customer service.  Beginning in Fiscal Year 2004, the Field Assistance Office began implementing the Embedded Quality Business Integration (EQBI).  The EQBI is an innovative approach to quality control for face-to-face interactions between assistors and taxpayers.  No other Federal Government agencies we contacted have developed a comprehensive quality measurement system that measures face‑to‑face assistance like the one the IRS is implementing. 

The new EQBI integrates an IRS-wide, customer-focused, standardized measurement system with centralized data collection tools and current technology, such as Contact Recording and Queuing Management.[1]  It will help address many concerns raised by the Office of Management and Budget, the Government Accountability Office, and the Treasury Inspector General for Tax Administration on the accuracy of tax law answers and the need for a quality measurement system with which to set goals and measure progress toward achieving those goals.

The Field Assistance Office began implementing the Embedded Quality component of the EQBI in October 2003 and began collecting data in April 2004.  Currently, the Embedded Quality requires TAC group managers to physically observe a statistical sample of assistors’ interactions with taxpayers that visit the TACs for help with a tax law question, an account issue, or the preparation of a tax return.  Group managers document the results of their observational reviews on a Data Collection Instrument that is entered into an electronic database, the Embedded Quality Review System (EQRS).[2] 

However, the EQRS data are not representative of the population of assistors, and internal controls are not sufficient to ensure the reliability of data.  For Fiscal Year 2004, group managers:

·         Did not complete observational reviews for 168 (11 percent) of 1,550 assistors required to be included in the sampling plan (or the statistical sample).

·         Included from 9 to 26 observational reviews in the sampling plan for 455 (29 percent) of 1,550 assistors that should not have been included.  These observations were in addition to those required by the sampling plan.

In addition, managerial observational reviews introduce bias, inhibiting the accurate assessment of employee performance.  To monitor the performance of assistors, managers currently must be physically present when assistors help taxpayers.  This physical presence establishes an artificial situation for both the employee and the taxpayer and inhibits the accurate assessment of performance in day-to-day contacts.

The EQRS data are also inconsistent and contain errors.  We tested a stratified statistical sample of 310 of the 11,839 Data Collection Instruments entered into the EQRS from April 19 through September 30, 2004, and found:

·         Thirty-eight of 228 Data Collection Instruments tested contained errors that affected Customer Accuracy.  Projecting the mean error rate to the total population, 2,500 (21.12 percent) of 11,839 Data Collection Instruments in the Field Assistance Office’s EQRS contain errors that affect Customer Accuracy.[3]

·         Most (306 of 310) Data Collection Instruments contained errors that affected Regulatory, Procedural, Professionalism, and/or Timeliness quality measures.  Projecting the mean error rate to the total population, 11,725 (99.04 percent) of 11,839 Data Collection Instruments in the EQRS contain errors that affect these 4 quality measures.[4]

The Embedded Quality is a new process for the Field Assistance Office.  It is in the first years of implementation and it is reasonable that there would be a considerable learning curve.  Specifically, sufficient procedures and internal controls were not developed to ensure the sampling plan was followed and only required observational reviews were included in the statistical sample.  In addition, insufficient resources were allocated to the Quality Assurance function staff.  Procedures and guidelines had not been developed to validate and monitor the results, and Quality Assurance function staffing was not sufficient to evaluate risks and the effectiveness of internal controls. 

The Field Assistance Office is moving in the right direction with the implementation of the Embedded Quality.  Current projections call for Contact Recording to be fully implemented in Fiscal Year 2008.  We believe the Embedded Quality with Contact Recording, when appropriately working and managed, can provide a consistent methodology for all managers to evaluate performance, establish baselines, and identify root causes of defects in employee interactions with taxpayers. 

Recommendations

The Commissioner, Wage and Investment Division, should limit the use of the Embedded Quality data until the data are validated as statistically representative of the population of assistors.  In addition, the Commissioner should establish, document, and implement a system of internal controls to ensure the sampling plan is followed and the results are statistically representative of the population of assistors.  This includes establishing a centralized process to monitor and ensure group managers are following the sampling plan methodology and only employees required to be observed are included in the sample.  Group managers should receive training on the importance of the Data Collection Instrument, not only to document the assistor/taxpayer interaction but also to substantiate the rating for use in subsequent reviews, reconciliations, and validations.  Finally, a system of internal controls should be implemented and documented to ensure the EQRS data are valid and reliable and have been checked and tested for significant errors.  This includes establishing written guidelines that clearly establish roles and responsibilities, a centralized process to conduct periodic statistical reviews and reconciliations, and the means by which to provide documentation on the effectiveness of the internal controls. 

Response

IRS management agreed with our recommendations and immediately took corrective actions.  The IRS made a decision to not use EQRS data as a quality measure for Fiscal Years 2005 and 2006.  The Fiscal Year 2006 goal will be to baseline EQRS quality scores and improve the integrity of the EQRS data. 

The IRS established an internal control system with guidelines that include formally established Embedded Quality Roles and Responsibilities, sampling plan requirements for the number of reviews per employee and time periods for input and sharing of review results, a requirement for Territory managers to perform weekly monitoring of the group managers, and a requirement for Area Offices to submit monthly variance reports listing assistors that did not require monthly reviews and why.  The guidelines also include a requirement for the headquarters quality staff to validate group manager coding and to share results with field offices, along with a standardized remarks section for Data Collection Instruments to permit verification by a third party and to permit Territory managers and quality staff to use them to provide feedback for managers. 

Lastly, the IRS delivered additional EQRS training and created a revised job aid to clarify attribute definitions, communicate EQRS changes, and reemphasize the importance of writing substantive remarks and improvement strategies.  Management’s complete response to the draft report is included as Appendix VII.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Scott A. Macfarlane, Acting Assistant Inspector General for Audit (Wage and Investment Income Programs), at (925) 210-7027.

 

 

Table of Contents

 

Background

Results of Review

The Field Assistance Office Is Implementing an Innovative Quality Measurement System

Embedded Quality Review System Data May Not Be Representative of the Population of Assistors

Recommendations 1 and 2:

Internal Controls Are Not Sufficient to Ensure the Reliability of the Embedded Quality Review System Data

Recommendations 3 and 4:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measures

Appendix V – The Embedded Quality Measurement System With Contact Recording

Appendix VI – Embedded Quality and Attributes

Appendix VII – Management’s Response to the Draft Report

 

 

Background

 

The Internal Revenue Service (IRS) estimated that it would take individual taxpayers from 3 hours and 46 minutes to prepare the simplest Tax Year 2004 tax return to over 27 hours to complete a more complex tax return with schedules.[5]  Mistakes and misinformation can easily contribute to noncompliance.  The IRS recognizes that providing quality customer service is the first step in achieving taxpayer compliance.

The IRS provides taxpayers the option of obtaining face‑to‑face assistance at over 400 Taxpayer Assistance Centers (TAC).  The IRS suggests that taxpayers visit the TACs when they have complex tax issues, need to resolve tax problems relating to their tax accounts, have questions about how the tax law applies to their individual income tax returns, or feel more comfortable talking with someone in person.  Approximately 2,200 IRS employees (called assistors) work in the TACs providing taxpayers personal, face‑to‑face assistance with tax matters that includes interpreting tax laws and regulations, preparing certain individual tax returns, resolving inquiries on taxpayer accounts, accepting payments, and providing various other services designed to minimize the burden on taxpayers in satisfying their tax obligations. 

Figure 1:  Percentages of Various Services Provided at Taxpayer Assistance Centers

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

According to the IRS, the TACs served approximately 7.3 million taxpayers during Fiscal Year 2004.  Figure 1 shows a breakdown of the types of services provided by assistors in the TACs during Fiscal Year 2004.[6]  

The Government Performance and Results Act of 1993 was established to improve the confidence of the American people in the capability of the Federal Government by holding Federal Government agencies accountable for achieving program results.[7]  This involved the initiation of program performance reform by setting program goals, measuring program performance against those goals, and reporting publicly on the progress. 

The Office of Management and Budget recently rated the IRS’ taxpayer service as part of the Fiscal Year 2006 Budget process, citing the program’s purpose is to reduce taxpayer burden by providing professional and courteous service to customers.  Included in the program rating were face-to-face assistance, toll‑free telephone assistance, correspondence, and the IRS web site, IRS.gov.  To complete the assessment, the Office of Management and Budget used the Program Assessment Rating Tool, which is a systematic method of assessing program performance across the Federal Government and a diagnostic tool with the main objective to improve performance in agency programs and link performance to budget decisions.  The Office of Management and Budget gave taxpayer service an adequate rating, stating that the IRS continues to have trouble with the accuracy of answers, needs long-term goals, and needs to improve its ability to determine the costs of its taxpayer service activities.

To measure its customer service, the IRS uses a quality measurement system called the Embedded Quality, which links employee performance to organizational results related to the quality of customer service.  Beginning in Fiscal Year 2004, the Field Assistance Office, the IRS office responsible for overseeing the TACs, began implementing the Embedded Quality Business Integration (EQBI).  The EQBI includes:

·         Embedded Quality that uses standardized review criteria to make contact evaluations easier and more consistent by linking criteria to Critical Job Elements and business measures. 

·         Contact Recording that captures the audio portion of the employee/customer interaction, synchronized with computer screen activity, for replay and quality review.

·         Queuing Management (Q-MATIC) that efficiently directs taxpayers to the right employee, based on tax issue.  Currently, Queuing Management is available in most TACs and will be available in all TACs by the end of Fiscal Year 2006.

·         Electronic Performance Based Individual Training that identifies employee training needs and delivers targeted training.  This portion of the EQBI is in its early stages and has not been implemented.

Figure 2:  The EQBI

Figure 2 was removed due to its size.  To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

See Appendix V for a flowchart of the Embedded Quality measurement system.

The Field Assistance Office began implementing the Embedded Quality component of the EQBI in October 2003 and began collecting data in April 2004.  The Embedded Quality requires group managers to observe assistors’ contacts with taxpayers and measure performance against predetermined standards.  Figure 3 shows the flow of the Embedded Quality process without Contact Recording.

Figure 3:  The Embedded Quality Process

Figure 3 was removed due to its size.  To see Figure 3, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

To accomplish this, group managers are to observe a statistical sample of assistors’ interactions with taxpayers that visit the TACs for help with a tax law question, an account issue, or the preparation of a tax return.  The three types of assistance are stratified into Embedded Quality product lines.  The Embedded Quality sampling plan is designed to provide statistically valid results.  Since managers cannot observe 100 percent of assistor interactions with taxpayers, a statistical sample allows for a reliable estimate, with a certain mathematical degree of confidence, that the Field Assistance Office can draw conclusions from based on observing only a portion of the total population of interactions between assistors and taxpayers.

Managers document the results of their observational reviews on a Data Collection Instrument, which is a checklist designed to assist group managers in rating and documenting interactions.  It is used to both provide feedback to the assistors and assist the group managers when entering the data into the Embedded Quality database known as the Embedded Quality Review System (EQRS). 

The Data Collection Instrument contains over 100 evaluative attributes[8] mapped to assistors’ Critical Job Elements and divided into 5 quality measures:

·         Customer Accuracy giving the taxpayer the correct answer with the correct resolution.

·         Regulatory Accuracy adhering to statutory/regulatory process requirements when making determinations on taxpayer accounts.

  • Procedural Accuracy adhering to nonstatutory/nonregulatory internal process requirements.
  • Professionalism promoting a positive image of the IRS by using effective communication techniques.

·         Timeliness resolving an issue in the most efficient manner with proper workload management and time utilization techniques.

See Appendix VI for details of the Embedded Quality and its attributes. 

Data from the EQRS are used to report Customer Accuracy, Professionalism, and Timeliness quality measures to the IRS Commissioner as part of the IRS’ balanced measures.[9]  The Customer Accuracy measure is also reported externally to IRS stakeholders (e.g., Congress and the Government Accountability Office (GAO)) and as part of the reporting requirement of the Government Performance and Results Act.  The Regulatory Accuracy and Procedural Accuracy quality measures are reported internally to IRS management to identify trends and training opportunities.

Field Assistance Office Quality Assurance function staff members visit the TACs and observe group managers conducting Embedded Quality observational reviews.  The Quality Assurance function staff also prepares a Data Collection Instrument for each taxpayer contact they observe.  The Quality Assurance function results are input to the National Quality Review System, which produces quality review reports designed to identify trends that may indicate problem areas, training needs, and opportunities for process improvements. 

This review was performed in the Field Assistance Office of the Wage and Investment Division in Atlanta, Georgia, during the period June 2004 through May 2005.  This audit focused only on the Embedded Quality component of the EQBI.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

 

 

Results of Review

 

The Field Assistance Office Is Implementing an Innovative Quality Measurement System

The EQBI is an innovative approach to quality control for face-to-face interactions between assistors and taxpayers.  No other Federal Government agencies we contacted have developed a comprehensive quality measurement system that measures face‑to‑face assistance like the one the IRS is implementing.  The new EQBI integrates an IRS-wide, customer-focused, standardized measurement system with centralized data collection tools and current technology, such as Contact Recording and Queuing Management.  It will help address many concerns raised by the Office of Management and Budget, the GAO, and the TIGTA on the accuracy of tax law answers and the need for a quality measurement system with which to set goals and measure progress toward achieving those goals.

Prior to Fiscal Year 2002, the IRS did not have an effective process in place to measure the progress made to improve customer service at the TACs.  In October 2001, the IRS hired an outside contractor to make anonymous visits to the TACs to assess the quality of service.  In October 2002, the IRS implemented a quality review program for the TACs using anonymous visits by IRS employees.  The results from these two approaches were not reliable and did not provide a statistical representation of the taxpayers that visit the TACs.

Consequently, the Field Assistance Office began developing the EQBI.  Beginning in Fiscal Year 2004, the Field Assistance Office:

·         Established a training program to train managers and educate employees on the benefits of the Embedded Quality.

·         Defined roles and responsibilities for each level of management and staff participating in the Embedded Quality process.  This included Area Office Directors, Territory Managers, and group managers.  The Field Assistance Office is divided into 32 Territories in 5 geographic Area Offices.

·         Conducted an Embedded Quality pilot from October through December 2003 with 14 group managers documenting observational reviews of over 140 front-line assistors.  Approximately 400 reviews were captured during the pilot, resulting in overall accuracy scores of 90 percent and above.

·         Rolled out the system IRS-wide and completed 11,839 observational reviews from April 19, 2004 (the date the Embedded Quality was implemented), through September 30, 2004 (the end of Fiscal Year 2004).

The picture was removed due to its size.  To see the picture, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

TAC group managers completed 11,839 observational reviews within the first 6 months of the Embedded Quality implementation.

The IRS reported assistors commented that the Embedded Quality helped them identify changes needed to better their performance.  Managers stated the EQRS’ powerful database and the template it generates lessened their burden by making it easier to evaluate performance and provide specific, targeted feedback to employees.

Embedded Quality Review System Data May Not Be Representative of the Population of Assistors

From an analysis of EQRS data, we determined that managers did not always follow the sampling plan.  Internal controls were not in place to ensure the required observational reviews were completed and only those required reviews were entered into the EQRS.  For Fiscal Year 2004, group managers:

  • Did not complete observational reviews for 168 (11 percent) of 1,550 assistors required to be included in the sampling plan (or the statistical sample).

·         Included from 9 to 26 observational reviews in the sampling plan for 455 (29 percent) of 1,550 assistors that should not have been included.  These observations were in addition to those required by the sampling plan.

Figure 4 presents the number of assistors and the number of observational reviews each received in Fiscal Year 2004.

Figure 4:  Fiscal Year 2004 Number of Assistors per Number of Observations

 

Figure 4 was removed due to its size.  To see Figure 4, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

The sampling plan was refined for Fiscal Year 2005 to provide additional guidance to ensure the integrity of the EQRS results.  Despite these refinements, an analysis of the EQRS for a limited period of Fiscal Year 2005, from October 1, 2004, through January 31, 2005, showed:

·         Twenty-one (13 percent) of 157 assistors did not receive the required minimum number of observational reviews. 

·         Fifty (32 percent) of 157 assistors had more than 1 observational review included in the sample for just the month of October 2004.  Figure 5 presents information on the selection of one operational review per month per employee. 

This happened because the program was new and sufficient procedures and internal controls had not been developed to ensure the sampling plan was followed and that only required observational reviews were included in the statistical sample.  Figure 5 presents a comparison of the Fiscal Year 2004 and Fiscal Year 2005 Sampling Plans.

Figure 5:  Comparison of Fiscal Years 2004 and 2005 Sampling Plans

Fiscal Year 2004 Sampling Plan

Fiscal Year 2005 Sampling Plan

Group managers were required to conduct a minimum of one observational review per month for each employee, plus two additional reviews, for a total of eight reviews by the end of September 2004. 

Group managers were required to select the first observational review per employee per month to be included in the statistical sample, for a total of 12 per year.

Observational reviews were to be entered into the EQRS.

The required observational reviews (those to be included in the statistical sample) were to be entered into the EQRS with the designation code “NR” for National Review. 

Group managers could also observe assistors more than required in the sampling plan (for example, because they identified performance issues).

While group managers were encouraged to complete more than the required number of reviews per employee, only the first observational review per employee per month was to be coded NR. 

No instructions were provided to differentiate between the observational reviews selected for the statistical sample and those that were not.

Only observational reviews coded NR were
to be used to calculate and report quality measures.

Source:  The IRS Field Assistance Office.

In addition, insufficient resources were allocated to the Quality Assurance function staff.  Staffing was not sufficient to evaluate risks and the effectiveness of the internal controls of the Embedded Quality process.  Also, procedures and guidelines had not been developed to validate and monitor the group managers’ adherence to the sampling plan.

For example, during our initial validation of the EQRS Fiscal Year 2004 data, we determined that 413 assistors were not included in the sampling plan and did not receive the required 8 observational reviews.  We provided the list to Field Assistance Office management.  They could not determine from the list which assistors were required to be included in the sample.  There was no consolidated schedule to validate which assistors were required to be included in the sample and how many observational reviews were required for each. 

The Field Assistance Office forwarded our list to its five Area Offices with instructions to explain why assistors were not observed and included in the sample.  After obtaining responses from all Area Offices, management explained and we determined that 245 of the 413 assistors were not required to have all 8 observational reviews because during the time period tested they were, among other reasons, in a nonwork status, attending training, or assigned to collection work.  However, the EQRS did not reflect this.

The Field Assistance Office had instructed Territory managers and Area Office analysts to monitor and ensure group managers follow the sampling plan and complete the required number of observational reviews.  The Field Assistance Office relied on them to follow procedures and did not do any independent verification.

An effective sampling plan requires objectivity and a means for establishing sample sizes and appraising sample results mathematically.  It should not be subject to potential bias.  Not following prescribed procedures produces a systematic bias that could compromise the randomness of the sample.

In addition, effective internal controls are necessary to ensure actions are taken to address areas of risk.  Internal controls should be an integral part of any system.  Controls should be designed to help ensure completeness, accuracy, authorization, and validity of all transactions.  Internal control systems need to be monitored–through ongoing monitoring activities, separate evaluations, or a combination of the two.  

When we shared our results with Field Assistance Office management, they established procedures that require each Area Office to submit monthly variance reports listing assistors that did not require monthly observational reviews and the reasons why.  In addition, some group managers were removed because they did not conduct the required observational reviews.

Recommendations

The Commissioner, Wage and Investment Division, should:

Recommendation 1:  Limit the use of the Embedded Quality data until the data are validated as statistically representative of the population of assistors.  The data should not be used to report balanced measures or to make significant business decisions.

Management’s Response:  The IRS made a decision to not use EQRS data as a quality measure for Fiscal Years 2005 and 2006.  The Fiscal Year 2006 goal will be to baseline EQRS quality scores and improve the integrity of the EQRS data.  The IRS will use our anonymous shopping scores as the quality measure for Fiscal Year 2006.

Recommendation 2:  Establish, document, and implement a system of internal controls to ensure the sampling plan is followed and the results are statistically representative of the population of assistors.  This includes establishing a centralized process to monitor and ensure group managers are following the sampling plan methodology and only employees required to be observed are included in the sample.

Management’s Response:  The IRS revised guidelines to include roles and responsibilities and sampling plan requirements for the number of reviews per employee and time periods for input and sharing of review results.  Territory managers will perform weekly monitoring of the group managers, and Area Offices are required to submit monthly variance reports listing assistors that did not require monthly reviews and why. 

Internal Controls Are Not Sufficient to Ensure the Reliability of the Embedded Quality Review System Data

The EQRS is not providing an accurate measure of the quality of service provided in the TACs.  The data are biased, inconsistent, and contain errors.  From April 19 through September 30, 2004, 11,839 Data Collection Instruments were entered into the EQRS.  We tested a stratified statistical sample of 314[10] Data Collection Instruments from across all product lines.  For Fiscal Year 2004:

·         Thirty-eight of 228 Data Collection Instruments (22 of 148 Tax Law, 12 of 49 Accounts, and 4 of 31 Tax Return Preparation) tested contained errors that affected Customer Accuracy.  Projecting the mean error rate to the total population, 2,500 (21.12 percent) of 11,839 Data Collection Instruments in the EQRS contain errors that affect Customer Accuracy.[11]

·         Most (306 of 310) Data Collection Instruments (157 of 160 Tax Law, 118 of 119 Accounts, and all 31 Tax Return Preparation) contained errors that affected Regulatory, Procedural, Professionalism, and/or Timeliness quality measures.  Projecting the mean error rate to the total population, 11,725 (99.04 percent) of 11,839 Data Collection Instruments in the EQRS contain errors that affect these 4 quality measures.[12]

To ensure the Embedded Quality data are valid and reliable, the data must be checked or tested for significant errors.  Periodic statistical reviews, reconciliations, or comparisons of data should be performed.  In addition, monitoring the effectiveness of internal controls should occur in the normal course of business.  Internal controls need to be clearly documented, and the documentation should be readily available for examination.

Managerial observational reviews introduce bias, inhibiting the accurate assessment of employee performance

To monitor the performance of assistors, managers currently must be physically present when assistors help taxpayers.  This physical presence establishes an artificial situation for both the employee and the taxpayer and inhibits the accurate assessment of performance in day-to-day contacts.  The GAO, in a November 2004 report, stated that observing taxpayer/employee interaction “. . . could yield biased data, because assistors will know they are being observed, which could influence their behavior.  Consequently, Embedded Quality data gathered by direct observation may not be representative of true performance.[13]

During a TIGTA audit of the TAC 2005 Filing Season,[14] auditors visited a judgmental sample of 50 TACs posing as taxpayers and asking assistors tax law questions.  Using the Embedded Quality method to score their responses resulted in a 39 percent accuracy rate compared to the accuracy rate of 92 percent using the results from managerial observational reviews during the 2005 Filing Season. 

The Field Assistance Office acknowledges that observational reviews could introduce bias into the results.  This bias would be eliminated once it implements Contact Recording, which is designed to remove the manager’s physical presence and allow him or her to select a recorded contact based on a statistically valid sampling plan.  In May and June 2005, the Field Assistance Office conducted a 60‑day pilot of Contact Recording in 7 TACs.  Results are being evaluated, and a second pilot is scheduled during the 2006 Filing Season.  Initial implementation is tentatively scheduled to begin in Fiscal Year 2006, with full implementation by Fiscal Year 2008.  We plan to begin a review of Contact Recording in late Fiscal Year 2006.

Inconsistent use of attributes resulted in inaccurate quality rates

Group managers did not consistently score attributes, nor did they always follow the Embedded Quality Job Aid[15] when preparing the Data Collection Instruments.  This resulted in the Field Assistance Office incorrectly calculating the rates used to report the five quality measures. 

Customer Accuracy Quality Measure 

Thirty-eight of the 228 Data Collection Instruments tested contained errors that affected Customer Accuracy.  Figure 6 shows the Customer Accuracy goals and reported rates for the IRS for Fiscal Year 2004, compared to the results of our statistical sample.

 

Figure 6:  Fiscal Year 2004 Comparison of Embedded Quality Accuracy Rates to TIGTA Statistical Sampling Results

Product Line

IRS  Customer Accuracy Goals

IRS Reported Customer Accuracy

TIGTA Calculated Customer Accuracy

Overstated  Customer Accuracy

Accounts

87%

96%

76%

20%

Tax Law

80%

94%

85%

 9%

Tax Return Preparation

99%[16]

96%

87%

9%

Source:  TIGTA statistical sample of 228 Data Collection Instruments selected from the period April 19 through September 31, 2004.

Regulatory, Procedural, Professionalism, and Timeliness Quality Measures 

Most Data Collection Instruments tested (306 of 310) contained errors that affected Regulatory, Procedural, Professionalism, and/or Timeliness quality measures. 

·         For 272 of 310 Data Collection Instruments, managers did not score attributes that should have been scored.  For example, the Data Collection Instrument showed the interaction involved the Accounts product line, but the group manager did not score all appropriate or required attributes for the Accounts product line (e.g., he or she did not score the attribute to indicate the employee properly verified the taxpayer’s identification before providing the tax information).

·         For 177 of 310 Data Collection Instruments, group managers scored attributes that should not have been scored.  For example, a group manager scored attributes relating to a tax law issue when the Data Collection Instrument showed that a tax law was not discussed with the taxpayer.

·         For 44 of 310 Data Collection Instruments, managers used the incorrect product line to document the observational review.  For example, group managers documented that the taxpayer visited the TAC to ask a tax law question but, after reviewing the Data Collection Instrument, it was apparent the taxpayer actually visited the TAC with an account question. 

·         For 21 of 310 Data Collection Instruments, group managers incorrectly scored attributes with a “yes” when they should have been scored with a “no,” or vice versa.  For example, group managers noted that assistors correctly applied the tax law when responding to taxpayers’ tax law questions.  However, the Data Collection Instrument showed assistors did not correctly apply the tax law.

The complexity of the tax law and the application of over 100 attributes contributed significantly to inconsistencies and errors.  The EQRS is a complex system and is in its first years of implementation in the Field Assistance Office.  The Quality Assurance function staffing has not been sufficient to identify and develop an effective internal control system that would include monitoring and periodic statistical assessments of the results. 

In addition, the Field Assistance Office modeled its Embedded Quality on the Toll-Free Telephone Assistance Program Embedded Quality.  Where applicable, attributes were modified, and in some instances created, to reflect specific procedures or services provided in the TACs.  However, in some cases, the Toll‑Free Embedded Quality attribute definitions did not apply, were insufficient, or did not fit with the services provided in the TACs.  This created some confusion, and a Field Assistance Office quality improvement team is currently addressing this issue.

The accuracy of the Data Collection Instruments should improve as the group managers’ learning curves decrease and as solutions from the quality improvement team’s various strategies are implemented and measured.  In addition, the Field Assistance Office revised the Data Collection Instrument and Embedded Quality Job Aid and is adding a coding consistency goal to Area Office Directors’ performance standards.

The Data Collection Instruments do not provide sufficient information to validate the results of the observational reviews

For 209 of 310 Data Collection Instruments (69 of 160 Tax Law, 111 of 119 Accounts, and 29 of 31 Tax Return Preparation) tested, the Data Collection Instruments do not provide sufficient information for independent reviewers to determine what actually took place during the interaction between the assistor and the taxpayer.  Group managers are not required to document in the narrative on the Data Collection Instrument all interactions between the assistors and the taxpayers.  Comments are generally required only to support when the assistors deviate from attributes or procedures.  Projecting the mean error rate to the total population, 9,799 (82.77 percent) of 11,839 Data Collection Instruments in the Field Assistance Office’s EQRS do not contain sufficient information for an independent reviewer to determine what took place between the assistor and the taxpayer.[17] 

We shared our results with the Field Assistance Office Quality Assurance function staff, who verified the coding inconsistencies and documentation issue.  They stated that, although the group managers had received training on how to score the attributes, they did not have specific training and guidelines on how to document the observational reviews. 

Until this review, the Data Collection Instruments have not been used to substantiate the results of the observational reviews.  The Data Collection Instrument was not developed to provide substantive documentation to support the rating of the attributes and the observational review.  It was designed more to prompt group managers on the attributes and to help them provide effective feedback to the assistor rather than to substantiate the rating.  However, at this time, the Data Collection Instrument is the only means to substantiate and support the rating of the observational review.

As part of the Embedded Quality internal control process, the Field Assistance Office uses dual monitoring to ensure coding consistency.  Quality Assurance function reviewers visit the TACs to monitor group managers during their observational reviews of assistors.  Both the reviewers and the group managers complete a Data Collection Instrument for each observational review.  After the observational reviews, the reviewers discuss the observational results with the group managers. 

In Fiscal Year 2005, the Field Assistance Office issued new standardized guidelines that require the Data Collection Instrument to include written documentation to support the Customer Accuracy quality measure rate.  This will facilitate an independent validation of the results for this measure to ensure the reliability of the data.

However, procedures do not require Data Collection Instruments to contain information that identifies taxpayers that visit the TACs for account inquiries and tax return preparation.  Field Assistance Office management stated that most group managers do not use the field and cited a concern with having taxpayer information on the Data Collection Instrument in case it was used in an employee grievance.  Without taxpayer identification information, an independent reviewer cannot research to determine if the assistor provided accurate account information or accurately prepared the tax return.  Once Contact Recording is implemented, the recording will provide this information.

Timely input of the Data Collection Instruments is not always assured

For 51 of 310 Data Collection Instruments (31 of 160 Tax Law, 16 of 119 Accounts, and 4 of 31 Tax Return Preparation) sampled, the Data Collection Instruments were not timely entered into the EQRS.  The time to input ranged from 4 to 71 days, or an average of 14 days, after the date of the observational reviews.  Projecting the mean error rate to the total population, 1,732 (14.63 percent) of 11,839 Data Collection Instruments in the Field Assistance Office’s EQRS were not timely entered. 

The Field Assistance Office did not have guidelines in Fiscal Year 2004 requiring Data Collection Instruments to be entered within a certain number of days.  However, beginning in Fiscal Year 2005, group managers are required to input all completed Data Collection Instruments into the EQRS within 3 business days of the observational review.  Therefore, we considered input to be timely using the Fiscal Year 2005 requirement of 3 business days.  Although the 3‑day requirement was in effect in Fiscal Year 2005, limited testing of 1 Field Assistance Office Area Office showed that, during Fiscal Year 2005, 53 (9 percent) of 579 Data Collection Instruments were not input within 3 business days of the observational review. 

Untimely recordation of observational reviews increases the risk that the data are unreliable.  GAO Standards for Internal Control in the Federal Government state, “Transactions should be promptly recorded to maintain their relevance and value to management in controlling operations and making decisions.  In addition, control activities help to ensure that all transactions are completely and accurately recorded.”  Further, the Office of Management and Budget Circular A-123, Management Accountability and Control, states transactions should be promptly recorded, properly classified, and accounted for in order to prepare timely and reliable reports.

Changes and deletions to the Embedded Quality Review System are not monitored or analyzed

Of 11,839 Data Collection Instruments input into the EQRS, 423 (4 percent) had been edited.  The group managers edited the Data Collection Instruments because they initially had input incorrect data or had to update the data with new information received after initial input.  In addition, 400 (3 percent) Data Collection Instruments had been deleted from the EQRS. 

The Field Assistance Office stated that the Embedded Quality implementation team did not identify deleted and changed data as a key component in the reliability of data in the EQRS.  Procedures require Territory Managers and Area Office analysts to monitor the Change and Delete reports; however, neither report was being reviewed.  In addition, the Quality Assurance staff had not reviewed either report because there were no procedures requiring a review. 

Risks to the integrity of the EQRS increase when changes and deletions are not reviewed or analyzed.  For Fiscal Year 2005, the Field Assistance Office had not updated its guidelines to include requirements that the Quality Assurance function staff review and monitor Change and Delete reports. 

Resource limitations limit the effectiveness of the Field Assistance Office Quality Assurance function

For 12 (32 percent) of 37 observational reviews tested in the National Quality Review System, Quality Assurance function reviewers scored the Customer Accuracy attribute differently from group managers.  The managers input the results of their observational reviews in the EQRS, and the resulting Customer Accuracy rate was 97 percent.  The Quality Assurance function reviewers input their results in the National Quality Review System, and the resulting Customer Accuracy rate was 79 percent.  The Quality Assurance function reviewers do not have authority to override the managers’ decisions. 

We could not determine the reasons for the differences, but we believe they include bias when an assistor’s immediate manager conducts the observational review, inconsistent scoring from different levels of knowledge and experience with the Embedded Quality and its attributes, or training issues.  Nevertheless, the Field Assistance Office has not had procedures in place or the resources to conduct analyses to identify such errors or trends and develop solutions.

Although the Field Assistance Office uses the National Quality Review System and has Quality Assurance function reviewers participating in observational reviews, it has not established a permanent independent Quality Assurance function staff.  The Field Assistance Office Quality Assurance function staff was comprised of experienced employees detailed from the field with no authority to override group manager decisions.  A lack of permanent staff and procedures prevents the establishment of a systematic process to monitor the Embedded Quality process and validate the data and results.

The lack of a formal review process contributed to the unreliability of quality measures because errors were not identified and corrective actions were not taken.  During Fiscal Year 2005, the Field Assistance Office has addressed these issues by starting projects that are focusing on problems such as coding consistency.  For example, the Field Assistance Office developed a new training module and job aids to assist group managers.  In addition, it has a quality improvement team studying the three specialized product lines:  Tax Law, Accounts, and Tax Return Preparation.  The team is analyzing the 100 attributes in an effort to reduce the number of attributes, thereby alleviating confusion and helping consistency.  The team will observe group managers’ adherence to established coding procedures, identify areas of variation, make changes, and monitor the changes.

Conclusion

The Field Assistance Office is moving in the right direction with the implementation of the Embedded Quality and efforts to improve the statistical validity and reliability of Embedded Quality data.  Current projections call for Contact Recording to be fully implemented in Fiscal Year 2008.  We believe the Embedded Quality with Contact Recording, when appropriately working and managed, can provide a consistent methodology for all managers to evaluate performance, establish baselines, and identify root causes of defects in employee interactions with taxpayers. 

The Embedded Quality system is a new process for the Field Assistance Office.  It is in the first years of implementation and it is reasonable that there would be a considerable learning curve.  However, we believe the lack of a strong internal control system to monitor and evaluate the effectiveness of processes and procedures has resulted in an over/understatement of reported quality measures.  As a result, the Field Assistance Office should not use Fiscal Years 2004 or 2005 data to report quality measures or to establish baselines to measure future improvement.

Recommendations

The Commissioner, Wage and Investment Division, should:

Recommendation 3:  Ensure group managers receive training on the importance of the Data Collection Instrument, not only to document the assistor/taxpayer interaction but also to substantiate the rating for use in subsequent reviews, reconciliations, and validations.

Management’s Response:  The IRS issued guidelines for Data Collection Instruments that standardize the remarks sections to permit verification by a third party and to permit Territory managers and quality staff to use them to provide feedback for managers.  It also delivered additional EQRS training and created a revised job aid to clarify attribute definitions, communicate EQRS changes, and reemphasize the importance of writing substantive remarks and improvement strategies. 

Recommendation 4:  Establish, document, and implement a system of internal controls to ensure the EQRS data are valid and reliable and have been checked and tested for significant errors.  This includes establishing written guidelines that clearly establish roles and responsibilities, a centralized process to conduct periodic statistical reviews and reconciliations, and the means by which to provide documentation on the effectiveness of the internal controls.

Management’s Response:  The IRS established an internal control system that includes:

·         Formally established Embedded Quality Roles and Responsibilities.

·         A sampling plan requirement for managerial review of at least one contact per employee per month and to provide employee feedback.

·         A requirement for Territory managers to perform weekly monitoring of group managers.

·         A requirement for the headquarters quality staff to validate group manager coding and to share results with field offices.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

Our overall objective was to determine whether the Embedded Quality measurement system is efficiently and effectively improving business results for the Wage and Investment Division Field Assistance Office.  To accomplish our objective, we:

I.                  Determined the purpose of the Embedded Quality, its role in the Internal Revenue Service Strategy and Program Plan, and its balanced measures.

II.                Determined if the Embedded Quality sampling plan will provide statistically valid quality measures.

A.    Obtained and reviewed a copy of the Embedded Quality sampling plan.

B.     Discussed the process the Field Assistance Office used to develop the Embedded Quality sampling plan.

C.     Consulted with the Treasury Inspector General for Tax Administration statistician to determine if the plan is statistically reliable and valid.

D.    Determined why the Field Assistance Office Embedded Quality sampling plan is not statistically valid and identified the potential effect on internal and external stakeholders.

III.             Determined if group managers are following the sampling plan.

A.    Determined the number of completed observational reviews per employee for Fiscal Year 2004 (April 19 – September 30, 2004).  

B.     Randomly selected one of five Area Offices to determine the number of completed observational reviews per employee for Fiscal Year 2005 (October 1 – December 31, 2005).[18]

C.     Compared the number of completed observational reviews to the required number of observational reviews according to the sampling plan.

D.    Determined why group managers were not following the sampling plan and identified the potential effect on internal and external stakeholders.

IV.             Evaluated the reliability of data captured in the Embedded Quality Review System (EQRS). 

A.    Selected from the EQRS a stratified statistical sample of 314[19] Data Collection Instruments[20] from a population of all 11,839 completed in Fiscal Year 2004 (April 19 – September 30, 2004).  We stratified the population by Embedded Quality product line;[21] this resulted in 2,488 Tax Law, 7,970 Accounts, and 1,381 Tax Return Preparation Data Collection Instruments.  We used a random number program to select a statistical sample of 162 Tax Law, 121 Accounts, and 31 Tax Return Preparation Data Collection Instruments.  We based our sample size on a 90 percent confidence and an expected error rate of 20 percent for Tax Law, 13 percent for Accounts, and 3 percent for Tax Return Preparation product lines.  The precision was ± 5 percent.

B.     Randomly selected one of five Area Offices to evaluate if the Data Collection Instruments were timely input into the EQRS for Fiscal Year 2005 (October 1, 2005, through December 31, 2005).[22] 

C.     Obtained and reviewed Quality Review Defect Reports showing the differences between the Quality Review Database version 2 Data Collection Instrument and the EQRS Data Collection Instrument.  Also, we reviewed all 37 Data Collection Instruments in the Quality Review Database version 2 for the same time period in Fiscal Year 2004.  These Data Collection Instruments were the results of the Field Assistance Office Quality Assurance function staff observing the managers during their observational reviews.

D.    Obtained EQRS reports to determine the volume of edited and deleted Data Collection Instruments during the review period.

E.     Determined why the database was not reliable and identified the potential effect on internal and external stakeholders.

V.                Contacted other Federal Government agencies (i.e., the Social Security Administration, the Department of Veterans Affairs, and the Department of Housing and Urban Development) with field offices that provide assistance to the public to determine how they measure quality.

 

Appendix II

 

Major Contributors to This Report

 

Scott A. Macfarlane, Acting Assistant Inspector General for Audit (Wage and Investment Income Programs)

Augusta Cook, Director

Frank Jones, Audit Manager

Tanya Boone, Lead Auditor

Deborah Drain, Lead Auditor

Jackie Forbus, Senior Auditor

Jerome Antoine, Auditor

Jean Bell, Auditor

Jerry Douglas, Auditor

Kathy Henderson, Auditor

Patricia Jackson, Auditor

Sylvia Sloan-Copeland, Auditor

Geraldine Vaughn, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Services and Enforcement  SE

Deputy Commissioner, Wage and Investment Division  SE:W

Director, Customer Assistance, Relationships, and Education, Wage and Investment Division  SE:W:CAR

Director, Strategy and Finance, Wage and Investment Division  SE:W:S

Acting Chief, Performance Improvement, Wage and Investment Division  SE:W:S:PI

Director, Field Assistance, Wage and Investment Division  SE:W:CAR:FA

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaison:  Acting Senior Operations Advisor, Wage and Investment Division  SE:W:S

 

Appendix IV

 

Outcome Measures

 

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration.  These benefits will be incorporated into our Semiannual Report to the Congress.

Type and Value of Outcome Measure:

·         Reliability of Information – Actual; 623 assistors received the incorrect number of observational reviews required by the sampling plan for the Embedded Quality Review System (EQRS) (see page 7).

For Fiscal Year 2004, group managers did not always follow the sampling plan that required eight observational reviews to be conducted for each assistor and entered into the EQRS.  As a result, managers:

  • Did not complete observational reviews for 168 (11 percent) of 1,550 assistors required to be included in the sampling plan (or the statistical sample).
  • Included from 9 to 26 observational reviews in the sampling plan for 455 (29 percent) of 1,550 assistors that should not have been included.

Methodology Used to Measure the Reported Benefit:

We sorted all of the Data Collection Instruments in the EQRS by location and employee name.[23]  We counted the number of observational reviews and determined how many employees received the required eight and how many received fewer or more than the required eight observational reviews for the period.

Type and Value of Outcome Measure:

·         Reliability of Information – Actual; 2,500 and 11,725 EQRS records affected (see page 11).

For the Customer Accuracy Quality Measure, group managers incorrectly scored Data Collection Instruments in the EQRS for each of the Embedded Quality product lines[24] (Tax Law, Accounts, and Tax Return Preparation).  As a result, we project the EQRS contained:

  • 370 Tax Law product line Data Collection Instruments with errors. 
  • 1,952 Accounts product line Data Collection Instruments with errors.
  • 178 Tax Return Preparation product line Data Collection Instruments with errors.

For the Procedural, Regulatory, Professionalism, and Timeliness Quality Measures, group managers incorrectly or insufficiently scored Data Collection Instruments in the EQRS.  As a result, we project the EQRS contained:

  • 2,442 Tax Law product line Data Collection Instruments with errors.
  • 7,903 Accounts product line Data Collection Instruments with errors.
  • 1,380 Tax Return Preparation product line Data Collection Instruments with errors.

Methodology Used to Measure the Reported Benefit:

We reviewed and compared how group managers scored each attribute to the required instructions listed in the Embedded Quality Job Aid [25] and internal manuals and to information obtained during discussions with Field Assistance Office staff.  For each product line, we identified the number of erroneous Data Collection Instruments and calculated an error rate[26] for the Customer Accuracy, Procedural, Regulatory, Professionalism, and Timeliness quality measures.  We calculated error rates by dividing the number of Data Collection Instruments with errors by the total number in the sample.  Also, we calculated the stratified mean error rate to project the errors found in the sample across the total population. 

For the Customer Accuracy Quality Measure, we projected 2,500 Customer Accuracy errors in the EQRS from our review of 228[27] Data Collection Instruments.  The EQRS uses only one attribute to score Customer Accuracy.  Specifically:

370 Tax Law product line Data Collection Instruments with errors:

o       Percentage of Tax Law Data Collection Instruments in the EQRS (2,488/11,839) = .2102.

o       The Tax Law product line sample error rate = (22/148) = .1486.

o       The Tax Law product line stratified mean error rate = .2102 x .1486 = .03124. 

o       The projected Tax Law product line errors in the EQRS = .03124 x 11,839 = 370.

1,952 Accounts product line Data Collection Instruments with errors:

o       Percentage of Accounts Data Collection Instruments in the EQRS (7,970/11,839) = .6732.

o       The Accounts product line sample error rate = (12/49) = .2449.

o       The Accounts product line stratified mean error rate = .6732 x .2449 = .16487. 

o       The projected Accounts product line errors in the EQRS = .16487 x 11,839 = 1,952.

178 Tax Return Preparation product line Data Collection Instruments with errors:

o       Percentage of Tax Return Preparation Data Collection Instruments in the EQRS (1,381/11,839) = .1166.

o       The Tax Return Preparation product line sample error rate = (4/31) = .1290. 

o       The Tax Return Preparation product line stratified mean error rate = .1166 x .1290 = .01504. 

o       The projected Tax Return Preparation product line errors in the EQRS = .01504 x 11,839 = 178.

For the Procedural, Regulatory, Professionalism, and Timeliness Quality Measures, we projected 11,725 errors in the EQRS from our review of 306 Data Collection Instruments.  The EQRS uses multiple attributes to calculate these four measures.  Specifically:

2,442 Tax Law product line Data Collection Instruments with errors:

o       Percentage of Tax Law Data Collection Instruments in the EQRS (2,488/11,839) = .2102.

o       The Tax Law product line sample error rate = (157/160) = .9813. 

o       The Tax Law product line stratified mean error rate = .2102 x .9813 = .20627. 

o       The projected Tax Law product line errors in the EQRS = .20627 x 11,839 = 2,442.

7,903 Accounts product line Data Collection Instruments with errors:

o       Percentage of Accounts Data Collection Instruments in the EQRS (7,970/11,839) = .6732.

o       The Accounts product line sample error rate = (118/119) = .9916.

o       The Accounts product line stratified mean error rate = .6732 x .9916 = .66754. 

o       The projected Accounts product line errors in the EQRS = .66754 x 11,839 = 7,903.

1,380 Tax Return Preparation product line Data Collection Instruments with errors:

o       Percentage of Tax Return Preparation Data Collection Instruments in the EQRS (1,381/11,839) = .1166.

o       The Tax Return Preparation product line sample error rate = (31/31) = 1.00. 

o       The Tax Return Preparation product line stratified mean error rate = .1166 x 1.00 = .11660. 

o       The projected Tax Return Preparation product line errors in the EQRS = .11660 x 11,839 = 1,380 (rounded down from 1,381).

 

Appendix V

 

The Embedded Quality Measurement System With Contact Recording

 

The Embedded Quality measurement system with Contact Recording (CR), shown below, captures the audio portion of the employee/customer interaction, synchronized with computer screen activity, for replay and quality review for the group managers to measure performance against predetermined standards.  

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

Appendix VI

 

Embedded Quality and Attributes

 

In October 2003, the Internal Revenue Service (IRS) Field Assistance Office began implementing the Embedded Quality component of the Embedded Quality Business Integration.  It began collecting data in April 2004.  The Embedded Quality requires group managers to observe assistors’ contacts with taxpayers and measure performance against predetermined standards. 

The group managers document the results of their observational reviews on a Data Collection Instrument.[28]  The Data Collection Instrument contains over 100 evaluative attributes identified with the reasons taxpayers visit Taxpayer Assistance Centers (TAC), whether for tax law, accounts, or tax return preparation assistance.  All attributes are mapped to assistors’ Critical Job Elements used to evaluate their performance. 

The evaluative attributes used regardless of the reason the taxpayer visits a TAC are professional greeting, employee identification, identification of the taxpayer’s issue, and deciding whether the assistor can address the taxpayer’s issue based on training or should transfer the issue to a designated area.  Other evaluative attributes that are specific to the reason a taxpayer visits a TAC include those in Chart 1.

Chart 1:  Assistance-Specific Embedded Quality Attributes

Tax Law Assistance

Accounts Assistance

Tax Return
Preparation Assistance

Apparent use of guides to provide correct response to taxpayer question.

Disclosure met (i.e., determined that taxpayer is authorized to have tax information).

Disclosure met (i.e., determined that taxpayer is authorized to have tax information).

Complete use of guides to provide correct response to taxpayer question.

Verified photo identification.

Verified photo identification.

Complete response given to taxpayer according to the guides.

Conducted complete research on account-related computer systems.

Verified taxpayer had all return preparation documents.

Interpreted/applied tax law correctly.

Verified full compliance to determine if taxpayer is compliant with all tax obligations.

Verified taxpayer met Field Assistance Office tax return preparation guidelines.

Obtained/determined tax law facts.

Determined taxpayer’s ability to pay if taxes owed.

Used tax return preparation forms, schedules, and worksheets.

Appropriate procedural action taken (e.g., did assistor provide taxpayer a publication at the end of the visit).

Determined enforcement actions needed (e.g., filing a lien).

Updated Taxpayer Identification Number, name, address, filing status, and filing requirements on computer systems.

Provided taxpayer with correct and complete response.

Determined if taxpayer can pay taxes in installments.

Input correct information into tax return preparation software.

Input visit to specialized system for future reference.

Input installment agreement into computer systems.

Returned original documents to taxpayer.

Prepared/input taxpayer request for specialized documents.

Discussed installment agreement rules with taxpayer.

Explained IRS time periods to taxpayer (when to expect refunds, etc.).

Listened effectively to taxpayer’s issue to provide correct response.

Followed internal processing guidelines for processing installment agreements.

Explained refund offset if taxpayer owes tax on another year or has other tax-related obligations.

Used appropriate talk time
(e.g., limited extraneous dialogue).

Secured sources for future enforcement (e.g., levies).

Explained completed tax return to taxpayer.

Used clear professional language (do not use jargon familiar to IRS employees).

Determined the cause for the taxes owed and discussed cures with taxpayer.

Provided a copy of return to taxpayer.

Source:  IRS Field Assistance Office.

The attributes are divided into five quality measures that are used to report Customer Accuracy, Professionalism, and Timeliness measures to the IRS Commissioner as part of the IRS’ balanced measures.[29]  The Customer Accuracy quality measure is also reported externally to IRS stakeholders (e.g., Congress and the Government Accountability Office) and as part of the reporting requirement of the Government Performance and Results Act.[30]  The Regulatory Accuracy and Procedural Accuracy measures are reported internally to IRS management to identify trends and training opportunities.  Chart 2 provides details.

Chart 2:  Business Results Reports to External Stakeholders[31]

 

Chart 2 was removed due to its size.  To see Chart 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Source:  Excerpt from the IRS presentation “TIGTA[32] Briefing of Embedded Quality,” dated July 22, 2002.

 

Appendix VII

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.



[1] Contact Recording captures the audio portion of the employee/customer interaction, synchronized with computer screen activity, for replay and quality review.  Queuing Management directs taxpayers to the right employee, based on the tax issue. 

[2] The Data Collection Instrument is a checklist designed to assist group managers in rating and documenting interactions.  It is used to both provide feedback to the assistors and assist the group managers when entering the data into the EQRS. 

[3] Only 228 of the 310 statistically sampled Data Collection Instruments had sufficient information to determine Customer Accuracy.  No percentage error rate is shown here because it is not a simple calculation of 38/228 due to the stratified statistical sample.  Instead, the mean error rate, which represents the weighted average for all 3 product lines combined, is shown as it applies to the projected total errors across the population (2,500/11,839).  See Appendix IV for a breakdown of our calculation of 2,500 total errors affecting Customer Accuracy.

[4] No percentage error rate is shown here because it is not a simple calculation of 306/310 due to the stratified statistical sample.  Instead, the mean error rate, which represents the weighted average for all 3 product lines combined, is shown as it applies to the projected total errors across the population (11,725/11,839).  See Appendix IV for a breakdown of our calculation of 11,725 total errors affecting Regulatory, Procedural, Professionalism, and/or Timeliness quality measures.

[5] Included in the estimate are the United States Individual Income Tax Return (Form 1040), Itemized Deductions (Schedule A), Interest and Ordinary Dividends (Schedule B), Capital Gains and Losses (Schedule D), and Earned Income Credit (Schedule EIC).

[6] Percentages in Figure 1will not equal 100 due to rounding.  “Other” services include date stamping tax returns, providing general information such as addresses of IRS offices, and directing taxpayers to other agencies.

[7] Pub. L. No. 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 U.S.C., 31 U.S.C., and 39 U.S.C.).

[8] Attributes identify specific aspects of a contact that need to be considered when measuring the quality of customer service.

[9] Three measures (employee satisfaction, customer satisfaction, and business results) the IRS uses to measure organizational and employee performance.

[10] Only 310 of 314 Data Collection Instruments in our statistical sample were available for review.

[11] Only 228 of the 310 statistically sampled Data Collection Instruments had sufficient information to determine Customer Accuracy.  No percentage error rate is shown here because it is not a simple calculation of 38/228 due to the stratified statistical sample.  Instead, the mean error rate, which represents the weighted average for all 3 product lines combined, is shown as it applies to the projected total errors across the population (2,500/11,839).  See Appendix IV for a breakdown of our calculation of 2,500 total errors affecting Customer Accuracy.  

[12] No percentage error rate is shown here because it is not a simple calculation of 306/310 due to the stratified statistical sample.  Instead, the mean error rate, which represents the weighted average for all 3 product lines combined, is shown as it applies to the projected total errors across the population (11,725/11,839).  See Appendix IV for a breakdown of our calculation of 11,725 total errors affecting Regulatory, Procedural, Professionalism, and/or Timeliness quality measures. 

[13] Tax Administration:  IRS Improved Performance in the 2004 Filing Season, But Better Data on the Quality of Some Services Are Needed (GAO-05-67, dated November 2004).

[14] Customer Accuracy at Taxpayer Assistance Centers Showed Little Improvement During the 2005 Filing Season (Reference Number 2005-40-146, dated September 2005).  The filing season is the period from January through mid-April when most individual income tax returns are filed.

[15] The Embedded Quality Job Aid provides operational definitions for the use of attributes.

[16] The Tax Return Preparation IRS Customer Accuracy Goal is based on the percentage of returns that contain math errors, missing schedules, and erroneous credits claimed for all tax returns prepared in the TACs.

[17] No percentage error rate is shown here because it is not a simple calculation of 209/310 due to the stratified statistical sample.  Instead, the mean error rate, which represents the weighted average for all 3 product lines combined, is shown as it applies to the projected total errors across the population (9,799/11,839). 

[18] Random sampling was used because we did not intend to project the results for this test.

[19] Only 310 of 314 Data Collection Instruments in our statistical sample were available for review.

[20] Group managers use a Data Collection Instrument to capture the results of their observational reviews of employees’ interactions with taxpayers that visit Taxpayer Assistance Centers. 

[21] A product line is a term for the type of work assistors perform when taxpayers visit Taxpayer Assistance Centers for face-to-face assistance.

[22] Random sampling was used because we did not intend to project the results for this test.

[23] The Data Collection Instrument is a checklist designed to assist group managers in rating and documenting interactions.  It is used to both provide feedback to the assistors and assist the group managers when entering the data into the EQRS. 

[24] A product line is a term for the type of work assistors perform when taxpayers visit Taxpayer Assistance Centers for face-to-face assistance.

[25] The Embedded Quality Job Aid provides operational definitions for how attributes may be used.

[26] An error rate is an estimate of the number of errors that exist in a population.  The error rates could not be rounded when reporting projected results from stratified samples.

[27] Only 228 of the 310 statistically sampled Data Collection Instruments had sufficient information to determine Customer Accuracy.

[28] Group managers use a Data Collection Instrument to capture the results of their observational reviews of assistors’ interactions with taxpayers that visit Taxpayer Assistance Centers.

[29] Three measures (employee satisfaction, customer satisfaction, and business results) the IRS uses to measure organizational and employee performance.

[30] Pub. L. No. 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 U.S.C., 31 U.S.C., and 39 U.S.C.).

[31] The reference to “the Service” in the chart refers to the IRS.

[32] Treasury Inspector General for Tax Administration.