TREASURY
INSPECTOR GENERAL FOR TAX ADMINISTRATION
Data Used to Report Toll-Free Telephone Quality Are Reliable, but Internal
Controls Need to Be Documented
July 21, 2006
Reference Number: 2006-40-107
This
report has cleared the Treasury Inspector General for Tax Administration
disclosure review process and information determined to be restricted from
public release has been redacted from this document.
Phone Number |
202-927-7037
Email Address | Bonnie.Heald@tigta.treas.gov
Web Site |
http://www.tigta.gov
July 21, 2006
MEMORANDUM FOR COMMISSIONER, WAGE AND INVESTMENT DIVISION
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Data Used to Report Toll-Free Telephone Quality Are Reliable, but Internal Controls Need to Be Documented (Audit # 200540008)
This report presents the results of our review to determine whether data used by the Wage and Investment Division to calculate the Toll-Free Accounts and Tax Law Customer Accuracy Rates were valid and reliable.
Synopsis
Each
year, millions of customers call the Internal Revenue Service (IRS) toll-free
telephone lines seeking assistance in understanding the tax law and meeting
their tax obligations. Toll-free telephone assistors provided over 26 million
tax account and 5 million tax law services[1] during Fiscal Year 2005. Because of the complexity of the tax
law and the need to serve millions of customers with tax questions, the IRS
divides the types of questions commonly asked into categories called
“applications.” Each application is
staffed with a group of assistors that have received specialized training to
assist customers with specific tax issues. To ensure assistors provide top-quality
service, the IRS subjects all telephone assistor contacts to numerous levels of
review. Quality reviews are used to
measure employee, operational, and organizational performance to identify
trends, problem areas, training needs, and opportunities for process
improvements.
Quality evaluations in
the NQRS database are generally valid and reliable but internal control
processes need to be documented to ensure effective and efficient operations.
Organizational
performance results are housed in the National Quality Review System (NQRS)
database. The IRS uses data from the
NQRS database to report Toll-Free Accounts and Tax Law Customer Accuracy Rates
to external stakeholders. The Centralized Quality Review System (CQRS) function
has implemented a variety of processes and internal controls to ensure consistent,
valid, and reliable data are input to the NQRS database. These
internal controls include numerous activities that help ensure management
directives are carried out and program objectives are met. Analysis of a sample of 114 Data Collection
Instrument records, transcripts, and supporting documentation showed that,
generally, the quality evaluations entered into the NQRS database by the CQRS
function reviewers are valid and reliable.
Although internal controls are sufficient to
ensure the overall quality results of the NQRS database, not all internal
controls are documented. Processes have been
developed over time and need to be documented.
For example, the CQRS function Program Manager has established numerous
reviews that provide critical internal controls to ensure the consistency of
coding and accuracy of the information entered into the NQRS database. The review requirements or procedures for the
reviews have not been documented. If the
Program Manager leaves the CQRS function or the organizational structure changes,
there will be no centralized and documented source for ensuring identification
and continuation of established processes, which could affect the efficiency
and effectiveness of the CQRS function operations.
In addition, in June 2006, the CQRS function plans to implement Contact
Recording[2] for call segments; recordings will be retained only at the call
site and generally will be erased within 6 months. Recordings will replace transcripts, and
transcripts will no longer be retained. Without
a longer retention period for the recordings, the performance measures for
Toll-Free Accounts and Tax Law Customer Accuracy Rates will generally not be
auditable.
There
are currently no clear guidelines on whether the transcripts or recordings used
for reporting quality come under the Federal Government record retention
requirements. The IRS is also
implementing Contact Recording in its Taxpayer Assistance Centers (local IRS
offices that provide face-to-face tax assistance). We plan to conduct a review of that
implementation and will address whether transcripts and recordings meet the
definition of a record that is subject to Federal Government retention
requirements. Therefore, we are making no
recommendations regarding record retention at this time.
Recommendation
The Commissioner, Wage and Investment Division, should
ensure internal controls are centrally documented to ensure the quality review
system continues effective and efficient operations.
Response
IRS management reviewed the draft report and agreed with our recommendation to ensure internal controls are centrally documented and indicated the CQRS Handbook was revised on May 30, 2006. Management’s complete response to the draft report is included as Appendix V.
Copies of this report are also being sent to the IRS managers affected by the report recommendation. Please contact me at (202) 622-6510 if you have questions or Michael E. McKenney, Assistant Inspector General for Audit (Wage and Investment Income Programs) at (202) 622-5916.
Contact Recording Is
Being Implemented, and Transcripts Will No Longer Be Retained
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – National Quality Review System Data Collection Instrument
Appendix V
– Management’s Response to the Draft Report
Each year, millions of
customers call the Internal Revenue Service (IRS) toll-free telephone lines
seeking assistance with understanding the tax law and meeting their tax
obligations. Toll-free telephone assistors provided over 26 million tax account and 5
million tax law services[3] during Fiscal Year 2005. Assistors
answer customer questions involving tax law and other related tax account conditions
such as refunds, balance-due billing activity, and changes to the amount of taxes
owed. For
example, if a customer calls to find out where to mail a tax return, the call
would be routed to an assistor who has been trained to handle IRS procedural issues. If a customer calls to find out the taxes due
on a business tax account, the call would be routed to an assistor who handles
balance‑due questions for the business customer.
Embedded Quality is a way of doing business that builds
commitment and capability of individuals to continually improve customer
service, employee satisfaction, and business results.
The IRS uses the Balanced Measurement System to measure organizational and employee performance. The Balanced Measurement System satisfies the strategic planning and performance measurement requirements of the Government Performance and Results Act of 1993.[4] In support of the Balanced Measurement System, the IRS uses a quality measurement system called Embedded Quality, which links employee performance to organizational results related to the quality of customer service. The Accounts Management function uses the Embedded Quality system to report Customer Accuracy, Professionalism, and Timeliness measures to the IRS Commissioner as part of the Balanced Measurement System. These measures are used at the strategic level to assess the effectiveness and overall progress of the IRS organization as a whole in achieving its strategic goal of improving customer service. In addition, the results are used as support for the budgeting process.
The Customer Accuracy measure is also reported externally to IRS stakeholders (e.g., Congress and the Government Accountability Office) and as part of the reporting requirement of the Government Performance Results Act. The Regulatory Accuracy and Procedural Accuracy measures are reported internally to IRS management to identify trends and training opportunities. These quality measures are a numeric indicator of the extent to which completed work meets prescribed standards. Figure 1 provides details of how business results are reported to stakeholders.
Figure 1: Business Results
Reported to Stakeholders[5]
Figure 1 was removed due to its size. To see Figure 1, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
The Embedded Quality system consists of two parts: employee performance and organizational results. The Centralized Quality Review System (CQRS) function in the Customer Account Services organization is responsible for collecting the performance data. Employee performance results data are housed in the Embedded Quality Review System database. The prime vehicle for collecting these data is through first-line manager reviews of Accounts Management function employees.
Organizational performance results data are
housed in the National Quality Review System (NQRS) database. Data from the NQRS database are used to report
Toll-Free Accounts and Tax Law Customer Accuracy Rates to external stakeholders. To collect these data, reviewers in the CQRS
function:
First.
Monitor
a statistical sample of incoming call segments[6] to each call site by application.
Second.
Document
key points of the conversations. Generally,
procedures require reviewers to remotely monitor a statistical sample of calls
using transcription call sheets, also called transcripts. Figure 2 presents an example of a transcript.
Figure
2: CQRS Call Transcript
Figure 2 was removed due to its size. To see Figure 2, please go to the Adobe PDF
version of the report on the TIGTA Public Web Page.
On the
transcripts, reviewers will type or write general identifying information (e.g.,
date, call site, application monitored, call start and stop time, and the
record number called the Data Collection Instrument (DCI) Number[7]). Reviewers
will also document the interaction between the assistors and customers.
Third.
Perform the necessary research using appropriate
references and tools to determine
whether assistors followed guidelines.
Fourth.
Create
new DCI records on the NQRS database to record and document whether the assistors
provided accurate answers. The DCI
records capture general information for identification purposes (DCI Header section)
to identify and evaluate compliance with quality standards (DCI Attribute section)
and to document synopses of the customers’ requests, assistors’ responses, and
overall results (DCI Remarks section).
See Appendix IV for an example of a DCI record.
Fifth.
File the
transcripts and related research materials to be retained for subsequent reviews. Each reviewer attaches the transcript and
related printouts of any research performed to a daily information sheet that
lists all completed calls reviewed.
These documents are filed and retained in the group for the current and
prior fiscal year and are available for subsequent reviews, including reviews
in response to a rebuttal of results.
The
five quality measures are: Timeliness,
Professionalism, Customer Accuracy, Regulatory Accuracy, and Procedural
Accuracy.
The NQRS database uses attributes to calculate five quality measures: Timeliness, Professionalism, Customer Accuracy, Regulatory Accuracy, and Procedural Accuracy. Each attribute is coded as “meeting the quality standard,” “not meeting the quality standard,” or “not appropriate.” Each coded attribute is the equivalent of an opportunity. When an attribute is coded as not meeting the quality standard, it is considered a defect. Coding of the Regulatory and Procedural Accuracy Attributes determines the coding results of the single Customer Accuracy Attribute (correct/complete response/resolution). Although a defect may be charged to a Procedural or Regulatory Accuracy Attribute, it may not always result in a Customer Accuracy Attribute defect.
The Customer Accuracy Attribute is based on whether the assistor provided the correct/complete response/resolution to the issue and, if appropriate, took the necessary actions to provide the response or resolution. The reviewer charges a defect if one of the following is present:
·
The action or response by the assistor is in
direct conflict with established IRS guidelines and procedures.
·
The reviewer can definitively determine whether
the customer was negatively affected by
the assistor’s response.
The NQRS Remarks section on the DCI record provides
a synopsis of the customer’s issue, the employee’s actions/response, and the evaluative
feedback. The volume of text entered in
this field is systemically limited; therefore a synopsis of only the key
points of the call should be entered.
The reviewer must provide feedback on any attribute coded as a defect
and identify the specific Internal Revenue Manual reference citing the error.
Results from the CQRS function’s independent
review of a statistical sample of calls are rolled into a weighted national
accuracy report, the Weighted Customer Accuracy Report, to provide an
assessment of the customer service provided by the assistors responding to
customer questions on the toll-free tax accounts and tax law telephone
lines. The Fiscal Year 2005 NQRS
Weighted Customer Accuracy Reports shows the
CQRS function reviewers sampled and monitored 35,336 toll-free tax account call
segments and evaluated 32,527 tax account call segments for accuracy. The accuracy rate was weighted by call site
and projected over a population of 26,067,230 tax account call segments to
produce the Customer Accuracy Rate of 91.5 percent. In addition, the CQRS function reviewers sampled
14,483 toll-free tax law call segments and evaluated 12,041 tax law call
segments for accuracy. The accuracy rate
was weighted by call site and projected over a population of 5,026,187 tax law
call segments to produce the Customer Accuracy Rate of 89.0 percent.
This
review was performed in the Customer Account Services organization of the Wage
and Investment Division in
The CQRS function has implemented a variety of
processes and internal controls to ensure consistent, valid, and reliable data
are input to the NQRS database. These internal controls include numerous
activities that help ensure management directives are carried out and program
objectives are met.
Our review of a sample of 114 DCI records,
transcripts, and supporting documentation showed CQRS function reviewers’
overall quality evaluations (correct/complete response/resolution) entered into
the NQRS database are valid and reliable.
The NQRS database accurately reflected the reviewers’ ratings of
their calls for 105 (92 percent) of the 114 DCI records reviewed. Specifically, the transcripts associated with
the DCI records provided the necessary information to identify the key events
of the conversations and supported the overall quality attributes entered on
the NQRS DCI records reviewed. For 9 (8 percent) of the 114 DCI records sampled, either the DCI record or
transcript contained errors, insufficient information to support the
evaluation, or opportunities to provide better customer service. However, only two of these related to
accuracy errors. This error rate would
not significantly affect the reliability of the statistics reported by the IRS
on the accuracy of toll-free responses. Although
we are making no recommendations regarding the errors, our concerns with the
sample cases reviewed are as follows:
·
Two DCI records were not properly coded
to reflect customer accuracy errors. In both cases, the customer was not provided
a correct and/or complete response. For
one DCI record, the CQRS function reviewer accurately identified and coded
the appropriate Procedural Accuracy Attributes with defects but did not code
the Customer Accuracy Attribute with a defect.
For the other DCI record, the CQRS function reviewer did not accurately
identify and code the Procedural Accuracy Attributes and Customer Accuracy Attribute
as errors.
· Four transcripts did not contain enough information to definitively conclude the assistor completely and accurately responded to the customers’ needs. Because we were not provided access to the recordings and we also did not monitor the calls, we had to rely exclusively on call transcripts to determine if the DCI records were accurate. The CQRS function reviewers cannot transcribe the calls verbatim. Some CQRS function reviewers transcribe more information than others. Some reviewers have the Probe and Response Guide (the Guide)[8] with them when they monitor the calls and identify whether probes or responses are/are not addressed while they are monitoring; other reviewers only listen and transcribe, choosing to research the Guide after the calls. Some reviewers listen and document at the same time, while others document everything after the call. Some reviewers transcribe almost every detail, while others limit their transcription to what they consider significant.
·
Three transcripts showed the assistor
could have provided better customer service. The
Embedded Quality process does not take into account the degree or level of
service provided to a customer. If the
assistor complies with required IRS guidelines, the CQRS function reviewer
records no defect and quality service is considered provided. Our review identified specific instances
where this assumption was not always correct.
For example, one transcript documented a customer asking for a Form 1040[9] and a form for dividends. The assistor prompted the customer by
referring to Schedule D.[10]
We believe the
assistor should have asked the customer what type of dividends he or she was
reporting before providing the schedule number. Dividends are reported on either Interest and
Ordinary Dividends (Schedule B) or Schedule D.
CQRS function management believes that, because the customer agreed to
what the assistor provided, not once but twice, the customer was
satisfied. The customer will not know
whether the correct form was received until he or she receives the form and completes
the tax return.
In addition, 13 DCI records contained Procedural
Attribute errors and 24 DCI records had errors in the DCI Header Section. However, these Procedural Attribute errors
did not affect the final Customer Accuracy Attribute.[11] CQRS function
management stated the general call information is not the main focus of their
review and guidelines probably should be clarified for the selection of some
header fields. The errors in the DCI
Header section also do not affect the accuracy of the calls. Nevertheless, these database errors may
produce doubt as to the reliability of the NQRS database and its use for other
reporting purposes and analyses.
The calls sampled and monitored were statistically
selected, were representative of the total population of services provided in each
call site, and produced valid projections
The IRS Office of Statistics of Income (SOI) helps
the CQRS function select a statistical sample of calls with which to report accuracy
results. It uses an algorithm to
determine a statistically valid sample size for each call site and call product
line (tax accounts or tax law). The
sample is further stratified by date and time for each reviewer. CQRS function management appropriately
followed the Office of SOI sampling plan and implemented an additional sampling
plan by application to ensure the call site sample also reflected the types of
services provided by the call site. CQRS
function management adhered to and exceeded the Fiscal Year 2005 sampling plan by
evaluating 32,527 tax account and 12,041
tax law call segments for accuracy.
Established processes and internal controls
ensure consistent and reliable results are documented in the NQRS database
The Customer
Account Services organization’s quality assurance manual provides procedures
for program quality reviews, call site quality reviews, and front-line manager
evaluative employee reviews. The
guidance is very broad to allow for all review levels. Specific day-to-day guidelines are not
provided in the manual. For
example, internal controls include:
·
Monitoring
adherence to the Office of SOI call site sampling plan provided and
implementing an additional sampling plan by application to ensure evaluated
call segments are in line with call traffic by service provided/application.
·
Ensuring
NQRS database integrity by limiting access to the database and properly
establishing user profiles to ensure appropriate capabilities and
authorizations for creating, editing, and deleting DCI records.
·
Developing
a web site for CQRS function managers and reviewers that contains the CQRS function’s
internal policies and procedures, Office of SOI information, application
schedules, monitoring instructions, general and specific attribute coding
guidelines, consistency exercises, group meeting minutes, forms, and training
links.
·
Establishing
various managerial reviews of individual reviewers to ensure compliance with
the sampling plan and attribute coding consistency by performing online or
joint monitoring with the reviewer and closed case reviews (similar to our
review of comparing the transcripts to the DCI records).
Although internal
controls are sufficient to ensure the overall quality results of the NQRS database,
not all internal controls are documented.
Numerous processes have been developed over time and need to be
documented. For example, the CQRS function
Program Manager has established numerous reviews that provide critical internal
controls to ensure the consistency of coding and accuracy of the information entered
into the NQRS database. The review
requirements or procedures for the reviews have not been documented. If the Program Manager leaves the CQRS
function or the organizational structure changes, there will be no centralized and
documented source with which to ensure identification and continuation of established
processes. This could affect the efficiency
and effectiveness of CQRS function operations.
According to the
Government Accountability Office, internal controls and all transactions and
other significant events need to be clearly documented, and the documentation
should be readily available for examination. The documentation should appear in management
directives, administrative policies, or operating manuals and may be in paper
or electronic form. All documentation
and records should be properly managed and maintained.[12]
The CQRS function has
used the method of evaluating telephone operations quality through the use of
coding attributes since 1998. Documentation
of internal controls and the experience and expertise gained by CQRS function management
should be used for identification of best practices and made available to other
functions as the Embedded Quality process is implemented by other functions and
programs.
Recommendation
Recommendation 1: The Commissioner, Wage and Investment Division, should ensure internal controls are centrally documented to ensure the quality review system continues effective and efficient operations.
Management’s Response: IRS management agreed with our recommendation and stated they completed the action on May 30, 2006, to document the additional managerial practices and internal controls in the CQRS Handbook.
Contact Recording Is Being Implemented, and
Transcripts Will No Longer Be Retained
In June 2006, the CQRS function plans to implement Contact Recording[13] for call segments; recordings will be retained only at the individual
call sites and generally will be erased within 6 months. Recordings will replace transcripts, and
transcripts will no longer be retained. Without
transcripts and/or a longer retention period for the recordings, the
performance measures for Toll-Free Accounts and Tax Law Customer Accuracy Rates
will generally not be auditable.
CQRS function management does not believe the recordings will be needed to support the NQRS database because the DCI records are being retained for 3 years. However, DCI records do not provide sufficient information with which to validate the results of a live or recorded contact. Without the paper transcripts or recordings, a third party will be unable to review or validate the performance measures obtained from the NQRS database. In fact, CQRS function managers currently use reviewers’ transcripts when evaluating CQRS function reviews. A paper transcript or the future recording will be needed to validate the information in the NQRS database.
If Contact
Recording had been operational in all call sites during Fiscal Year 2005,
approximately 31 million call segments would have been recorded. The CQRS function reviewers evaluated 32,527
tax account and 12,041 tax law call segments for Customer Accuracy. Approximately 45,000 recordings out of the
population of approximately 31 million recorded tax account and tax law
call segments would need to be retained to support the 91.5 percent
Toll-Free Accounts and 89.0 percent Toll-Free Tax Law Customer Accuracy Rates
reported in Fiscal Year 2005. The CQRS function currently has approximately
45,000 paper transcripts stored in its offices; those transcripts will be
destroyed in October 2006. Once Contact
Recording is fully operational, CQRS function management plans to no longer
create or maintain transcripts.
There are currently no clear guidelines on whether the transcripts or recordings used for reporting quality come under the Federal Government record retention requirements. The IRS is also implementing Contact Recording in its Taxpayer Assistance Centers (local IRS offices that provide face-to-face tax assistance). We will be conducting a review of that implementation and will address whether transcripts and recordings meet the definition of a record that is subject to Federal Government record retention requirements. Therefore, we are making no recommendations regarding record retention at this time.
Appendix I
Detailed Objective,
Scope, and Methodology
The overall objective of this review was to determine whether data used by the Wage and Investment Division to calculate the Toll-Free Accounts and Tax Law Customer Accuracy Rates were valid and reliable. To accomplish our objective, we:
I. Determined the purpose of the Centralized Quality Review Staff (CQRS) function and its role in providing information for the Internal Revenue Service (IRS) Wage and Investment Division Strategy and Program Plan and the IRS Balanced Measurement System.[14]
II. Determined whether the sampling plan used by the CQRS function will provide statistically valid quality measures by obtaining and reviewing the Fiscal Year 2005 sampling plan, consulting with a contract statistician, and determining whether the plan was being followed.
III. Determined whether the Data Collection Instrument (DCI) records[15] on the National Quality Review System (NQRS) database[16] accurately document the monitored calls so quality measures are accurately calculated.
A. Accessed the NQRS database to identify the population of 17,952 Accounts-General DCI records and 8,396 Tax Law-General DCI records that were used to calculate Customer Accuracy during the period January 1 through June 30, 2005, and extracted limited header information (DCI Number, CQRS Reviewer Number, Closing Date, and Review Date).
B. Determined a minimum statistical sample size of 98 Accounts-General DCI records and 98 Tax Law-General DCI records using a sampling methodology similar to that of the IRS (90 percent confidence level, 90 percent expected accuracy rate, and +5 percent precision margin).
C. To identify the cases for each sample (Sample DCI Numbers), generated 125 random numbers between 1 and 17,952 to select the Accounts-General DCI records, generated another 125 random numbers between 1 and 8,396 to select the Tax Law-General DCI records, and obtained 250 corresponding call transcripts prepared by CQRS function reviewers.
D. Evaluated the reliability of the selected DCI records and determined whether third-party reviewers have sufficient facts to arrive at the same conclusions by reviewing and comparing 114 transcripts to the related DCI Header, Attribute, and Remarks sections. Transcripts were not reviewed in selection order rendering our sample judgmental. Based on the results of reviewing a judgmental sample of 50 Accounts-General and 64 Tax Law-General transcripts and DCI records and identified internal controls established, we discontinued our review.
E. Identified potential errors in transcripts and DCI records and discussed potential errors with CQRS function management to obtain agreement to the facts.
IV. Evaluated CQRS function managerial oversight of the review process (i.e., what internal controls are in place to validate the results and ensure the reliability of the data).
A. Determined the process used to assign reviewers to product lines and applications and whether the process adhered to the sampling plan. We also determined how section managers ensured reviewers correctly documented DCI records and evaluated whether the process was effective to ensure valid results.
B. Determined the procedures used to ensure adherence to the sampling plan (by reviewer, application, and call site).
C. Determined the procedures used to control call site rebuttals and evaluated whether the process was effective to document changes to the NQRS database.
D. Determined the process used to modify and delete DCI records.
V. Determined what NQRS database reports and processes are used to ensure reliable and valid results.
A. Generated select reports, evaluated them for accuracy, and identified the specific DCI records that comprise data captured in the reports.
B. Reviewed DCI records’ attributes to determine whether they were calculated correctly and rolled up into the reports.
C. Traced the sampled DCI records (from Step III.C.) to determine whether they were captured correctly in respective reports.
D. Identified variances and discussed with program analysts the process used to correct discrepancies and determined the impact on the reliability of accuracy measures.
VI. Evaluated internal controls over the NQRS database.
A. Identified the types of validity and maintenance tests performed and how program calculations were validated (e.g., customer accuracy rates, precision margin).
B. Determined whether internal controls, all transactions, and other significant events were clearly documented and the documentation was readily available for examination.
Appendix II
Major Contributors
to This Report
Michael
E. McKenney, Assistant Inspector General for Audit (Wage and Investment Income
Programs)
Augusta
R. Cook, Director
Paula
W. Johnson, Audit Manager
Lynn
Faulkner, Lead Auditor
Jack
Forbus, Senior Auditor
Sharon
Shepherd, Senior Auditor
Jerry
Douglas, Auditor
Kathy
Henderson, Auditor
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Deputy Commissioner
for Services and Enforcement SE
Chief, Agency-Wide
Shared Services OS:A
Chief Financial
Officer OS:CFO
Deputy
Commissioner, Wage and Investment Division
SE:W
Associate Chief
Financial Officer for Corporate Performance Budgeting OS:CFO:CPB
Director, Business
Systems Planning, Wage and Investment Division
SE:W:BSP
Director, Customer
Account Services, Wage and Investment Division
SE:W:CAS
Director, Strategy and Finance, Wage and Investment Division SE:W:S
Chief, Performance Improvement, Wage and Investment Division SE:W:S:PI
Chief,
Program Coordination and Support, Wage and Investment Division SE:W:CAS:PCS
Director,
Accounts Management, Wage and Investment Division SE:W:CAS:AM
Director,
Joint Operations Center, Wage and Investment Division SE:W:CAS:JOC
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis RAS:O
Office of Management Controls OS:CFO:AR:M
Audit Liaisons:
Chief, Agency-Wide Shared Services OS:A:F
Chief Financial Officer OS:CFO
Chief Information Officer OS:CIO:SM:PO
Senior Operations Advisor, Wage and Investment
Division SE:W:S
Appendix IV
National Quality Review System Data
Collection Instrument
The Centralized Quality Review Staff function
is responsible for collecting data that will provide a basis for measuring and
improving program effectiveness.
Organizational performance results data are housed in the National
Quality Review System (NQRS) database.
Data from the NQRS database are used to report Toll-Free Accounts and
Tax Law Customer Accuracy Rates to external stakeholders.
Centralized Quality Review Staff function
reviewers remotely monitor a statistical sample of incoming call segments to
evaluate the accuracy of the actions taken and responses provided by assistors to
customers and enter the results into the NQRS database using a Data
Collection Instrument record. The NQRS database uses attributes to calculate
five quality measures: Timeliness,
Professionalism, Customer Accuracy, Regulatory Accuracy, and Procedural Accuracy.
Figure 1 provides an example
of a Data Collection Instrument record of a call relating to a request for tax
forms. The Quality Assurance Metrics section
provides the calculation of the ratio of defects to opportunities. For example, three Procedural Accuracy
Attributes were coded (Attribute Code 003, 624, and 706 as shown on the last
page). The reviewer marked each
attribute as “Yes,” indicating the assistor met each quality standard. Therefore, the assistor received 100 percent
for Procedural Accuracy.
Figure 1 was removed
due to its size. To see Figure 1, please
go to the Adobe PDF version of the report on the TIGTA Public Web Page.
Appendix V
Management’s Response to the Draft Report
The response
was removed due to its size. To see the
response, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
[1] A service is a telephone interaction between an assistor and a customer. The call is answered by an assistor. The customer’s question is addressed, so a service is provided. If the same customer has an additional question or issue and is transferred to another area/assistor, an additional service is provided.
[2] Contact Recording captures and records employee/customer interaction, synchronized with computer screen activity, for replay and quality review.
[3] A service is a telephone interaction between an assistor and a customer. The call is answered by an assistor. The customer’s question is addressed, so a service is provided. If the same customer has an additional question or issue and is transferred to another area/assistor, an additional service is provided.
[4] Pub. L. No. 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 U.S.C., 31 U.S.C., and 39 U.S.C.).
[5] The reference to “the Service” in Figure 1 refers to the IRS.
[6] The IRS manages call traffic by key segments in the “life” of a call (e.g., the segment from the time a screener answered the call to when it was transferred to an assistor or from the time an assistor connected to the customer to the time the customer was transferred or the call ended).
[7] The DCI Number is systemically generated from the NQRS database when the CQRS function reviewer creates a new record to input his or her evaluation of a monitored call.
[8] This Guide is used by assistors to respond to telephone tax law inquiries on the topics covered in the Guide.
[9]
[10] Capital Gains and Losses (Schedule D).
[11] Overall, 83 DCI records were fully supported by the transcripts, and 31 DCI records contained 1 or multiple accuracy (9), attribute (13), or header (24) errors.
[12] Standards for Internal Control in the Federal Government (GAO/AIMD-00-21.3.1, dated November 1999).
[13] Contact Recording captures and records employee/customer interaction, synchronized with computer screen activity, for replay and quality review.
[14] The Balanced Measurement System includes three measures balanced among customer satisfaction, employee satisfaction, and business results (comprised of work quantity and quality measures) and reflects the IRS’ mission and strategic goals and priorities.
[15] A DCI is used to identify and document the quality
standard attributes satisfied or not satisfied by the assistors to determine
whether the answers provided were accurate.
[16] Organizational performance results are housed in the NQRS database. The IRS uses data from the NQRS database to report Toll-Free Accounts and Tax Law Customer Accuracy Rates to external stakeholders.