The Internal Revenue Service Should Ensure That Its Data on
the Treasury’s Performance Reporting System Have Been Verified and Validated
November 2001
Reference Number: 2002-10-027
This report has cleared the Treasury Inspector
General for Tax Administration disclosure review process and information
determined to be restricted from public release has been redacted from this
document.
November
30, 2001
MEMORANDUM FOR
COMMISSIONER ROSSOTTI
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for Audit
SUBJECT: Final Audit Report - The Internal Revenue
Service Should Ensure That Its Data on the Treasury’s Performance Reporting
System Have Been Verified and Validated (Audit # 200110007)
This
report presents the results of our review of the Internal
Revenue Service’s (IRS) controls over data submitted to the Executive
Management Support System (EMSS) and the Organizational Performance Division
(OPD) to ensure valid and reliable information are made available for use in
executive management decisions. Our
overall objective was to assess the reliability of the performance data the
Department of the Treasury receives from the IRS.
In summary, we found that submitting
the data to the Treasury’s Performance Reporting System through the EMSS Office
or OPD does not introduce great risk that the data might be corrupted. However, we are concerned about the
definitions of some measures and the lack of documented validation and
verification of the data.
Management’s Response: IRS management
agrees with the recommendations presented and will require each Operating
Division Commissioner to approve the year-end data submitted for reporting to
the Department of the Treasury. In
addition, the IRS is requiring submission of supporting documentation on
reported values of all critical measures and indicators and is working to
develop datamarts to house the data and make it available for use by the
Business Performance Management System web-based report applications. In the interim,
the OPD will develop a transmittal document to be used by the Divisions when
submitting their monthly data. A
designated official in each division will sign and forward the transmittal
indicating the data have been verified.
Management’s complete response to the draft report is included as
Appendix IV.
Copies
of this report are also being sent to the IRS managers who are affected by the
report recommendations. Please contact
me at (202) 622-6510 if you have questions or Daniel R. Devlin, Assistant
Inspector General for Audit (Headquarters Operations and Exempt Organizations
Programs), at (202) 622-8500.
Data Reliability
Rests With Each Operating Division and Unit
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Management’s Response to the Draft Report
The Government Performance and
Results Act of 1993 (GPRA) requires that federal agencies establish performance
measures. Additionally, the GPRA requires the measures be meaningful and that the
agencies must “describe the means to be used to verify and validate measured
values.” The acting Chief Financial
Officer for the Department of the Treasury requested a review of the Internal
Revenue Service’s (IRS) GPRA performance data that are submitted to the
Treasury’s Performance Reporting System (PRS).
The IRS submits its performance measures and workload indicators
to the Treasury’s PRS on a mid-year and annual basis. In Fiscal Year (FY) 2000, the IRS reported 105 performance
measures and workload indicators; in FY 2001, the number was reduced to 65.
Within the IRS, the operating divisions and functional units produce their performance measures and workload indicators and report them to the Executive Management Support System (EMSS) Office and the Organizational Performance Division (OPD) monthly.
To assess the reliability of the data the Department of the
Treasury receives from the IRS, we reviewed the controls established within the
EMSS Office and the OPD to verify and validate the performance data. We held discussions with EMSS and OPD staffs
in the Manhattan, New York, and the National Headquarters offices,
respectively. In addition, we met with
several IRS staff members who are responsible for gathering and reporting
operating division performance and workload indicators (hereafter we refer to
these as data providers) in Washington, D.C., and Atlanta, Georgia. We conducted our fieldwork from May through
June 2001 in accordance with Government
Auditing Standards.
Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
We evaluated the controls within the EMSS Office and the OPD and interviewed data providers. We determined that they do not have a validation process for the performance measures and workload indicators received from the operating divisions and functional units. Instead, the EMSS staff, OPD staff, and data providers relied on the operating divisions and functional units to have properly validated the data. Additionally, the EMSS and OPD staffs and the data providers do not have a formal verification process in place. The staffs and the data providers do use an informal process to try to ensure that all the data submitted to the Treasury’s PRS are accurate and accounted for.
FY 2000 PRS data
In FY 2000, all the IRS performance measures and
workload indicators were taken from the EMSS and forwarded to the
Treasury. The data were submitted
electronically by the data providers to the EMSS staff and arrived in any
number of formats including, but not limited to, ASCII, Word, Excel, Access, or
e-mail text. The EMSS staff then
converted the data for input into the EMSS model.
The EMSS staff did not have any formal review
procedures. Instead, they conducted an
informal verification by “eyeballing” the data, which included seeing if it
appeared accurate, checking for obvious errors, and performing trend
analyses. If errors or anomalies were
detected, the EMSS staff sent the data back to the data provider for
correction.
FY 2001 PRS data
In FY 2001, the process changed. The IRS decided to upgrade the old EMSS to a new web-based application. In the new process, the data providers are required to submit the performance data in a “Monthly Business Performance Summary” spreadsheet to the OPD following the guidelines developed by the OPD.
Once the data are received in the OPD, the analysts also visually examine the figures for accuracy and obvious errors and then perform trend analyses. If any problems are found, the data are sent back to the data providers for correction. Once the correct figures are input on the spreadsheet, a senior manager reviews the data using the same informal technique.
Under both methods of gathering and forwarding performance data, the responsibility for verification and validation of the data rests with the operating divisions and functional units.
Submitting the data to the Treasury’s PRS through the EMSS Office or the OPD does not introduce great risk that the data might be corrupted. We identified one confirmed instance in which data forwarded to the Treasury’s PRS differed from source data, although the difference was very slight (the number reported was 65.30 percent; it should have been 66.96 percent). Also, we identified two instances where the performance measures reported on the Treasury’s PRS could not be supported. The IRS documentation available for the two measures did not match the figures recorded on the Treasury’s PRS. In our opinion, the discrepancies noted did not constitute a material deficiency.
However, without documented procedures for the staff to follow, there is an increased possibility of having inconsistencies in the methodology used to gather and verify data. In addition, there is an increased risk that inaccurate data could be submitted to the Treasury’s PRS.
The operating divisions and functional units need to ensure that their performance data are properly qualified
The Treasury Inspector General for Tax Administration (TIGTA) has issued several audit reports within the last 2 years that have identified problems with some IRS performance measures. A common finding was that the measures, as reported, needed to be properly qualified. For example, the IRS measure of Examination Customer Satisfaction would by definition imply all Examination customers, when only individual taxpayers involved in the examination process were surveyed. The survey population excluded corporate, estate, excise, and gift tax returns examined. Similar conditions were found to exist in the six other audits of customer satisfaction surveys.
More recently, two reports have been issued on the IRS’ Toll-Free telephone service performance measures. In one report, the TIGTA reported that the indicators for toll-free quality did not include all elements of the IRS’ toll-free system. Items such as questions answered by a return telephone call or electronic mail (known as r-mail), calls from Spanish-speaking taxpayers, and calls to the IRS’ automated telephone systems were excluded. In the other report, the TIGTA reported that the measures did not address how long customers waited to receive assistance, service levels for assistor-answered calls and automated-answered calls are not separated, and the three principal Customer Account Services toll-free telephone lines are also not reported on separately.
The operating divisions and functional units have concentrated their efforts on reorganizing and staffing. There is no requirement to have operating divisions and functional units provide documentation to the data provider to show that the performance measures and workload indicators have been properly verified and validated.
Last year, the IRS Deputy Commissioner (Modernization) informed selected units that, “Concerns about the timeliness, consistency, and validity of information . . . caused me to institute a mandate that all information be provided systemically.” The Deputy Commissioner also mandated that the individual units will be responsible for validating their data. Although the Department of the Treasury is receiving the IRS’ critical performance measures and workload indicators on a mid-year and annual basis, it has no assurance that the data have been validated.
The OPD plans to address the issue of performance data in part by working with a vendor to assess how IRS performance data are collected and reported. Also, the OPD plans to provide updated instructions to all IRS operating divisions and units in the fall of 2001. The instructions are intended to require the divisions and units to document any verification procedures developed. This is expected to improve the quantity and quality of information provided in the data dictionary. In addition, the OPD will work with the divisions to identify areas that are critical to data gathering and document processes that are in place to ensure the verification of the data gathering methods used.
1.
The Director, OPD, should
continue to develop and recommend to the IRS Commissioner procedures that each
operating division and functional unit will follow on a recurring basis to
ensure its performance measures and workload indicators are properly verified
and validated. This verification and
validation should also be properly documented.
Any data limitations identified during this process should be properly
disclosed.
Management’s Response: IRS management
has implemented procedures for reporting year-end data (issued on September 26,
2001) requiring each Division Commissioner to approve the data and supporting
documentation submitted for reporting on critical measures to the Department of
the Treasury. In addition, the OPD is
working with the divisions to improve how measures are validated.
2. In the interim, the OPD staff should have the operating divisions and functional units review the consolidated monthly performance spreadsheet and indicate their agreement that the figures in the spreadsheet are correct before they are forwarded to the Department of the Treasury.
Detailed Objective, Scope, and
Methodology
Our overall objective was to assess the reliability of the data the Department of the Treasury receives from the Internal Revenue Service’s (IRS) Executive Management Support System (EMSS) and the Organizational Performance Division (OPD). To do so, we reviewed the controls established by the EMSS Office and the OPD to verify and validate the IRS performance data received from the data providers in the IRS operating divisions and units. The scope of this audit did not include transactional testing of data source systems within the divisions or units. To accomplish our objective, we conducted the following tests:
I.
Determined
the process used to identify the performance measures and workload indicators
to be included in the EMSS model and the approval process (verification and
validation) used prior to the data being submitted to the Department of the
Treasury’s Performance Reporting System (PRS).
A.
Interviewed the
acting EMSS manager and gained an understanding of how the data are handled
when received from the data providers.
1.
Reviewed procedures
for data received from the data providers.
a)
Obtained a copy of
the Data Flow and Data Model Description document and evaluated it. Determined if any changes had been made.
b)
Identified the
measures in the Data Flow and Data Model Description documents and determined
if any are related to the critical 65 performance measures and workload
indicators submitted to the Department of the Treasury.
2.
Evaluated the
methodology used in the EMSS Office to validate and verify data and determined
if it was adequate.
3.
Identified
three data developers in the EMSS Office and the projects they are responsible
for.
B.
Interviewed
three of the EMSS personnel (data developers) and gained an understanding of
the process used to validate and verify the data received from the data
providers.
1.
Identified
critical measures associated with the data developers that were interviewed.
2.
Reviewed
procedures used to verify and validate the performance measures and workload
indicators received from the data providers.
a)
Determined
the format the performance measures and workload indicators were in when
received by the EMSS staff.
b)
Evaluated
the procedures used to load the performance measures and workload indicators in
the EMSS model.
C.
Interviewed
personnel in the OPD to understand how the performance measures and workload
indicators are verified, validated, and approved prior to being submitted to
the Department of the Treasury’s PRS.
1.
Determined who
receives the performance measures and workload indicators in the OPD and what
their responsibilities are.
2.
Evaluated
procedures for verifying and validating performance measures and workload
indicators once received from the data providers.
3.
Evaluated the
approval process used in the OPD for the performance measures and indicators
before they are submitted to the Department of the Treasury’s PRS.
4.
Evaluated how the
performance measures and workload indicators are submitted to the Department of
the Treasury’s PRS, once approved.
D.
Interviewed seven data providers (within the offices
of the National Taxpayer Advocate, Wage and Investment Division, Tax Exempt and
Government Entities Division, and Small Business/Self-Employed Division) and
evaluated their processes for verifying and validating the performance measures
and workload indicators prior to submitting them to
the EMSS Office and OPD.
1.
Identified the
methodology used to calculate performance measures and workload indicators.
2.
Evaluated
procedures used for verifying and validating performance measures and workload
indicators.
3.
Determined
the method and format used to submit performance measures and workload
indicators to the EMSS Office and
OPD.
4.
Determined which of
the seven data providers had critical performance measures and workload
indicators reported in the Department of the Treasury’s PRS and traced the
measures and indicators back to the data provider’s source documents to verify
the accuracy of the measures and indicators reported.
5.
Identified
discrepancies and determined their cause and effect.
II.
Determined
the process used within the Department of the Treasury to receive the EMSS data and to upload it into the PRS.
A.
Interviewed
Department of the Treasury PRS personnel to gain an understanding of the
process for receiving the performance measures and workload indicators from the
IRS.
B.
Reviewed procedures
for receiving performance measures and workload indicators from the IRS.
C.
Determined if any
analysis is performed on the data received from the IRS.
D.
Identified the
cut-off period for inclusion of performance measures and workload indicators in
the Department of the Treasury’s PRS.
E.
Obtained a copy of
the Department of the Treasury’s PRS Fiscal Year (FY) 2001 mid-year and
FY 2000 year-end reports and determined whether all the IRS critical
measures and workload indicators had been accounted for.
Appendix II
Major Contributors to This Report
Daniel R. Devlin, Assistant
Inspector General for Audit (Headquarters Operations and Exempt Organizations
Programs)
John R. Wright, Director
Kevin Riley, Audit Manager
Ken
Henderson, Senior Auditor
Charles Ekunwe, Auditor
Gene Luevano, Auditor
Appendix III
Commissioner N:C
Deputy Commissioner N:DC
Commissioner, Large and Mid-Size Business Division LM
Commissioner, Small Business/Self-Employed Division S
Commissioner, Tax Exempt and Government Entities Division T
Commissioner, Wage and Investment Division W
Director, Organizational Performance Division N:CFO:O
Director, Systems Development M:I:SD
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaisons:
Large and
Mid-Size Business Division LM
Legislative Affairs CL:LA
National Taxpayer Advocate TA
Small
Business/Self-Employed Division S
Tax Exempt
and Government Entities Division T
Wage and Investment Division W
Appendix
IV
Management’s Response to the
Draft Report
The
response was removed due to its size.
To see the complete response, please go to the Adobe PDF version of the
report on the TIGTA Public Web Page.