The Internal Revenue Service Should Ensure That Its Data on the Treasury’s Performance Reporting System Have Been Verified and Validated

 

November 2001

Reference Number:  2002-10-027

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

November 30, 2001

 

 

MEMORANDUM FOR COMMISSIONER ROSSOTTI

 

FROM:     Pamela J. Gardiner /s/ Pamela J. Gardiner

                Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - The Internal Revenue Service Should Ensure That Its Data on the Treasury’s Performance Reporting System Have Been Verified and Validated (Audit # 200110007)

 

This report presents the results of our review of the Internal Revenue Service’s (IRS) controls over data submitted to the Executive Management Support System (EMSS) and the Organizational Performance Division (OPD) to ensure valid and reliable information are made available for use in executive management decisions.  Our overall objective was to assess the reliability of the performance data the Department of the Treasury receives from the IRS.

In summary, we found that submitting the data to the Treasury’s Performance Reporting System through the EMSS Office or OPD does not introduce great risk that the data might be corrupted.  However, we are concerned about the definitions of some measures and the lack of documented validation and verification of the data. 

Management’s Response:  IRS management agrees with the recommendations presented and will require each Operating Division Commissioner to approve the year-end data submitted for reporting to the Department of the Treasury.  In addition, the IRS is requiring submission of supporting documentation on reported values of all critical measures and indicators and is working to develop datamarts to house the data and make it available for use by the Business Performance Management System web-based report applications.  In the interim, the OPD will develop a transmittal document to be used by the Divisions when submitting their monthly data.  A designated official in each division will sign and forward the transmittal indicating the data have been verified.  Management’s complete response to the draft report is included as Appendix IV.

Copies of this report are also being sent to the IRS managers who are affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs), at (202) 622-8500.

Table of Contents

Background

Data Reliability Rests With Each Operating Division and Unit

Recommendation 1:

Recommendation 2:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Management’s Response to the Draft Report

Background

The Government Performance and Results Act of 1993 (GPRA) requires that federal agencies establish performance measures.  Additionally, the GPRA requires the measures be meaningful and that the agencies must “describe the means to be used to verify and validate measured values.”  The acting Chief Financial Officer for the Department of the Treasury requested a review of the Internal Revenue Service’s (IRS) GPRA performance data that are submitted to the Treasury’s Performance Reporting System (PRS).  The IRS submits its performance measures and workload indicators to the Treasury’s PRS on a mid-year and annual basis.  In Fiscal Year (FY) 2000, the IRS reported 105 performance measures and workload indicators; in FY 2001, the number was reduced to 65.

Within the IRS, the operating divisions and functional units produce their performance measures and workload indicators and report them to the Executive Management Support System (EMSS) Office and the Organizational Performance Division (OPD) monthly. 

To assess the reliability of the data the Department of the Treasury receives from the IRS, we reviewed the controls established within the EMSS Office and the OPD to verify and validate the performance data.  We held discussions with EMSS and OPD staffs in the Manhattan, New York, and the National Headquarters offices, respectively.  In addition, we met with several IRS staff members who are responsible for gathering and reporting operating division performance and workload indicators (hereafter we refer to these as data providers) in Washington, D.C., and Atlanta, Georgia.  We conducted our fieldwork from May through June 2001 in accordance with Government Auditing Standards.

 

Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

Data Reliability Rests With Each Operating Division and Unit

We evaluated the controls within the EMSS Office and the OPD and interviewed data providers.  We determined that they do not have a validation process for the performance measures and workload indicators received from the operating divisions and functional units.  Instead, the EMSS staff, OPD staff, and data providers relied on the operating divisions and functional units to have properly validated the data.  Additionally, the EMSS and OPD staffs and the data providers do not have a formal verification process in place.  The staffs and the data providers do use an informal process to try to ensure that all the data submitted to the Treasury’s PRS are accurate and accounted for.

FY 2000 PRS data

In FY 2000, all the IRS performance measures and workload indicators were taken from the EMSS and forwarded to the Treasury.  The data were submitted electronically by the data providers to the EMSS staff and arrived in any number of formats including, but not limited to, ASCII, Word, Excel, Access, or e-mail text.  The EMSS staff then converted the data for input into the EMSS model. 

The EMSS staff did not have any formal review procedures.  Instead, they conducted an informal verification by “eyeballing” the data, which included seeing if it appeared accurate, checking for obvious errors, and performing trend analyses.  If errors or anomalies were detected, the EMSS staff sent the data back to the data provider for correction.

FY 2001 PRS data

In FY 2001, the process changed.  The IRS decided to upgrade the old EMSS to a new web-based application.  In the new process, the data providers are required to submit the performance data in a “Monthly Business Performance Summary” spreadsheet to the OPD following the guidelines developed by the OPD.

Once the data are received in the OPD, the analysts also visually examine the figures for accuracy and obvious errors and then perform trend analyses.  If any problems are found, the data are sent back to the data providers for correction.  Once the correct figures are input on the spreadsheet, a senior manager reviews the data using the same informal technique.

Under both methods of gathering and forwarding performance data, the responsibility for verification and validation of the data rests with the operating divisions and functional units. 

Submitting the data to the Treasury’s PRS through the EMSS Office or the OPD does not introduce great risk that the data might be corrupted.  We identified one confirmed instance in which data forwarded to the Treasury’s PRS differed from source data, although the difference was very slight (the number reported was 65.30 percent; it should have been 66.96 percent).  Also, we identified two instances where the performance measures reported on the Treasury’s PRS could not be supported.  The IRS documentation available for the two measures did not match the figures recorded on the Treasury’s PRS.  In our opinion, the discrepancies noted did not constitute a material deficiency.

However, without documented procedures for the staff to follow, there is an increased possibility of having inconsistencies in the methodology used to gather and verify data.  In addition, there is an increased risk that inaccurate data could be submitted to the Treasury’s PRS.

The operating divisions and functional units need to ensure that their performance data are properly qualified

The Treasury Inspector General for Tax Administration (TIGTA) has issued several audit reports within the last 2 years that have identified problems with some IRS performance measures.  A common finding was that the measures, as reported, needed to be properly qualified.  For example, the IRS measure of Examination Customer Satisfaction would by definition imply all Examination customers, when only individual taxpayers involved in the examination process were surveyed.  The survey population excluded corporate, estate, excise, and gift tax returns examined.  Similar conditions were found to exist in the six other audits of customer satisfaction surveys.

More recently, two reports have been issued on the IRS’ Toll-Free telephone service performance measures.  In one report, the TIGTA reported that the indicators for toll-free quality did not include all elements of the IRS’ toll-free system.  Items such as questions answered by a return telephone call or electronic mail (known as r-mail), calls from Spanish-speaking taxpayers, and calls to the IRS’ automated telephone systems were excluded.  In the other report, the TIGTA reported that the measures did not address how long customers waited to receive assistance, service levels for assistor-answered calls and automated-answered calls are not separated, and the three principal Customer Account Services toll-free telephone lines are also not reported on separately.

The operating divisions and functional units have concentrated their efforts on reorganizing and staffing.  There is no requirement to have operating divisions and functional units provide documentation to the data provider to show that the performance measures and workload indicators have been properly verified and validated.

Last year, the IRS Deputy Commissioner (Modernization) informed selected units that, “Concerns about the timeliness, consistency, and validity of information . . . caused me to institute a mandate that all information be provided systemically.”  The Deputy Commissioner also mandated that the individual units will be responsible for validating their data.  Although the Department of the Treasury is receiving the IRS’ critical performance measures and workload indicators on a mid-year and annual basis, it has no assurance that the data have been validated.

The OPD plans to address the issue of performance data in part by working with a vendor to assess how IRS performance data are collected and reported.  Also, the OPD plans to provide updated instructions to all IRS operating divisions and units in the fall of 2001.  The instructions are intended to require the divisions and units to document any verification procedures developed.  This is expected to improve the quantity and quality of information provided in the data dictionary.  In addition, the OPD will work with the divisions to identify areas that are critical to data gathering and document processes that are in place to ensure the verification of the data gathering methods used.

Recommendations

1.      The Director, OPD, should continue to develop and recommend to the IRS Commissioner procedures that each operating division and functional unit will follow on a recurring basis to ensure its performance measures and workload indicators are properly verified and validated.  This verification and validation should also be properly documented.  Any data limitations identified during this process should be properly disclosed.

Management’s Response:  IRS management has implemented procedures for reporting year-end data (issued on September 26, 2001) requiring each Division Commissioner to approve the data and supporting documentation submitted for reporting on critical measures to the Department of the Treasury.  In addition, the OPD is working with the divisions to improve how measures are validated. 

2.      In the interim, the OPD staff should have the operating divisions and functional units review the consolidated monthly performance spreadsheet and indicate their agreement that the figures in the spreadsheet are correct before they are forwarded to the Department of the Treasury.

Management’s Response:  IRS management will develop a transmittal document to be used by the Divisions for submission of their monthly data.  The transmittal will consist of a statement that the data being transmitted have been verified and will be accompanied by an electronic signature of an official designated by each division.  Additionally, IRS management plans to automate the data capture and reporting processes through development of functional datamarts.  These datamarts are designed to accept and house data from designated systems and make it available for use by the Business Performance Management System web-based report applications.

Appendix I

 

Detailed Objective, Scope, and Methodology

 

Our overall objective was to assess the reliability of the data the Department of the Treasury receives from the Internal Revenue Service’s (IRS) Executive Management Support System (EMSS) and the Organizational Performance Division (OPD).  To do so, we reviewed the controls established by the EMSS Office and the OPD to verify and validate the IRS performance data received from the data providers in the IRS operating divisions and units.  The scope of this audit did not include transactional testing of data source systems within the divisions or units.  To accomplish our objective, we conducted the following tests:

I.                    Determined the process used to identify the performance measures and workload indicators to be included in the EMSS model and the approval process (verification and validation) used prior to the data being submitted to the Department of the Treasury’s Performance Reporting System (PRS).

A.     Interviewed the acting EMSS manager and gained an understanding of how the data are handled when received from the data providers.

1.      Reviewed procedures for data received from the data providers. 

a)            Obtained a copy of the Data Flow and Data Model Description document and evaluated it.  Determined if any changes had been made.

b)            Identified the measures in the Data Flow and Data Model Description documents and determined if any are related to the critical 65 performance measures and workload indicators submitted to the Department of the Treasury.

2.      Evaluated the methodology used in the EMSS Office to validate and verify data and determined if it was adequate.

3.      Identified three data developers in the EMSS Office and the projects they are responsible for.

B.     Interviewed three of the EMSS personnel (data developers) and gained an understanding of the process used to validate and verify the data received from the data providers.

1.      Identified critical measures associated with the data developers that were interviewed.

2.      Reviewed procedures used to verify and validate the performance measures and workload indicators received from the data providers.

a)            Determined the format the performance measures and workload indicators were in when received by the EMSS staff.

b)            Evaluated the procedures used to load the performance measures and workload indicators in the EMSS model.

C.     Interviewed personnel in the OPD to understand how the performance measures and workload indicators are verified, validated, and approved prior to being submitted to the Department of the Treasury’s PRS.

1.      Determined who receives the performance measures and workload indicators in the OPD and what their responsibilities are.

2.      Evaluated procedures for verifying and validating performance measures and workload indicators once received from the data providers.

3.      Evaluated the approval process used in the OPD for the performance measures and indicators before they are submitted to the Department of the Treasury’s PRS.

4.      Evaluated how the performance measures and workload indicators are submitted to the Department of the Treasury’s PRS, once approved.

D.     Interviewed seven data providers (within the offices of the National Taxpayer Advocate, Wage and Investment Division, Tax Exempt and Government Entities Division, and Small Business/Self-Employed Division) and evaluated their processes for verifying and validating the performance measures and workload indicators prior to submitting them to the EMSS Office and OPD.

1.      Identified the methodology used to calculate performance measures and workload indicators.

2.      Evaluated procedures used for verifying and validating performance measures and workload indicators.

3.      Determined the method and format used to submit performance measures and workload indicators to the EMSS Office and OPD.

4.      Determined which of the seven data providers had critical performance measures and workload indicators reported in the Department of the Treasury’s PRS and traced the measures and indicators back to the data provider’s source documents to verify the accuracy of the measures and indicators reported.

5.      Identified discrepancies and determined their cause and effect.

II.                 Determined the process used within the Department of the Treasury to receive the EMSS data and to upload it into the PRS.

A.     Interviewed Department of the Treasury PRS personnel to gain an understanding of the process for receiving the performance measures and workload indicators from the IRS.

B.     Reviewed procedures for receiving performance measures and workload indicators from the IRS.

C.     Determined if any analysis is performed on the data received from the IRS.

D.     Identified the cut-off period for inclusion of performance measures and workload indicators in the Department of the Treasury’s PRS.

E.      Obtained a copy of the Department of the Treasury’s PRS Fiscal Year (FY) 2001 mid-year and FY 2000 year-end reports and determined whether all the IRS critical measures and workload indicators had been accounted for.

Appendix II

 

Major Contributors to This Report

 

Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

John R. Wright, Director

Kevin Riley, Audit Manager

Ken Henderson, Senior Auditor

Charles Ekunwe, Auditor

Gene Luevano, Auditor

Appendix III

 

Report Distribution List

 

Commissioner  N:C

Deputy Commissioner  N:DC

Commissioner, Large and Mid-Size Business Division  LM

Commissioner, Small Business/Self-Employed Division  S

Commissioner, Tax Exempt and Government Entities Division  T

Commissioner, Wage and Investment Division  W

Director, Organizational Performance Division  N:CFO:O

Director, Systems Development  M:I:SD

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  N:ADC:R:O

Office of Management Controls  N:CFO:F:M

Audit Liaisons:

            Large and Mid-Size Business Division  LM

Legislative Affairs  CL:LA

National Taxpayer Advocate  TA

            Small Business/Self-Employed Division  S

            Tax Exempt and Government Entities Division  T

Wage and Investment Division  W

 

Appendix IV

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.