Important Progress Has Been Made in Using Research to Improve Programs for Large Businesses, but Challenges Remain

 

August 2004

 

Reference Number:  2004-30-130

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

 

August 12, 2004

 

 

MEMORANDUM FOR COMMISSIONER, LARGE AND MID-SIZE BUSINESS DIVISION

 

FROM:     Gordon C. Milbourn III /s/ Gordon C. Milbourn III

                 Acting Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - Important Progress Has Been Made in Using Research to Improve Programs for Large Businesses, but Challenges Remain  (Audit # 200330015)

 

This report presents the results of our review of the Large and Mid-Size Business (LMSB) Division Research Unit.  The overall objective of this review was to assess the progress the LMSB Division is making in (1) addressing the concerns raised in past studies of the Internal Revenue Service’s (IRS) research operations, (2) integrating its research operation into key business decision-making processes, (3) using research for improving performance on strategic initiatives, and (4) aligning its research processes with best practices adopted by other research programs.

In summary, studies of the IRS between 1995 and 2001 identified concerns about whether its research programs were generating research that was both relevant and effective.  Four of the five studies we reviewed, for example, discussed the need to increase customer involvement in the research process so that the areas of greatest interest and need were being addressed, and to establish measures for evaluating the effectiveness of research results.  Four of the five studies also surfaced concerns about whether the research staff had the appropriate mix of skills and whether the processes used for managing the research were adequate.

In conjunction with the agency’s overall reorganization effort, the IRS developed a blueprint in March 2000 to guide how its research programs would be modernized by establishing research units in each of its operating divisions.  We determined that the LMSB Division Research Unit is providing the organizational structure and customer focus to address areas with the greatest interest and necessity, while implementing the IRS’ vision for research as outlined in the blueprint.  To date, LMSB Division executives have focused considerable effort and made important progress to integrate the Research Unit’s results into key decision-making processes and to improve the effectiveness of strategic initiatives for large businesses.  As reflected in its strategic planning documents, the LMSB Division expects the Research Unit to provide, and relies on it for, decision support to improve its programs, particularly those that enforce and encourage voluntary compliance, as well as to help answer questions about how best to spend its over $700 million annual budget.  For example, the Research Unit provides analysis for the LMSB Division’s strategic assessments that identifies trends, issues, and problems along with proposed operating priorities.  Further, it is developing up-to-date information for understanding large businesses’ compliance with the tax laws and is using this information to build and refine automated, risk-based scoring systems for identifying and prioritizing potentially noncompliant large businesses for examination of their tax returns. 

In regard to its Research Unit employees, the LMSB Division, like many areas within the Federal Government, faces the challenge of acquiring needed skills and competencies to improve program effectiveness.  To meet this challenge, a Training and Organizational Assessment Project was completed in Calendar Year (CY) 2002 that identified potential gaps in training and skills for the LMSB Division’s research staff.  Additionally, business cases were made early in CY 2004 for hiring a statistician, an economist, and other individuals with expertise in research disciplines to maintain and develop tools for working with an integrated network of compliance databases that are being created.

While important progress has been made to use research for improving programs devoted to large businesses, challenges remain concerning internal operational processes and procedures in three areas.  First, the LMSB Division processes for managing research do not always follow best practices, particularly in the area of enhancing the credibility of research results.  Office of Management and Budget guidelines, for instance, indicate that having the capability of replicating research can enhance the credibility of results, increase opportunities for improvement, and avoid perceptions of bias.  However, we determined that key source documents needed to independently confirm the validity of the LMSB Division research results were not consistently maintained in project files.

Second, the LMSB Division could improve the process for evaluating the effectiveness of research efforts.  Although the LMSB Division can show benefits from some of its research projects, it does not have a systemic process to measure research outcomes against intended results.  As a result, the LMSB Division cannot readily determine whether specific projects are achieving their objectives or answer important questions such as “What impact is the research having?”  Finally, research projects could be better managed by using project management techniques.  These techniques are designed to develop plans and control systems to ensure research is completed on time and within budget so that needed information is available for executive decision-making purposes.

Given the expectations of the LMSB Division Research Unit and the significant level of resources that are depending on its work, we made three recommendations to enhance the processes for managing the research.  The Director, Strategy, Research, and Program Planning, LMSB Division, should develop, define, and adopt (1) procedures that will ensure all research support information is consistently maintained in project files, (2) a systematic process for measuring research efforts, and (3) guidelines for determining whether individual projects and the related activities are on time and within budget.

Management’s Response:  IRS management agreed with our findings, and their corrective actions are responsive to our recommendations.  The Commissioner, LMSB Division, responded that research processes and procedures will be improved by developing and issuing guidelines for maintaining project documentation, measuring research project results, and implementing a Time Reporting System.  Management’s complete response to the draft report is included as Appendix VI.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Philip Shropshire, Acting Assistant Inspector General for Audit (Small Business and Corporate Programs), at (215) 516-2341.

 

Table of Contents

Background

The Large and Mid-Size Business Division Is Implementing the New Research Vision As Planned

The Large and Mid-Size Business Division’s Processes for Managing Research Do Not Always Follow Best Practices

Recommendations 1 and 2:

Recommendation 3:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Organizational Chart of the Internal Revenue Service’s Research Operation

Appendix V – The Internal Revenue Service’s Quality Standards for Research Projects

Appendix VI – Management’s Response to the Draft Report

 

Background

Studies of the Internal Revenue Service (IRS) between 1995 and 2001 identified concerns about whether its research programs were generating research that was both relevant and effective.  Four of the five studies we reviewed, for example, discussed the need to increase customer involvement in the research process so that the areas of greatest interest and need were being addressed, and to establish measures for evaluating the effectiveness of research results.  Four of the five studies also surfaced concerns about whether the research staff had the appropriate mix of skills and whether the processes used to manage the research were adequate.

In conjunction with the agency’s overall reorganization effort, the IRS developed a blueprint in March 2000 to guide how its research programs would be modernized by establishing research units in each of its operating divisions.  The units are intended to provide the organizational structure and customer focus for dealing with past concerns over whether research was addressing areas with the greatest interest and necessity.

As planned, the IRS organized its research operation in a decentralized manner with a research unit in each of its four operating divisions and the National Headquarters (NHQ) Office.  A director who is also part of a high-level leadership group called the Servicewide Research Council (SRC) heads each research unit.  In the Large and Mid-Size Business (LMSB) Division, the research director is an executive-level manager who has responsibility for a staff of researchers located in various offices across the country. Appendix IV shows the IRS’ decentralized research operation structure.  It also shows that the Director, NHQ Office of Research, does not have line authority over the research teams in the operating divisions.  Rather, the Commissioners and Directors of Research in each operating division control budget and resources needed for their respective research teams.

The research staff in the NHQ Office of Research works through the SRC to develop and maintain national research databases and conducts long-term research projects of agency-wide interest.  In the operating divisions, researchers concentrate on their specific groups of taxpayers in projects that have relatively short time periods, such as a year.  With a geographically dispersed research staff, the SRC’s role is to support, guide, and coordinate research activities across organizational boundaries.

Specific activities of the SRC may include identifying and meeting training and data needs, as well as promoting and marketing research products to internal and external stakeholders.  The SRC, through its Quality Assurance Council (QAC), also monitors and provides feedback to researchers on the quality of completed projects.  To define research quality, the QAC uses 11 standards for research planning and 8 standards for research reporting.  Each standard also has several key elements that elaborate on the overall standard.  Appendix V has more details on the QAC standards and their associated elements.

This review was performed at the LMSB Division Headquarters in Washington, D.C., and offices in Pittsburgh, Pennsylvania; Buffalo, New York; and Oakland, California, during the period September 2003 through February 2004.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

The Large and Mid-Size Business Division Is Implementing the New Research Vision As Planned

The LMSB Division Research Unit is providing the organizational structure and customer focus to address areas with the greatest interest and necessity, while implementing the IRS’ vision for research as outlined in the blueprint.  To date, the LMSB Division Commissioner and executives have provided the general direction for the Research Unit, while focusing considerable effort and making important progress in integrating research results into key decision-making processes and improving the effectiveness of strategic initiatives for large businesses.  For example, the Research Unit:

• Provides analysis for the LMSB Division’s strategic assessments that identifies trends, issues, and problems along with proposed operating priorities. 
• Identifies potential issues and risk information involving multinational entities to improve international tax administration.
• Evaluates how best to identify and address compliance risks involving complex organizational structures that cross tax periods and entity types.
• Is developing up-to-date information for understanding large businesses’ compliance with the tax laws and using this information to build and refine automated, risk-based scoring systems for identifying and prioritizing potentially noncompliant large businesses for examination of tax returns.  These automated systems will allow the LMSB Division to replace the practice of simply relying on the experience and judgment of examiners to manually screen large business tax returns for noncompliance.

Overall, the Research Unit is implementing the vision, under the direction of LMSB Commissioner and other executives, given to it in conjunction with the IRS’ overall reorganization effort. 

As reflected in its strategic planning documents, the LMSB Division expects and relies on its Research Unit’s results for decision support to improve its programs, particularly those that enforce and encourage voluntary compliance, as well as to help answer questions about how best to spend its over $700 million annual budget. Strategic planning documents state:

[The] LMSB [Division] relies upon the work of our Research component for systems to identify areas of compliance risk, emerging issues, the size and scope of compliance issues, and patterns of non-compliant behavior within our customer base using internal and external information.  Our strategic assessment identified the critical need for a system to deliver a predictive model identifying: 1) taxpayers who have the potential to be influenced by pre-filing products, 2) returns with unreported income, 3) returns likely to include participation in abusive corporate tax shelters, and 4) other factors which are indicators of compliance risk. Research is working to develop a prototype to respond to this need.  Prototype accomplishments could be used immediately and final prototype results will fit within current Modernization blueprints.  In [Fiscal Year] FY 2004 Research will continue work on improving the Form 1065 scoring model and programs to improve our ability to estimate voluntary compliance levels.

At the time of our review, the LMSB Division had also begun to address concerns regarding the skills and competencies of the research staff.  The Research Unit has an annual budget of approximately $7 million and 54 full-time equivalents (FTE) on staff.  Our analysis of the distribution of skills across the staff showed a mix of specialized skills in economics, statistics, computer research, and program analysis.  Researchers have extensive institutional knowledge considering that, on average, they have worked in IRS operations for approximately 20 years. 

In regard to its Research Unit employees, the LMSB Division, like many other areas within the Federal Government, faces the challenge of acquiring needed skills and competencies to improve program effectiveness.  To meet this challenge, a Training and Organizational Assessment Project was completed in Calendar Year (CY) 2002 that identified potential gaps in training and skills for the LMSB Division’s research staff.  Additionally, business cases were made early in CY 2004 for hiring a statistician, an economist, and other individuals with expertise in research disciplines to maintain and develop tools for working with an integrated network of compliance databases that are being created.

While important progress has been made to use research for improving programs devoted to large businesses, challenges remain concerning internal operational processes and procedures in three areas.  First, the LMSB Division processes for managing research do not always follow best practices, particularly in the areas of enhancing the credibility of research results.  Second, the LMSB Division could improve the process for evaluating the effectiveness of research efforts.  Finally, research projects could be better managed by using project management techniques.

Our discussions with LMSB Division officials indicate that the weaknesses we identified in the processes for managing research were due to focusing on other priorities brought on by the transition from the old IRS structure to the new one.  During the transition, for example, officials were focused on simultaneously managing ongoing research projects while assigning staff to the new operating divisions and laying the groundwork for the long-term modernization of the IRS research program.  Consequently, the research staff has been relying on old work processes that were not as effective as they should be, according to prior studies. 

The Large and Mid-Size Business Division’s Processes for Managing Research Do Not Always Follow Best Practices

Among the well-established best practices in the research community is that research should have the capability of being reproduced.  Office of Management and Budget (OMB) guidelines, for example, indicate that having the capability of replicating research can enhance the credibility of results, increase opportunities for improvement, and avoid perceptions of bias. Similarly, the IRS provides through its Records Management Program that, because of the unique nature of research records, most are required to be maintained for several years, while many should be kept permanently.

The credibility of research results can be enhanced by ensuring source information used in research is consistently maintained in project files

We used the IRS’ SRC quality standards and criteria developed by the OMB to evaluate the design, data collection and analysis techniques, and reports from a judgmental sample of six research projects.  Our results showed that, overall, many aspects of the research processes were documented and that the SRC quality standards were followed.  For example, the:

·        Questions to be answered and the basis for the research were stated clearly in the design of the projects.

·        Methods and sources for collecting the data needed for the projects were described and appeared appropriate.

·        Procedures for forming and processing automated data were outlined.

·        Methodological strengths and limitations were identified, and the costs and benefits for resolving the projects’ questions were analyzed and presented.

However, in reviewing the research project files, we determined that key source documents needed to independently confirm the validity of the LMSB Division research results were not consistently maintained in the project files.  As a result, questions can be raised about whether the research is as credible as it could be and free of any bias.

For example, the project files supporting the report entitled Partnership Selection System for LMSB - A Data Mining Proof-of-Concept did not contain the source data collection checksheets.  The checksheets were an integral part of the project because they documented the decision-making process (judgments) used by experienced examiners in reviewing 2,086 returns and deciding to select (or not select) the returns for examination.  The examiner judgments, as reflected on the checksheets, were ultimately programmed into an automated data mining process that is currently identifying the LMSB Division partnership returns for examination.

Additionally, the project files for the report on High Range Corporation Return Workload Selection System Development did not have the source Audit Information Management System (AIMS) database used in the project.  The AIMS database was a critical component of the project because it contained results from prior examinations of large corporate returns. The mathematical formulas, which relied extensively on the AIMS database, were developed for deciding which large corporate returns should be examined.  According to documentation in the file, the researchers had to overcome problems with the source AIMS database obtained for the project. 

To overcome the problems, the researchers eliminated over 15,000 records they determined would not be needed for the formulas and used the remaining 16,415 records in developing the formulas.  Although the reasons for removing the records were documented, the absence of the source AIMS database makes it difficult, if not impossible, to validate that all the data intended to be used in developing the formulas were, in fact, used.

The effectiveness of research efforts needs to be measured

Although the LMSB Division can show benefits from some of its research projects, the effectiveness of the overall research program needs to be measured. For example, the projects developing the automated systems that are replacing the practice of using examiners to identify tax returns for examination are saving the time and resources that examiners would spend manually screening tax returns.  While this and other research projects can demonstrate the effectiveness of research efforts, the LMSB Division does not have a systemic process to measure research outcomes against intended results.  As a result, the LMSB Division cannot readily determine whether specific projects are achieving their objectives or answer important questions such as “What overall impact is the research having?”

Both the Government Accountability Office (GAO) and the Treasury Inspector General for Tax Administration (TIGTA) have previously reported that establishing performance measures is critical to the success of any significant project and is a requirement under the Government Performance and Results Act of 1993 (GPRA).  Successful projects rely heavily upon performance measures to achieve objectives, quantify problems, evaluate alternatives, allocate resources, track progress, and learn from mistakes.

The Committee on Science, Engineering, and Public Policy reported in 1999 on the difficulties the research community in the Federal Government was having in developing meaningful ways to measure the effectiveness of research programs.  Despite the difficulties, the Committee concluded that, regardless of their nature, all research programs could be evaluated on a regular basis and in accordance with the GPRA.  Further, the Committee recommended choosing methods of evaluating research efforts that best match the character of the research. 

Results of applied research can be evaluated, according to the Committee, in terms of progress being made toward achieving specific outcomes.  For example, if the LMSB Division had an objective of finding less expensive ways to identify noncompliance on tax returns, it could measure the results of research aimed at decreasing the cost of screening returns for examination. Where there are ongoing efforts aimed at increasing knowledge through basic research, the Committee found that a peer review process is widely used as a best practice to evaluate three aspects of the GPRA: (1) the quality of the research, (2) the relevance of the research to an agency’s mission, and (3) if the research is at the forefront of work being done in the field. 

The LMSB Division has numerous research projects seeking to further understand the compliance risks with large business for which a peer review could be used to evaluate its efforts.  For example, the LMSB Division has projects on assigning global indexes to tax returns to assess the operating environment and to reveal the state of the multinational operations of its customers, evaluating the effects of foreign tax credits over time, and studying how some large businesses maintain low tax rates despite high published earnings.  We recognize that a peer review process will involve additional expenditures for the LMSB Division, particularly if outside experts are involved.  However, these costs could be relatively low if, instead of hiring outside experts, the IRS operating divisions coordinated in conducting peer reviews of each other’s research units. 

Project management guidelines can be strengthened

As outlined in the Standards for Internal Control in the Federal Government, managers need access to reliable and timely operational data to meet their responsibility of ensuring the effective use of resources.  To help research managers in the IRS operating divisions fulfill this responsibility, the SRC has established standards for planning and reporting on individual research projects.  Among other things, the standards recommend tracking time frames for completing activities, producing deliverables, and achieving milestones given the estimated resources that will be applied.  Tracking such information directly relates to the effective stewardship of resources by helping to answer questions such as “How long are the different steps in the research process taking?” and “Are the time frames that have been set for the process being met?”  These techniques are designed to develop plans and control systems to ensure research is completed on time and within budget so needed information is available for executive decision-making purposes.

To monitor the progress of research projects in the LMSB Division, procedures require project team leaders to periodically submit status reports to their managers that describe the progress being made, problems being encountered, and expected completion dates.  In addition, the Research Unit maintains a project management information system that could be used to determine whether projects and specific activities (planning, execution, and reporting) are being completed within budget and on time.  However, guidelines needed to determine whether projects and their activities are on time and within budget have yet to be developed.  Although some of this information can be found in status reports, these documents are submitted only periodically.  Consequently, an important project management tool that managers could use to enhance the monitoring of multiple projects is not available.

Recommendations

Given the expectations of the LMSB Division Research Unit and the significant level of resources that are depending on its work, we are making recommendations to enhance the processes for managing research.  The Director, Strategy, Research, and Program Planning, LMSB Division, should develop, define, and adopt:

1.      Procedures for the research staff that will ensure support information needed to replicate the research process is consistently maintained in project files and in accordance with the IRS Records Management Program.

Management’s Response:  The Commissioner, LMSB Division, responded that guidelines will be developed and issued on research documentation to ensure research staff members understand what documentation should be consistently maintained in project files.

2.      A framework for systemically measuring completed and ongoing research against intended results.

Management’s Response:  The Commissioner, LMSB Division, responded that guidelines will be issued for measuring research against intended results.  The guidelines will reemphasize the need for a final report on each completed research project and require that each report contain a section outlining the original project objective(s) and the results of the project as compared to the objective(s).

3.      Guidelines for determining whether individual projects and their related activities (planning, execution, and reporting) are on time and within budget.

Management’s Response:  The Commissioner, LMSB Division, responded that a Time Reporting System using Microsoft Project will be implemented to allow for monitoring the planning, execution, and labor costs of all research projects.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

Our objective was to assess the progress the Large and Mid-Size Business (LMSB) Division is making in (1) addressing the concerns raised in past studies of the Internal Revenue Service’s (IRS) research operations, (2) integrating its research operation into key business decision-making processes, (3) using research for improving performance on strategic initiatives, and (4) aligning its research processes with best practices adopted by other research programs. To meet our objective, we relied on the IRS’ internal management reports and databases.  We did not establish the reliability of these data because extensive data validation tests were outside the scope of this audit and would have required a significant amount of time.  To accomplish our objective, we:

              I.      Reviewed five studies of the IRS to identify past concerns with its research operations and obtained information from LMSB Division officials on actions taken to address the concerns.  The studies included the Government Accountability Office reports entitled IRS Has Made Progress but Major Challenges Remain (GAO/GGD-96-109, dated June 1996) and Internal Revenue Service – Status of the Modernized Research Operation (GAO-01-565R, dated April 2001); the IRS Inspection (now the Treasury Inspector General for Tax Administration [TIGTA]) report entitled Review of the National Office Research Analysis/District Office Research Analysis and Compliance Research Information System Implementation (Reference Number 056406, dated September 1995); the IRS discussion document entitled Research, dated March 2000; and the TIGTA report entitled The Internal Revenue Service Needs to Improve Control of Its Compliance Research Program (Reference Number 2000-40-068, dated May 2000).

           II.      Obtained information from IRS officials, including representatives of the LMSB Division’s Research Unit, on how the IRS’ research program is organized and on the LMSB Division’s processes for planning, conducting, monitoring, and evaluating research projects. 

         III.      Analyzed the Treasury Integrated Management Information System (TIMIS) and the IRS Administrative Corporate Education System (ACES) to assess skill and experience levels of the LMSB Division research staff.

        IV.      Reviewed various publications on best practices and standards in the research community from the Committee on Science, Engineering, and Public Policy; Office of Management and Budget (OMB); and IRS Office of Research, Analysis, and Statistics.

           V.      Used standards developed by the OMB and IRS Quality Assurance Council to evaluate the design, data collection and analysis techniques, and reports from a judgmental sample of 6 out of 64 research projects that were underway as of June 30, 2003. Judgmental sampling was used to minimize time and travel costs.

 

Appendix II

 

Major Contributors to This Report

 

Philip Shropshire, Acting Assistant Inspector General for Audit (Small Business and Corporate Programs)

Frank Dunleavy, Audit Manager

William Tran, Lead Auditor

Denise Gladson, Auditor

Ali Vaezazizi, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Services and Enforcement  SE

Deputy Commissioner, Large and Mid-Size Business Division  SE:LM

Director, Strategy, Research, and Program Planning SE:LM:SR

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaison:  Commissioner, Large and Mid-Size Business Division  SE:LM

 

Appendix IV

 

Organizational Chart of the Internal Revenue Service’s Research Operation

 

The diagram below shows the structure of the research operation.  It shows that the Director, National Headquarters Office of Research, does not have line authority over the research teams in the operating divisions.  Rather, the Commissioners and Directors of Research in each operating division control budget and resources needed for their respective research teams.

 

The diagram was removed due to its size.  To see the diagram, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

Appendix V

 

The Internal Revenue Service’s Quality Standards for Research Projects

 

The Servicewide Research Council, through its Quality Assurance Council (QAC), monitors and provides feedback to researchers on the quality of their projects. To define research quality, the QAC uses 11 standards for research planning and 8 standards for research reporting.  Each standard also has several key elements that elaborate on the overall standard.  Table 1 provides a summary of the quality standards and related elements for research planning.  Table 2 summarizes the quality standards and related elements for research reporting.

 

Source:  The Internal Revenue Service (IRS) Servicewide Research Council.

 

Source:  The IRS Servicewide Research Council.

 

Appendix VI

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.