This report presents the results of our review of the Information Systems (IS) organization’s efforts to develop business results measures. In summary, the IS organization’s Performance Assessment Office (PAO) has accomplished a significant task in a limited amount of time by working with the IS line organizations and Booz-Allen & Hamilton to identify business results measures and diagnostic indicators. The PAO also developed a data dictionary and the policies and procedures for maintaining the IS organization’s share of the Internal Revenue Service’s (IRS) Balanced Measurement System. However, to meet the needs of the IS organization and IRS management, the PAO needs to work with the IS line organizations to complete development of the business results measures and diagnostic indicators. The PAO also needs to present business results measures accurately and completely in its business performance review and consider developing additional business results measures to address identified high-risk management challenges.

Executive Summary

Objective and Scope

Background

Results

Conclusion

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – The Information Systems Organization’s Business Results Measures and Diagnostic Indicators

Appendix V – Removed and Reconsidered Diagnostic Indicators

Appendix VI – Possible Business Results Measures for Addressing Fiscal Year 2001 Management Challenges

Appendix VII – Management’s Response to the Draft Report

Executive Summary

The Government Performance and Results Act of 1993 (GPRA) requires the Internal Revenue Service (IRS) to submit annual performance plans to the Office of Management and Budget (OMB) and the Congress. The plans include performance goals and prior performance reports that compare actual performance to annual goals. To implement the GPRA requirements, the IRS developed the Balanced Measurement System composed of three types of performance measures: employee satisfaction, customer satisfaction, and business results. The IRS has contracted with independent vendors to manage the employee satisfaction and customer satisfaction measures. Each IRS business unit and functional organization is responsible for establishing and compiling business results measures.

The objective of this audit was to determine if the Information Systems (IS) organization’s business results measures are adequately defined, meet legal standards, and support the IRS’ Balanced Measurement System. To accomplish this objective, we assessed the effectiveness of the procedures the IS organization developed to collect, monitor, and analyze business results measures information and set baselines for each business results measure.

In March 1999, the IRS began working with Booz-Allen & Hamilton (BAH) to develop a new set of organizational performance measures for the IS organization. BAH delivered its proposal for measuring the IS organization’s performance in October 1999. The IS organization’s Performance Assessment Office (PAO) further refined these measures and, in March 2000, the Chief Information Officer (CIO) and the Commissioner approved the IS organization’s measures.

After the IS balanced measures approval, the PAO and the IS line organizations (the actual IS operating divisions and offices) began gathering data and developing the data analysis to establish performance baselines for its 16 business results measures and 44 diagnostic indicators that support these measures. The PAO prepared the Information Systems Balanced Measures Data Dictionary and a Policies and Procedures document as references for the IS managers who will collect data and for PAO personnel who will monitor and report on the business results measures. The November 2, 2000, IS Business Performance Review (BPR) included the first presentation of the business results measures as an attribute of IS performance.

While much has been accomplished, additional work is needed to meet the business measurement needs of IS and IRS management. The PAO needs to work with the IS line organizations to complete development of the business results measures and diagnostic indicators, present business results measures accurately and completely in its business performance review, and consider developing business results measures to address identified high-risk management challenges.

The GPRA requires that the annual performance plans include a description of the means used to verify and validate business results data. The PAO and the IS line organizations have had difficulty accomplishing this activity because the systems the IS organization uses to collect data were not designed to calculate the business results measures.

Our analysis showed that 3 of the 16 business results measures did not include complete data for adequate reporting. Without completed business results measures, the PAO cannot provide data to IS managers that could be used to improve organizational performance. The business results measures will be used to determine if proposed service levels to the IRS’ business units and functional organizations are achieved.

The PAO is in the process of reviewing the 44 IS diagnostic indicators to evaluate which indicators will provide adequate support for the business results measures. At the time of our review, 26 of the 44 (59 percent) diagnostic indicators did not have data sources available to provide information to support the business results measures.

Also, there were 24 diagnostic indicators presented in BAH’s October 1999 proposal that the PAO removed from use by March 2000. The PAO removed these indicators because of inadequate data sources or a vague understanding of the indicator’s significance. Our analyses and conversations with the PAO managers and analysts indicated that 18 of these removed indicators have value and potential data sources to support the business results measures. Without good diagnostic indicators, managers will be unable to determine the reasons for sub-optimal performance identified in the business results measures.

The IS organization submits a quarterly BPR report to the Commissioner which includes the IS business results measures. The BPR provides the Commissioner and IS management with performance data so that they can make informed decisions about the organization’s activities.

The IS organization’s first quarterly BPR was presented to the Commissioner in November 2000. The IS organization intends to use the 16 business results measures in this BPR to set baselines for future years’ results. This BPR showed that all business results measures had data sources, baselines/targets, and available data. However, our analysis showed that one measure was not clearly reported, three other measures did not include all relevant data in the presentation, two measures did not have complete data available to calculate and meet the measure’s definition, and one measure was not presented at all. Without accurate and complete business results measures in its BPR, IS and IRS management could make inappropriate decisions.

The Fiscal Year 2001 Congressional Justification, which proposes agency budget initiatives, includes a performance plan that describes 13 management challenges/high-risk areas for the IRS. The IS organization shares responsibility for 3 of the 13 challenges identified. However, the IS business results measures do not specifically address these 3 challenges. The IS organization’s responsibilities are: accuracy of the inventory system for automated data processing equipment, information systems security controls, and updating programs for filing season readiness.

By not including specific performance measures in the performance plan, the IS organization is not reporting to the Congress what it is doing to address these challenges. Without specific measures, it is difficult, if not impossible, for the Congress to assess progress in addressing major management problems and to hold agencies accountable.

To improve the ability of IS managers to assess program performance and make more informed decisions in directing operations, the CIO should direct the PAO to further develop the IS business results measures and diagnostic indicators. To accomplish this, the PAO should work with the IS line organizations to research information systems data sources for use in compiling the measures and indicators. The PAO also needs to follow GPRA guidance and ensure that it documents the verification and validation of these data to provide assurance of its accuracy and completeness.

To ensure that the BPR provides relevant information for management decision-making purposes, the CIO should direct the PAO to complete efforts to clearly and fully define the IS business results measures and include all relevant data for measure calculation. It also needs to disclose the data sources and limitations of the business results measures in the BPR presentation.

To meet performance assessment requirements of the Congress, General Accounting Office, OMB, and Department of the Treasury, the CIO should develop business results measures that assess the IS organization’s performance in meeting the management challenges. The IS organization and the PAO should reassess the measures annually, developing measures, as necessary, that address new challenges as they arise.

Management’s Response: Management agreed that they need to perform additional work to further develop business results measures and improve their data to meet the business needs of the IS organization and IRS management. Management’s complete response to the draft report is included as Appendix VII.

Objective and Scope

The objective of this audit was to determine if the Information Systems (IS) organization’s business results measures are adequately defined, meet legal standards, and support the Internal Revenue Service’s (IRS) Balanced Measurement System. To accomplish this objective, we assessed the effectiveness of the procedures the IS organization developed to collect, monitor, and analyze business results measures information and set baselines for each business results measure. We performed this audit as part of our regular coverage of the IS organization’s program management. We reviewed legal requirements, baseline data collected, and diagnostic indicators to assess the business results measures development.

We gathered data and interviewed IS executives, managers, and staff in the IRS’ New Carrollton, Maryland, office. We conducted our audit fieldwork and analysis from October 2000 through January 2001. This audit was performed in accordance with Government Auditing Standards.

Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

Background

The Government Performance and Results Act of 1993 (GPRA) requires the IRS to submit annual performance plans to the Office of Management and Budget (OMB) and the Congress. The plans include performance goals and prior performance reports that compare actual performance to the annual goals.

To implement the GPRA requirements, the IRS developed a Balanced Measurement System composed of three types of measures: employee satisfaction, customer satisfaction, and business results. The IRS has contracted with independent vendors to manage the employee satisfaction and customer satisfaction measures. Each IRS business unit and functional organization is responsible for establishing and compiling business results measures.

Business units and functional organizations also have the responsibility to identify diagnostic indicators to support each business results measure. Diagnostic indicators are output measures used to analyze factors that influence a business result’s measurement. Managers can use the indicators for strategy selection, workload planning, or performance monitoring.

In March 1999, the IRS began working with Booz-Allen & Hamilton (BAH) to develop a new set of organizational performance measures for the IS organization. BAH delivered its proposal for measuring IS performance in October 1999. The IS organization established the Performance Assessment Office (PAO) to help refine and implement the Balanced Measurement System. In March 2000, the Chief Information Officer (CIO) and the Commissioner approved the 16 IS business results measures proposed by the PAO. The IRS is planning to use four of these measures in its Fiscal Year (FY) 2002 annual performance plan.

Implementing and maintaining the IS business results measures requires the involvement and coordination of several levels and components of the IS organization. The IS organization’s managers "own" the measures related to their responsibilities. They collect refined data for the measurements and indicators and transmit the data to the PAO. The IS organization plans to use its Balanced Measurement System to develop and support Service Level Agreements (SLA). SLAs are agreements between the IRS’ business unit or functional organization executives and IS managers that define the levels of unique services the IS organization will provide to the business unit or functional organization. The PAO will identify, analyze, and report on the measures, and the IS organization Division Information Officers (DIO) will develop and monitor the SLAs.

Results

The PAO has accomplished a significant task in a limited amount of time. After the IS balanced measures approval, the PAO and the IS line organizations (the actual IS operating divisions and offices) began gathering data and developing the data analysis to establish performance baselines for its 16 business results measures. The PAO prepared the Information Systems Balanced Measures Data Dictionary and a Policies and Procedures document as references for the IS’ functional and operational managers who will collect data and for PAO personnel who will monitor and report on the business results measures. Also, the PAO and the IS line organizations began compiling information for 44 diagnostic indicators to support these measures. The November 2, 2000, IS Business Performance Review (BPR) included the first presentation of the business results measures as an attribute of IS performance. Appendix IV presents the IS organization’s business results measures and diagnostic indicators.

While much has been accomplished by the PAO, additional work is needed to meet the business measurement needs of IS and IRS management. The PAO needs to work with the IS line organizations to complete development of the business results measures and diagnostic indicators. The PAO also needs to present business results measures accurately and completely in its business performance review and consider developing additional business results measures to address identified high-risk management challenges.

The Information Systems Organization Needs to Complete Development of the Business Results Measures

To meet GPRA requirements, the PAO and the IS line organizations began gathering data for the 16 IS business results measures. Our analysis showed that 3 of the 16 business results measures did not include complete data for adequate reporting. These business results measures are not completely developed because the PAO and IS line organizations are in the process of identifying appropriate data to meet the measures’ definitions.

• Percentage of Systems Data Integrity – This measure’s definition is, "To ensure IS Systems and processes result in high quality data for business users." The PAO is currently reporting on the accuracy of the initial manual data input to information systems at the service centers. This presentation does not show the quality of the data for business users.
• Number of Transactions Processed by Transaction Type – This measure’s definition is, "Number of Transactions processed across all computing platforms by Operating Unit, Time Period, and System Type (production, admin, etc.)." The PAO has not identified the data sources to provide results for this measure.
• Number of Desktop Users Supported – This measure’s definition is, "Count of # of desktop users supported by Operating Unit, Time Period, and Desktop Category." The PAO has not identified data sources to provide results for this measure.

In addition, the PAO and the IS line organizations have had difficulty meeting the GPRA’s requirement to provide a description of the means used to verify and validate business results data. These offices have encountered problems verifying and validating data because the systems the IS organization uses to collect data were not designed for calculating the business results measures. The PAO is working with the IS line organizations responsible for the systems to collect reliable data for the measures.

For example, the Request for Information Services (RIS) Tracking and Reporting System (RTRS) is used to track RISs in the IRS. The PAO identified the RTRS as a data source for three of the IS organization’s business results measures. Initially, the PAO identified problems with the data coming from the RTRS. Many fields were missing data or had data conflicting with other fields.

Currently, the PAO is receiving monthly extracts from the RTRS. According to the PAO staff, the quality of the information in the RTRS has improved significantly. The PAO plans to have a workshop with the RTRS coordinators and users to emphasize the need for accurate and complete input for each RIS.

Without completed business results measures, the PAO cannot provide data to the IS organization’s managers that could improve organizational performance. In addition, incomplete measures will affect the SLAs currently under development by the DIOs. The SLAs will provide the business units and functional organizations with an agreement on the service that they can expect from the IS organization. The business results measures will be used to determine if proposed service levels are achieved.

To help ensure that operational performance is adequately measured, the CIO should direct the PAO to:

1. Identify data sources for the three business results measures that do not have data sources [Percentage of Systems Data Integrity, Number of Transactions Processed by Transaction Type, and Number of Desktop Users Supported].

Management’s Response: Management agreed that data sources are insufficient and must be augmented with automated sources that the PAO is evaluating to support the three business results measures.

• For the Percentage of Systems Data Integrity measure, the PAO is evaluating the use of IRS balancing reports and other systems’ data sources to better define the measure and to calculate data integrity using reject and error counts on outputs.
• For the Number of Transactions Processed by Transaction Type measure, the PAO is evaluating other sources of data on mainframes at the computing centers to collect the total number of transactions by system.
• For the Number of Desktop Users Supported measure, the PAO is researching the use of the Information Technology Asset Management System, an inventory database replacing the Inventory Network Operations Management System, to continue collecting desktop data. For the long term, the PAO is evaluating software called Tivoli and working with the Division Information Officers/Business Systems Planners Council to identify the number of users by desktop type.

1. Continue to work with the IS operating functions to perfect the data used to compile all business results measures. The PAO should also follow GPRA guidance and ensure that it documents the verification and validation of these data to provide assurance of its accuracy and completeness.

Management’s Response: The PAO is seeking to improve the quality and accuracy of the business results measures’ data by employing an enhanced validation and verification process using generally accepted accounting principles as a basis. The PAO will develop a repeatable process to meet these requirements. Once the validation and verification of data is complete, the PAO will document the results and publicize them on the balanced measures web site.

The Information Systems Organization Needs to Complete Development of the Diagnostic Indicators

IS managers can use diagnostic indicators to analyze the factors that affect changes in the business results measures. These analyses allow managers to "get behind the numbers" and identify improvement opportunities. For example, the three diagnostic indicators -- Number of Organizational Moves Planned, IS Support Staff to Customer Ratio, and Percentage of Variability of Desktop Configurations -- can explain aberrations in the business result measure, Number of Desktop Modifications Completed.

The IS Telecommunications Division has taken the lead in developing its own diagnostic indicators. This Division provided input to the PAO of possible indicators with available data sources and methods to measure service levels. The PAO plans to perform an analysis of these diagnostic indicators and adopt any that show a direct relationship to the business results measures. This input by IS line organizations to the PAO is valuable in identifying good diagnostic indicators that already have a data source available.

The PAO is in the process of reviewing the 44 IS diagnostic indicators to evaluate which indicators will provide adequate support for the business results measures. At the time of our review, 26 of the 44 (59 percent) diagnostic indicators did not have data sources available to provide information to support the business results measures.

There were also 24 diagnostic indicators presented in BAH’s October 1999 proposal that the PAO removed from use by March 2000. The PAO removed these indicators because of inadequate data sources or a vague understanding of the indicator’s significance. Our analyses and conversations with the PAO managers and analysts indicated that 18 of these removed indicators have value and potential data sources to support the business results measures. The PAO managers and analysts agree that these indicators were removed without considering all potential data sources.

Managers can use diagnostic indicators to understand underlying factors that cause changes in the business results measures and identify "root causes" for performance levels. Without good diagnostic indicators, managers will be unable to determine the reasons for sub-optimal performance identified in the business results measures. By removing some of the diagnostic indicators identified early in the measure development process, the PAO was not considering some potentially valuable performance indicators.

For example, the business results measure reporting the Average Issue Resolution Time had a diagnostic indicator that captured the Number of Calls Received. This indicator was initially removed because the PAO did not consider the impact these data had on the business results measure. They are reconsidering the use of this indicator because of the potential impact these data have on determining the scope of helpdesk demand.

Recommendation

To help ensure that operational performance is adequately measured, the CIO should direct the PAO to:

2. Complete its review of the data sources for the diagnostic indicators. The PAO should review the 18 diagnostic indicators that were eliminated earlier in the process to determine if there are data now available to develop these diagnostic indicators and determine if there is a relationship between the proposed indicators and the business results measures that they are meant to support.

Management’s Response: The PAO is developing a "Value-Chain of Diagnostic Indicators" (an Interconnectivity Matrix) showing the relationship between the business results measures and their associated diagnostic indicators across MITS operating functions, platforms, and applications. The MITS operating functions will provide the PAO with information needed to populate the Interconnectivity Matrix, including appropriate diagnostic indicators, data sources, and data. The PAO will complete the review of the 18 diagnostic indicators eliminated earlier in the process to allow it to determine the relationship between the proposed indicators and the business results measures they support. By populating the Matrix with valid diagnostic indicators that support the business results measures, the PAO will ensure operational performance is accurately measured.

The Information Systems Organization Needs to Present Accurate and Complete Business Results Measures in Its Business Performance Review

The IS organization submits a quarterly BPR report to the Commissioner which includes the IS business results measures. The BPR provides the Commissioner and IS management with the performance data so that they can make informed decisions about the organization’s activities.

The IS organization’s first quarterly BPR was presented to the Commissioner in November 2000. The IS organization intends to use the 16 business results measures in this BPR as baselines for future years’ results. This BPR showed that all business results measures had data sources, baselines/targets, and available data.

However, our analysis showed that the presentation of seven measures was not accurate or complete because source data were limited or because of the way data were interpreted:

• One measure was not clearly reported.
• Three measures did not present all relevant data in the presentation.
• Two measures did not have complete data available to calculate and meet the measure’s definition.
• One measure was not presented at all.

Without accurate and complete business results measures in its BPR, IS and IRS management could make inappropriate decisions.

Percentage of Systems Data Integrity – This measure’s definition is, "To ensure IS Systems and processes result in high quality data for business users." The BPR result reported was a percentage of records not requiring correction of data entry errors at the service centers. Although the definition of this measure is interpretive, the result reported does not meet the scope of the business results measure’s definition. This result involved only a body of records not requiring further data validation.

Additionally, the BPR reported this percentage without disclosing that the PAO had not yet determined how it would specifically define data integrity. Without a clear and relevant definition, IS and IRS management could misinterpret the BPR result in the decision-making process.

To obtain the results for three of the business results measures, IS management decided that they should use only the filing season RISs because they wanted to focus on filing season readiness. The PAO did not disclose in the BPR results that it presented only filing season data. Therefore, management could be misled by the results and make incorrect decisions based on them.

To illustrate, the business results measure for Systems Delivery Commitments Met (percent of RISs delivered by agreed date) reported data for filing season RISs only, without disclosing that all other RISs were not included in the calculation. If that result showed performance at an acceptable level, management may make a decision to maintain the current programming staff level. However, the result could show an unacceptable level of performance, if the calculations included information from all the RISs. Therefore, management may actually need to increase programming staff.

Only filing season RIS data were reported in the BPR for the following business results measures:

• Average RIS Response Time.
• Systems Delivery Commitments Met.
• Number of RISs Analyzed.

Number of Transactions Processed by Transaction Type – This measure’s definition is, "Number of Transactions processed across all computing platforms by Operating Unit, Time Period, and System Type (production, admin, etc.)." The PAO reported the "Number of Sample Transactions" because it had data only for transactions processed from a sample of 360 desktop workstations. The PAO is working on identifying a data source to provide complete and informative data to report this measure.

Number of Desktop Users Supported – This measure’s definition is, "Count of # of desktop users supported by Operating Unit, Time Period, and Desktop Category." The PAO reported the "Number of Desktop Users" for "Number of Desktops." The result is not reported as defined because data were not available to identify the number of employees using the desktop workstations. The PAO is working on identifying a data source to provide complete and informative data to report this measure.

Number of Requirements Delivered – This measure’s definition is, "Number of projects delivered in the prior 6 months." This result was not reported because the method to calculate the measure was not in place for an adequate period. The PAO plans to report this business results measure in future BPRs.

To help ensure that the BPRs provide relevant information for management decision-making purposes, the CIO should direct the PAO to:

4. Complete efforts to clearly and fully define the IS business results measures and identify and include all relevant data for measure calculation.

Management’s Response: To provide relevant information for management decisions, the PAO will continue to define the MITS business results measures and include them on all future BPRs. The PAO will document the updated definitions by adding them to its Data Dictionary and balanced measures web site.

5. Disclose the data sources and limitations for the business results measures in the BPR presentation.

Management’s Response: To ensure data in the BPR is current, accurate, and relevant for management decisions, the PAO will disclose data sources and limitations for all business results measures reported in the BPR. The PAO will continue to update the Data Dictionary, policies, and procedures to reflect data sources and changes.

The Information Systems Organization Should Use Business Results Measures to Report Accomplishments in Meeting Its Management Challenges to the Congress

In July 1999, the General Accounting Office (GAO) issued a report on the Department of the Treasury’s FY 2000 Performance Plan. In the report, the GAO found that the FY 2000 plan did not include performance goals to address all management challenges and high-risk areas.

In July 1999, the OMB issued guidelines to all Federal agencies that provided general guidance on developing performance goals. The guidelines state that, "Performance goals for management problems should be included in the annual plan."

In January 2000, the Department of the Treasury provided guidance to all of its agencies suggesting that performance plans use performance measures to address the management challenges facing the agency. In October 2000, the Chairman of the Congress’ Governmental Affairs Committee issued a report that stated, "Agencies should incorporate performance measures for major management challenges into the performance agreements of agency leaders and program managers."

In April 1999, the IRS’ Organization Performance Division (OPD) provided direction to the IRS, including the PAO, on developing the business results measures. Since this direction was issued prior to the above agency and Congressional concerns and guidance, the OPD did not include in its direction that the IRS should consider developing measures to address the management challenges/high-risk areas.

The Fiscal Year 2001 Congressional Justification, which proposes agency budget initiatives, includes a performance plan that describes 13 management challenges/high-risk areas for the IRS. The IS organization shares responsibility for 3 of the 13 challenges identified. However, the IS organization’s business results measures do not specifically address the challenges.

The three challenges for which IS has responsibility are:

By not having specific performance measures included in the performance plan, the IS organization is not reporting to the Congress what it is doing to address these challenges. Without specific measures, it is difficult, if not impossible, for the Congress to assess progress in addressing major management problems and to hold agencies accountable. In addition, without measures to address the management challenges, IS management will not have a reference to determine if they are improving in these high-risk areas.

Recommendations

To help ensure that the IS organization provides relevant information and support to accomplish the IRS’ management challenges, the CIO should direct the PAO to:

6. Develop business results measures that assess the IS organization’s performance in meeting the management challenges. The IS organization and the PAO should reassess the measures annually, developing measures, as necessary, that address new challenges as they arise. See Appendix VI for possible measures to consider.

Management’s Response: Using a repeatable process for determining business results measures for performance of MITS products and services, the PAO will work with MITS line organizations to develop business results measures for high-risk management challenges. This repeatable process will guide the PAO in reassessing measures annually and help to address new challenges as they arise.

7. Coordinate its efforts with the IS line organizations to develop business results measures and identify data sources addressing the management challenge on the accuracy of the inventory system for automated data processing equipment.

Management’s Response: To help ensure MITS provides relevant information to address the accuracy of the inventory systems for ADP equipment, the PAO will coordinate with the Director, Enterprise Systems and Asset Management (ESAM) to define business results measures. The Director, ESAM, has identified Tivoli software and the Information Technology Asset Management System as potential data sources for data collection, analysis, and reporting. When these systems "stand up," the PAO will have a constant and consistent flow of data to support the business results.

8. Coordinate its efforts with the IS Security Privacy and Oversight Office to develop business results measures and identify data sources addressing the management challenge on the adequacy of the information system security controls.

Management’s Response: To help ensure MITS provides relevant information to address the adequacy of the information systems’ security controls, the PAO will complete its work with the Director, Office of Security, Privacy, and Oversight, to develop business results measures.

Conclusion

While much has been accomplished, additional work is needed to meet the business measurement needs of IS and IRS management. The PAO needs to completely develop the business results measures. These measures are important indicators of the IS organization’s level of performance to the Commissioner in the quarterly BPRs, to the IRS business units in annual SLAs, and to the Congress in the IRS annual performance plan.

When the business results measures and the diagnostic indicators are completely developed, they will provide IS managers with the information needed to determine if their work units’ performance is satisfactory. If the results are not meeting expectations, the managers can use the diagnostic indicators to help determine what improvements they need to consider.

Detailed Objective, Scope, and Methodology

The objective of this audit was to determine if the Information Systems (IS) organization’s business results measures are adequately defined, meet legal standards, and support the Internal Revenue Service’s (IRS) Balanced Measurement System. To accomplish this objective, we assessed the effectiveness of the procedures the IS organization developed to collect, monitor, and analyze business results measures information and set baselines for each business results measure. To achieve this objective, we performed the following tests:

1. To determine whether IS business results measures met legal standards, we:
1. Assessed whether IS business results measures met the requirements for performance plans in the Government Performance and Results Act of 1993 (GPRA), § 1115, that requires annual performance plans to:

1. Establish performance goals.
2. Express such goals in an objective, quantifiable, and measurable form.
3. Briefly describe the operational processes, skills and technology, and resources required to meet the performance goals.
4. Establish performance indicators to be used in measuring or assessing the relevant outputs, service levels, and outcomes.
5. Provide a basis for comparing actual program results with the established performance goals.
6. Describe the means to be used to verify and validate measured values.

1. Reviewed Office of Management and Budget (OMB) Circular A-11 Part 2, Section 220 to determine if the IS business results measures met the OMB requirements. This Circular contains guidance on how agencies should prepare their performance plans to comply with the GPRA.
2. Reviewed 26 Code of Federal Regulations (CFR) Part 801, Annual Performance Plans and Reports, to determine if the IS business results measures met CFR requirements.
3. Evaluated the annual performance plan guidance provided to the IS organization by the Organizational Performance Management Executive and the Chief Financial Officer and assessed the status of the IS Performance Assessment Office’s (PAO) input to the IRS annual performance plan.
4. Evaluated compliance with legal standards for the annual performance plan.
0. Determined if the IS organization included a required description of the means used to verify and validate measured values.
1. Reviewed contract task order documentation to determine if the IS organization needed to disclose a non-Federal party’s contribution to performance plan preparations as a result of Booz-Allen & Hamilton’s (BAH) participation.

1. To evaluate the IS organization’s procedures to collect, monitor, and analyze business results information and efforts to set baselines for each measure, we:
1. Determined if the business results information sources provide reliable data.
1. Interviewed IS line organization (the actual IS operating divisions and offices) managers responsible for collecting, validating, and transmitting business results data to the PAO to determine how the business results measures were collected, validated, and transmitted.
2. Interviewed employees who administer the computer programs and related databases to determine what steps they take to gather data for the business results measures.
3. Evaluated the effectiveness of the input/output, processing, and managerial controls used for the collection of data.
2. Determined if the procedures will ensure there is adequate information to set effective baselines. The PAO collected business results information to set initial baselines for the budget performance plan and Service Level Agreements (SLA). The definition, source, and methodology used to collect the information is documented in the Information Systems Balanced Measures Data Dictionary.
1. Interviewed the IS line organization managers responsible for collecting and analyzing the business results data to determine how the baselines were developed.
2. Interviewed the PAO employees responsible for processing the IS line organizations’ information and setting the baselines for each business results measure to determine how the baseline data were accumulated and how the baseline was set.
3. Evaluated the Information Systems Balanced Measures Data Dictionary methodology established to administer the balanced measures and baselines related to each business results measure.
3. Determined if management plans will provide adequate monitoring and oversight of the business results measures.

1. Interviewed the IS line organization managers responsible for collecting and analyzing the business results data to determine how they plan to conduct and document the required monthly reviews.
2. Evaluated the adequacy of the planned quarterly Business Performance Review (BPR) and reviewed the results of the IS organization’s first BPR.
3. Interviewed the PAO staff regarding annual reviews to be performed.

1. Determined if the IS organization’s business results measures and diagnostic indicators provide the needed support for the IRS’ goals.

1. Identified the management challenges the IS organization reported to the Congress in its Fiscal Year 2001 budget request and determined if the IS organization’s business results measures addressed the challenges and established means to improve in the areas identified.
2. Reviewed the diagnostic indicators to determine if they were relevant to the IS business results measures.
3. Traced all diagnostic indicators from the original BAH deliverable to the current Policies and Procedures document.
0. Identified diagnostic indicators that were not included in the current Policies and Procedures guideline.

1. Determined if the corresponding business results measure was still in place.
2. Obtained and reviewed supporting documentation for the indicators that were not included by the PAO.
3. Interviewed PAO staff to identify the reason why the indicators were not included (i.e., cost, purpose, and benefits).

1. Determined the status of baselines for the diagnostic indicators.

1. Evaluated the PAO’s plans to collect information and set baselines for the diagnostic indicators.
2. Interviewed IS managers to determine how they plan to collect data to set baselines for the diagnostic indicators.

1. Evaluated the diagnostic indicator review conducted by the PAO.

1. Analyzed review documentation and determined if the review adequately supported the addition or removal of indicators.
2. Interviewed PAO staff and IS line organization representatives to determine if the line organizations had adequate input during diagnostic indicator development.

1. Interviewed Division Information Officers regarding SLA development status and evaluated plans to use the business results measures.
2. Determined if the IS balanced measure website (http://measures.is.irs.gov/testweb/), provided adequate communication to IS customers.

1. Interviewed PAO staff to determine if the website was the primary means of communicating the business results measures or if the IS organization used any other methods.
2. Reviewed the website to determine if all the business results measures’ results were included.

Major Contributors to This Report

Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)

Scott A. Macfarlane, Director

Edward A. Neuwirth, Audit Manager

Eulala Davis, Senior Auditor

Beverly Tamanaha, Senior Auditor

George Franklin, Auditor

Suzanne Noland, Auditor

Report Distribution List

Commissioner N:C

Director, Legislative Affairs CL:LA

Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O

Director, Strategic Planning and Client Services M:SP

Chief Counsel CC

Office of Management Controls N:CFO:F:M

National Taxpayer Advocate TA

Audit Liaison:

Deputy Commissioner for Modernization & Chief Information Officer M

The Information Systems Organization’s Business Results Measures and Diagnostic Indicators

Possible Business Results Measures for Addressing Fiscal Year 2001 Management Challenges

The following are possible measures that the Performance Assessment Office (PAO) could consider to address the Fiscal Year 2001 management challenges:

• Percentage of automated data processing (ADP) equipment on inventory system.
• Percentage of annual physical inventories of ADP equipment performed.
• Percentage of ADP equipment accurately valued on inventory system.

• Number of security reviews performed.
• Number of security incidents reported.
• Percentage of desktops upgraded to the common operating environment.

There are currently three business results measures that the PAO could use to address this management challenge. In these measures, the PAO could separate filing season Requests for Information Services (RIS) results from the results including all other RISs and report them separately. The current measures are:

• Average RIS Response Time Categorized by RIS Complexity.
• Systems Delivery Commitments Met.
• Number of RISs Analyzed Categorized by Complexity.

Management’s Response to the Draft Report

The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.