Improvements in the Measures Program for Information
Technology Services Would Further Increase Its Value to Stakeholders
March 2003
Reference Number:
2003-20-069
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
March
4, 2003
MEMORANDUM FOR
DEPUTY COMMISSIONER FOR MODERNIZATION & CHIEF INFORMATION OFFICER
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Acting Deputy Inspector
General for Audit
SUBJECT: Final Audit Report – Improvements in the Measures Program for
Information Technology Services Would Further Increase Its Value to
Stakeholders (Audit # 200220035)
This
report presents the results of our review of the business results measures used
by the Modernization, Information Technology and Security (MITS) Services
organization. The overall objective of
this review was to determine whether the business results measures were
adequately defined, met legal standards, and supported the Internal Revenue
Service’s (IRS) balanced measurement system.
The Government Performance and Results Act of 1993 (GPRA) was enacted by
the Congress to hold Federal agencies accountable for achieving business results. The GPRA requires agencies to identify their
high-risk management challenges, set performance goals, and adopt measures to
assess their performance. To comply
with this law, the MITS Services organization established a system of balanced
measures that included a set of business results measures.
We
also followed up to determine whether the MITS Services Performance Assessment
Office (PAO) addressed the issues reported in a previous Treasury Inspector
General for Tax Administration review of MITS Services business results
measures. In our previous review, we
reported that some measures did not have data sources and diagnostic
indicators, the Business Performance
Review (BPR)
presented some measures that were inaccurate or misleading, and the measures
did not address the management challenges that the IRS reported to the
Congress.
In
summary, we determined the PAO has made progress in developing and using MITS
Services business results measures.
Most of the measures are reported in the MITS Services quarterly BPRs in
colorful graphs in which performance goals can easily be compared to actual
performance. A Service Level Agreement
was completed with all of the IRS business organizations to which MITS Services
provides computer services. The data
sources needed to calculate the measures were identified and described in a
data dictionary. Actions were initiated
to develop measures to address the critical management challenges of computer
security and business systems modernization.
We also determined that MITS Services management is using the measures
to improve their operations.
While
the PAO has made improvements to the system of business results measures,
several issues we reported in our previous review have not been fully
addressed. For example, the helpdesk
tickets, which are used to calculate five of the measures, are unreliable. A significant percentage of these tickets,
recorded in the Information Technology
Asset Management System to track computer problems, are misclassified or inaccurately
recorded. Also, a valid sampling plan
was not used to calculate another measure.
Additionally,
MITS Services did not accurately or clearly report two measures in its third
quarter Fiscal Year 2002 BPR presentation.
Without an accurate calculation and clear reporting of the business
results measures, IRS management and oversight bodies do not have reliable
performance data to make decisions.
Although
good procedures for verifying and validating the measures have been developed,
these procedures are not being followed.
Not verifying and validating the measures increases the likelihood that
errors might not be detected, and management may take actions based on
inaccurate information.
Further,
one of the nine measures is not an appropriate business results measure because
the information it tracks is outside of the control of the MITS Services
organization. Therefore, MITS Services
is being measured on something that is not a factor of its performance.
Lastly,
measures have not been developed to report on the accuracy of the IRS’
inventory system for automated data processing equipment. This issue is
consistently of interest to oversight bodies, including the Congress, and a key
goal for which measures should be developed.
To improve the accuracy
and reliability of the measures, we recommended live monitoring of the
personnel responsible for recording the helpdesk tickets and monthly reviews of
the tickets until the inaccuracies are resolved. In addition, a formal sampling plan should be prepared and followed
for the calculation of one measure, and the PAO should disclose all limitations
and potential inaccuracies in the measures that are reported in the BPR and
budget submissions.
The measures that are
calculated based on helpdesk tickets should be calculated using all of the
relevant helpdesk tickets, and be based on the core or specific hours requested
by the users of the computer systems.
To comply with the GPRA and ensure the measures report accurate results,
ongoing verification and validation activities should be conducted for all
business results measures.
Finally, one
of the measures should be used as a diagnostic indicator rather than a business
results measure, and MITS Services
should develop a measure to address the challenge of creating and maintaining
an accurate automated data processing inventory system.
Management’s
Response: The IRS’ Director of Resources Allocation
and Measurement agreed with our recommendations and indicated that verification
and validation procedures would be implemented to ensure that the calculated
data being reported in the BPRs is accurate and complete. A formal documented sampling plan will be
implemented for the “Percent Subsecond System Response Time” measure. In addition, the PAO will also disclose the data limitations and potential
inaccuracies reported in the measures.
To
ensure the “Number of Hours of Availability” and “Percent Systems Availability”
measures are properly calculated and reported, the PAO will begin counting the
unscheduled system downtime recorded on all of the helpdesk tickets. Furthermore, the PAO will calculate these
measures based on the specific hours requested by the users. The PAO will also begin reporting the
“Number of RISs Received for Filing Season” solely as a diagnostic indicator. Lastly, to address the management challenge
of creating and maintaining an accurate automated data processing inventory
system, the PAO will develop a business results inventory measure. Management’s complete response to our report
is included as Appendix V.
Copies of this
report are also being sent to the IRS managers who are affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Gary V. Hinkle, Acting Assistant
Inspector General for Audit (Information Systems Programs), at (202) 927-7291.
Progress
Has Been Made in Developing and Reporting Business Results Measures
Two of the Nine Measures Did Not Accurately or Clearly Represent the Performance of Operations
One of the Business Results Measures Is Not Appropriate
Measures Have Not Been Developed to Report on the Accuracy of the Inventory System
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Summary
of Issues Identified With the Nine Business Results Measures
Appendix V – Management’s Response to the Draft Report
The Modernization, Information Technology and Security (MITS) Services organization has adopted a system of balanced measures to assess the performance of its major programs and progress in achieving goals. The system is composed of three types of measures: employee satisfaction, customer satisfaction, and business results. We focused our review on the business results measures because this type of measure is designed to specifically assess the key services and programs of the MITS Services organization.
MITS Services established its system of measures to comply with the Government Performance and Results Act of 1993 (GPRA) and other guidance issued by the Office of Management and Budget (OMB). The GPRA was enacted by the Congress to hold Federal agencies accountable for achieving results. The GPRA requires agencies to identify their high-risk management challenges, set performance goals, and adopt measures to assess their performance. The goals and measures must be objective, quantifiable and measurable, and provide sufficient detail to allow comparison of the goals to the actual performance so that Internal Revenue Service (IRS) management and the Congress can determine the results of IRS programs.
The OMB issued guidance to Federal agencies through its Circular A-11 on how to comply with the GPRA. The OMB instructed agencies to strike a balance between too few and too many measures. Agencies should include enough measures to show, in a substantive way, how well the agency is doing in carrying out its programs. Furthermore, measures usually have a numerical target level or goal that facilitates an assessment of whether the goals were actually achieved.
The MITS Services Performance Assessment Office (PAO) works with other segments of the MITS Services organization to identify business results measures and diagnostic indicators. Diagnostic indicators support or help explain the business results measures and allow management to better understand the numbers. For example, the “Ticket Activity” measure is the total number of helpdesk tickets opened and closed during a given period of time. The “Number of Computer Users” and “Number of Computer Network Devices” are two diagnostic indicators that support or help explain the results of the “Ticket Activity” measure.
In our previous review of MITS Services business results measures, we reported the PAO achieved significant accomplishments in the development of business results measures, diagnostic indicators, a data dictionary, and a Business Performance Review (BPR) process. We also reported that some measures did not have data sources and diagnostic indicators, the BPR presented some measures that were inaccurate or misleading, and the measures did not address the management challenges that were reported to the Congress.
Subsequent to the issuance of our
previous report, the MITS Services revised, eliminated, and added to its set of
16 business results measures. The new
set of business results measures consists of nine measures grouped into three
categories:
·
User Support – Four measures
report on the timeliness of the resolution of computer problems reported to the
helpdesk by IRS employees.
·
Data Access and Communication
– Three measures report on the functioning and responsiveness of the IRS’ major
computer systems.
·
Applications Support – Two
measures report on the timeliness of completion of programming changes that
impact the IRS’ filing season.
In this review, we followed up on
the issues reported in our previous audit and conducted tests to determine
whether the new set of business results measures were adequately defined, met
legal standards, and supported the IRS’ balanced measurement system.
We conducted this review from July to October 2002 at the MITS Services offices in New Carrollton, Maryland. Our review was conducted in accordance with Government Auditing Standards. Details on our audit objective, scope, and methodology are presented in Appendix I. Major contributors to the report are listed in Appendix II.
Several important actions have been taken since our last review of the MITS Services balanced measures. The IRS Commissioner holds quarterly BPR meetings, and those reviews involve preparation and presentation of the business results measures along with other key data for management decision-making. In the BPR presentations, most of the MITS Services measures are presented in colorful graphs and charts in which goals can easily be compared to actual performance. Trends are clearly identified and discussed at the bottom of each graph. This method of reporting the measures complies with the requirements of the GPRA, the OMB, and the Department of the Treasury.
The MITS Services organization also recently completed a Service Level Agreement with the IRS business organizations to which MITS Services provides computer services. The Agreement is effective for Fiscal Year (FY) 2003 and represents a groundbreaking achievement for MITS Services. The Agreement contains the services and service levels that MITS Services agreed to provide to the IRS business units. The business results measures will be used to measure the achievement of the targeted service levels.
Data sources needed to calculate the measures have been identified for all nine business results measures, and these sources are clearly described in the data dictionary. The data dictionary also provides the methods used to calculate each measure.
The PAO is taking actions to define and baseline all of the diagnostic indicators needed to support the business results measures. It plans to begin reporting this information beginning in early 2003.
Other actions have been taken to measure the effectiveness of MITS Services actions to address the critical management challenges reported to the Congress. Measures to address the challenges of computer security and business systems modernization are now being reported in the BPRs and in Strategy and Program Plans.
In addition to making progress in developing business results measures, management is taking actions based on the results reported in the measures. For example, the “Tickets Resolved on Time” measure reports the percentage of computer helpdesk tickets that are resolved in a timely manner. The performance goal for this measure is 80 percent. The second and third quarter FY 2002 BPRs reported that this performance goal was not being met. Therefore, the Director of the MITS Services End User Equipment and Services organization has identified and is implementing steps to improve the abilities of its helpdesk technicians to address these tickets in a timely manner. One of these steps involves consolidating helpdesks to achieve a more consistent service level.
The MITS Services organization has made significant progress in improving its business results measures. However, we identified several issues, some of which were discussed in our previous report, that need to be addressed to ensure the nine business results measures are accurately and clearly presented, and to address other key concerns. The issues identified with the nine business results measures are discussed below and a summary of the issues is presented in Appendix IV.
In our previous review, we reported that seven business results measures reported in the BPR were not accurate or complete and could be misleading. We recommended the PAO disclose the data sources and limitations for all of the business results measures. In this review, we found similar issues with five of the nine new measures.
The underlying data is unreliable
We found that inaccuracies in the underlying data compromise the reliability of five measures. Helpdesk tickets, recorded in the Information Technology Asset Management System (ITAMS) to document and track computer problems, are used to calculate the following five business results measures:
· “Percent Tickets Resolved on Time” – percentage of helpdesk tickets closed within established time standards.
· “Ticket Activity” – total number of tickets opened and closed during a given time period.
· “Percent Resolution at First Contact” – percentage of tickets that can be resolved by helpdesk technicians at first contact that are, in fact, resolved by the technicians.
· “Percent Systems Availability” – percentage of the total scheduled hours in a day that a system is available for use.
· “Number of Hours of Availability” – average number of actual hours per day that a system is available for use.
In our review, we found that the helpdesk tickets used in the calculation of the above five measures were often misclassified, and many tickets remained open after the computer problems were resolved. In addition, many tickets were closed with the ITAMS system default date rather than with the actual ticket closure date. As a result, the calculations of the above five business results measures are incorrect.
For example, the End User Equipment and Services organization selected and analyzed a statistical sample of 718 Priority 3 helpdesk tickets. The analysis determined 47 percent of the tickets recorded in the ITAMS between May and June 2002 were inaccurately classified. Approximately 10 percent of the tickets should have been classified as higher Priority 1 or 2 tickets.
Two additional studies also determined that the helpdesk tickets were being inaccurately classified. One study was conducted on the Priority 2 tickets that were recorded in the IRS Martinsburg Computing Center in January 2002 after the PAO Manager questioned the high number of Priority 2 tickets. The study determined that 7 (37 percent) of 19 tickets recorded as Priority 2 were inaccurate. These seven tickets should have been classified as another priority.
Another study was conducted on Priority 1 and 2 tickets recorded in the ITAMS during the months of January through September 2002. A formal report of the results of this study was not prepared. However, the End User Equipment and Services office informed us that they found a significant number of the Priority 1 and 2 helpdesk tickets were inaccurately classified.
The correct recording and classification of helpdesk tickets is critical to the accurate calculation and reliability of the business results measures. For example, unscheduled downtime of a major IRS computer system should be recorded on a Priority 1 helpdesk ticket. To calculate the “Number of Hours of Availability” measure, the PAO counted all of the unscheduled downtime recorded on Priority 1 tickets. This total was then subtracted from the total hours the system was scheduled to be up and running. If the unscheduled downtime of the computer system was misclassified and recorded on a Priority 2 or Priority 3 ticket, this data would not be used in the calculation of the measure. Consequently, the “Number of Hours of Availability” measure would report better results than were actually achieved.
The ITAMS was deployed in October 2001, and there was a transition period between the old system and the new system. This transition resulted in some issues with the helpdesk ticket data and in a learning curve for assistors.
Additionally, the inaccurate classification and recording of helpdesk tickets was due to inadequate discipline and accounting skills among the employees responsible for recording the tickets. On-line monitoring of the assistors as they recorded the tickets would help ensure their accuracy and improve this inadequate discipline.
Lastly, prior to selecting the measures, a thorough feasibility analysis to determine the accuracy and reliability of the underlying data was not conducted. This analysis would have detected the significant number of inaccuracies and unreliability of the helpdesk ticket data.
A valid sampling methodology was not used
The reliability of the “Percent Subsecond System Response Time” measure was also questionable. This measure reported the percentage of time that three major IRS computer systems respond to a user in 1 second or less. The systems measured were the Automated Collection System (ACS), Corporate Files On Line (CFOL), and the Integrated Data Retrieval System (IDRS). MITS Services set a performance goal for each of these computer systems. For example, when an IRS employee submits a request or types a command into the IDRS, the goal for this system is to respond in 1 second or less 98 percent of the time.
MITS Services calculates the “Percent Subsecond System Response Time” measure using the Attachmate software tool, which is an automated response time measurement tool. Since it would be costly and impractical to install Attachmate on all of the IRS workstations that use the three systems, the Attachmate tool is installed on a sample of computer workstations. To properly calculate this measure and project the results of the sample over the population of workstations that access the systems, Attachmate must be installed on a statistically valid sample of workstations.
However, a statistically valid sample of workstations was not used to calculate the “Percent Subsecond System Response Time” measure. The computer workstations monitored by Attachmate were selected in part based on specific requests by the IRS Commissioner who wanted Attachmate installed at small and large sites located at various distances from the IRS’ Computing Centers. The managers at each site judgmentally selected the workstations on which Attachmate would be installed. The selection was further complicated by the fact that Attachmate could not be installed on all types of computer terminals. Therefore, the results reported by this measure are unreliable because they cannot be statistically projected.
The PAO does not believe a statistically valid random sample is needed. It stated the initial selection of computer terminals was random, and there is a point of diminishing returns where using a statistically valid sample to calculate the measure would not significantly improve its accuracy. It also stated that given its physical, logistical, and budgetary constraints, a pure random sample is not possible.
However, when we discussed the business results measures with management from the Enterprise Operations organization within MITS Services, they expressed concerns about relying on the accuracy of this measure to make business decisions because of the clustered locations of the terminals selected. It is critical to ensure that measures are accurate so that executive management in the business area and other oversight organizations can have sufficient confidence in them to make appropriate business decisions.
Management Actions: MITS Services management is taking actions to address the inaccurate helpdesk tickets. The End User Equipment and Services organization updated a key document titled the Probe and Response Guide with helpful scenarios and guidance that can assist IRS employees and contactors to accurately record helpdesk tickets. This guide will be stressed during bi-weekly conference calls between managers and employees responsible for recording helpdesk tickets.
Training was also provided to new employees and contractors assigned to the Enterprise Service Desk. Lastly, a workshop was held for key officials to identify and address inaccuracies in the helpdesk tickets and the measures based on these tickets.
To improve the accuracy and reliability of the data used for the MITS Services business results measures, the Deputy Commissioner for Modernization & Chief Information Officer should direct:
1. The End User Equipment and Services organization to conduct live monitoring of the helpdesk assistors and perform monthly reviews of helpdesk tickets until the inaccuracies related to the recording of service tickets are resolved. These reviews should be conducted on a sample of tickets recorded by each helpdesk. Formal written reports should be prepared to document the results of the reviews. Additional formal training should be conducted in those locations where problems persist.
Management’s Response: The End User Equipment and Services organization, with support from the PAO, will implement verification and validation processes.
2. The PAO to prepare a formal sampling plan for the “Percent Subsecond System Response Time” measure that focuses on the computer terminals used to calculate the measure. The sampling plan should identify the population size, confidence level, precision, sample size, and other sampling criteria. The computer terminals on which Attachmate is installed should be randomly selected. In addition, the population of terminals, on which Attachmate cannot be installed, should be identified and, if significant, disclosed when this measure is reported in the BPR and in the IRS’ GPRA submissions.
Management’s Response: The PAO, working with the End User Equipment and Services and Enterprise Operations organizations, will implement a formal documented sampling plan for the “Percent Subsecond System Response Time” measure.
3. The PAO to disclose all limitations of and potential inaccuracies in measures that are included in reports, such as the BPR and budget submissions, until those limitations and inaccuracies are addressed.
Management’s Response: The PAO will disclose the data limitations and potential inaccuracies in its reports, such as the BPR and budget submissions, until the limitations are resolved.
In our previous review of MITS Services business results measures, we recommended the PAO complete its efforts to clearly and fully define the business results measures, and identify and include all relevant data for measure calculation. This action was necessary to ensure the BPR provides relevant information for management decision-making. In response to our report, MITS Services stated it would ensure data in the BPR were accurate and relevant for management decisions.
However, in our current review, we identified issues with the accuracy and clarity of the reporting of two of the measures - “Number of Hours of Availability” and “Percent Systems Availability.” Improvements in the accuracy and clarity of these two measures is critical to ensure IRS management and oversight bodies have accurate performance data upon which to make decisions.
Measures were improperly calculated
The “Number of Hours of Availability” and “Percent Systems Availability” measures were improperly calculated due to a decision by the PAO to not count the unscheduled system downtime recorded on Priority 2 and 3 helpdesk tickets.
As previously discussed, a significant number of helpdesk tickets recorded in the ITAMS were miscategorized or inaccurate. Some helpdesk tickets coded as Priority 3 tickets should have been coded as a higher priority, such as a Priority 1 or 2. For example, if a computer system unexpectedly shut down during the middle of the day and users were unable to access the system to do their jobs, this problem should have been recorded on a Priority 1 ticket, but may have been recorded as a Priority 2 or 3 ticket. Unscheduled system downtime should be counted and factored into the calculation of the measures regardless of whether it was accurately classified on a helpdesk ticket.
For the third quarter FY 2002 BPR, the PAO calculated the performance by first counting the downtime recorded on Priority 1, 2, and 3 helpdesk tickets, and then by counting only the downtime recorded on Priority 1 tickets. Although the inaccuracies in the helpdesk tickets were widely known, the results of the second count, using only Priority 1 tickets, showed better results and were the only numbers reported for this measure in the BPR presentation.
The following table shows a comparison of the differences in the calculation of the performance measure results using all the helpdesk tickets and only the Priority 1 tickets. If all helpdesk tickets showing system downtime had been used in the calculations, the BPR would have shown that targets were missed for four of the five systems measured.
Table 1: Average Hours Per Day for Systems
Availability
|
Computer System |
All Helpdesk Tickets Counted |
Only Priority 1 Tickets Counted |
Performance Targets (customer
expectations) |
|
ACS |
19.5 |
24.0 |
18.5 |
|
CFOL |
23.7 |
24.0 |
24.0 |
|
IDRS |
16.8 |
20.4 |
17.1 |
|
ISRP |
12.8 |
18.8 |
12.9 |
|
SCRIPS |
13.9 |
18.0 |
14.1 |
Sources: MITS Services Cumulative Year-to-Date by
Month Report,
3rd Quarter Fiscal Year 2002 BPR, the Service Level Agreement, and
the Strategy and Program Plan.
Systems performance was measured after the hours requested by
users
The “Number of Hours Availability” and “Percent Systems Availability” measures were also inaccurate because the PAO measured the performance of the above computer systems outside or after the core hours requested by the users. Late in FY 2002, MITS Services signed a Service Level Agreement that provided for certain levels of service for the ACS, CFOL, and IDRS users during specific periods of the day. The following table presents the specific hours requested by the users.
Table 2: Specific Hours Requested by the Users
|
Computer System |
Sunday |
Monday – Friday |
Saturday |
|
ACS |
0 Hours |
19 Hours |
16 Hours |
|
CFOL |
24 Hours |
24 Hours |
24 Hours |
|
IDRS |
8 Hours |
19 Hours |
17 Hours |
Source: The MITS Service Level Agreement, Attachment 1.
In its calculation of the “Number of Hours of Availability” and the “Percent Systems Availability” measures, the PAO did not count only the specific hours requested by the users. The PAO counted all 24 hours in the day (see Table 1, third Column). This practice distorted the intent of the measure, which was to report whether MITS Services provided the level of service it agreed to with its customers. If MITS Services did not keep a key tax processing system running during the core work hours requested by the users, this poor performance might not be reflected in the measure because MITS Services measured the performance of the system after regular work hours when the users are off-duty and did not have a need for the system.
To be meaningful, these two measures should be calculated using the computer system’s performance during the core hours specifically requested by the users. Goals should be established and measurements against those goals should be based on these core hours.
Management indicated that because the hours of availability needed by the users vary depending on the season, the calculations to determine these measures become complex. As a result, in the past, they have chosen to simplify the calculations and use a 24-hour schedule.
The presentation of measures in the Business
Performance Review was not clear
The “Number of Hours of Availability” and the “Percent Systems Availability” measures were also reported differently in the BPR from the other business results measures. They were presented in a statistical table on the last page of the measures section in the BPR. Unlike the other measures, which were presented in graphs and actual performance could be easily compared to performance goals, the presentation of these two measures was unclear. Trends could not be identified, and there was no explanation or discussion of the results.
Management indicated they made a conscious decision not to present these measures graphically. They stated that it was their option how to present the various measures in the BPR, and they determined to report these measures in the tabular format.
To ensure that all measures accurately and clearly represent the performance of MITS Services operations, the Deputy Commissioner Modernization and Chief Information Officer should direct the PAO to:
4. Count the unscheduled downtime recorded on all helpdesk tickets and factor this total into the calculations of the “Number of Hours of Availability” and “Percent Systems Availability” measures. The PAO should continue this practice until the misclassification and inaccurate recording of the helpdesk tickets is resolved.
Management’s Response: The PAO will work with the End User Equipment and Services and Enterprise Operations organizations to count the unscheduled downtime until the misclassification and inaccurate recording of the helpdesk tickets is resolved.
5. Calculate the “Number of Hours of Availability” and “Percent Systems Availability” measures based on the performance of the computer systems during the core or specific hours requested by the users.
Management’s Response: The PAO will work with the End User Equipment and Services and Enterprise Operations organizations to calculate the scheduled hours of availability measures based on the Master Service Level Agreement.
In our previous review, we reported the PAO had difficulty verifying and validating data used for the business results measures. The GPRA requires agencies to describe the means to be used to verify and validate their measures. In response to our report, MITS Services indicated it would employ an enhanced validation and verification process using generally accepted accounting principles as a basis.
The PAO developed good verification and validation procedures and documented them in its policies and procedures guide. The procedures require independent verification and validation work to be conducted on the calculation of the measures. Analysts who perform the verification and validation are required to keep records of the work and maintain these records in the form of working papers. These working papers are required to contain sufficient detail to enable an experienced analyst with no previous connection with the verification and validation activity to follow the working papers and arrive at the same conclusions and judgments. Analysts performing the verification and validation work should prepare written reports detailing the results of the work and submit the reports to the PAO manager. If followed, these procedures would help ensure the accuracy of the measures.
However, we found that there is no substantial ongoing verification and validation for seven of the nine business results measures. Management indicated that this validation did not occur because they were still in the process of correcting errors in the underlying data due to the transition period between the old helpdesk system and the new system. This transition resulted in issues with the accuracy of the helpdesk ticket data. As a result, time constraints did not permit them to follow the procedures as documented.
The following measures were not validated as the procedures describe:
· Ticket Activity.
· Mean Queue Time.
· Percent Resolution at First Contact.
· Percent Tickets Resolved on Time.
· Percent Subsecond System Response Time.
· Percent Systems Availability.
· Number of Hours of Availability.
The only verification and validation activity that is conducted on an ongoing basis for these measures is a customer survey. Although a survey does provide some indication of the accuracy of the business results, it is not adequate in itself to provide assurance that supporting data is accurate and that results are accurately calculated and presented.
Without verification and validation, there is an increased likelihood that errors might not be detected, measures may be improperly calculated, or errors in the underlying sources could go undetected. As a result, management or oversight bodies could take inappropriate actions based on the inaccurate results reported in the measures.
Management Actions: The End User Equipment and Services organization and PAO management recently conducted a helpdesk ticket workshop to identify some of the issues with inaccuracies with the helpdesk tickets. This workshop also discussed the need to establish a cyclical ticket review process. These activities should be a good start towards developing an ongoing verification and validation activity for those measures based on helpdesk ticket accuracy.
To further ensure the accuracy of the MITS Services business results measures, the Deputy Commissioner Modernization and Chief Information Officer should direct the PAO to:
6. Perform ongoing verification and validation activities for all of the business results measures and supporting data reported in the MITS Services BPRs.
Management’s Response: The PAO, in cooperation with MITS Services providers, will implement verification and validation processes.
The “Number of RISs Received for Filing Season” measure is not an appropriate measure because it does not describe performance by the MITS Services organization. This measure reports the number of RISs the MITS Services Business Systems Development organization receives each month with requested operational dates within the IRS’ annual filing season.
The GPRA indicates that measures should assess results in achieving performance goals, and OMB Circular A-11 indicates that measures should show how the agency is doing in achieving its programs. Because many factors outside the control of the MITS Services organization directly influence the number of RISs received during the IRS’ annual filing season, it is inappropriate for MITS Services to set a performance goal for this measure.
However, this measure provides valuable insight into why fluctuations occur in another measure, the “Systems Delivery Commitments Met” measure, which reports the number of RISs completed on time compared with those completed late. Therefore, we believe the “Number of RISs Received for Filing Season” measure should be changed from a business results measure to a diagnostic indicator.
When we discussed this measure with the PAO, the director indicated that it was selected because the business area was firm in its stance that the number of RISs received was an important indicator of its ability to perform.
If the “Number of RISs Received for Filing Season” continues to be reported as a business results measure, it could cause the Business Systems Development Office to be evaluated on things beyond its control. For example, if this measure shows an increase or decrease, IRS management, the IRS Oversight Board, or the Congress could misinterpret this measure and assume MITS Services is responsible for correcting these aberrations.
To ensure that the MITS Services’ performance is accurately measured, the Deputy Commissioner Modernization and Chief Information Officer should direct the PAO to:
7. Move the “Number of RISs Received for Filing Season” from the nine business results measures and report it solely as a diagnostic indicator. This diagnostic indicator should be used to help management better understand the other business results measures.
Management’s Response: The PAO, with concurrence from the Director, Business Systems Development, will report the “Number of RISs Received for Filing Season” solely as a diagnostic indicator.
The challenge of maintaining an accurate inventory of automated data processing equipment has impacted MITS Services for many years. The problem results in inadequate financial accountability and the inability to identify, safeguard, and manage IRS assets. Although this is not one of the current management challenges as reported to the Congress, this issue is a consistent area of interest of oversight bodies, including the Congress, and a key goal for which measures should be developed. In our previous review, we recommended the PAO coordinate the efforts and work with the appropriate MITS Services organizations to develop business results measures to measure the progress in achieving this goal.
In this follow-up review, we did not identify any measures under development or in testing that would address this issue. Management indicated that their attempts to develop an inventory measure have been unsuccessful because the underlying data was unreliable and error-prone. In addition, the ITAMS, from which these measures would be calculated, is not completely automated in the inventory area.
Without a specific measure, MITS Services cannot report its progress in creating and maintaining an accurate inventory of automated data processing equipment. Therefore, it is difficult for IRS management or the Congress to assess the progress in addressing this challenge, and management does not have a reference to determine whether the IRS is improving in this area.
To measure the MITS Services’ progress in developing an accurate inventory system, the Deputy Commissioner Modernization and Chief Information Officer should direct the PAO to:
8. Develop a business results measure and identify data sources to address the management challenge of creating and maintaining an accurate automated data processing inventory system.
Management’s Response: The PAO, working with the End User Equipment and Services organization, will develop a business results inventory measure and identify data sources.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine if the business results measures established for the Modernization, Information Technology and Security (MITS) Services organization were adequately defined, met legal standards, and supported the Internal Revenue Service’s (IRS) balanced measurement system. We also evaluated the effectiveness of the actions taken to correct the issues identified in our previous report. To accomplish these objectives we performed the following tests:
I.
Determined the relevance,
usefulness, and legal compliance of MITS Services business results measures.
A. Evaluated
the Business Performance Review process to determine whether it provides useful
information for management decision-making.
B. Secured and reviewed the IRS’ Fiscal Year (FY) 2003 Annual Performance Plan and the FY 2003 Congressional Justification to verify whether any MITS Services business results measures were included and reported.
C. Reviewed the revised set of nine MITS Services business results measures to determine if they met legal standards.
II.
Determined whether the MITS Services has adequately
addressed the findings reported in our previous audit report on MITS Services
business results measures.
A. Determined whether data sources were identified for the nine business results measures.
B. Evaluated whether the Performance Assessment Office (PAO) actively conducts an ongoing verification and validation of the business results measures data to provide assurance of its accuracy and completeness.
C. Determined whether the PAO ensured that the appropriate diagnostic indicators were matched to the nine new MITS Services business results measures.
D. Evaluated whether the PAO clearly and fully defined the MITS Services business results measures and included them in Business Performance Reviews.
E.
Determined
the status of actions taken by the PAO to coordinate its efforts with the MITS
Services line organizations to use a repeatable process to develop business
results measures for high-risk challenges reported to the Congress, including
the security of the IRS’ information systems, systems modernization, and
accuracy of the automated data processing inventory system.
Appendix II
Major Contributors to This Report
Gary V. Hinkle, Acting Assistant Inspector General for
Audit (Information Systems Programs)
Scott A. Macfarlane, Director
Tammy L. Whitcomb, Audit Manager
W. Allen Gray, Senior Auditor
Michelle Griffin, Senior Auditor
Paul M. Mitchell, Senior Auditor
George L. Franklin, Auditor
Albert C. Greer, Auditor
Appendix III
Acting Commissioner N:C
Chief Financial Officer N:CFO
Chief, Information Technology Services M:I
Director, End User Equipment and Services M:I:EU
Director,
Enterprise Operations M:I:EO
Director, Financial Management Services M:FM
Director, Portfolio Management M:R:PM
Director, Resources Allocation and Measurement M:R
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and
Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:F:M
Audit Liaisons:
Chief, Information Technology
Services M:I
Program
Manager, Program Oversight & Coordination
M:R:PM:PO
Appendix IV
Summary
of Issues Identified With the Nine
|
Business Results Measures |
Underlying Data Unreliable |
Invalid Sampling Methodology Used |
Inadequate Verification and Validation of Data |
Not Accurately and Clearly Presented |
Does Not Measure MITS Services Performance |
|
User Support |
|
|
|
|
|
|
Percent Tickets Resolved on Time |
X |
|
X |
|
|
|
Ticket Activity |
X |
|
X |
|
|
|
Mean Queue Time |
|
|
X |
|
|
|
Percent Resolution at 1st Contact |
X |
|
X |
|
|
|
Data Access and Communications |
|
|
|
|
|
|
Percent Systems Availability |
X |
|
X |
X |
|
|
Number of Hours Availability |
X |
|
X |
X |
|
|
Percent Subsecond Response Time |
|
X |
X |
|
|
|
Applications Support |
|
|
|
|
|
|
Number of RISs Received |
|
|
|
|
X |
|
System Delivery Commitments Met |
|
|
|
|
|
Appendix V
Management’s Response to the Draft Report
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version
of the report on the TIGTA Public Web Page.