Employee Training Data Have Not Improved Enough to Determine Training Cost or Effectiveness

TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

Employee Training Data Have Not Improved Enough to Determine Training Cost or Effectiveness

September 29, 2006

Reference Number: 2006-10-176

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

Phone Number | 202-927-7037

Email Address | Bonnie.Heald@tigta.treas.gov

Web Site | http://www.tigta.gov

September 29, 2006

MEMORANDUM FOR CHIEF HUMAN CAPITAL OFFICER

FROM: Michael R. Phillips /s/ Michael R. Phillips

Deputy Inspector General for Audit

SUBJECT: Final Audit Report – Employee Training Data Have Not Improved Enough to Determine Training Cost or Effectiveness (Audit #200510032)

This report presents the results of our review of the Internal Revenue Service’s (IRS) implementation of the Enterprise Learning Management System (ELMS) to determine whether the anticipated functionality and benefits expected from the System were achieved. This audit was conducted as part of our Fiscal Year 2005 annual audit plan and serves as a follow-up review to a prior audit on training conducted in Fiscal Year 2003.[1]

Impact on the Taxpayer

The ELMS is the system of record for employee learning and is used for managing all aspects of training activity in the IRS. Incorrect and incomplete training data from the ELMS limit the IRS’ efforts to fully evaluate the effectiveness of its training efforts. This makes it more difficult to determine if any particular training program(s) should be expanded or reduced and to determine which employees may need additional or different training, so they can provide the quality service taxpayers expect from the IRS.

Synopsis

The IRS had implemented some of the corrective actions proposed in response to our prior audit of training information and made the new training system, the ELMS, available to all IRS employees in January 2005. However, the IRS has declined to build validity checks into the ELMS to prevent errors, and some of the functionality expected to be delivered in December 2004 has yet to be implemented for use by the IRS business units, which has delayed some of the benefits. In addition, the new System still does not enable the IRS to generate reports of cost data or reports to fully assess IRS training efforts. As of July 2006, the ELMS had not been fully implemented and was unable to deliver some of the functionality and benefits that had been expected by December 2004. The delays resulted from a significant increase in new requirements for the ELMS. The increase in the number of requirements caused the percentage of customizations[2] and configuration changes[3] to the standard software to increase from less than 10 percent to about 40 percent. Furthermore, the IRS did not capture and report all of the costs associated with implementation of the ELMS. These conditions, in addition to the discontinuation of the IRS’ previous training data system, leave the IRS with limited ability to manage and track its training programs.

Recommendations

We recommended the Chief Human Capital Officer (CHCO) add validity checks to the ELMS and work with the Chief Financial Officer to identify a practical solution that will enable the Integrated Financial System[4] to track IRS training costs. In addition, we recommended the CHCO provide a revised schedule, based on the maximum benefit provided to the IRS, for the System functionalities that have not been made available yet and develop a process for tracking all costs for developing and implementing new systems so the full cost of the project can be reported.

Response

The CHCO agreed with all of our recommendations. The CHCO will identify ELMS fields that could be customized or configured to support point-of-entry data validation and will work with the Chief Financial Officer to identify training costs and determine how those costs would most appropriately be captured. Furthermore, the CHCO will review and update the ELMS Master Project Plan to ensure critical IRS business requirements not previously implemented are prioritized in coordination with the business units. This action is subject to IRS and Department of the Treasury Governance approval and resource availability. Lastly, the CHCO will work with the Chief Financial Officer organization to identify all statute-required project costs for nonmajor IRS Internal Management Systems. If sufficient funds are available and agreement is reached to capture the cost in the Integrated Financial System, a development and release strategy will be in place by December 31, 2007. Management’s complete response to the draft report is included as Appendix IV.

Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Nancy A. Nakamura, Acting Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs), at (202) 622-8500.

Table of Contents

Background

Results of Review

Recommendations 1 and 2:

Significant Functionality Expected in the Enterprise Learning Management System Has Been Delayed, and Development Costs Have Been Understated

Recommendations 3 and 4:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Management’s Response to the Draft Report

Abbreviations

ACES	Administrative Corporate Education System
CFO	Chief Financial Officer
CHCO	Chief Human Capital Officer
ELMS	Enterprise Learning Management System
FY	Fiscal Year
HCO	Human Capital Office
IFS	Integrated Financial System
IRS	Internal Revenue Service
LMS	Learning Management System
RTM	Requirements Traceability Matrix

Background

Management and tracking of employee training within the Internal Revenue Service (IRS) involve a significant number of records because the IRS can have as many as 105,000 employees with training histories and must track an estimated 4.4 million training courses that have been taken by these employees. Training data are also an important part of the IRS’ Human Capital Strategy because the data help the IRS ensure employees stay current in their job skills, develop new skills to move to other business units, and have skills needed for higher grades with more responsibility. The IRS training management information system has maintained training data for the IRS, including courses taken by employees, since the 1990s. The IRS Fiscal Year (FY) 2006 training budget was $140 million. Historical training cost data for prior years are shown in Figure 1.

Figure 1: IRS Training and Training Travel Budgets for
FYs 2003 – 2006

Fiscal Year	Training (in millions)	Training Travel (in millions)	Totals (in millions)
2003	$42	$55	$97
2004	$36	$58	$94
2005	$33	$64	$97
2006*	$44	$96	$140

Source: IRS Chief Financial Officer (CFO). * Data are through March 2006.

Previously, we reported[5] that the Administrative Corporate Education System (ACES) was used to schedule training, document the training that has been provided, and assist in budget tracking and projection for all IRS employees. However, the IRS business units did not use the ACES consistently to assist in budget tracking and projection, the Office of Chief Counsel generally did not use the ACES, and there were errors in and omissions from the ACES. Additionally, the Office of Strategic Human Resources[6] staff did not have the cost data needed to assess training efforts. Based on these issues, we recommended all IRS units use the same training tracking system with built-in validity checks and the IRS use a course numbering system that prevented duplications. We also recommended the IRS develop the cost data needed to report on its training efforts.

At that time, the IRS was in the process of procuring and developing a new Enterprise Learning Management System (ELMS) that would replace the ACES. The ELMS was intended to address most of the issues we had raised and to provide learning delivery, instructor-led course tools and resources, content, performance, competency management,[7] and collaborative learning.[8] The IRS retired the ACES in March 2005 because it determined the ELMS had the capability to perform all of the functions of the ACES.

The ELMS consists of two primary commercial off-the-shelf applications: the Learning Management System (LMS) and the Learning Content Management System. The LMS is expected to provide comprehensive learning management capabilities for employees, managers, and training specialists including tailored instruction, competency management, training administration, and launching of e-learning content.[9] The Learning Content Management System is expected to provide a structured environment to develop, store, maintain, and reuse content.

In Calendar Year 2003, the Office of Management and Budget noted that both the IRS and the Office of Personnel Management were requesting funds for their individual e‑learning[10] projects. Consequently, the Office of Management and Budget designated the Office of Personnel Management as the executive agent responsible for developing a Government-wide e-learning program as one of several E-Government initiatives under the President’s Management Agenda. The IRS was required to combine its ELMS development effort with the Government-wide project and work under the direction of the Office of Personnel Management. Additionally, the application and data associated with the ELMS would reside on computer systems maintained by the Office of Personnel Management. Within the IRS, the Learning and Education function in the Human Capital Office (HCO) is responsible for the management of employee training.

Section 201 of the Federal Workforce Flexibility Act of 2004[11] requires that each Federal Government agency, on a regular basis, assess how each of its training plans accomplishes or effectively promotes the agency’s specific performance plans and strategic goals. Agencies should modify those training plans or programs to ensure they continually meet and support specific agency-established performance plans and strategic goals. While implementing procedures for this law have not yet been published by the Office of Personnel Management, we believe the IRS should start tracking accurate data showing not only the type of training provided but the direct and related support costs of each course. The IRS should also develop some systemic method(s) to assess course quality to effectively evaluate its training efforts. The data taken together should allow the IRS to compute a return on training investment and aid the IRS in deciding which courses or training methods provide appropriate value in support of the IRS’ overall strategic plan.

This audit was conducted as part of our FY 2005 annual audit plan and serves as a follow-up review to our prior audit on training conducted in FY 2003. This review was performed in the Office of the Chief Human Capital Officer (CHCO) at the IRS National Headquarters in Washington, D.C., and the Dallas, Texas, post-of-duty during the period October 2005 through July 2006. The audit was conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.

Results of Review

Although the IRS purchased the ELMS and made it available to all IRS employees in January 2005, all of the planned functionality has not yet been fully implemented for use by the business units and it does not enable the IRS to address some of the recommendations we made in a prior report. These conditions leave the IRS with a limited ability to track and evaluate its training efforts. ELMS deficiencies include the lack of built-in validity checks to prevent errors and the inability to produce cost data or to generate reports to assess IRS training efforts. To produce cost data and to generate reports to assess IRS training efforts would require a manual collection and review of the data.

Validity checks were not built into the System

The IRS has completed some of the corrective actions recommended in our prior report. For example, the ELMS course numbering system eliminated the use of general course numbers,[12] and the old ACES course numbering pattern is not a part of the ELMS. Although the ELMS is able to generate unique course catalog numbers (so duplication of course numbers is prevented), the IRS did not modify the ELMS to include systemic validity checks as we recommended. The IRS Learning and Education function staff declined to develop validity tests and advised us the ELMS required fewer manual entries, which they believed would reduce the risk of input errors. In addition, they planned to design reports to assist with ensuring the accuracy of data entry.[13]

With the new ELMS, quarterly “item compliance reports” on key data are to be issued to the business units by the HCO staff. The business units will have 15 workdays to update and correct the data. These reports are expected to identify instances in which required data are missing and should find other errors, such as the input of a travel cost over $5,000 for a 1-day class or an unreasonable start date (e.g., April 26, 1990). However, the reports developed as of July 2006 address only the accuracy of the information in the training course catalog, not the accuracy of employee training data.[14]

We reviewed a copy of an item compliance report dated June 30, 2006, that identified 1,475 courses with incomplete information. The most common errors involved some type of missing data. For example, 361 courses (24 percent) did not identify the audience (types of employees) for whom the training courses are intended, and 813 courses (55 percent) did not include a contact person from whom employees could get additional information about the course. Validity tests to ensure data fields are not blank when added to the ELMS can prevent incomplete information from being entered.

Historically, the HCO has made IRS employees responsible for reviewing their training histories. If they find any errors, employees have been instructed to prepare and submit an ELMS Learning/Teaching History (Form 12201) to their business unit training administrators, who will make corrections to the training histories. However, if an employee does not review and submit changes to the business unit training administrator, the information will remain inaccurate in the ELMS. For example, as many as 10,000 employees in one of the IRS business units entered information about a particular course in the ACES. In some cases employees entered the course multiple times and in other cases employees entered the course number as training hours (4 hours was the maximum that should have been claimed), which further compounded the errors. The cumulative effect of these errors is that as much as 90 percent of the hours associated with that category of training may be inaccurate and has been carried over to the new ELMS.

While the new System is expected to limit the amount of manual data entry required, relying on a manual review of individual training records by employees is the least effective internal control. We believe it will benefit the IRS to have in place a systemic mechanism that detects data errors before they are posted to the ELMS rather than relying on compliance reports and employee reviews to identify errors that have been entered into the System.

Cost data to assess training efforts are still incomplete

The HCO staff experienced delays in the following areas: (1) addressing concerns we previously raised about the IRS’ inability to provide (in a reportable form) cost data needed to assess its own training efforts fully and (2) developing a methodology to allocate overhead costs. We reported that the IRS cannot provide detailed cost information down to the course or student level. This cost information is important because the IRS uses a number of vendor contracts in its training efforts, and, without detailed information, the IRS cannot validate the contract costs by tracing costs to the specific training courses or students. These costs include:

Fees paid to private contractors/instructors.
Course books/materials.
Equipment.
Classroom or hotel rentals.
Course development.

To address our concerns, the IRS planned to ensure its Integrated Financial System (IFS)[15] and the new LMS collectively would provide appropriate cost data. Corrective actions were to be completed by October 2005. However, as of May 2006, the HCO staff were still working to produce the required reports. The HCO staff stated that the detailed cost information needed must be obtained from systems controlled by the IRS CFO, the most important of which is the IFS. Although the CFO is working with the HCO to identify which capabilities currently in the IFS may assist in extracting training cost data, the CFO had indicated as of July 2006 that, according to the information available at the time, the IFS will not be reprogrammed or interfaced with the ELMS.

Recommendations

Recommendation 1: The CHCO should ensure that, to the extent possible, validity checks are built into the ELMS to ensure the data entered into the System are accurate.

Management’s Response: IRS management agreed with our recommendation. The CHCO will identify ELMS fields that could be customized or configured to support point-of-entry data validation. A cost/benefit analysis will be conducted and completed by April 30, 2007. If justified by the analysis and supported by the business divisions, the ELMS Master Plan will be revised to reflect a development and release strategy by September 30, 2007. However, this will be subject to coordination with the Department of the Treasury LMS Governance Committee if it is determined that potential changes would affect the consolidated LMS Architecture.

Recommendation 2: The CHCO should work with the CFO to identify a practical solution that will enable the IFS to track IRS training costs. The solution should be used IRS-wide and be accessible to all business units. Training costs should be assigned to the individual course level, if possible.

Management’s Response: IRS management agreed with our recommendation. The CHCO will work with the CFO and the IFS Program Office to identify additional training costs and to determine how those costs would most appropriately be captured. The CHCO has submitted a Change Request to the Office of the Associate CIO for Business Systems Modernization to initiate this effort. If an agreement can be reached to capture the cost in the IFS and funding is available, the CHCO will have a development and release strategy in place by December 31, 2007.

Significant Functionality Expected in the Enterprise Learning Management System Has Been Delayed, and Development Costs Have Been Understated

The ELMS is the enterprise e-learning infrastructure[16] for the IRS; it is expected to provide comprehensive learning management capabilities for employees, managers, and training specialists including tailored instructions, competency management, training administration, and launching e-learning content. The IRS has reported that the ELMS has achieved functionality in managing the areas of curriculum and certification, registration, standard reporting, training event management, training records, and parts of content management. We selected a sample of 32 of the 165 functional requirements scheduled for ELMS Releases 1 and 2 and verified that 31 of the requirements were implemented. The other requirement could not be verified because it was not fully implemented. As of July 2006, the functionality of competency management and most of the content management and custom reporting had not yet been realized. In addition, the IRS has not identified when it will address competency management, and the Learning Content Management System project is handling most of the content management.

In Calendar Year 2002, the HCO, with vendor support, prepared an LMS Statement of Work that was submitted to multiple vendors to find a company that could provide an LMS for the IRS. The IRS business units provided input on the functionality requirements the System should offer. The business units’ input resulted in the creation of an LMS functional requirements traceability matrix[17] (RTM) that was comprised of 241 requirements. Through a series of releases, the following six areas originally were expected to be produced by the LMS:

Learning delivery.

Instructor-led course tools and resources.
Content management.
Performance management.
Collaborative learning.
Competency management.

At that time, the IRS expected to implement a commercial off-the-shelf system with little or no customization. The IRS received a proposal from the eventual winner of the LMS contract indicating a commercial off-the-shelf product typically could be installed and implemented in 2 months to 5 months, although the contract ultimately required the ELMS to be fully implemented by December 2004.

An analysis of the February 2005 RTM indicated the number of requirements had increased from 241 to 466 (93 percent increase)[18] between October 2002 and February 2005. The significant increase in requirements, and the resultant design changes based on the new requirements, were identified through a series of workshops that involved the ELMS staff, the HCO, the IRS business units, the ELMS vendor, and other stakeholders. The increase in the number of requirements caused the percentage of customizations[19] and configuration changes[20] to the standard software to increase from less than 10 percent to about 40 percent and added to the time needed to implement the System. As a result of the increase in vendor customizations and IRS internal configurations, every time the IRS upgrades the ELMS application, these requirements will need to be upgraded individually, resulting in additional time and costs.

Additionally, the project team explained that the decision by the Office of Management and Budget to direct the IRS to combine its project with the Government-wide e-learning initiative contributed to the project’s overall delay. The requirement to place IRS data in servers outside of the IRS’ own domain added about 3 months to the project as the team worked to address security concerns that resulted from this decision.

In July 2006, the IRS reported that three releases of the ELMS had been implemented. However, we were unable to determine the exact extent of the functionality that had been implemented because the ELMS staff had not kept the RTM current. The ELMS staff advised us that IRS resource constraints prevented them from keeping the RTM current. The last known independent validation of the RTM was conducted in July 2005.

In July 2005, an IRS subcontractor completed a validation of all the ELMS requirements (324) that were scheduled for implementation in Releases 1 and 2. At that time, Release 3 had not been implemented. We reviewed the number of functional requirements (165) in Releases 1 and 2 to determine what the IRS expected from the ELMS and whether the functionality had been achieved. The subcontractor validated 133 of the 165 (81 percent) functional requirements to ensure they were implemented as planned. The other 32 requirements (19 percent) had not been validated due to one of the following reasons:

Thirteen requirements were postponed to a future release.
Twelve requirements failed the validation.
Seven requirements were not included in the validation.

The ELMS was expected to increase the number of e-learning hours delivered to 30 percent of total training in FY 2003 and to 40 percent in FY 2005. Additionally, the System was to reduce travel costs associated with training. Neither goal has been accomplished. Since FY 2002, the number of e-learning hours delivered has increased by only about 3 percent from FY 2002 through FY 2004 (the last full year of data available), and travel costs have increased. Figure 2 shows the number of training hours counted as e-learning and the total number of all training hours incurred over the 3½ years from FY 2002 to the middle of FY 2005.

Figure 2: Total Training Hours Compared to e-Learning Hours Delivered for
FY 2002 – February 2005

	FY 2002	FY 2003	FY 2004	FY 2005 (as of 2/05)
Total Training Hours	7,286,439	6,938,486	6,406,366	1,887,728
e-Learning Total Hours	1,271,033	1,519,926	1,262,374	371,025
Percentage of e-Learning to Total	17%	22%	20%	20%

Source: The ACES.

Actual travel expenses associated with training have increased by about $7 million over the last 4 years and are budgeted at $96 million in FY 2006. Figure 3 shows the amount of training travel expenses incurred over the last 4 years.

Figure 3: Actual Training Travel Expenses for FYs 2002 – 2005

FY 2002	FY 2003	FY 2004	FY 2005
$54,618,000	$48,718,000	$57,902,000	$61,936,000

Source: IRS CFO.

An important anticipated benefit of the ELMS is its ability to provide the HCO and business unit management with training data through the design, development, and implementation of custom reporting. Many of the custom reports included in the System’s development (such as a report showing the numbers and percentage of training hours delivered and their methods of delivery) have not been implemented. The IRS expects it will need to purchase a third-party report writing application and will then need to train some number of IRS employees in the use of the application to generate custom reports as needed.

Another significant anticipated benefit of the ELMS that has not been implemented is the ability for the training staff to better evaluate employee competencies. Competency is determined by comparing the measurable skills, abilities, and personality traits that identify successful employees against defined roles within an organization. Defining competencies and then measuring an employee against them aids in determining potential gaps in an individual’s skills and possible gaps in IRS training plans. Obtaining the tools needed to assess competencies and skills gaps is a significant goal of the HCO, as noted in the IRS Human Capital Strategic Plan.

The creation and delivery of online content[21] was to be addressed in the new LMS. However, some of the requirements under content management have been included in the LMS, while others associated with content authoring are expected to be part of a second system called the Learning Content Management System that is expected to be fully integrated with the ELMS. Widespread use of the System by subject-matter experts to create online content has not been achieved because the IRS is still training employees to use the content authoring features.

Development costs have been understated

The overall cost to implement the ELMS was estimated at about $20 million through 2009 and included budgetary resource expenditures associated with development and implementation of the System. Although these budgeted costs were identified in the final ELMS cost/analysis Business Case,[22] the actual resource expenditures incurred on the ELMS project have not been tracked or recorded by the HCO. As many as 95 employees worked on various ELMS implementation subteams from January 2004 to January 2005. Some of the team members worked part time on the ELMS project and had other duties outside the team. However, the amount of time these employees worked on the ELMS was not formally tracked or recorded. Total salaries for 9 full-time team members amounted to over $726,000 during the development period, and none of their salaries or travel costs have been recorded as a cost of the ELMS project. The ELMS Project Offices estimated that the 86 part-time team members spent about one-third of their time that year working on the project. We estimate one-third of their salary costs would total $ 2.4 million. Furthermore, after January 2005, 15 employees worked part time on the ELMS implementation. The Enterprise Life Cycle[23] document states the project manager is responsible for managing performance, expenditures, schedule, and communications of the project. In addition, the project manager must collect and update performance statistics for technical solutions, costs, resources, and schedule activities.

Because all the costs were not tracked, it is difficult for the IRS to determine the total development cost and to know whether it is complying with Section 5122 of the Clinger-Cohen Act.[24] This Act requires that the project “. . . provide the means for senior management personnel of the executive agency to obtain timely information regarding the progress of an investment in an information system, including a system of milestones for measuring progress, on an independently verifiable basis, in terms of cost, capability of the system to meet specified requirements, timeliness, and quality.”

Recommendations

Recommendation 3: The CHCO should propose a revised schedule for implementation of ELMS System functionalities that were not made available to IRS business units by July 2006. Per the Exhibit 300 cost justification submitted to the Office of Management and Budget, all functionality was to be implemented by December 2004. The revised schedule should be prioritized based on the maximum benefit provided to the IRS and in coordination with business units’ requirements.

Management’s Response: IRS management agreed with our recommendation. The CHCO will review and update the ELMS Master Project Plan to ensure critical IRS business requirements not previously implemented are prioritized in coordination with the business units and will be scheduled for implementation in future ELMS Releases, following completion of the Learning and Education Redesign technology upgrades and the Department of the Treasury LMS Consolidation. Reprioritized requirements will be subjected to IRS and Department of the Treasury Governance approval and will be incorporated in future releases based on resource availability. After approval is granted, the CHCO will prepare a reprioritized Master Plan by December 31, 2007, reflecting a development and release strategy.

Recommendation 4: The CHCO should develop a process for tracking all costs for developing and implementing new systems, including salary and travel costs incurred by IRS employees as well as any contractor costs and hardware/software costs, so the full cost of the project is apparent to IRS executives and other stakeholders.

Management’s Response: IRS management agreed with our recommendation. The CHCO will work with the CFO organization to identify all statute-required project costs for nonmajor IRS Internal Management Systems. If sufficient funds are available and agreement is reached to capture the cost in the IFS, a development and release strategy will be in place by December 31, 2007.

Appendix I

Detailed Objective, Scope, and Methodology

The overall objective of this audit was to evaluate the effectiveness of the ELMS[25] implementation by assessing whether the anticipated functionality and benefits were achieved. In addition, we followed up on the IRS corrective actions taken in response to recommendations made in a prior audit report.[26] To accomplish this objective, we:

I. Determined whether training data and costs were reflected in the ELMS accurately.

A. Determined that the IRS has officially retired its training system (the ACES)[27] and replaced it with the ELMS, as planned. From the ACES, we selected a random sample of 30 IRS employees with training histories from a population of 293 with training histories (these are the same employees reviewed in the prior training audit with training histories established on the ACES as of September 18, 2002). We used a random sample to ensure each employee had an equal chance of being selected.

B. Determined how the IRS keeps track of training costs to ensure they are all accounted for.

C. Determined whether the data provided by the ELMS allow the IRS to more accurately report on training, its effectiveness, and its costs.

II. Identified the IRS’ original expectations for the ELMS and compared them to the System’s functionality as of July 2006.

A. Determined the basis for the IRS developing the ELMS.

B. Determined if any System functionality has not been implemented and assessed the impact on the IRS’ ability to produce complete, accurate, and timely training data. We selected a judgmental sample of 32 LMS[28] functional requirements from a population of 165 functional requirements expected to have been implemented in ELMS Releases 1 and 2 to determine their functionality. We used judgmental sampling because of limited resources and because we were drawing conclusions only about the sampled items.

III. Determined whether the ELMS project is being developed on schedule and on budget.

A. Determined whether the ELMS development and rollout are on schedule.

B. Determined whether IRS management is reporting effectively the cost of implementing the ELMS project.

C. Determined whether management decisions have affected the implementation plan.

IV. Determined whether the benefits anticipated when the project was approved have been realized.

Appendix II

Major Contributors to This Report

Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

Nancy A. Nakamura, Acting Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)

Carl L. Aley, Director

Kevin P. Riley, Audit Manager

Ken Henderson, Lead Auditor

Charles Ekunwe, Senior Auditor

Michael Della Ripa, Auditor

Stephen E. Holmes, Auditor

James Allen, Information Technology Specialist

Appendix III

Report Distribution List

Commissioner C

Office of the Commissioner – Attn: Chief of Staff C

Deputy Commissioner for Operations Support OS

Chief Financial Officer OS:CFO

Chief Counsel CC

National Taxpayer Advocate TA

Director, Office of Legislative Affairs CL:LA

Director, Office of Program Evaluation and Risk Analysis RAS:O

Office of Internal Control OS:CFO:CPIC:IC

Audit Liaisons:

Chief Financial Officer OS:CFO

Chief Human Capital Officer OS:HC

Appendix IV

Management’s Response to the Draft Report

The response was removed due to its size. To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

[1] Information on Employee Training Is Not Adequate to Determine Training Cost or Effectiveness (Reference Number 2003-10-212, dated September 2003).

[2] Customizing requirements requires the assistance of the vendor and results in additional costs.

[3] Configuration changes to requirements can be performed by the IRS.

[4] The Integrated Financial System is the IRS financial management system that provides detailed financial, cost accounting, property accounting, and procurement data to authorized users.

[5] Information on Employee Training Is Not Adequate to Determine Training Cost or Effectiveness (Reference Number 2003-10-212, dated September 2003).

[6] The Office of Strategic Human Resources was replaced by the Office of the Chief Human Capital Officer in February 2004.

[7] Competency management is the process that allows an organization to track an employee’s skills, behavior, and knowledge and assess their competency gaps, to help improve employee performance as well as meet the business goals of the organization.

[8] Collaborative learning is the process of learning through the exchange and sharing of information and opinions among individuals.

[9] This is course or instructional material that can be used and reused that is distributed by a learning management system or learning content management system.

[10] This is learning that is facilitated and supported using information and communication technologies like the Internet and email.

[12] Courses are assigned a unique number for tracking purposes and are logged into a catalog.

[13] September 24, 2003, Memorandum for Treasury Inspector General for Tax Administration From the IRS CHCO.

[14] Our review of a random sample of 30 IRS employees with training histories from a population of 293 did not reveal any errors or obvious omissions in the training histories.

[15] The IFS is the IRS’ core accounting system of record; when fully implemented, it is expected to provide budget preparation, general ledger, accounts payable, accounts receivable, financial reporting, and purchasing functions.

[16] The infrastructure addresses the organization’s existing culture, governing principles, processes, and structures (including technical architecture) that will contribute to e-learning success or failure.

[17] A functional RTM serves as the tracking mechanism to ensure requirements are fully tested and incorporated into the appropriate implementation phase.

[18] The requirements are grouped into six categories, and functional requirements make up the majority.

[19] Customizing requirements requires the assistance of the vendor and results in additional costs.

[20] Configuration changes to requirements can be performed by the IRS.

[21] Online content refers to commercial off-the-shelf products, custom development, testing, integration, and maintenance services.

[22] Business cases are also known as Exhibit 300, Capital Asset Plan and Business Case Summary.

[23] The Enterprise Life Cycle defines the processes, products, techniques, roles, responsibilities, policies, procedures, and standards associated with planning, executing, and managing business change.

[24] Federal Acquisition Reform Act of 1996 (Information Technology Management Reform Act of 1996), Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).

[25] The ELMS was intended to provide learning delivery, instructor-led course tools and resources, content, performance, competency management, and collaborative learning.

[26] Information on Employee Training Is Not Adequate to Determine Training Cost or Effectiveness (Reference Number 2003-10-212, dated September 2003).

[27] The ACES was used to schedule training, document the training that has been provided, and assist in budget tracking and projection for all IRS employees.

[28] The LMS is expected to provide comprehensive learning management capabilities for employees, managers, and training specialists including tailored instruction, competency management, training administration, and launching of e-learning content.