Executive Summary

Objective and Scope

Background

Results

conclusion

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Management’s Response to the Draft Report

Appendix V – Earned Value Management Process

Appendix VI – Glossary of Terms

Executive Summary

The Internal Revenue Service (IRS) reorganized Information Systems (IS) in 1996 to better manage contracts and improve systems development oversight. The reorganization was in response to problems encountered with Tax Systems Modernization (TSM). The congress and the General Accounting Office have had concerns over the ability of the IRS to oversee systems modernization projects.

The IRS also made a commitment to dramatically improve telephone service to taxpayers. Part of the improved service will be achieved by implementing a call routing system that allows calls to be routed to the next available customer service representative anywhere in the country. The customer Service call Router (cScR) project is one of the first IS projects to be managed after the IS reorganization. The use of modern call routing technology is necessary to improve customer service.

The objective of the audit was to assess the effectiveness of the cScR project. Specific objectives were to determine if the cScR project met the business needs of customer Service and whether the project was in conformance with prescribed project management procedures.

Results

While the cScR project did show that the call router technology was adaptable to the IRS, the project management process needs improvement to ensure that business needs are met in accordance with stated performance, cost, and schedule goals. The overall cost of the cScR project will likely be significantly higher than originally budgeted, and the contractor’s pilot evaluation report indicates that the desired improvements in customer service may not be fully achieved.

The IRS has made significant efforts within IS to develop the organizational structure, policies, and Systems Life cycles (SLc) to better manage development projects. However, we have two concerns with IRS’ overall project management process based on our review of the cScR pilot.

Industry best practices require that a business case for an information technology investment be continuously updated to reflect changed conditions. Apparently, the first business case for the cScR project was used only to receive funding. It does not address:

• Using a prototype.
• changing costs/benefits.
• changing schedules.

IS management needs to ensure that the business case is used as a guiding document to ensure business needs are met in accordance with cost, schedule, and performance plans.

The Government Program Management Office is the organization within IS that is responsible for ensuring that systems development projects meet business needs. However, our audit determined that some of customer Service’s performance measures were not included in the tests of the cScR project.

The IRS can improve its project management process by taking actions that will ensure that business needs are met in accordance with cost, schedule and performance goals. These actions should include better use of the business case as a guidance document and ensuring that project offices follow prescribed guidelines.

IS management agrees with the facts cited in the report and is taking appropriate corrective actions. They are completing a plan of action relative to the development of the Systems Life cycle. In addition, an updated approach to project management is being developed in partnership with the Prime Systems Integration Services contractor (PRIME). Management's complete response to the draft report is included as Appendix IV.

Objective and Scope

The overall objective of this audit was to assess the effectiveness of the customer Service call Router Project (cScR). During 1996, the Internal Revenue Service (IRS) restructured its Information Systems (IS) organization to better manage contracts and improve systems development oversight. The cScR project is one of the first IS projects to be managed after the reorganization. The review was conducted between March and November 1998. This audit was performed in accordance with Government Auditing Standards.

The specific objectives were to determine (1) if the cScR project meets the business needs of customer Service, and (2) whether the project was in conformance with prescribed project management procedures.

Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

Background

Two of the major areas that the congress has charged the IRS to significantly improve are:

• Modernizing information systems.
• Improving customer service.

These areas require extensive attention by IS management in order to meet stakeholder and customer expectations.

The IRS has made significant efforts to correct serious management and technical weaknesses in its modernization effort. In response to recommendations by the General Accounting Office (GAO), the IRS made changes to the internal structure of the IS organization. In 1996, the IRS established the Government Program Management Office (GPMO) to oversee the increased use of contractors for system development and program integration.

The primary mission of the GPMO is to:

• Manage the design, development, and deployment of technical information systems solutions.
• Manage and monitor systems development activities to ensure delivery adheres to system and program objectives and standards.
• Manage the activities of all development teams through the establishment of project offices.

The GPMO also coordinates the development of business requirements and, after contract award, manages the contractors.

The Director of the GPMO reports directly to the chief Information Officer. Included in the GPMO are project offices to manage the development of each systems modernization project. Each project office is established under the direction of a qualified Project Director.

One of the reasons that taxpayers have indicated dissatisfaction with the IRS is the difficulty they have in contacting the IRS by telephone. The congress has major concerns with the low level of telephone accessibility. For the 1997 filing season, GAO estimated that taxpayer accessibility was only 52 percent.

At congressional hearings in March 1998, the commissioner introduced a pilot technology program, the Intelligent call Router (IcR). With this IcR, calls can be directed to the next available assistor anywhere in the country. By acquiring this technology, the IRS projected that its "toll-free access rate" would increase from 65 percent in Fiscal Year (FY) 1997 to 86 percent in FY 1999.

The IRS called the IcR pilot the cScR project. This is one of the first projects under the new Modernization Blueprint managed by the GPMO.

Results

While the cScR project did show that the call router technology was adaptable to the IRS, the management process of the GPMO needs improvement to ensure that business needs are met in accordance with stated performance, cost, and schedule goals. This conclusion supports the GAO report that states:

We have two concerns in this area:

• The customer Service call Router Project Did Not Follow All the Requirements of a Disciplined Project Management Process.
• The Government Program Management Office Needs to Better Manage Systems Development Projects.

The customer Service call Router Project Did Not Follow All the Requirements of a Disciplined Project Management Process

The GAO noted that the IRS did not have a process to manage systems modernization projects as investments rather than expenses. The GAO also stated that the IRS lacked comprehensive decision criteria for controlling and evaluating systems modernization projects.

current IRS policies and procedures require the use of a disciplined, decision-making process for planning, managing, and controlling the acquisition of information systems and services. In order to assess the merits of new systems before implementing them, the IRS generally produces a business case.

A business case is a management tool that documents the essential aspects of an information technology project. A business case should:

• Identify the organizational needs that the project meets or proposes to meet.
• Provide information on the benefits, costs, and risks of the project.
• Establish proposed project development time frames and delivery schedules.
• Evaluate all risks and alternatives.

Best practices in the management of information technology investments require that a business case be continuously updated to represent the current situation.

On May 14, 1997, customer Service was granted approval to receive $8.4 million in FY 1997 systems modernization funds towards a projected $27.5 million total cost (through FY 2003) for the installation and operation of the cScR project.

The May 1997 business case was not used as a control to aid IS management in directing and monitoring this initiative during its implementation. Apparently, the document was prepared only to obtain approval from the Investment Review Board (IRB) for this $27.5 million project. The IRB approved this technology investment based on total anticipated benefits of $90.3 million through FY 2003.

Several significant issues were identified in regards to the business case that need to be addressed.

The business case does not contain any reference to a prototype. The business case was prepared in anticipation of the IRS purchasing specific commercial-off-the-shelf call routing software and hardware. There is no reference in the business case either to a pilot program or to any preliminary testing of the software. However, the business case does state that:

cScR project costs will exceed the projected total cost estimates. The IRS' May 1997 business case cost/benefit analysis on the cScR project showed that benefits exceeded costs. The IRS had projected the benefits of the system to be about $90 million (through FY 2003). Included in the benefits were $65 million in increased productivity and $20 million in savings due to reductions in telephone related costs. The projected cost of the system for this same period was about $27 million, which made the ratio between benefits and projected costs to be about 3.3 to 1.

The full cost schedule is shown below:

The cost summary designated all $10.1 million of the investment costs and $1.9 million of the Operations and Maintenance (O&M) costs to be spent in FYs 1997 and 1998. This $12 million represented nearly 44 percent of the total estimated expenditures. The audit showed that FYs 1997, 1998, and 1999 budgets had requested (in total) $27.2 million in funding for the cScR project. This amount is an increase of over $12 million from the total costs originally projected for the same period. So, if O&M costs remain consistent, the original total cost projection for the cScR project will be exceeded by FY 2000.

The schedule for implementation of the cScR project was inaccurate and was not updated. The schedule and milestones in the business case noted that the national rollout would start in March 1998. However, the national rollout did not begin until January 1999.

The anticipated benefits of implementing the cScR project’s technology are not fully supported. The business case presents an estimated benefit totaling $90.3 million (through FY 2003). This calculation is based in part, on benefits that would arise from:

• An increase in customer satisfaction (level of access).
• A reduction in taxpayer burden.
• Savings due to reductions in telephone and equipment costs.

However, the pilot evaluation report, as prepared by the contractor, indicates that, during the test period, the baseline sites actually performed better than the prototype sites in four of the six performance measures, as shown in the table below. If use of the cScR project’s technology does not improve call center performance, customer satisfaction will not increase, and taxpayer burden will not be reduced. In addition, the projected return on investment would be adversely affected.

(From September 24, 1998, Presentation of Results)

1. The IRS needs to use the business case as a tool to better manage information systems development projects. The business case needs to be used as the guiding document to ensure that business needs and goals are met.

Management’s Response: A plan of action is being completed for the strategies and dates relative to the development and distribution of the Systems Life cycle (SLc).

The Government Program Management Office Needs to Better Manage Systems Development Projects

The business case for the cScR project establishes that:

The GPMO was established to oversee the increased use of contractors for systems development and program integration. The primary mission of the GPMO is to deliver a system that meets performance, schedule, and cost requirements. The GPMO also coordinates the development of business requirements; and, after contract award, manages the contractors.

The SLc defines the policies, processes, and products for managing information technology investments from conception, development, and deployment through maintenance and support. The SLc provides the framework for a disciplined approach to systems development and management.

An analysis of the various documents and reviews essential to the management of the cScR project indicated that responsibility for many of the documents and reviews was entrusted to the contractor.

At least 27 of the 76 essential documents/reviews of the cScR project were shown as "to be provided by the vendor" or "internal to the vendor’s own process." The products to be supplied by the contractor included the integration testing reports and a pilot evaluation report. The contractor was also noted as having the responsibility to conduct and complete a risk assessment and a release test readiness review. All of these are significant features of the SLc methodology.

In the May 1996 TSM Progress Report submitted to the congress, the Secretary of Treasury stated:

By not being sufficiently involved in the completion of various required products and reviews, and allowing the contractor to carry out IRS requirements, the GPMO did not assume a management role in the development of the cScR project. The GPMO cannot acquire the skills and experience needed to implement a mature, disciplined, and repeatable software acquisition process unless they are actively involved.

The GPMO did not adequately fulfill its project manager role in two areas:

• Obtaining complete agreement between customer Service and the contractor on the performance indicators/measures to be used.
• Ensuring that only agreed upon measures were reported and used to evaluate the effectiveness of the cScR project.

Several of the documents required by the SLc were reviewed. The Metrics Working Paper (Pilot Evaluation Plan) identifies, details, and defines the performance indicators that would be used to measure the effectiveness of the cScR system.

The contractor prepared the Metrics Working Paper and stated that the measures shown in the table below would be used to evaluate the success of the cScR project.

The audit showed that, on several occasions, customer Service had requested an additional performance measure, Actual ASA (Absolute Values), be used in the cScR project. The contractor’s Presentation of Results does not include this performance measure. This audit also showed that the contractor had agreed, in October 1997, to add another performance measure, Total Length of call. This performance measure also does not appear in the Presentation of Results. However, the document does include a performance measure labeled Agent Available Time that confirms increased efficiency is achieved with call router technology.

The GPMO, as project manager, was responsible for ensuring that the cScR project was conducted in a test environment that would provide consistent data to measure against the desired business results. This audit identified several areas that may have had a negative effect on the study’s results.

Site management practices at the prototype sites were not uniform during the test period. customer Service operations at the prototype sites were allowed to continue in a "business-as-usual" manner. Work practices, such as agent skill groupings, may have affected the results of the project. Work practices varied between pilot and control sites, and pilot sites were not even consistent among themselves. Therefore, conclusions should not have been drawn based on the results achieved.

In addition, computer information discrepancies between the controller and the pilot sites affected data that was necessary for analysis of the project’s performance. Both customer Service and the contractor noted problems in the database and experienced delays in fine-tuning the computer program. customer Service was confident that reliable and consistent statistics would become available as of June 1, 1998. However, the Presentation of Results contains a measure (ASA for the 1040 product line) that produced positive results while using data collected from April 13, 1998 to June 25, 1998.

Issues, such as site staffing, local operating procedures, and the time frames for valid data collection, should have been resolved by the GPMO before the cScR project was allowed to begin. The contractor reiterated an issue with the pilot environment with this statement from the Presentation of Results:

The Internal Revenue Manual requires that computer security be considered in the SLc process to ensure that security controls are incorporated into new information systems and in significant modifications of existing systems. IRS security requirements call for all information systems to protect taxpayer data and to ensure that taxpayer information is disclosed only to authorized parties.

Pending accreditation, an Interim Authority to Operate (IAO) is permitted for a new system that is in an advanced test phase requiring actual operational data for final design. A security survey (risk assessment) and security plan are required before approval of an IAO. customer Service granted an IAO for the cScR project on March 9, 1998.

The contractor prepared the risk assessment and its related documents. The insufficient involvement by IRS personnel raises a great deal of concern. A risk assessment documents security risks such as security threats, vulnerabilities, and current and proposed safeguards. The contractor is not ultimately responsible for the security of taxpayer data. Instead, this is the sole responsibility of the IRS.

1. The GPMO needs to design and implement management controls to ensure that its project offices are following prescribed guidelines. The project offices need to ensure that stated business needs are met in all performance, cost, and schedule parameters. A disciplined process, such as Earned Value Management, may assist the project offices in managing development contracts. See Appendix V for an overview of the Earned Value Management process.

Management’s Response: A program level approach is being developed in partnership with the Prime Systems Integration Services contractor (PRIME).

conclusion

The IRS has made improved customer service a priority and it has achieved results in improving toll-free telephone level of access. However, to successfully integrate modern technology into current programs at an acceptable cost, the IRS must make continued improvements in its project management capabilities.

Detailed Objective, Scope, and Methodology

The overall objective was to provide an assessment of the customer Service call Router Project (cScR). To accomplish the objective, the following audit tests were performed:

1. Determining whether an adequate business case had been developed to cost justify the need for the new call router initiative by:
0. Reviewing the business case to determine if any alternatives were listed.
1. Assessing whether the documented costs of the new system were justified by the documented benefits.
2. Reviewing the concept of Operations, systems requirements, and the Statement of Work for the cScR project to determine whether the project met the documented business needs.
3. Reviewing documented requirements to ensure proper controls exist regarding taxpayer privacy and security.
4. Reviewing General Accounting Office (GAO) reports and Audit reports pertaining to telephone systems and services.
5. Interviewing personnel from the Illinois Institute of Technology Research Institute to gain an understanding of telecommunications.

1. Determined whether the management and development of the project was in conformance with prescribed Information Systems (IS) project management procedures. This was accomplished by:
1. Reviewing GAO reports on topics such as project management, Tax Systems Modernization, and IS Technology.
2. Determining whether the new Systems Life cycle methodology (or some other structured methodology) was followed during project development and implementation.
3. Determining what role the Government Program Management Office, Program Management and control Division, played from a contractor management perspective. To do so, we determined whether plans were in place to:
1. Manage the design, development, and deployment of the technical information systems solutions.
2. Manage and monitor the contractor’s systems development activities to ensure delivery of a quality product.
3. Assure the project is completed within planned time schedules and costs.
4. Determining if the Systems Standards and Evaluation Office provided guidance and oversight to assure applicable security standards were followed.
5. Determining if the Performance Management Office provided necessary assistance in analyzing business cases and baselining systems performance and quality.
6. Determining if the Architecture, Engineering, and Infrastructure Division had been involved in analyzing and managing the system design to assure it complied with architecture plans.
7. Verifying that a pilot evaluation plan had been prepared, and reviewed the plan to determine whether it ensured that a proper evaluation of the pilot was conducted.

Major contributors to This Report

Scott E. Wilson, Associate Inspector General for Audit (Information Systems Programs)

Kerry Kilpatrick, Director

Philip Shropshire, Deputy Director

Robert Irish, Audit Manager

Bernard Kelly, Senior Auditor

Joan Raniolo, Senior Auditor

Paul Baker, Auditor

Virginia Korzec, Auditor

Patrick Madigan, Auditor

John Quartapella, Auditor

Stephen Wybaillie, Auditor

Report Distribution List

chief Information Officer IS

Deputy chief Information Officer for Systems IS

Assistant commissioner (Program Management and Architecture) IS:PM

Assistant commissioner (Program Evaluation and Risk Analysis) M:OP

Director, Office of Information Resources Management IS:IR

National Director for Legislative Affairs cL:LA

Office of Management controls M:cFO:A:M

Audit Liaison – (chief Information Officer) IS:S:O

Management’s Response to the Draft Report

Earned Value Management Process

Earned value is a management technique that relates resource planning to schedules and technical cost and schedule requirements. All work is planned, budgeted, and scheduled in time-phased "planned value" increments constituting a cost and schedule measurement baseline. The two major objectives of an earned value system are:

• To encourage contractors to use effective internal cost and schedule management control systems.
• To permit the government to be able to rely on timely data produced by those systems in determining a product’s contract status.

As work is performed, it is "earned" on the same basis it was planned, in dollars or in another quantifiable unit such as labor hours. A comparison of planned value with earned value would measure the dollar volume of work planned versus the equivalent dollar volume of work accomplished. Any difference is called a "schedule" or "accomplishment" variance. A comparison of the earned value with the actual cost incurred for the work performed would provide an objective measure of planned and actual cost. Any difference is called a cost variance.

Glossary of Terms

Baseline Site: A location used as a control site for comparison purposes during the testing of a systems development project.

Best Practices: The processes, practices, or systems identified in public or private organizations that performed exceptionally well and are widely recognized as improving an organization’s performance and efficiency in specific areas. Successfully identifying and applying best practices can reduce business expenses and improve organizational efficiency.

Business case: A structured proposal for business improvement that functions as a decision package for organizational decision-makers. A business case includes an analysis of business process performance and associated needs or problems; proposed alternative solutions, assumptions, and constraints; and a risk-adjusted, cost-benefit analysis.

capital Asset: Land, structures, equipment, and intellectual property, including software, which are used by the Federal Government and have an estimated useful life of two years or more.

concept of Operations (cONOPS): A narrative description of how a proposed automated system will interact with people and other technology to accomplish the business objectives. cONOPS are intended for communication to customers and other interested parties about how new systems will actually function in the business.

Integration Testing: Testing that verifies the inter-module communication links until all interfaces between the software (or hardware) units meet the requirements.

Investment Review Board: Decision-making body, made up of senior program, financial, and information managers, that is responsible for making decisions about information technology projects and systems.

Modernization Blueprint: Four-part document that details Internal Revenue Service's (IRS) information technology plan for the future. The four parts are: (1) systems life cycle (SLc), (2) business requirements, (3) functional and technical architecture, and (4) sequencing plan.

Pilot Evaluation Report: Document that provides the test results and analysis in order to support a go/no-go decision for rollout.

Post Routing: The transfer of a taxpayer's telephone call from one assistor to another due to either the technical nature of a question or the complexity of the caller's problem.

Product Lines: The three toll-free telephone lines (1-800-829-XXXX) used by the IRS' customer Service organization for taxpayer contacts. Each line involves a specific type of inquiry as shown below.

• 1040 - General Taxation questions
• 4262 - Refund Inquiries
• 8815 - Account Notice questions

Prototype: A preliminary type, form or instance of a system that serves as a model for later stages or the final, complete version of the system.

Release Test Readiness Review: Review conducted to ensure agreement is reached on the acceptability of the integration tests conducted, procedures for constructing the software components, and the results of the integration tests.

Risk Assessment: The process of identifying threats and vulnerabilities of information systems or applications and evaluating alternatives for mitigating or accepting the resulting appropriate judgements about system controls and risks.

Systems Life cycle: Provides guidelines for ensuring that systems are developed, acquired, evaluated, and operated in an efficient manner, within prescribed budget and schedule constraints, and responsive to mission requirements.