Executive Summary

Objective and Scope

Background

Results

Conclusion

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Variance Calculations

Executive Summary

The Internal Revenue Service (IRS) is currently in the early phases of its effort to modernize its outdated, paper-intensive tax processing systems. This multi-billion dollar effort, known as Business Systems Modernization, is projected to last up to 15 years. The IRS created the Business Systems Modernization Office (BSMO) to oversee the modernization effort and hired a contractor, Computer Sciences Corporation (CSC), to help design and integrate the projects. One of the initial modernization projects is the Customer Relationship Management Examination (CRM Exam) project.

The CRM Exam project was initiated to correct long-standing weaknesses in the IRS’ ability to efficiently and accurately compute complex corporate taxes. The CRM Exam project is not a complex system development project; however, the experience gained from planning, developing, and releasing low-risk projects can help the BSMO and the CSC improve as future, more complex projects are initiated.

The objective of our audit was to determine whether the CRM Exam project team had implemented processes to deliver intended taxpayer benefits in a reasonable time and at a reasonable cost. To accomplish our objective, we reviewed the CSC’s delivery of goods and services and evaluated the project team’s compliance with critical processes established to enable project success.

The CRM Exam project is one of the first modernization projects to complete the planning phases. The project team expects to deliver the CRM Exam application to Examination personnel beginning in the latter half of Fiscal Year 2001. As part of this and the other early modernization projects, the BSMO developed and revised most of the key processes necessary for project success. As the project progressed, we noted improvements in areas such as contract management, quality review of products delivered by the contractor, and project sponsorship by IRS business executives. However, the CRM Exam project team was not able to complete the CRM Exam project planning phases in a timely and cost-effective manner. In addition, there were several key development processes that were not effectively implemented by the project team. Improvements in these key processes will be needed to successfully deliver future modernization projects.

The BSMO and the CSC overestimated their ability to deliver the CRM Exam project timely and within budget. The actual completion date and cost for the planning phases varied from original estimates provided to the Congress by 7 months (70 percent increase) and approximately $2.5 million (115 percent increase). The majority of these variances occurred early in the project and more recent cost and schedule estimates have been significantly closer to actual performance. While these delays and cost overruns were significant, the IRS took actions during our review that should help address the factors contributing to the delays and overruns.

Configuration management involves establishing proper control over approved project documentation, hardware, and software and assuring that changes are authorized, controlled, and tracked. While the BSMO and the CSC did develop policies and procedures for configuration management, the project team did not ensure the processes were properly followed. Access to official documents was not restricted and procedures for making and approving changes to the documents were not properly implemented. Without this control, it will become harder to determine which document or configuration item is the official baselined document. This could lead to project teams following the wrong set of requirements or agreements while developing the projects. During our audit, the BSMO and the CSC initiated corrective actions regarding configuration management processes.

Risk management procedures provide guidelines for identifying, tracking, and reporting risks. However, the CRM Exam Risk Management Plan did not include key indicators that could be used to identify and track the status of risks. Additionally, risk reduction plans and issue statements were not always clear and specific. Inadequate identification and monitoring of potential risks can lead to schedule delays and additional costs. The BSMO and the CSC have begun initiating corrective actions regarding risk management processes.

During the last phase of planning, the requirements for security and the integration with other projects were not stable. The BSMO did not include requirements to develop critical security documents in the initial contract for the final planning phase. Also, the IRS made a decision early in the final planning phase to implement the project without integration with other IRS systems. However, discussions about whether to integrate continued throughout the final planning phase because the "no integration" decision was not properly communicated to all stakeholders, including the IRS Commissioner. While the instability of the requirements did not significantly affect the CRM Exam project, unclear requirements could lead to project delays and additional costs to complete future modernization projects.

The BSMO has recently been focusing on issuing contracts where payments are based on contractor performance rather than simply on the hours expended by the contractor. The BSMO has also improved its ability to ensure specific contract requirements are agreed to prior to tasking the contractor to begin work. However, the most recent contract with the CSC that we reviewed did not apply performance-based contracting methods and did not properly define the requirements of the next phase of the project’s development. As a result, the contractor was being paid based on hours expended, instead of results achieved, and the IRS and the CSC had not agreed to responsibilities for the next project phase.

The BSMO and the CSC use earned value measurement, a best practice method of periodically comparing actual cost and schedule results to budgeted results. While the BSMO is improving its ability to monitor the performance of the CSC, further steps can be taken to improve earned value measurement. We determined that earned value data did not include all costs and had not been validated. When all project costs are not included and validated, actual return on investment and earned value cannot be calculated accurately.

The Project Manager was using a schedule to manage the project team’s tasks. The schedule listed the tasks that needed to be completed by the project team. Each task was identified with a specific identification number, and had an assigned start date, finish date, and estimated duration. However, near-term tasks were not assigned to individual team members, and the schedule did not factor in or allow for adequate reserve or recovery time. Without implementing more effective techniques in allowing reserve time and allocating task assignments to individual team members, the BSMO and the CSC could continue to overestimate their ability to deliver projects on time and within budget.

This audit was performed in conjunction with several other modernization project audits. The conditions described above were also identified in the other audits. Because these conditions were identified in multiple projects, we believe that corrective actions should be taken by the BSMO at the program level rather than by the individual project teams. Consequently, we are not making any recommendations in this audit report. We plan to issue a separate report later this year with recommendations for corrective actions that the BSMO can take at the program level to address the conditions we identified.

Management’s response was due on August 13, 2001. As of August 22, 2001, management had not responded to the draft report.

Objective and Scope

This audit was one of a series of audits to evaluate the Internal Revenue Service’s (IRS) Business Systems Modernization projects. The overall objective of our audit was to determine whether the Customer Relationship Management Examination (CRM Exam) project team had implemented processes to deliver intended taxpayer benefits in a reasonable time and at a reasonable cost. To accomplish our objective, we:

• Determined whether the contractor hired by the IRS was delivering high-quality requested goods and services in a timely and cost-effective manner.
• Determined whether the project was following critical processes that have been established to enable project success.

In addition, we reviewed project dependencies and sponsorship provided by the IRS’ Large and Mid-Size Business (LMSB) Division, which will be the primary recipient of the CRM Exam application.

We conducted this audit from November 2000 through March 2001, in the National Headquarters’ Business Systems Modernization Office (BSMO) and at the Computer Sciences Corporation (CSC) office. This audit was performed in accordance with Government Auditing Standards.

Details of our audit objective, scope, and methodology are presented in Appendix I. Major contributors to this report are listed in Appendix II.

Background

The IRS is currently in the early phases of its effort to modernize its outdated, paper-intensive tax processing system. This multi-billion dollar effort, known as Business Systems Modernization, is projected to last up to 15 years. The IRS created the BSMO to oversee the modernization effort and hired the CSC to help design and integrate the various projects. One of the first modernization projects is the CRM Exam project.

The overall objective of the CRM Exam project is to modernize processes, policies, and technology to enable taxpayer examinations to be conducted more quickly with higher customer satisfaction. The CRM Exam project team plans to purchase a commercial software application that will address IRS deficiencies in computing complex corporate taxes.

As part of the overall Business Systems Modernization program, the BSMO and the CSC are subject to specific funding and systems development processes. The Congress places funds for the IRS’ systems modernization activities in an Information Technology Investment Account (ITIA). The IRS must submit an expenditure plan requesting that funds from the ITIA be withdrawn for its use. The expenditure plan must be reviewed by the General Accounting Office and approved by the Department of the Treasury, the Office of Management and Budget, and the Congress.

The BSMO and the CSC are required to follow the Enterprise Life Cycle (ELC). The ELC is a structured business systems development method that requires specific work products to be developed during different phases of the life of a project.

Results

The CRM Exam project team recently completed the planning phases of the project, making it one of the first modernization projects to pass this critical juncture. The project team is now preparing to test the software, train LMSB Division employees, and distribute the software for use. The application deployment is planned to begin in the latter half of Fiscal Year 2001.

The BSMO has established a quality review process that involves IRS executives validating products received from the CSC prior to authorizing payment. The BSMO is also improving its contract management capabilities. It is evaluating the use of incentives in the contracts it issues to contractors for specific products (called task orders).

The BSMO has also made progress in correcting previously reported issues concerning issuing better defined task orders where payments are based on performance rather than just a level of effort. The BSMO issued a defined, performance-based task order for the final CRM Exam project planning phase.

While much has been accomplished on the CRM Exam project, we did identify some issues that affected its efficiency. The project has been ongoing for 18 months and is one of the least demanding modernization projects currently in process. However, it incurred significant delays and cost increases throughout the planning phases. We also found that the project team had not fully implemented several key ELC processes.

As an early project in a much larger effort, it should be expected that experience gained from the CRM Exam project could be used to improve future projects. To learn from this experience, the IRS has tasked the MITRE Corporation to gather and document lessons learned from this project.

The Project Experienced Schedule Delays and Significant Cost Increases During the Planning Phases

The Clinger-Cohen Act requires federal agencies to have processes and information in place to help ensure that information technology projects are being implemented at acceptable costs, and within reasonable and expected time frames, and are contributing to tangible, observable improvements in mission performance.

However, the BSMO and the CSC overestimated their ability to deliver the CRM Exam project timely and within cost estimates. From the start of the project to the completion of planning, the project team significantly exceeded its estimated completion dates and costs.

The CRM Exam project exceeded the estimates in the first 2 ITIA expenditure plans by 7 months and approximately $2.5 million (see Table 1). Per the BSMO, the reason for the inaccuracies is that the IRS must provide ITIA estimates to the Congress before the costs and schedules have been validated through negotiations with the CSC.

Differences of this magnitude indicate that the BSMO and CSC’s collective ability to realistically estimate the costs and completion dates of projects is still maturing. However, continued increases in costs and delays in completing projects could erode Congressional confidence in the IRS’ ability to deliver modernized systems.

After ITIA expenditure plans are approved, the BSMO negotiates with the CSC about the work to be performed, which is documented in a task order. During the planning phases, three task orders were issued for the CRM Exam project. These task orders contained additional refinements to the initial cost and completion dates previously submitted in the ITIA expenditure plans.

Task order cost information was more accurate than ITIA expenditure plan estimates; however, task order figures for completing project planning activities were exceeded by 7 months and $1.7 million (see Table 2).

The Clinger-Cohen Act sets a tolerance that projects exceeding 10 percent of their cost and/or schedule should be evaluated to determine if they should continue. While cost and schedule variances for the CRM Exam project were significantly over the 10 percent threshold, we agree with IRS’ decision to continue the project as a means to gain experience and learn lessons for future modernization projects.

A review of task order cost and schedule variances revealed that the majority of the cost and schedule slippage occurred during the initial task order time period. Since moving toward performance-based contracting and fixed price task orders, the project has steadied (see Charts 1 and 2).

While cost and schedule variances have decreased, the last task order for the final planning phase was over schedule and cost. The IRS agreed it had contributed to some of the delays in completing the final project planning phase and agreed to pay the CSC approximately $245,000 after the final planning phase was completed.

The IRS did not timely provide government-furnished resources and information to the CSC as required.

• The LMSB Division staff resources were provided late to the project team. For example, revenue agents were provided almost 2 months after the date contained in the Memorandum of Agreement between the IRS and the CSC.
• The IRS did not involve the National Treasury Employees Union (NTEU) in the up-front planning sessions and establishment of requirements. As a result, the NTEU needed more time to review the IRS’ resource commitments to the project.
• The LMSB Division took about 2 months to provide the CSC with the number of software packages to purchase. The delay in receiving the number of packages was reported by the CSC as a project issue in August 2000. The CSC advised the BSMO that the missing number affected the timely completion of costing information needed in the Baseline Business Case, a key document needed to successfully complete planning.

These difficulties contributed to the project team completing planning activities and delivering several key planning documents later than scheduled.

Management Actions: During our review, the IRS took numerous actions to enhance project sponsorship by the LMSB Division and address the factors contributing to the delays. We believe these actions properly address the causes of previous schedule delays and cost increases.

• In September and October 2000, the CSC met with the LMSB Division to determine what information and resources needed to be supplied to the project team.
• In December 2000, an executive steering committee was charged with the responsibility of overseeing the CRM Exam project for the LMSB Division.
• By February 2001, a charter for this executive steering committee was developed which included top LMSB Division and NTEU executives as members.

Configuration Management Processes Were Developed, But Not Consistently Followed

Configuration management involves identifying critical project items (documents, software, and hardware), controlling changes to those items, and recording and reporting any changes to the items. The ELC requires that configuration management procedures be implemented throughout the life of the project.

On modernization projects, one of the more important purposes of configuration management is to assist project management in controlling the content of the developing system. Key activities necessary for proper control include establishing baselines for approved items and assuring that changes to baselined items are authorized, controlled, and tracked. As a result, an effective configuration management process assists project management in developing systems that meet the intended IRS business needs.

If configuration management procedures and baselines are not effectively implemented, the BSMO cannot assure that the systems being developed will have the intended functionality. In this environment, items could be inappropriately manipulated, which could affect the validity of performance measurements and the functionality of systems.

We determined that the project team had not fully implemented an effective configuration management process.

• Baselines were not established.
• Access to the configuration management repository where items were stored was not properly restricted.
• Procedures for change management were not properly implemented.

The CSC had not established the required baselines for project documentation as required by configuration management procedures. Several versions of documents were maintained in the configuration management repository; however, the baseline files were not separately identifiable. Configuration management procedures require that a Baseline Contents List (BCL) be prepared for each baseline item. However, on one BCL, we identified six Preliminary Business Cases, two Project Management Plans, two Quality Management Plans and two Risk Management Plans. The baseline document could not be identified on the BCL.

The configuration management repository was not restricted as required by ELC procedures. To protect the integrity of the products under configuration control, configuration management procedures require that access be restricted to only those employees who need access.

Numerous employees had inappropriate access to the products maintained on the configuration management repository database. For example, the Project Manager and the CRM Exam project team both had the ability to post documents to the configuration management repository database. In this environment, the integrity of the baseline files could be affected because unauthorized documents could be saved to the configuration management repository.

We determined that BSMO oversight reviews did not initially include project processes and procedures for configuration management. In addition, the CSC had not conducted any baseline reviews of the project and had cancelled other project level inspections or reviews. Therefore, the lack of access controls was not detected until our review was conducted.

Change management procedures were not properly implemented to ensure that changes to configured items were approved and tracked. These procedures require that when baselines are established, a change request is prepared and approved by the Configuration Control Board.

Change requests were not part of the baseline process at the beginning of our review, and change requests were not used to establish or revise the baselines. When an organized method of determining the impact of proposed changes to baselines does not exist, the risk increases that unauthorized changes could be made to critical project software, hardware, or documentation.

Management Actions: Before we completed our audit, the BSMO and the CSC had initiated corrective actions regarding configuration management processes.

The BSMO implemented Configuration Management Program Instructions. In addition, the BSMO conducted a review and issued a report in February 2001 on the lack of effective configuration management processes. The report confirmed our observation that project baselines were not established for some modernization projects and configuration management inspections (such as baseline configuration audits) were not performed.

The CSC hired a Director to oversee the Configuration Management Office and developed revised procedures for products to be delivered to the IRS. The revised procedures require that baselines be established using formal change requests after products are approved. Also, the CSC provided documentation indicating that change access to the configuration management repository had been restricted to the configuration management staff.

Risk Management Processes Were Developed, But Not Consistently Followed

Risk management is the process of identifying, analyzing, and tracking risks and assessing the probability that risks will occur and their potential consequences. A risk is a potential event that, if it occurs, will adversely affect the ability of the project to meet its objectives. Once a risk has an impact on the project or a previously unidentified event has an impact on a project, the event is referred to as an issue.

The CRM Exam Risk Management Plan includes the guidelines to follow in identifying and reporting on risks and issues. In prior audit reports, we made recommendations to improve the risk management process.

The BSMO and the CSC have not fully implemented risk management procedures. We identified several concerns regarding risk management. The BSMO identified similar weaknesses in its analysis of project risks.

• The Risk Management Plan should include tolerance levels that would create a risk or issue.
• Risk reduction activities should be started and completed timely.
• Issue statements should be more comprehensive.

The Risk Management Plan contains guidance for identifying project risks and issues. Key indicators of project risks and issues include significant cost and schedule variances from the project plan. Significant variances in project cost or schedule may indicate that a new risk or issue exists or efforts to reduce a known risk are not effective.

According to the BSMO’s risk management procedures, project teams should use standard tolerances for cost and schedule variances (earned value reporting) or establish project-specific tolerances. Whichever method is used to determine these tolerances, the project team should document the tolerances in the Risk Management Plan. While the CRM Exam project did have a Risk Management Plan, it did not contain any tolerances that could be used by team members to identify new risks or monitor known risks and issues. Although we did not find evidence that the lack of risk tolerance guidelines had a negative impact on the CRM Exam project, incomplete guidance on other projects could preclude project teams from timely beginning actions to address potential risks.

Actions taken to reduce known risks should be documented in a Risk Mitigation Plan. The plan should describe the risk, the impact of the risk on the project, and the estimated date that the risk will adversely affect the project. The plan should also document the actions that will be taken to deal with the risk, when these actions must be completed, and who is responsible for the implementation and resolution of these actions. Risk management procedures also state that a risk that has already occurred should be handled as an issue.

The project team actively used risk mitigation plans to manage project risks; however, we noted several areas where the risk reduction procedures were not followed. BSMO and CSC personnel did not ensure that risk reduction activities were given priority. This was evidenced by the fact that the project team did not timely plan or execute several actions to reduce risks and did not always convert risks to issues timely. To effectively reduce the adverse effects of project risks, the project must plan and initiate actions as soon as practical. Delays in implementing actions to reduce risks increase the likelihood that risks will have an adverse effect on the project.

Through a review of project risk mitigation plans, we determined that:

• Risk reduction activities were not always started or completed by the planned start and completion dates.
• Risk reduction activities were scheduled to be completed after the date the risk would begin to adversely affect the project.
• Risks were not treated as issues after the probable risk impact date had passed. Actions taken to reduce risks are often different from actions taken to handle an adverse event that has already occurred. Therefore, misclassification of an issue as a risk could lead to inappropriate actions taken by the project team.

In our opinion, issue statements should include enough information for the IRS and the CSC to be able to act upon them. However, the CRM Exam Project Manager did not ensure that issue statements included pertinent information.

Issue statements did not give sufficient information to understand what had occurred and what corrective actions would be appropriate. In the following two examples, the CSC claimed that the problems identified could delay the completion of planning activities but it did not give sufficient information to clarify the issues.

• An issue statement claiming that IRS resources were not available did not include the job titles or skills of the needed workers.
• An issue statement concerning additional design requirements imposed by the IRS’ Architecture and Engineering Organization did not list the additional design requirements.

Unclear issue statements impede the ability to adequately track risks and issues.

Management Actions: After our audit fieldwork was completed, the IRS and the CSC provided information that they agreed with our concerns and were implementing the following actions:

• The CSC developed a revised set of risk management procedures that were accepted by the IRS on March 28, 2001.
• The IRS accepted an Issues Management and an Action Item Management approach on March 12, 2001.
• A process action team was formed in April 2001 to develop a proposed concept for an executive risk management review board.

Significant System Requirements Were Not Stable During the Final Project Planning Phase

The Reporting Compliance Sub-Executive Steering Committee determined in its "lessons learned" session that the requirements for the CRM Exam project kept changing. We also noted instances where high-level requirements, such as security and integration needs, were not stable during planning activities.

Best practices require that system requirements be common and unambiguous. Requirements management requires establishing and maintaining agreements among the project team, the end user, and the contractor team.

At the beginning of the final planning phase, the BSMO accepted a CSC plan to complete project planning activities although the plan did not include all requirements for security documents. A task order was later modified to include a requirement to deliver three security documents.

In May 2000, the CSC identified several potential interfaces between the CRM Exam project and other IRS systems. These interfaces would allow users to work more efficiently by eliminating the need to manually input data into multiple systems. Early in the final planning phase (June 2000), the LMSB Division concluded that the CRM Exam project would not interface with other IRS systems. The CSC noted that the decision would reduce the estimated benefits of the CRM Exam project because the lack of interfaces would require more effort on the part of users. The CSC also noted that IRS executives, including the IRS Commissioner, could reject the solution if it did not include interfaces with other IRS systems.

In December 2000, the IRS Commissioner requested additional information about the benefit of the project without integration. After reviewing the information, IRS executives approved continuing the CRM Exam project without interfaces. However, a separate project was initiated to develop interfaces with other IRS systems.

Discussions about whether to integrate continued throughout the final planning phase because the "no integration" decision was not properly communicated to all stakeholders, including the IRS Commissioner. In our opinion, high-level requirements should remain stable during planning activities. If requirements change, the changes need to be properly communicated to decision-makers to prevent confusion or rework.

Unclear requirements can lead to project delays and additional costs to complete future modernization projects. Also, systems could be developed that may not meet the needs of the business processes they are intended to support.

Contract Management Capabilities Have Improved, But Further Improvements Can Be Made

Government policy states that agencies should negotiate a contract type and price that will result in reasonable contractor risk and provide the contractor with the greatest incentive for efficient and economical performance. Performance-based contracting is a best practice for achieving this purpose. Additionally, the Office of Management and Budget informed agencies that during Fiscal Year 2002 they should migrate toward awarding contracts following performance-based contracting techniques.

The BSMO has made improvements in defining task order requirements and using performance-based contracting techniques since the early stages of the project; however, further improvements are needed.

In April 2000, the BSMO issued a performance-based task order for the CRM Exam project. While this task order initially did not adequately define requirements, the BSMO later modified the task order to better define requirements. In July 2000, the BSMO issued a firm fixed price task order that was properly defined and used performance-based contracting methods. Based on our analysis of these two task orders, the IRS was improving its contract management capabilities. However, the next task order issued for the CRM Exam project was not properly defined and did not use performance-based contracting concepts.

In February 2001, after the CRM Exam project team had completed planning activities, the BSMO and the CSC entered into a firm fixed price level of effort contract to begin purchasing software and conducting training for 30 days at a cost of approximately $300,000. Once the BSMO and the CSC determined that they could not agree on requirements for the next stage of the project, the task order was extended for another 30 days and the cost was increased by approximately $400,000. At the end of our audit, a performance-based task order defining the work to be completed for the next phase of the project had not been issued.

The February 2001 task order did not use performance-based contracting methods because the BSMO and the CSC had not agreed on the scope of activities for the next project phase prior to completing planning activities. While the BSMO recognizes that this is not the most productive way of awarding task orders, the current task order was awarded to keep experienced contract employees working on the project while the BSMO and the CSC negotiated the scope and cost of the next phase of activities.

Management Actions: The BSMO is currently developing procedures to require defined task orders prior to allowing project teams to progress into the next phase. The procedures documenting this new effort were not available for our review prior to the completion of our audit work. Also, the IRS informed us that the February 2001 task order was modified in April 2001 to better define requirements.

Earned Value Data Should Include All Costs and Be Validated

In our first review of the Business Systems Modernization program, we reported that the IRS had not developed a process for performance monitoring. Since that time, the BSMO has shown significant progress in its ability to measure the status of modernization projects. One example is the use of earned value management, a best practice method of periodically comparing actual cost and schedule results to budgeted results.

The BSMO and the CSC use earned value measurements to track the status of modernization projects. While the BSMO is improving in its ability to track the progress of modernization projects, earned value data would be more useful to management if the following factors were included:

• Costs over the project’s life cycle.
• Costs that are not included in the ITIA.
• Actions to validate the CSC’s baseline costs.

ELC guidelines state that earned value should measure the cost of the project over its life cycle. However, the CSC did not follow this guidance. The CSC currently measures earned value over the current milestone. As a result, data collected are not as useful as they could be for measuring the total cost and schedule performance of the project. Measuring earned value over a longer period would help the BSMO and IRS executives determine whether the project is progressing according to the course established when the original decision was made to initiate the project.

Management Actions: The BSMO has recognized that, to be of the greatest value, the baseline period of time or cost that earned value data are compared against should include the entire project life cycle or as much of that as possible. As a result, the BSMO has tasked the CSC to change the way it captures project measures.

Costs that are not covered by ITIA funds, such as the salary of IRS employees assigned to the project team, are not accurately tracked or reported by the BSMO. Consequently, these costs are not included in any earned value analysis. The ELC indicates that an appropriate measure of total project cost should include any indirect costs.

The BSMO has not yet developed an effective process to track costs outside of the ITIA funds. BSMO personnel indicated that it is difficult to ensure IRS employees, especially those who do not work full-time on the project team, use the correct codes for charging their time. Without an accurate accounting of the IRS’ internal costs associated with the project, actual return on investment and earned value analyses cannot be accurately calculated.

Since the costs of IRS staff assigned to the project represent a significant portion of the total project cost, the BSMO needs to routinely monitor these costs to ensure they do not materially exceed estimates established during planning.

Management Action: In Fiscal Year 2001, the IRS reemphasized a practice designed to capture all obligations and commitments that can be directly attributed to a modernization project. However, as of January 2001, the IRS was not able to provide accurate cost information for IRS staff assigned to the CRM Exam project.

In response to an earlier audit report, the BSMO stated that it would review the CSC’s Program Management Control System (PMCS) process by August 2000. The PMCS is a management tool that computes the earned value measures, cost and schedule variances, and related trends from the actual cost, current status, and baseline cost and schedule information. The BSMO and the CSC rely on PMCS reports for management analysis.

The CSC is required to provide earned value data that are 100 percent accurate. A validation of the PMCS would help ensure that this goal is being met. However, the BSMO has yet to validate the PMCS because the CSC has not provided the information necessary for this review.

The success of any technology initiative is contingent upon management’s ability to make sound decisions based on accurate information. Management’s decisions are only as good as the information being used to make those decisions.

The BSMO needs to validate earned value baseline costs to assure that accurate measures of performance are being derived. Baseline costs form the foundation for measuring the performance of the project. Without assurance that baselines are valid, the BSMO could receive inaccurate earned value data that could lead to incorrect business decisions.

Project Management Processes Can Be Improved

The CRM Exam Project Manager used a Work Breakdown Structure (WBS) to manage the project team’s tasks. The WBS listed the tasks that were required to be completed by the project team. Each task was identified with a specific WBS identification number and had an assigned start date, finish date, and estimated duration. However, near-term tasks were not assigned to individual team members, and the WBS did not factor in or allow for adequate reserve or recovery time in the schedules.

For example, the project team broke down the tasks in the WBS to various levels of detail and, at the lowest level, the Project Manager made assignments to groups of team members who would complete the task. However, we were unable to obtain names of the individuals in each of these groups who were assigned responsibility for the completion of the tasks that were due within the next few days or weeks. Accountability for completion of project tasks is critical to ensuring that projects are completed on schedule and in a quality manner. Although the Project Manager is assigned overall accountability for all the tasks, he or she must delegate responsibility and accountability for the numerous tasks to the individual team members.

We discussed this issue with the Project Manager and other CSC officials, and they indicated that it was not their practice to assign tasks down to individual staff members. They stated they do not find that information useful on the WBS. However, we believe that accountability should be documented in the WBS or elsewhere by team leaders and Project Managers for tasks due in the very near future. Identifying specific team members can become critical if a person has specific skills and is working on more than one project. The Project Manager needs to be sure that this person will be available when needed, and this becomes difficult when team members are not specifically identified on the WBS.

We also found that the WBS did not contain adequate reserve or recovery time allocated to address unplanned events that could occur. The Project Manager indicated that the CSC does not separately allocate time to recover from unplanned events.

Although reserve time is not always required on a project, it may be necessary if the current schedule is very aggressive or if a project team faces schedule risks. This issue becomes even more critical as project teams move towards development and deployment of systems that affect IRS personnel and taxpayers. Without adequate reserve or recovery time, the IRS and the CSC could continue to exceed cost and schedule goals.

Conclusion

The IRS plans to begin deploying the CRM Exam project in the latter half of Fiscal Year 2001. The project will be one of the first low-risk projects deployed by the Business Systems Modernization program. While the IRS and the CSC have exceeded their cost and schedule goals on this project, the experience gained can be used to improve other ongoing or planned projects. To accomplish this, the IRS has tasked the MITRE Corporation to gather and document lessons learned from the CRM Exam project and one other project that has completed the planning stages. Learning from these projects will lead to an increased capability to initiate, develop, and release projects in an efficient, consistent manner.

Detailed Objective, Scope, and Methodology

The overall objective of our audit was to determine whether the Customer Relationship Management Examination (CRM Exam) project had implemented processes to deliver intended taxpayer benefits in a reasonable time and at a reasonable cost. To accomplish this objective, we:

1. Determined whether the contractor was delivering high-quality requested goods and services in a timely and cost-effective manner.

Timeliness Tests

1. For the last approved Enterprise Life Cycle (ELC) milestone (MS) (i.e., MS 2, task order 35), used the milestone exit documentation, Core Business Systems Executive Steering Committee information on milestone approval, and discussions with the Project Manager to determine whether:
1. The project met all exit criteria.
2. All conditional approvals were satisfied timely.
2. For the current milestone (i.e., MS 3, task order 47), determined whether the contractor met deliverable due dates for the prior 6-month period (May 2000 through October 2000).
1. Determined how many of the deliverables were delivered by the scheduled due dates by comparing task orders to receipt documentation.
2. If the required deliverables were late or had not been delivered yet, determined the reasons for the delays, the impact on the project schedule (due dates, cost increases, etc.), and any risk or issues attributed to this late or non-delivery.
3. Used the project schedule and discussions with the Project Manager to determine whether any delays in deliverables had affected the implementation of the software in Fiscal Year 2001 for the Large and Mid-Size Business (LMSB) Division.

Quality Tests

3. Determined whether there was an approved ELC tailoring plan incorporated into the current task order.
4. Determined whether there was a validation process in place to cover each deliverable prior to the payment for the task order.
5. For the last approved milestone (MS 2), determined whether both the modernization contractor’s Quality Assurance and a qualified designated IRS official performed a validation for each deliverable prior to payment for the task order.

Cost Tests

6. Determined the cost of the work performed by reviewing task orders covering the past year (task orders 23, 35, 47, and 62) and the Preliminary Business Case and through discussions with the Project Manager.
1. Determined whether the current task order was written using firm fixed price mechanisms. If not, determined the reason why.
2. Determined whether the MS 3 and MS 4 task orders were properly defined, agreed to, and approved by all appropriate parties prior to the start of work on the task orders. If not, determined how much delay occurred between beginning the work and defining and signing the task orders.
3. Determined if there was any effect on the project in terms of costs that could be attributed to not including both incentives and disincentives in the task orders.
4. Determined whether modifications to task orders had been approved to increase the original agreed upon firm fixed price cost. If so, identified what additional deliverables were requested. Determined if management reviewed the reasonableness of the cost increases by performing impact assessments or involving the change control board.
5. Determined whether there were any duplicate deliverables or requirements included in the prior task order (35) and the current task order (47).
6. Determined how Earned Value (EV) Management was tracked for the project and the status of adhering to the cost estimates. Determined if EV monitoring:
1. Considered the full cost of the project over its life cycle and not just the cost of the current task order.
2. Showed how the dollar amount spent thus far on the project compared to the projected costs in the Preliminary Business Case.
3. Identified what percentage of the projected costs had been realized thus far.
4. Showed whether the project was still on track to meet its projected return on investment as reported in the Preliminary Business Case.
2. Determined whether the CRM Exam project team was following critical processes that had been established to enable project success.
1. Determined whether the project was adhering to ELC requirements.
1. Reviewed the status of configuration management.
1. Determined whether a configuration management plan had been developed for the project. If so, reviewed the configuration management plan to determine whether it included the requirements in the ELC configuration management template.
2. Determined whether a process had been established for change requests to be initiated and approved/disapproved. If so, reviewed a sample of change requests to determine whether approvals were documented, changes were not made prior to documented approval, and appropriate levels of approval were required. (This step involved documentation only. No software or hardware changes were reviewed.)
3. Determined how changes were communicated to the IRS (users and program management) and the Integrated Product Team.
4. Determined whether a repository for configuration management project documentation had been established. If so, documented the location and content of the repository, who maintained the system, and who had access.
5. Determined whether a repository for system modules had been developed for version control. If it was too early in the ELC for this to be required, determined if this was addressed in the configuration management plan.
6. Determined whether the document list and the configuration items list had been defined and baselined. If so, determined if there was a process for ensuring these lists were complete.
2. Reviewed the status of risk management.
1. Determined whether the project used risk inventory and assessment worksheets to document risks.
2. Determined whether the project measured the potential quantitative and qualitative effects of the risk. Also, determined if the most critical risks were raised to an appropriate level of IRS management in a timely manner.
3. Determined whether all critical documented risks (for which a risk reduction strategy had been chosen) were addressed with risk reduction plans. Reviewed risk reduction plans for all critical risks to determine whether the plans appeared to effectively address the risks and whether the actions contained in the plans were being tracked and were on schedule. If not, documented the delays in completing risk reduction actions for the associated risk and the potential effect of not completing the risk reduction action.
4. Reviewed all issue worksheets prepared for the project between July 2000 and February 2001 to determine whether project issues were effectively documented, assigned, tracked until completion, and timely addressed.
5. Obtained and reviewed the Work Breakdown Structure (WBS) to determine if the project was behind schedule and, if so, how far. If the project was more than 5 percent behind, determined whether this was identified as a risk to the project and was accurately reported to the appropriate officials.
3. Reviewed the status of requirements management.
1. Determined whether a formal process (workshops, etc.) was used to gather system requirements focusing on stakeholders and individuals at both the executive and technical level of the LMSB Division.
2. Determined whether requirements were analyzed for consistency, completeness, and feasibility. Reviewed this analysis to determine whether it appeared comprehensive.
3. Determined whether the systems requirements were documented and approved by IRS officials in the LMSB Division and in the Information Technology Services function.
4. Determined whether the project had documented requirements traceability matrices.
5. Reviewed Project Integration Architecture Team meeting documentation and all other requirement briefings to identify issues and proper resolution.
6. Determined what process was in place to measure the impact on cost and schedule if requirements had been changed during the project life cycle.
4. Reviewed the status of project staffing.
1. Compared the organization chart for the Integrated Product Team to the Project Management Plan. Determined the actual versus planned staffing. Identified any significant vacancies and trends.
2. Determined whether the project had been able to effectively recruit, obtain, and retain sufficient staff to accomplish its tasks. Identified any significant staff turnover through interviews with the Project Manager.
3. Determined whether all tasks scheduled to start in the near-term (e.g., within 90 days) were assigned to a project staff person by reviewing Tax Administration Alerts.
4. Determined whether a skills assessment had been performed for the Integrated Product Team and a training plan developed to address any skills deficiencies.
5. Reviewed the status of software project tracking and oversight.
1. Reviewed the Performance Measurement Plan and determined if it adequately documented how project performance would be gauged (i.e., did it identify who should do what, what measures would be used, how frequently they would be measured, etc.).
2. Obtained access to the project weekly/monthly measures and determined if project measures were being tracked as described in the Performance Measurement Plan and/or the preliminary or baselined business case.
2. Determined whether the project was adhering to Modernization Blueprint Requirements.
1. Reviewed Blueprint 2000 to determine if it referenced the CRM Exam project.
2. Determined if the Preliminary Business Case or Baseline Business Case documented how CRM Exam fits into the IRS’ architecture of the future.
3. Determined the risks related to project dependencies and sponsorship.
1. Determined whether project dependencies were effectively and efficiently managed to minimize project delays.
1. Identified any IRS projects and external dependencies by reviewing the Integrated Master Schedule, weekly management reports (Alerts report, Dependency Matrix, Slippage report), the WBS, and the monthly Program Management Reviews and discussing dependencies with the Project Manager.
2. For any dependencies identified, determined how the Project Manager identified slippage of tasks in the dependent projects and how the Project Manager coordinated with the dependent projects.
3. Determined what changes the Project Manager made in the CRM Exam project schedule, risks, and issues based on the dependencies.
4. Determined what progress had been made in determining the security certification requirements for the CRM Exam software package.
5. Determined the status of a decision on whether systems acceptance testing would be performed.
2. Determined whether the project’s sponsor met his/her responsibilities and time commitments and assisted in bringing the project to its successful completion.
1. Interviewed the sponsor’s representative and the Project Manager to determine the sponsor’s commitment, responsibilities, and methods of being involved in the project and assuring that delays were minimized.
2. Determined if the sponsor had been involved in obtaining information the project needed from the LMSB Division to move the project forward and maintain the schedule.

Major Contributors to This Report

Scott E. Wilson, Assistant Inspector General for Audit (Information Systems Programs)

Scott A. Macfarlane, Director

Troy D. Paterson, Audit Manager

James A. Douglas, Senior Auditor

George L. Franklin, Senior Auditor

Michelle Griffin, Senior Auditor

Wallace C. Sims, Senior Auditor

Charlene L. Elliston, Auditor

Sylvia Sloan-Copeland, Auditor

Report Distribution List

Commissioner N:C

Associate Commissioner, Business Systems Modernization M:B

Deputy Associate Commissioner, Program Management M:B

Deputy Associate Commissioner, Program Planning and Control M:B

Deputy Associate Commissioner, Systems Integration M:B

Director, Legislative Affairs CL:LA

Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O

Chief Counsel CC

National Taxpayer Advocate TA

Office of Management Controls N:CFO:F:M

Audit Liaison:

Associate Commissioner, Business Systems Modernization M:B

Variance Calculations

Schedule variances were calculated as follows:

Sum of (Milestone exit date related to each task order – initial period of performance listed on each task order) / Sum of (Initial period of performance listed on each task order).

Cost variances were calculated as follows:

Sum of (Costs listed on vouchers paid for each task order – initial cost ceiling listed on each task order) / Sum of (Initial cost ceiling listed on each task order).