Project Management Techniques Need to Be Followed to Effectively Develop the Tax Exempt Determination System
Reference Number: 2003-10-103
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
May 12, 2003
MEMORANDUM FOR COMMISSIONER, TAX EXEMPT AND GOVERNMENT ENTITIES DIVISION
FROM: Gordon C. Milbourn III /s/ Gordon C. Milbourn III
Acting Deputy Inspector General for Audit
SUBJECT: Final Audit Report - Project Management Techniques Need to Be Followed to Effectively Develop the Tax Exempt Determination System (Audit # 200210028)
This report presents the results of our review to determine if Tax Exempt and Government Entities (TE/GE) Division management was following accepted systems development guidelines to ensure that defined goals will be achieved in the Release I development stage of the Tax Exempt Determination System (TEDS).
The TEDS is being designed to provide increased systems capability and much needed improvements in overall system performance and reliability. TE/GE Division management set the goal of having the TEDS Release I operational in Fiscal Year (FY) 2002 in order to provide better customer service and process the increased number of applications from Employee Plans customers in FYs 2002 and 2003.
In summary, our review showed that Business System Planning management’s planned use of a modified Enterprise Life Cycle and additional project management techniques for the development of the TEDS complied with Internal Revenue Service (IRS) guidelines for the initial stages of systems development. However, TEDS project management did not implement these techniques as planned. Project management techniques are used to ensure that the development of computer systems is efficient and effective. We identified the following:
· Ineffective requirements gathering led to delays in project development.
· Ineffective performance monitoring did not ensure project objectives were completed on schedule or within budget.
· Ineffective risk management did not identify potential problems that could affect completion of the TEDS development and did not always develop or monitor plans to ensure the risks were minimized and/or eliminated.
· Ineffective configuration management did not ensure that all development team members were continuously working on the most recent version of custom software or system design elements.
As a result, delays occurred, and TE/GE Division management did not meet its goal of having the TEDS Release I in place in FY 2002. Accordingly, the TE/GE Division did not realize the planned FY 2002 benefits of $1 million in cost savings and providing better service to its customers, and it is at risk of not fully realizing planned benefits for FY 2003. Furthermore, expenses that were not accurately estimated in the Baseline Business Case (BBC) are now being incurred to develop the new system.
We recommended that TEDS project management fully implement their planned project management techniques for future releases of the TEDS. In addition, for Release I, a feasible schedule for the completion of the project should be developed, and the BBC should be updated to show variances in cost, benefit, schedule, and contractor performance and to show that the TEDS will be developed using multiple releases.
Management’s Response: IRS management agreed with our recommendations. Specifically, they agreed to fully implement their planned project management techniques, including a review of Risk and Configuration Management procedures. The TE/GE Program Configuration Control Board has approved and baselined a new project schedule. The BBC will be updated to conform with the requirements of the IRS’ Investment Decision Management Business Case Procedure and will include an update of the release plan and schedule, cost, and benefit estimates. Management’s complete response to the draft report is included as Appendix IV.
Copies of this report are also being sent to the IRS managers who are affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs), at (202) 622-8500.
The Tax Exempt and Government Entities (TE/GE) Division processes and controls Employee Plans and Exempt Organization (EP/EO) determination letter applications using the EP/EO Determination System (EDS). However, TE/GE Division management determined that the EDS is an outdated technology that lacks the capacity to: handle the required workload, fulfill statutory IRS responsibilities under the Internal Revenue Code, and meet expectations of the EP/EO customers.
In July 2000, an Improvement Project Plan was approved for the Tax Exempt Determination System (TEDS) as an IRS Tier B portfolio project. In November 2001, the TE/GE Division’s Office of Business Systems Planning (BSP) developed a concept of operations document for the TEDS to replace the EDS. In December 2001, the Investment Executive Steering Committee approved the multiple release strategy for the TEDS. The TEDS is being designed to provide increased systems capability and much needed improvements in overall system performance and reliability. It will automate certain manual processes and thereby reduce or eliminate employee labor expense. The current plans are to implement the TEDS in seven releases, with each release providing additional capabilities to the system.
Due to several law changes, the TE/GE Division anticipated receiving a significant increase in the number of EP restatement applications during Fiscal Years (FY) 2002 and 2003. TE/GE Division management set the goal of having the TEDS Release I operational in FY 2002 in order to provide better customer service and process the increased number of applications from EP customers.
The IRS uses the Baseline Business Case (BBC) to provide the basis for making investment-funding decisions and for monitoring and evaluating investment performance. IRS policy generally requires that the BBC be updated at Milestone Four. However, revisions to the BBC may occur at any time during the project and do not have to coincide with a milestone. Reasons for revising the BBC include changes to the project scope or cost increases of greater than 10 percent to reach the next milestone. Revisions are necessary to ensure that the most accurate information is available for making investment decisions and monitoring the progress of the system’s development. The BBC also satisfies the Clinger-Cohen Act that requires business benefits from information technology (IT) investments be reported to the Office of Management and Budget and the Congress annually and be compared to the projected business benefits for each IT investment.
In the TEDS BBC dated June 2001, the TE/GE Division projected that the total non-recurring costs for the project would be $6.8 million. The BBC also projected that $1 million would be saved during FY 2002, with an additional $2.3 million saved in FY 2003, because of the increased efficiency of the TEDS over the EDS.
The IRS adopted the Enterprise Life Cycle (ELC) as a process to plan and manage the development of new computer systems. The ELC is a system of reviews, checkpoints, and milestones used to ensure that projects are efficiently and effectively planned, designed, developed, and implemented. TEDS project management planned to follow the ELC, as well as implement several project management techniques, to help ensure that the development of the TEDS was both efficient and effective. Those techniques included the following:
· Requirements management – the process of gathering requirements, recording them in a document, tracking the design and code (software programming) against them, and managing changes to them for the rest of the project. It is important to identify the majority of the system requirements early in the development process because they provide a basis for planning and tracking the project, and they provide developers with a foundation upon which to base the design and computer code for the new software. Also, identifying the requirements early in the development process avoids the problems associated with adding requirements later, which could result in significant and costly software reprogramming and revising the BBC if costs increase more than 10 percent above the approved budget.
· Performance monitoring – a comparison of the project baseline (projected budget and schedule) to actual performance so that deviations can be identified and corrective actions taken to keep the project on track to meet key milestone dates. It is performed over the life of the project and is important to help determine if project objectives are being achieved. For example, if a vendor has been paid more than what was budgeted, management should be able to recognize this variance through cost analysis.
· Risk management – the continual process of identifying potential problems that could impede development of a new computer system and potential solutions to resolve or lessen the severity of the problems. It is performed over the life of the project and is important to prevent delays and a disruption of the development cycle.
· Configuration management – the process of identifying, controlling, and approving changes to system documentation, custom computer source code, or off-the-shelf software. It is performed over the life of the project and is important to help ensure that the entire development team is working from the most current version of this information. Ineffective configuration management can contribute to project delays and costly rework to correct work accomplished on unapproved or incorrect versions, or it may result in a system that does not operate properly because developers worked from different versions of information.
TE/GE Division management hired two private contractors to assist with the development of the TEDS. One contractor was hired as the Project Management/Systems Integrator and the other contractor was hired to write the custom computer software.
We performed audit work in the TE/GE Division and the Modernization, Information Technology and Security Services (MITS) offices in the National Headquarters in Washington D.C., from March through December 2002 in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
The TE/GE Division’s Office of BSP management’s planned use of a modified ELC and additional project management techniques for the development of the TEDS complied with IRS guidelines for the initial stages of systems development. However, we determined that due to a very aggressive development schedule, the development team did not follow through with these project management techniques. As a result, delays occurred, and TE/GE Division management did not meet its goal of having the TEDS Release I in place in FY 2002. Accordingly, the TE/GE Division did not realize the planned FY 2002 benefits of $1 million in cost savings and providing better service to its customers, and it is at risk of not fully realizing planned benefits for FY 2003.
A significant change occurred during development of the TEDS. During the System Design phase (Milestone Three), TE/GE Division management decided to change the release strategy for the TEDS development from a single release project (full functionality all at once) to a multiple release strategy (functionality added in successive stages). Instead of revising the BBC at this point, TEDS project management decided to wait and revise the BBC at the next regularly scheduled update (Milestone Four). In our opinion, the BBC should have been updated as soon as possible to include the new release strategy and the additional development costs for the TEDS Release I. Because TEDS project management did not update the costs anticipated for Release I, additional expenses that greatly exceeded 10 percent of the funds approved in the initial BBC were incurred during the development of the TEDS Release I.
The following paragraphs outline the project management techniques planned by BSP management and summarize the results of our assessment of the development team’s efforts to implement the planned actions.
The requirements-gathering process was not effective or efficient and did not ensure early identification of all, or substantially all, the requirements needed to develop the TEDS. Weaknesses in this process potentially contributed to project delays and increased costs.
The ELC and project management
guidelines provide for a Requirements Management Plan to guide an organization
in identifying the requirements necessary to develop a computer system. A formal plan should be used to identify the
system requirements early in the development process. It is important to do this early in the process because the
requirements provide a basis for planning and tracking the project, and they
provide developers with information to begin their system and software design.
The project management contractor developed a Requirements Management Handbook with guidelines for identifying system requirements. However, it was not completed until May 16, 2002, which was toward the end of the requirements-gathering process. As a result, the development team did not have the benefit of these guidelines to help them in identifying system requirements.
In June 2001, the Configuration Control Board (CCB) approved the initial 178 high-level requirements for the entire TEDS project. Based on these requirements, the project management contractor developed the system architecture and the BBC.
However, 3 subsequent requirements-gathering efforts were required to identity and refine over 500 requirements for the TEDS Release I. The document containing the final requirements was approved on June 7, 2002, 12 months after the initial requirements were approved.
The effect of weaknesses in the requirements-gathering process is reflected in the additional 12 months and increased expenses needed to gather adequate requirements before software programming could begin. This delay contributed to not completing the development of the TEDS in time to realize the anticipated cost savings or the increase in customer service that will be needed to handle the additional EP restatement applications.
The use of performance monitoring was not effective to identify problems with the development schedule or surface deviations from the estimated project costs. Weaknesses in this process contributed to TE/GE Division executives not being aware of the extent of delays and increased costs.
Performance monitoring and reporting involves collecting and disseminating analytical information to provide stakeholders with knowledge of how resources are being used to achieve project objectives. The TEDS Project Team held regular weekly performance meetings to discuss progress and potential problems. In addition, they developed a performance monitoring plan but did not implement the performance metrics to monitor the development process. Several of the planned performance metrics would have provided management with information needed to monitor the progress of the development process. For example:
· Cost Analysis Report – compares actual expenses to budgeted expenses. TEDS project management could have used this to determine if the project was being developed within estimated costs.
· Schedule Analysis Report – compares actual completion dates to target completion dates. TEDS project management could have used this to identify deviations and implement corrective solutions in case of delays.
When the project management contractor developed the system architecture and the BBC, the cost estimates in the BBC were based on the initial strategy of developing the TEDS in one release. Estimated costs for the delivery of the single-release system were projected at $6.8 million, including $4.5 million for contractor development and integration. The TEDS was approved for development by the IRS’ Chief Financial Officer based on the BBC.
Due to funding constraints, a new release strategy was subsequently approved to develop the TEDS in seven stand-alone releases. As stated earlier, the development team decided not to revise the BBC to reflect cost and scheduling changes based on the new release strategy. As of November 21, 2002, $7.6 million had been paid to both contractors for work completed on Release I. This is significantly more than the $4.5 million initially proposed in the BBC for contractor work and is more than the initial estimate of $6.8 million for the entire TEDS project.
Furthermore, TEDS project management did not establish baseline costs and scheduling for Release I. As a result, TE/GE Division management could not compare planned to actual costs and planned to actual schedule completion dates, which is a key part of performance monitoring.
In addition, BSP management advised us that they do not match the BBC estimates with the funds approved and spent each year for system development. As a result, TE/GE Division management does not have the information to readily determine if the TEDS is being developed within the costs and scheduling constraints of a valid BBC. Consequently, TE/GE Division management cannot determine accountability for the delayed completion dates and increased costs.
BSP management stated that, to continue development of the TEDS, payment for the remaining Release I development will be spread out over several fiscal years as money becomes available. TE/GE Division management approves funds for systems development projects each year, but these funds are not tied to specific deliverables. For example, BSP management informed us that funds were allocated for FY 2003 to complete development of Release I and start development of Release II. However, there was no break down between Release I and Release II costs. As a result, with the TEDS Release I not yet operational as of the end of our fieldwork, TE/GE Division management has continued to spend FY 2003 funds to address the development problems.
TEDS project management advised us that emphasis was placed on meeting an aggressive project completion schedule and that project monitoring was not performed because of a lack of resources. The emphasis placed on meeting the aggressive schedule, without appropriate monitoring of the estimated costs and the quality of work, contributed to the development team focusing primarily on the schedule goal and not on the collection and dissemination of performance information. As a result of not having an up-to-date BBC and of not using performance-monitoring techniques, neither TE/GE Division management nor we can determine the variance between the planned and actual budget and schedule.
The risk management process was not effective to identify and resolve risks that could affect completion of the TEDS development. Weaknesses in this process may have contributed to project delays and increased costs.
Risk management is the process of identifying, analyzing, controlling, and resolving items that may have an effect on the success of the system under development. Risk Response Planning includes assigning responsibility or ownership for resolving identified risk. The owner should report periodically to the project manager and the risk team leader on the effectiveness of the resolution plan, any unanticipated effects, and any corrections needed to resolve the risk.
A formal risk management plan was drafted by the project management contractor and a risk database was established to capture ways to resolve identified risks. However, the plan was not approved or baselined by the TEDS CCB. The TEDS project management advised us that the procedures for the plan were not followed due to a lack of resources and that actions taken were not always documented. Nonetheless, TEDS project management cannot be assured that risks, when identified, are resolved.
The TEDS project management created an atmosphere that allowed members of the team to identify and surface risks for resolution. However, our review of the database identified risks that were put under control but did not have associated plans to resolve the risks. Examples include the following:
· “If the DIO [Deputy Information Officer] staff is not trained on the COTS [commercial-off-the-shelf] products that are being used to implement TEDS, then the O&M [Operation and Maintenance] phase of TEDS may require increased time to fix application errors.”
· “The databases used for development are smaller than the database used in the production environment and the testing of TEDS is not representative of the production database then there may be an impact on the performance of TEDS.”
· “If Product Assurance team or the TEDS Test Team does not have the design document in a timely manner and they are unable to develop the requisite test data, then testing will be incomplete.”
In addition, we determined there were other risks identified by the TEDS Project Team that were not monitored to ensure the risks were minimized and/or eliminated. Examples include the following:
· “The Mid-America Application Center is working on LINUS and has an individual discussing the LINUS and TEDS interface with TEDS Development Team member. If the interface with the TEDS and LINUS is not completed by June 14, then there may be a slip in the schedule impacting the Pilot date.”
· “The TEDS Release I requirements may not be complete, testable, or clear and the test team may not fully understand the requirements in order to develop test scripts and designate expected results then testing will be slow and require rewrite of the project requirements after development has occurred.”
· “The TEDS Project Team is developing the requirements for TEDS based on the initial requirements lack of clarity. If the requirements are not completed in a timely [manner] and require additional discussion with the User Group for review and validation then there will be a schedule impact.”
Weaknesses in the risk management process may have contributed to delays and increased costs to complete the TEDS. The TEDS Project Team identified issues that could have caused disruption of the system’s development; however, the risks were not properly controlled or resolved. At the end of our fieldwork, TEDS project management was working to resolve problems that prevented the system from processing test data. We could not determine if these problems had been identified as a risk; therefore, we could not determine if the actual delays experienced would have been resolved before the system was tested.
Configuration management is the process used to control, manage, and maintain a historical trail of changes to hardware, software, documentation, and other work products. Configuration management includes the following:
· Baseline Management – the process of establishing project baselines and controlling changes to the configuration items such as user requirements, interface specifications, or test plans.
· Change Management – the process of managing changes to ensure that they are properly identified, documented, evaluated for impact, approved by the appropriate CCB, and incorporated into the project development.
· Software Version Control – the process of providing identification control for each version of software, the capability to recreate any version of software, and a point of reference for identifying and managing changes to system configurations.
· Document Version Control – the process of providing a unique identification for each version of the document and ensuring that the correct and most current version of the document is in use at all times.
The contractors developed two configuration management plans that established processes to place configuration items under configuration control. The plans provided for the use of an automated tool set (Rational Tool Suite) that would be used to maintain control over configuration items.
· The project management contractor developed a plan to provide configuration management control over the entire TEDS project. Microsoft Outlook was used to maintain control over configuration items.
· The software development contractor developed a plan to provide configuration management control over the system design documents and the custom software code. Microsoft Visual Source Safe was used to control configuration items.
The plans prepared by both contractors provided a process for identifying, documenting, monitoring, evaluating, controlling, and approving changes. However, the processes identified in the plans were not fully implemented as documented.
· The project management contractor did not conduct regular audits to verify the status of items under control and to monitor compliance with approved standards, expected delivery content, and completeness of baselines.
· The software development contractor’s interface programming was not under formal configuration control. However, one of the programmers maintained an informal control system where he maintained versions of computer code written for the interface programs.
· Changes to baseline documents were not always traced to other related or affected documents to ensure that the related changes were made. The project management contractor advised us that, due to inadequate resources, this process was not always followed.
· Several configuration items were not under configuration management control, including:
Ř Scanner software.
Ř Communications Software.
Ř October 2001 Baselined Requirements.
Ř Requirements Document v1.2.
· There were no effective automated tools (e.g., Rational Clear Case Software) used to assist in the configuration control process as required by the approved plan.
· There was no improvement plan in place to upgrade the configuration control process as suggested by project management best practices.
TEDS project management and the project management contractor advised us that due to the aggressive development schedule, they did not have the resources available to ensure that all configuration management actions took place. As a result, TE/GE Division management does not have assurance that the system was developed effectively and efficiently with the entire development team working from the most recent version of information.
Project management techniques such as those planned for use in the development of the TEDS have been proven to be valuable and useful. For the TEDS, not implementing these management controls has contributed to the following:
· TE/GE Division management did not have access to information for use in making important decisions.
· Contractors could not begin custom programming because of delays in identifying system requirements.
· TE/GE Division management did not realize the planned benefits of $1 million in cost savings by using the TEDS to process the EP restatement applications.
· TE/GE Division management did not realize the planned increase in customer service by using the TEDS to process the EP restatement applications.
· The TEDS will cost much more than was originally budgeted and approved.
TEDS project management should take the following actions for the TEDS Release I and all future releases:
1. Identify and explain cost, benefit, schedule, and contractor performance variances in the BBC update for Release I at Milestone Four. Specifically, this should include reporting actual business benefits realized as compared to projected benefits.
Management’s Response: TE/GE Division management will update the Release 1 BBC at Milestone Four. This updated BBC will conform with the requirements of the IRS’ Investment Decision Management Business Case Procedure dated December 1, 2001, including analysis of business benefits resulting from the deployed system. TE/GE Division management has obtained expert consultant services to assist in this BBC update.
2. Update the BBC to show that the TEDS will be developed using multiple releases. The costs, scheduling, and performance for each release should be shown separately in the BBC. However, if a separate investment decision is needed on any future release, that release should be considered a separate project and have a separate business case.
Management’s Response: TE/GE Division management will update the TEDS BBC showing that the TEDS will be developed in multiple releases. The Release 2 analysis will include an update of the release plan and schedule, cost, and benefit estimates.
3. Establish and baseline a schedule for the remainder of Release I that is feasible both technically and in terms of resources. The schedule should provide the basis for measuring and reporting schedule performance.
Management’s Response: The TE/GE Program Configuration Control Board approved and baselined a new TEDS Release 1 Project Schedule on March 18, 2003.
4. Provide for periodic reports that monitor whether the project is adhering to its established baselines. Status information should include cost and scheduling variance analysis comparing actual project results to planned or expected results.
Management’s Response: The TEDS project management contractor will track the actual and planned completion dates of major tasks and deliverables, and weekly schedule variance reports will be submitted to the TEDS Project Manager. In addition, the software developer will submit a monthly report showing the number of hours worked against each work request. TEDS project management will compare the actual hours worked against each work request to the planned hours for that work request.
5. Fully implement the planned risk management process, including documenting any mitigation plans and the steps taken to resolve identified risks.
Management’s Response: The TEDS Project Risk Review Board will meet to review, and if appropriate, revise the TEDS Risk Management Plan and procedures. The approved Risk Management Plan will be fully implemented, including review of all open and newly identified risks, related mitigation plans, and action items.
6. Fully implement the planned configuration management process to control and manage any changes to items that should be under change control, including project design documentation, computer hardware, or software.
Management’s Response: All TEDS baselined design documentation and requirements, and the documented configuration of the TEDS hardware and software, have been placed under CM. All changes to documentation and physical architecture under CM are reviewed, approved, and recorded as Change Requests according to the TEDS Configuration Management Plan.
The Contractor Development team is exercising internal CM for both the Tier-2 and Tier-3 software deliverables. All changes made to the software during the Systems Acceptance Testing were approved by the staff of the Division Information Officer. All Tier-2 custom code and documentation artifacts were given to the IRS Product Assurance, Source Code/Document Control Branch for configuration management control, in accordance with established IRS procedures.
Our overall objective was to determine if Tax Exempt and Government Entities (TE/GE) Division management was following accepted systems development guidelines to ensure that defined goals will be achieved in the Release I development stage of the Tax Exempt Determination System (TEDS).
To accomplish this objective, we performed the following work:
A. Identified the level of experience and training received by key persons (e.g., Project Manager) involved in the development of the TEDS.
B. Interviewed the Project Manager and other applicable project team members to determine if they were following a written comprehensive project plan.
C. Reviewed the project plan to determine if it had a Work Breakdown Structure schedule.
D. Reviewed the project plan to determine if a risk management process was in place to identify, quantify, respond to, and control the risks in the project.
E. Reviewed the project plan to determine if the performance measurement process was effectively used to monitor the project’s progress.
F. Reviewed the project plan to determine if there was a configuration management process for identifying, documenting, monitoring, evaluating, controlling, and approving all changes made during the life cycle.
G. Reviewed the project plan to determine if a Requirements Development Plan had been prepared that documented how requirements would be developed.
A. Determined whether the project team was adhering to the Enterprise Architecture (EA) requirements.
B. Determined if the project documents showed how the TEDS project fit into the IRS architecture of the future.
C. Obtained and reviewed the key EA deliverables as well as meeting minutes and other key briefings where the TEDS project and Blueprint EA information were discussed. Determined whether the information provided was sufficient to enable the project’s direction to be consistent with that of the IRS’ overall modernization efforts.
D. Determined if the TEDS Project Team had coordinated its development efforts, and resolved any differences, with the Tier 2 Consolidation Project Office.
Daniel R. Devlin, Assistant Inspector General for Audit (Headquarters Operations and Exempt Organizations Programs)
Nancy A. Nakamura, Director
Gerald T. Hawkins, Audit Manager
Tom Seidell, Acting Audit Manager
Barry G. Huff, Senior Auditor
Andrew J. Burns, Auditor
Perrin T. Gleaton, Auditor
Deputy Commissioner N:DC
Associate Commissioner, Business Systems Modernization M:B
Director, Business Systems Planning, Tax Exempt and Government Entities Division T:BSP
Chief, Information Technology Services M:I
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:AR:M
Director, Communications and
Liaison, Tax Exempt and Government Entities Division
Chief, Information Technology Services M:I
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.