Business Cases for Information Technology Projects Need Improvement

 

April 2005

 

Reference Number: 2005-20-074

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

 

April 29, 2005

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:     Pamela J. Gardiner /s/ Pamela J. Gardiner

                 Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - Business Cases for Information Technology Projects Need Improvement (Audit # 200420037)

 

This report presents the results of our review of the Internal Revenue Service’s (IRS) compliance with the Office of Management and Budget (OMB) and Clinger-Cohen Act of 1996 requirements to plan, manage, and control information technology (IT) investments.  Twice yearly, the OMB requires agencies to complete an OMB Circular A-11 Exhibit 300, Capital Asset Plan and Business Case, for each major IT investment. These documents are essentially business cases used by agencies to request funds, monitor the progress of projects, and improve management decision making over expensive IT investments.

In summary, the IRS prepares about 30 business cases each year.  We reviewed the business cases prepared for Budget Years (BY) 2005 and 2006 for four IT projects.  Two of the projects were under development and two were in operation.

All four business cases we reviewed contained deficiencies, did not comply with the OMB and Clinger-Cohen Act requirements and, in some cases, did not comply with the IRS’ own Exhibit 300 Business Case Guide.  As a result, we believe senior IRS executives and Department of the Treasury and OMB officials should not rely on the data in these business cases to manage and fund the projects.  Inaccurate information in business cases can distort viable analysis and provide IRS executives with a false assessment of the actual progress and costs of projects.  Senior IRS executives advised us they recognize the reporting problems and will make changes to the BY 2007 submissions. 

We found project costs were not reported accurately in three of the four business cases we reviewed.  One project omitted $38.5 million in direct labor costs in determining the total amount spent on the project and also did not report required project-specific security costs such as costs for system administrators and background checks.

For financial systems, the OMB recommends agencies consider the three best alternatives when selecting vendors for IT products.  In 1 of the business cases we reviewed for a project expected to cost over $1 billion, only 1 viable alternative was provided.  Contractors were responsible for this procurement and did not provide the IRS sufficient information for preparing the alternatives analysis.    

We also found the Earned Value Management (EVM) data for the two projects under development were incomplete, inaccurate, and outdated, making it difficult for senior IRS executives and Department of the Treasury and OMB officials to measure and assess the overall progress of IT projects.  Budgeted costs were inaccurate, baseline information changed from 1 year to the next, project teams could not provide supporting documentation, and IT project costs were outdated.  A problem identified in the software used by the IRS also contributed to inaccuracies in the EVM sections of the business cases.

Additionally, in-depth reviews required by the E-Government Act of 2002 to identify smarter and more cost effective methods for delivering performance were not documented for two projects already in operation.  One of these IT projects requested an increase in operational costs even though it continues to experience a significant reduction in use.

The Capital Planning and Investment Control (CPIC) office did not always provide adequate guidance to Project Managers.  More significantly, Project Managers did not ensure business cases were prepared accurately and in compliance with guidance given by the OMB and IRS.  Overall, we noted a general view by the Project Managers involved that the sole purpose of the business case was to obtain funding for projects.  Insufficient effort was made to prepare accurate business cases which could be relied upon to better manage IT projects as the Clinger-Cohen Act of 1996 intended.

To make business cases more reliable and useful, we recommended the Chief Information Officer (CIO) hold Project Managers and the CPIC office accountable for following existing guidance in computing costs for IT projects, ensure work performed by contractors is adequately reviewed before acceptance, modify the Enterprise Life Cycle (ELC) to require a comprehensive evaluation of all vendor proposals, and revise the Exhibit 300 Business Case Guide to require an assessment of three viable alternatives for IT investments.  Additionally, the CIO should hold IT Project Managers accountable to ensure all sections of the business case are consistent, accurate, and complete.  For projects affected by the EVM software problem, we recommended the CIO correct all software problems that lead to incorrect earned value calculations.  For operational systems, the CIO should ensure Project Managers document the results of their E-Government reviews in their business cases.

Management’s Response:  The IRS CIO agreed with all of our recommendations and indicated some corrective actions have already been taken.  The CIO designated Project Managers as the accountable individuals for all data contained in their business cases, designated appropriate persons to review work performed by contractors, revised the ELC to require a comprehensive package evaluation and selection report on all commercial off-the-shelf products that are considered for new IRS systems, and modified the IRS Exhibit 300 Guide to require the three best viable alternatives be presented in applicable business cases.  Furthermore, the CPIC office will provide training and guidance documents to Project Managers.  To address inaccurate EVM data in business cases, the IRS will work with the necessary parties to cause a permanent change to the software used to report EVM data in the BY 2008 business cases.  To ensure accurate EVM data in BY 2007 business cases, the IRS will implement a manual temporary solution.  Lastly, the CIO will require operational Project Managers to include, in their business cases, the results of E-Government reviews performed on their projects.

The CIO disagreed with our conclusion that executives from the IRS, Department of the Treasury, and the OMB should not rely upon data in the IRS’ business cases.  In addition, the CIO did not concur with the outcome measures reported in Appendix IV of the report.  Management’s complete response to the draft report is included as Appendix VII.

Office of Audit Comment:  Our conclusion that executives from the IRS, Department of the Treasury, and the OMB should not rely upon data in the IRS’ business cases is based on the multiple errors we found in the sampled business cases and the lack of compliance with OMB requirements.  The outcome measures claimed in the report are based on errors and misstatements in the business cases that hinder the ability of decision-makers to rely on these documents.  The outcome measures are not based on cost savings. 

While management agreed with our recommendation to include the results of E-Government reviews in their business cases, they stated that these reviews are not required for projects that are scheduled for replacement by modernization projects.  We disagree with this position since the IRS’ modernization efforts are scheduled over several years and we believe there could be many opportunities to improve the efficiency of existing projects in the interim.  We plan to conduct a follow-up review to determine the effectiveness of all of IRS management’s corrective actions and to evaluate the accuracy of the business cases for the BY 2007 budget submission. 

We will include this issue in our follow-up review and, accordingly, we do not intend to elevate this matter to the Department of the Treasury. 

 

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs) at 202-622-8510.

 

Table of Contents

Background

Procedures for Preparing Business Cases Have Improved, but Managers Are Not Complying

Project Costs Were Not Reported Accurately

Recommendations 1 and 2:

Relevant Cost and Benefit Information Was Omitted in Alternatives Analysis Reporting

Recommendations 3 and 4:

Progress on Development Projects Was Measured Inaccurately

Recommendations 5 and 6:

Business Cases for Operational Projects Did Not Demonstrate Results of an In-depth E-Government Review

Recommendation 7:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measures

Appendix V – Summary of Business Case Deficiencies

Appendix VI – Description of Information Technology Investments Selected for Review

Appendix VII – Management’s Response to the Draft Report

 

Background

The Clinger-Cohen Act of 1996 requires Federal Government agencies to focus on the results they achieve through their information technology (IT) investments.  Agencies are required to put their technology investment decisions in a true business context and analyze investments for their return on investment.

In October 2000, the Senate Governmental Affairs Committee stated the Clinger-Cohen Act was the result of the Committee’s reviews of failed computer system acquisitions, such as the Internal Revenue Service’s (IRS) Tax Systems Modernization project.  The Committee further reported Federal Government agencies were not using sound business procedures before investing in information technology.

The Clinger-Cohen Act requires agencies to implement processes and maintain information needed to help ensure IT projects are completed at acceptable costs, are within reasonable and expected time periods, and are contributing to tangible, observable improvements in mission performance.  The Department of the Treasury has not yet completed and formalized its capital planning process; however, information is available to assist bureaus.

The Office of Management and Budget (OMB) published Circular A-11, Preparation, Submission and Execution of the Budget, to assist Federal Government agencies in complying with the Clinger-Cohen Act.  This detailed guidance includes two key sections applicable to IT capital planning.  Section 300, Planning, Budgeting, Acquisition, and Management of Capital Assets, provides guidance on how business cases should be prepared.  Section 53, Information Technology and E-Government, provides guidance on how to prepare an agency’s IT Investment Portfolio.

Twice yearly, agencies must complete OMB Circular A-11 Exhibit 300, Capital Asset Plan and Business Case, for each major IT investment and submit it to the OMB.  The OMB reviews the initial submission and returns it to the agency for revision and resubmission.  Exhibit 300 is essentially the business case for a specific IT project.  It is used by the OMB to allocate funds for IT investments and should also be used by agencies to manage their investments.  The business case must clearly demonstrate past and projected costs of the project along with what the agency achieved or expects to achieve with the investments.  In Budget Year (BY) 2004, the IRS prepared 30 business cases.

We judgmentally selected four IT investment projects and reviewed their business cases for BYs 2005 and 2006.  Two of the projects, the Custodial Accounting Project (CAP) and the Integrated Financial System (IFS), were in the development stage and two, the Integrated Submission and Remittance Processing (ISRP) system and the Counsel Automated Systems Environment (CASE), were operational.  See Appendix VI for an explanation of each of these four IT investment projects.

We reviewed the business cases to determine if they were accurate and in compliance with OMB requirements.  We focused on the following key sections of the business cases:

·         Summary of Spending table - A summary of planned, actual, and expected costs for the life of the project.

·         Alternatives Analysis - An analysis of alternatives to the computer system under development.

·         Earned Value Management (EVM) - For development projects, a comparison of planned cost and schedule goals to actual cost and schedule results to date.

This review was performed in the office of the Modernization and Information Technology Services organization at the IRS Headquarters offices in Washington, D.C., and New Carrollton, Maryland, during the period June through December 2004.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.  Management’s complete response to the draft report is included as Appendix VII.

Procedures for Preparing Business Cases Have Improved, but Managers Are Not Complying

In early 2003, the IRS made a substantial effort to improve business cases after the OMB assigned failing scores to all but one of the IRS’ BY 2004 business cases.  Specifically, the IRS:

·         Prepared the Exhibit 300 Business Case Guide to provide detailed instructions for completing business cases.

·         Required subject matter experts to review a specific part of each business case and provide feedback to the IT Project Manager.

·         Required the Capital Planning and Investment Control (CPIC) office to perform an overall review of each business case and provide feedback to the IT Project Manager.

·         Required the Project Manager for each IT investment to conduct a final overall review to ensure the business case is correct.

However, these procedures are not operating effectively.  IT Project Managers did not take ownership of their IT development projects and did not provide sufficient emphasis to ensure the business cases presented accurate information.  Specifically, documentation did not exist to support important data on the business cases or, when we found documentation, it did not always support the information provided on the business cases.  Several of the errors we identified could have been identified with a careful review of only the business case itself, without going to supporting documents.  We noted several instances where IT Project Managers relied on contractors’ input without questioning the information presented.

All four business cases for BYs 2005 and 2006 contained deficiencies and did not comply with either the Clinger-Cohen Act or OMB requirements and, in some cases, the IRS’ own Exhibit 300 Business Case Guide.  Business case deficiencies are summarized in Appendix V.

We also noted a general view on the part of Project Managers and their staffs that the business cases were solely a budget document designed to obtain funding from the OMB.  Insufficient attention was given to ensuring the accuracy of the information so the business cases could also be used to manage projects.  For example, business cases should be used to ensure spending on IT projects will provide a return on investment equal to or better than alternate uses of funding.  Business cases should also be used to identify projects that are performing poorly and those that no longer fulfill mission requirements.

Additional emphasis is needed to improve the accuracy and completeness of the business cases.  We anticipate that, by addressing the recommendations included in the following sections, the IRS will provide the needed emphasis.

We have been assured by senior IRS executives that actions are being taken to improve IRS processes for developing business cases.  The IRS expects these actions will significantly improve its BY 2007 submissions to the OMB.

Project Costs Were Not Reported Accurately

The Summary of Spending table presents a high-level overview of budgeted, actual (to the extent these are known), and expected costs for an entire project on a year-by-year basis.  Errors in the Summary of Spending table hinder the ability of decision makers to rely on this information to allocate funds or to use the business case as a management tool.

For BYs 2005 and 2006, the IRS inaccurately reported project costs in three of the four business cases.  Multiple errors were reported in the Summary of Spending tables.  Specifically, management and overhead costs were not allocated to IT projects, and IT Project Managers made errors in calculating security costs.  In addition, the business case for the ISRP omitted direct labor costs and some hardware and software costs.

The IRS CPIC office did not allocate management and overhead labor costs to development projects

IRS management and overhead labor costs of $79.4 million were not allocated to IT development projects for BY 2005.  As a result, the costs for each IT development project were understated.  The IRS did not follow the OMB and its own guidance which require such costs to be allocated among the IT development projects.

Inaccurate security costs were reported for three projects

Many IT projects rely on the hardware, software, and staff support services that constitute the underlying infrastructure security for IRS systems.  Consequently, rather than trying to determine precisely how much each system uses the infrastructure security, IRS security specialists allocated infrastructure security costs based on a percentage of IT operating costs.

The IRS determined its security infrastructure costs were $45.3 million at the time it prepared the BY 2005 business cases, or 3.08 percent of total costs for each project.  Each project was to have this security percentage applied to its total operating costs to produce a basic infrastructure security cost.  Project-specific security costs should be added to the basic infrastructure security cost to calculate a total project cost to report in the business case.

In reviewing the ISRP expenditures, we identified security costs that were not included, such as security administrators and background checks for contractors.

In addition, CAP project BY 2005 security costs were overstated by $11.2 million.  The CAP BY 2005 business case reported security costs would be $17.9 million (29.5 percent) of total project costs of $60.6 million.  Based on available documentation, actual security costs totaled $6.7 million (11 percent) of the total project costs.

IFS project BY 2005 security costs were also reported inaccurately.  The business case reported security costs in 1 section as $7.18 million and in another section as $4.14 million.  IFS staff indicated they may have calculated the infrastructure cost percentage against only some of the project costs.  However, since the IRS relied on a contractor and could not provide support for either calculation, we could not identify where the actual error occurred.

The BY 2006 business cases also underreported security costs for both the CAP and IFS.  Following IRS guidance described above, security costs should have been at least 3.08 percent of total project operating costs.  However, the CAP business case reported no security costs, and the IFS business case reported only 1 percent ($217,000) of total project costs.

ISRP system total project costs were understated

OMB guidance directs full life-cycle costs, including labor costs, to be included in total project costs.  The ISRP business case did not properly report $38.5 million in IRS labor costs in BY 2005.  The total project cost is a significant factor in evaluating a business case.  The ISRP business case disclosed the IRS labor costs as a separate line item, but did not include them in the total project costs as required by OMB guidance.  The ISRP project team advised us these costs were omitted because they misunderstood guidance given to them.

In addition, the ISRP system BY 2006 business case did not include $4.5 million related to system hardware and software costs used by the ISRP system.  The costs, representing 32 percent of maintenance costs of the project, were instead included in agency-wide infrastructure costs.  As a result, project costs were understated.  Since the costs were directly attributable to the ISRP system, they should have been reported on the ISRP system business case as required by the OMB.

Project management for all three projects was not aware of applicable OMB and IRS guidelines for computing costs.  For both the CAP and IFS projects, management relied on contractors to make key security calculations without adequately monitoring and reviewing the contractors’ work.  IRS reviews of the business case information did not identify obvious errors such as security costs that were below the basic security cost allocation.

Recommendations

The Chief Information Officer (CIO) should:

1.      Hold Project Managers and the CPIC office accountable for following existing guidance in computing the costs of IT projects.

Managements Response:  The CIO has designated Project Managers in Business Systems Development (BSD) and Business Systems Modernization (BSM) offices as the individuals accountable for all data contained in their operational and developmental business cases.  The CPIC office will provide training and guidance documents as appropriate.

The CIO disagreed with our conclusion that executives from the IRS, Department of the Treasury, and the OMB should not rely upon data in the IRS’ business cases.  In addition, the CIO did not concur with the outcome measures of $339 million reported in Appendix IV of the report.  The premise that dollar savings could result from the findings is not accurate.

Office of Audit Comment:  Our conclusion that executives from the IRS, Department of the Treasury, and the OMB should not rely upon data in the IRS’ business cases is based on the multiple errors we found in the sampled business cases and the lack of compliance with OMB requirements.  The outcome measures claimed in the report are based on errors and misstatements in the business cases that hinder the ability of decision-makers to rely on these documents.  The outcome measures are not based on cost savings.

2.      Ensure work performed by contractors is adequately reviewed before acceptance.

Managements Response:  The Associate CIO, BSM, has taken corrective actions to address this recommendation.  Work performed by contractors is reviewed by the assigned Integrated Project Team, Project Manager, Acquisitions Project Manager, and stakeholders who have the expertise to affirm the deliverable is accurate and complete.

Relevant Cost and Benefit Information Was Omitted in Alternatives Analysis Reporting

The Alternatives Analysis is a key component of the business case.  It provides estimated cost and benefit information on viable alternatives to assist management in determining the most effective approach for the project.  The Alternatives Analysis for the IFS and CAP projects did not contain critical data for both BYs 2005 and 2006 reporting.

The IFS Alternatives Analysis did not compare viable options 

The IFS project business case did not present the three best alternatives, as required by the OMB.  For financial systems, the OMB has stated agencies must use commercial off-the-shelf (COTS) software from among a number of predetermined vendors.  For the IFS, the most viable alternatives would have been COTS software products described in OMB guidance.  Only one of the alternatives presented in the business case included a COTS software product.

The other two alternatives presented in the business case were a “do nothing” option of patching the current system and a “build from scratch” option.  Neither option was a viable alternative.  The IRS did not include cost data for these options but merely stated costs would be “astronomical” or “prohibitive.”  The business case itself stated these two alternatives were not viable.  The IRS estimates, over the project’s life cycle, IFS development costs will be $531 million and total costs will be over $1 billion.

Two obstacles precluded the full completion of the Alternatives Analysis.  First, the IRS did not have cost information on the COTS software alternatives that were considered for the IFS.  The IRS delegated the COTS software product selection to a contractor, the Computer Sciences Corporation (CSC).  The CSC was tasked with issuing a request for vendor proposals, evaluating the proposals, and selecting the best vendor package.  However, after making the selection, the CSC did not share detailed cost information with the IFS project team claiming this information was proprietary.  CSC did share summary information on the vendor proposals with select high-level IRS executives in early 2002. 

Further, the IFS project team stated that, initially, the CSC had a contract requirement to deliver a package evaluation and selection report that included cost information on all alternatives.  This requirement was removed in February 2002.

The IRS Enterprise Life Cycle (ELC) methodology states a package evaluation report for COTS software products will report the results of package selection and include information on vendors contacted.  The CSC and the IRS did comply with the ELC.  The CSC provided the IRS with cost and additional information on the winning vendor and summary information on the other vendors who submitted proposals. 

We believe the ELC requirements are not adequate on this issue.  If the IRS does not require contractors, such as the CSC, to provide cost and other relevant information on proposals from all vendors, it cannot ensure the Federal Government’s interests are being protected in these selections.  Also, it will not be able to provide the OMB with information it needs to ensure the IRS is making prudent investment decisions.

The second obstacle encountered for the IFS project was the IRS Exhibit 300 Business Case Guide does not follow OMB guidance and could encourage nonviable alternatives to be reported.  OMB guidance requires agencies to identify all viable alternatives and then select and report details on the top three viable alternatives.  In contrast, the IRS Exhibit 300 Business Case Guide specifies the first of three alternatives will be to do nothing.

The IRS did not conduct a cost-benefit analysis for the CAP

As part of the Alternatives Analysis, the business case must provide an analysis of the costs and benefits expected from the project.  The OMB allows Federal Government agencies to use a cost-effectiveness approach, rather than a cost-benefit analysis, to justify an investment in a new computer system when the new system is intended to satisfy a legislative mandate.  A project that is justified by cost effectiveness would not need to show the same level of monetary benefits (i.e., benefits that exceed costs).  Other nonmonetary benefits, such as improved mission performance or increased quality, speed, and flexibility, may play a more important role in justifying the investment.

The IRS did not calculate the net present value (NPV), return on investment (ROI), or other cost-benefit indicators for the CAP, citing the cost-effectiveness approach.  The Government Accountability Office (GAO) supported the IRS’ decision to proceed without conducting a cost-benefit analysis.

At the time of the GAO decision, the total estimated CAP project cost over a 10-year life was approximately $150 million.  A more recent estimate is the CAP project will cost $367 million over its first 10 years.  It is not clear whether the Congress and the GAO would have approved a cost-effectiveness justification for the CAP at the current estimated cost.  However, since the IRS has cited the cost-effectiveness approach for the CAP on every business case since 2001, we are not making any specific recommendations on this issue.

For future IT investments, we suggest the IRS use the cost-effectiveness approach only in rare circumstances.  A cost-benefit analysis is a central provision of the Clinger-Cohen Act.  This analysis documents the locations and areas where cost savings can be achieved after a system is deployed.  OMB Circular A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs (dated October 29, 1992) states, while cost-benefit indicators are not always computable, efforts to measure them can produce useful insights even when the monetary values of some benefits or costs cannot be determined.  The OMB requirements emphasize quantifying benefits and costs is worthwhile, even when it is not feasible to assign monetary values.

Recommendations

The CIO should:

3.      Modify the ELC to require a comprehensive package evaluation and selection report, including costs, on all COTS software products that are considered for new IRS systems.

Management Response:  The Associate CIO, BSM, implemented a revised ELC directive that was signed on August 24, 2004.  The directive states an evaluation and selection report, including costs on all COTS products that are considered for new IRS systems, is a mandatory delivery requirement at milestones II and III.

4.      Revise the Exhibit 300 Business Case Guide to require the three best viable alternatives be presented in the business case.  In cases where authoritative guidance directs that COTS software products be used, three COTS software alternatives should be compared in the Alternatives Analysis.

Management Response:  The Director, CPIC, modified the Exhibit 300 Guide on February 1, 2005.  This guide was modified to require that for Development, Modernization, and Enhancement investments the three best viable alternatives be presented in the business case.

Progress on Development Projects Was Measured Inaccurately

EVM data in the CAP and IFS business cases were incomplete, inaccurate, and outdated.  EVM is a method of analysis that provides management with an objective measurement of how well an IT investment in the development stage is progressing in terms of planned costs and schedule.  Earned value compares the work the project team has finished so far with the estimates made at the beginning of the project.  If budget and schedule variances are not properly tracked, underlying problems may not be promptly discovered and addressed by management.  The OMB requires agencies to use an industry-compliant EVM system to obtain specific timely information on the progress of investments.

We identified multiple problems in the EVM sections of the two development project business cases we reviewed.  The IRS did not adequately review the business cases and data provided by contractors.

The Budgeted Cost of Work Scheduled (BCWS) was inaccurate for both the CAP and IFS projects

The CAP BY 2005 business case reported the budgeted costs for the project as $77.5 million.  Based on the limited amount of documentation provided to us by the IRS, we determined the budgeted costs should have been $46.7 million.

The IRS made five separate errors in its calculation of the budgeted costs for the CAP.  Specifically, the IRS:

  • Double-counted $87.1 million in planned costs.
  • Did not account for Infrastructure Shared Services (ISS) costs.
  • Multiplied the BCWS by a Performance Cost Index (PCI) ratio.
  • Calculated the BCWS using a percentage complete factor.  The percentage of the work completed should be used to calculate the Budgeted Cost of Work Performed, not the BCWS.
  • Did not use a work breakdown structure to calculate the BCWS.  The work breakdown structure is a key support document needed to calculate the BCWS.

Additionally, the IFS BY 2005 business case reported the planned costs for the project as $58.1 million.  Based on limited information provided to us by the IRS, the costs should have been $111.8 million.

Original baseline budget information changed from 1 year to the next for both the CAP and IFS

The original baseline represents the approved cost, schedule, and performance goals from the first time a project goes through the Federal budget process.  The original baseline information should never change, so progress can be measured in future years.

The baseline budget estimates were reported inaccurately in both the CAP and IFS projects’ BY 2006 business cases.  In its BY 2006 business case, the CAP project changed its original baseline in four separate areas.  For example, the BY 2005 business case reported its baseline for the total costs of major components to be $8.4 million and completion to be November 29, 2000.  However, the BY 2006 business case shows the baseline costs as $16.2 million and the completion date as November 30, 2004.

Baseline budget estimates for the IFS project were reduced from $247 million in 2005 to $227 million in 2006 because $20 million in costs were inadvertently omitted.

The CAP and IFS project teams could not provide supporting documentation for much of the EVM data

The CAP and IFS project teams could not provide supporting documentation for EVM data reported in the BYs 2005 and 2006 business cases.  Furthermore, IRS officials assigned to the project teams could not explain how EVM numbers were calculated.  They referred us to contractors and other IRS employees for explanations of the calculations.  For BY 2005, EVM data for both business cases were completed by a contractor who is no longer assigned to the IRS and an IRS employee who was not available during our review.  OMB guidance requires the retention of supporting documentation to permit review and verification of key management reports.

Earned value data were not updated before submission to the OMB

Business cases are officially submitted to the OMB twice yearly.  In making critical oversight and funding decisions, the OMB needs to have the most recent data possible.

Earned value data reported in the CAP and IFS projects’ BY 2005 business cases were out of date when this information was submitted in November 2003.  CAP and IFS project data were dated February 28, 2003, and May 30, 2003, respectively.  This information should have been calculated as of October 31, 2003.  Based on information available at the time of our review, we could not determine if business case data would have been significantly different if they had been correctly reported.

Cost data were reported inconsistently between sections of the same business case

The Summary of Spending table, Alternatives Analysis, and EVM sections all serve different purposes, but they often require the same cost information.  Information should be reported consistently among the sections of a business case.  IRS guidance also states information in these sections should match.

In the CAP BY 2005 business case, the EVM section indicates the IRS will spend $286 million through September 30, 2006, while the Summary of Spending table indicates $314 million for the same period, a $28 million difference.  The difference was due to IRS labor costs being omitted from the EVM section.

In addition, IRS labor costs for the CAP Alternatives Analysis and the Summary of Spending table did not agree.  The Alternatives Analysis section reported $21.5 million in IRS labor costs, while the Summary of Spending table reported $52 million for the same period, a $30.5 million difference.

The IFS BY 2005 business case had similar discrepancies. Specifically, the EVM section omitted about $35 million in IRS labor costs that had been included in the Summary Spending table.

The IRS could not verify contractor EVM systems were compliant with industry standards

The OMB requires agencies to demonstrate and explain the process they use to verify their contractors’ project management systems follow the current industry standard.  The IRS did not comply with this requirement.

The IRS could not verify contractor or IRS EVM systems were compliant with industry standards at the time the IRS submitted its business cases.  The CAP project BY 2005 business case reported the IRS would review its contractor’s EVM system to ensure compliance with the industry standard.  However, the IRS did not follow through and perform this review.  The CAP project BY 2006 business case reported the contractor’s EVM system complied with industry standards.  However, we were not provided with any documentation to support that statement.

For the IFS, the IRS had undertaken efforts to ensure contractor compliance with standards.  The IRS had identified several instances of contractor noncompliance that had not been corrected at the time BY 2005 data were submitted.  This information was not disclosed in the IFS project BY 2005 business case.

The Department of the Treasury (Treasury) ProSight Portfolios (ProSight) system contributed to inaccuracies in business cases

A problem was identified in the Treasury’s ProSight application that contributed to inaccuracies in the EVM section of the IRS’ business cases.  The Treasury required bureaus to use the ProSight application to report business case information effective for the BY 2006 submission.  Information was due to the Treasury in August 2004.  At that time, Treasury officials noticed some data in business cases were inaccurate.  Following our discussion with Treasury officials regarding the cause of the error, they reported the problem to the OMB to ensure the OMB was aware of it.

The problem occurred only when projects had already changed their budget at least once and were proposing an additional budget change.  For such projects, one table compared actual costs to proposed budget data.  The table should have been comparing actual costs to current OMB-approved budget data.

As a result, earned value data were inaccurate and could not be relied upon to make management decisions.  While this error affected only those projects requesting more than one budget change from the OMB, such projects are the ones likely to be experiencing prolonged difficulties with budget and schedule overruns.  Therefore, it is even more important that the EVM data for these projects be stated correctly.

Recommendations

The CIO should:

5.      Hold Project Managers accountable to ensure all sections of the business case are consistent, accurate, complete (including IRS overhead and IRS labor costs), and supported by documentation.

Management Response:  The CIO has designated Project Managers in BSD and BSM offices as the individuals accountable for all data contained in their business cases, including all costs.  The CPIC office will also provide training and guidance documents as appropriate.

6.      Correct all software problems that lead to incorrect earned value calculations so accurate cost and schedule variance information is provided.  Accurate cost and schedule variance calculations and metrics, as required by the OMB, should be included elsewhere in the EVM section.

Management Response:  The Director, CPIC office, will work with the necessary parties to cause a permanent change to correct EVM modeling in the application software used for data reporting in the FY 2008 business cases.  In addition, the IRS will prepare a manual temporary work around for EVM calculations in the FY 2007 business cases.

Business Cases for Operational Projects Did Not Demonstrate Results of an In-depth E-Government Review

The E-Government Act of 2002 requires a comprehensive review and analysis to be performed on legacy systems and IT investments to identify strategies for smarter and more cost-effective methods of delivering performance.  The CPIC office has affirmed the need for all IT investments to undergo an E-Government review.

The business cases for the two operational projects we reviewed, the ISRP and the CASE, did not demonstrate an in-depth E-Government review was conducted.  As a result, important information that could help management improve performance may not be considered.  Opportunities to share hardware or software, use emerging technology, and provide better service to the public and other Federal Government agencies may not be identified.

IRS guidance for conducting E-Government reviews was not adequate.  The IRS required projects to conduct an operational analysis that addressed a project’s performance in three areas:  budget and milestones, requirements, and customer expectations and needs.  The CPIC office staff stated these were the only elements required to be reported on the business case for operating projects.

In addition to this information, the OMB requires:

  • A review of the current way of doing business and performing the function, aimed at identifying improvements and cost savings.

·         A recent alternatives analysis performed with a future focus.

The ISRP system and CASE project business cases did not go beyond answering the basic questions identified in the IRS guidance.  Consequently, important information affecting project performance was not analyzed or reported.

For example, the ISRP system business case projected an increase in operational costs of about $5 million over the next 6 years.  The ISRP system, which is used to process paper-based taxpayer documents, should experience a declining workload over time, as more and more taxpayers file electronically.  In the next 6 years, the IRS estimates the annual number of paper returns will decrease from 72.5 million in Fiscal Year 2004 to 51.5 million in Fiscal Year 2010.  This trend was not reflected in the ISRP system operational analysis.

Recommendation

The CIO should:

7.      Require the Project Managers of operational systems to document the results of their E-Government reviews in their projects’ business cases.

Management Response:  Management agreed with our recommendation but in their response took the position that E-Government reviews are not required for existing projects that are scheduled for replacement by modernization projects, and stated the OMB approved this position.  For operational projects not scheduled for replacement by a modernization project, the CPIC team will provide training and guidance on how to conduct an E-Government review and analysis.  Project Managers will be required to report the results of the E-Government review in their business cases.

Office of Audit Comment:  We disagree with management’s position that E-Government reviews are not required for projects that have been scheduled for replacement by modernization projects.  Since the IRS modernization efforts are scheduled to be implemented over several years, we believe the IRS could have many opportunities to improve the efficiency of existing projects before they are modernized.  We found no documentation that the OMB had approved the IRS’ position.  We will revisit this issue in our audit of the BY 2007 submission.  During the follow-up review, we will also evaluate management’s progress in developing E-Government reviews for those projects not scheduled for replacement by modernization projects.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to determine whether the Internal Revenue Service (IRS) is planning, managing, and controlling its information technology (IT) investments in compliance with the Office of Management and Budget (OMB) and Clinger-Cohen Act of 1996 requirements.  To accomplish this objective, we:

I.       Judgmentally selected 4 of 30 IT investments; 2 were development projects and 2 were classified as operational.  The two development projects were the Custodial Accounting Project and the Integrated Financial System.  The two operational projects were the Integrated Submission and Remittance Processing system and the Counsel Automated Systems Environment (see Appendix VI for a description of these projects).  We selected these four projects to avoid duplicating the work of other Treasury Inspector General for Tax Administration audit groups and based on the significant funds expended by the IRS on these projects.

II.     For these four projects, reviewed the OMB Circular A-11 Exhibit 300, Capital Asset Plan and Business Case, prepared by the IRS prior to authorizing the IT investments and determined whether the business cases conformed to the OMB business case requirements.

A.    Reviewed OMB Circular A-11 Exhibit 300 and consulted with the OMB and Department of the Treasury to determine the requirements for an agency to be in compliance.

B.     Obtained the Budget Years (BY) 2004 and 2005 business cases, reviewed the OMB comments and concerns, and determined whether the IRS addressed them.

C.     Evaluated the BYs 2005 and 2006 budget submissions to the OMB and determined whether the business cases were completed in compliance with OMB requirements.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Program)

Stephen R. Mullins, Director

Thomas Polsfoot, Audit Manager

W. Allen Gray, Senior Auditor

Mary L. Jankowski, Senior Auditor

Jody Kitazono, Senior Auditor

Thomas Nacinovich, Senior Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Acting Chief, Security Services  OS:CIO:S

Director, Financial Management Services  OS:CIO:FM

Acting Director, Portfolio Management  OS:CIO:R:PM

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaison:  Chief Information Officer  OS:CIO

 

Appendix IV

 

Outcome Measures

 

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration.  These benefits will be incorporated into our Semiannual Report to the Congress.

Type and Value of Outcome Measure:

§         Reliability of Information – Potential; $133.6 million.  Project costs were not reported accurately (see page 4).  The inaccurate estimates and the misstated costs hindered the ability of decision-makers to rely on the business cases to allocate funds or use the information as a management tool.  Reliability of Information outcome measures are reported on an absolute basis; i.e., both overestimated and underestimated amounts are reported as positive amounts.

Methodology Used to Measure the Reported Benefit:

1.      Overhead Costs:  Budget Year (BY) 2005 costs not allocated
to projects; project costs were underestimated (see page 5).                          $79.4 million

2.      Security infrastructure costs were inaccurate (see page 5).                            $11.2 million

3.      Integrated Submission and Remittance Processing costs
were understated (see page 6).

a)      BY 2005 labor costs were understated                                                     $38.5 million

b)      BY 2006 hardware and software costs                                                      $4.5 million

Total                                                                                                                    $133.6 million

Type and Value of Outcome Measure:

§         Reliability of Information – Potential; $205.8 million.  Budget and schedule variances were not properly tracked and underlying problems may not be promptly discovered and addressed by management (see page 11).

Methodology Used to Measure the Reported Benefit:

1.      Budgeted cost of work inaccurate.

a)      Custodial Accounting Project (CAP) BY 2005 business case (see page 12).

i)        Reported                                                               $77.5 million

ii)      Treasury Inspector General for
Tax Administration (TIGTA) determination        $46.7 million

iii)    Difference                                                                                           $30.8 million

b)      Integrated Financial System (IFS) BY 2005 business case (see page 13).

i)        Reported                                                               $58.1 million

ii)      TIGTA determination                                         $111.8 million

iii)    Difference                                                                                           $53.7 million

2.      Baseline budget information changed.

a)      CAP baseline for major components (used this as an example) (see page 13).

i)        BY 2005 business cases                                          $8.4 million

ii)      BY 2006 business cases                                        $16.2 million

iii)    Difference                                                                                             $7.8 million

b)      IFS baseline budget estimates (see page 13).

i)        BY 2005                                                                $247 million

ii)      BY 2006                                                                $227 million

iii)    Difference                                                                                           $20.0 million

3.      Inaccurate reporting between sections of the same business case.

a)      CAP BY 2005 business case through September 30, 2006 (see page 14).

i)        Earned Value Management (EVM)                       $286 million

ii)      Summary of Spending Table                                 $314 million

iii)    Difference                                                                                           $28.0 million

b)      CAP – Internal Revenue Service labor costs (see page 14).

i)        Alternatives Analysis                                            $21.5 million

ii)      Summary of Spending                                          $52.0 million

iii)    Difference                                                                                           $30.5 million

c)      IFS BY 2005 business cases

i)        Amount not shown in EVM included in the
Summary of Spending Table                                                              $35.0 million

Total                                                                                                                    $205.8 million

Grand Total ($133.6 million plus $205.8 million)                                               $339.4 million

 

Appendix V

 

Summary of Business Case Deficiencies

Tables 1 and 2 below present a summary of the Budget Year 2005 and 2006 business case deficiencies for the development and operational projects we reviewed.

Table 1:  Business Case Deficiencies for Development Projects

 

Business Case Deficiencies

Custodial Accounting Project

Integrated Financial System

2005

2006

2005

2006

Allocation of Management and Overhead Costs Inaccuracies1

 

 

Security Costs Inaccurate

Alternatives Analysis Insufficient

 

 

Cost/Benefit Analysis Not Prepared

 

 

Earned Value Metrics Inaccurate

No Supporting Documentation for Earned Value Management (EVM) Data

EVM Data Outdated1

 

 

Cost Data Inconsistent1

 

 

Earned Value Systems Not Compliant

1 We did not evaluate these items for Budget Year 2006 due to time constraints.

Source:  Treasury Inspector General for Tax Administration’s (TIGTA) analysis of business cases and supporting documentation.

 

Table 2:  Business Case Deficiencies for Operational Projects

 

Business Case Deficiencies

Integrated Submission and Remittance Processing

Counsel Automated Systems Environment

2005

2006

2005

2006

Direct Labor, Hardware and Software Costs Omitted

 

 

 

Security Costs Inaccurate1

 

 

 

Results of E-Government Review Not Demonstrated

1 We did not evaluate this item for Budget Year 2006 due to time constraints.

Source:  TIGTA’s analysis of business cases and supporting documentation.

 

Appendix VI

 

Description of Information Technology Investments Selected for Review

Counsel Automated Systems Environment (CASE)

The CASE is a collection of office automation tools, customized computer programs, hardware and software, and staff dedicated to support the ongoing maintenance of the CASE.  The Internal Revenue Service (IRS) Office of Chief Counsel uses the CASE to provide IRS attorneys with a case tracking and management system.

Custodial Accounting Project (CAP)

The CAP is planned to provide a single, integrated data repository of taxpayer account information, integrated with the general ledger and accessible for management analysis and reporting.  The CAP will allow the IRS to generate timely, reliable financial reports and other financial information that the Congress and IRS management need to oversee and manage the agency.  The project will solve several internal management problems.

Currently, the IRS’ financial systems environment is a collection of information systems that provide limited functionality and are time-consuming to use, expensive to maintain, and not compliant with Federal financial law and regulation in several areas.  Additionally, the systems have a high cost of ownership, constrain the quality of service, and provide little information to support improvement in financial business operations or managerial decision making.

Integrated Financial System (IFS)

The IFS will replace multiple steady state financial systems with a single, integrated commercial off-the-shelf Enterprise Resource Planning software package that will comply with all Federal laws and accounting standards and rules.  The IFS will enable the IRS to integrate the majority of its financial processes, share common data and practices across the entire organization, and produce and access information in a real-time environment.  The IFS will help the IRS face several challenges in its internal management operations.

Integrated Submission and Remittance Processing (ISRP) System

The ISRP system is a mission critical, steady state system that provides both individual and business taxpayers a method to file paper tax returns, provides a method to process remittances received from taxpayers, and allows the IRS to process various tax forms at nine Submission Processing sites throughout the country.  Many individuals still choose to file paper tax returns.  Currently, approximately 72.5 million (55 percent) of all individual returns are filed by paper.  The IRS uses the ISRP system to process this massive volume of paper.

 

Appendix VII

 

Management’s Response to the Draft Report

 

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.