TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

Strengthened Management Processes Are Needed to Assure the Usefulness of Products and Services Received Through the Infrastructure Shared Services Task Order

 

 

 

March 2006

 

Reference Number:  2006-20-063

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

Phone Number   |  202-927-7037

Email Address   |  Bonnie.Heald@tigta.treas.gov

Web Site           |  http://www.tigta.gov

 

March 30, 2006

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:                            Michael R. Phillips /s/ Michael R. Phillips

                                         Deputy Inspector General for Audit

 

SUBJECT:                    Final Audit Report – Strengthened Management Processes Are Needed to Assure the Usefulness of Products and Services Received Through the Infrastructure Shared Services Task Order (Audit # 200520018)

 

This report presents the results of our review of the Fiscal Year (FY) 2004/2005 Infrastructure[1] Shared Service (ISS) task order.[2]  The overall objective of this review was to determine whether the Internal Revenue Service (IRS) is receiving full value from the PRIME contractor[3] under the terms and conditions of the FY 2004/2005 ISS task order.

Synopsis

The IRS is currently in the midst of a multiyear, multibillion dollar effort to update its core business systems, known as Business Systems Modernization (BSM).  According to IRS estimates, this effort involves integrating thousands of hardware and software components over 15 years.  The success of the BSM program depends on establishing a strong foundation from which to build business applications supporting tax return processing and other critical mission-related functions.  In February 2002, the Business Systems Modernization Office began building this foundation by establishing the ISS program to install the infrastructure.  These critical infrastructure areas are being relied upon by other modernized projects and systems.  For FYs 2004 and 2005, the IRS issued a task order to the PRIME contractor to work with the IRS to maintain the current infrastructure environment and adapt to new changes and upgrades that are inherent in information technology components.

We determined the IRS has improved its ability to perform detailed reviews of the contractor’s invoices, the PRIME contractor is making corrections to work items as requested by the IRS, and significant work items and special projects provided through the ISS task order are providing value to the IRS.  While value is being received for many of the work items and special projects in the FY 2004/2005 ISS task order, we determined the IRS is not obtaining the anticipated value for some work items and special projects, and it is difficult to determine the value being received for other work items and special projects.

We reviewed a judgmental sample of 27 work items produced by the PRIME contractor and received by the IRS between February 2004 and May 2005 and determined the IRS received the anticipated value for 12 work items and received value for an additional 3 work items as part of different projects.  However, the value received for the remaining 12 work items was more difficult to determine.  Initially, we concluded the IRS did not receive full value for these work items.  Upon meeting with IRS officials in September 2005, December 2005, and January 2006, the IRS provided additional information concerning whether these remaining work items provided value.  The IRS agreed seven work items did not provide the IRS with their anticipated value.  However, the IRS stated it believed that, while the original anticipated value may not have been obtained, the IRS did receive value from the remaining five work items.  In addition, we reviewed all 11 special projects performed under the FY 2004/2005 ISS task order and determined 2 (18 percent) did not provide the anticipated value to the IRS.[4]

The IRS does not individually evaluate the value obtained from specific work items or special projects during the completion of activities within the FY 2004/2005 ISS task order.  We determined the IRS could do more to proactively assess if value is being received from the PRIME contractor.  Currently, the IRS does not have all the information necessary to perform value measurements because 1) users/owners are not documenting the purpose and planned results for work items, 2) the IRS does not have cost estimates for individual work items, and 3) the IRS is not performing quality reviews of deliverables or making overall performance assessments of the PRIME contractor.

The IRS has spent Federal Government resources for Infrastructure Shared Services work items that are not being used.

Recommendations

To ensure users/owners are appropriately identified and actively participate in measuring the value received from work items delivered by the PRIME contractor, we recommended the Chief Information Officer (CIO) implement a consistent process to formally document users/owners and require users/owners to 1) document the purpose, 2) create success criteria, and 3) document whether each work item met the stated purpose and defined success criteria.  To ensure adequate cost estimates are available, we recommended the CIO require the PRIME contractor to provide original and revised (if applicable) cost estimates for all task order activities.  To ensure quality ratings and measurements are performed, we recommended the CIO require all Acquisition Project Managers and Project Account Managers[5] to follow the ISS task order monitoring plan.  Finally, to ensure an overall contractor performance evaluation process is available to create a rating for the entire PRIME contract, we recommended the Business Systems Modernization Office complete activities to implement an overall contractor performance evaluation process.

Response

IRS management fully agreed with three of our five recommendations and partially agreed with two recommendations.  To ensure users/owners are appropriately identified and actively participate in measuring the value received from work items delivered by the PRIME contractor, the IRS will modify templates to define expected outcomes and ensure users/owners are identified and documented.  This will allow the IRS to determine whether stated objectives were met.  To ensure adequate cost estimates are available, the IRS will ensure certain cost estimate revisions are documented as separate estimates.  To ensure quality ratings and measurements are performed and an overall contractor performance evaluation process is available to create a rating for the entire PRIME contract, the IRS will perform quarterly surveillance of expected outcomes, has implemented a contractor performance evaluation process, and is completing an evaluation for the PRIME contractor.  Management’s complete response to the draft report is included as Appendix VIII.

Office of Audit Comment

The CIO stated the partial agreement for two recommendations reflected the IRS’ view that the cost to fully implement the recommendations outweighs the benefits.  However, the IRS believes the planned corrective actions reflect a cost-effective approach to addressing the recommendations.

For our recommendation to require users/owners to 1) document the purpose, 2) create success criteria, and 3) document whether each work item met the stated purpose and defined success criteria, the IRS will modify its templates to define expected outcomes.  However, the IRS stated it did not plan to implement our recommendation at the work item level.  Based on a preliminary version of this report, IRS officials explained they planned to evaluate logical groupings of work items against expected outcomes instead of defining expected outcomes for every work item.  During our preliminary discussions, we agreed the IRS’ planned corrective action would meet the intent of our recommendation.

For our recommendation to require the PRIME contractor to provide original and revised cost estimates for all task order activities, the IRS stated it would strengthen its processes for certain cost estimate revisions.  However, the IRS does not believe a separate accounting of each task order activity is needed.  As stated in the report, the IRS process for initiating work items and special projects within the contract does not require estimates of hours or dollar amounts for specific work items.  For example, one group of products and services requested from the PRIME contractor was support for project management, business management, risk management, and quality management, as well as the creation of a Project Management Plan.  The request included an estimate of over 3,500 hours.  Since the estimate is for all products and services, it is not possible to determine whether the PRIME contractor planned to spend a relatively small or large amount of time creating the Project Management Plan.  As a result, the IRS can not determine the relative cost of each work item or service, which makes it difficult to judge the value being received for any specific item within the contract.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

 

Table of Contents

 

Background

Results of Review

The Modernized Infrastructure Is Being Maintained by the Internal Revenue Service Infrastructure Shared Services Team

The Internal Revenue Service Did Not Receive the Anticipated Value for All Work Items It Received

Recommendations 1 and 2:

Recommendation 3:

Recommendations 4 and 5:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measure

Appendix V – Major Components and Special Projects for the Fiscal Year 2004/2005 Infrastructure Shared Services Task Order

Appendix VI – Enterprise Life Cycle Overview

Appendix VII – Review of Value Received From Sample Work Items

Appendix VIII – Management’s Response to the Draft Report

 

 

Background

 

The Internal Revenue Service (IRS) is currently in the midst of a multiyear, multibillion dollar effort to update its core business systems, known as Business Systems Modernization (BSM).  According to IRS estimates, this effort involves integrating thousands of hardware and software components over 15 years.  The IRS hired the Computer Sciences Corporation as its PRIME contractor to head an alliance of leading technology companies in assisting with the BSM program.  The IRS also established the Business Systems Modernization Office (BSMO) to manage the BSM program and oversee the work of the PRIME contractor.

The success of the BSM program depends on establishing a strong foundation from which to build business applications supporting tax return processing and other critical mission-related functions.  In February 2002, the BSMO began building this foundation by establishing the Infrastructure Shared Services (ISS) program to install the following infrastructure areas:

1)      The Security and Technology Infrastructure Release provides the hardware and software necessary to deploy and run a BSM project.  It provides key features such as web servers, security, messaging, and directory services.

2)       Enterprise Systems Management provides the overlying management capabilities to the Security and Technology Infrastructure Release and network and systems management to improve infrastructure availability and performance.  It also manages hardware and software inventories and monitors the performance of various systems.

3)      The Development, Integration, and Test Environment provides a complete project development and testing environment meant to simulate actual operating conditions.  It provides labs to evaluate potential solutions and their impact on business processes (Solutions Demonstration Lab), support all phases of information systems development (Virtual Development Environment), and test systems integration and final acceptance (Enterprise Integration and Test Environment).

These critical infrastructure areas have been implemented and are being relied upon by other modernized projects and systems.  For Fiscal Years (FY) 2004 and 2005, the IRS issued a task order[6] to the PRIME contractor to work with the IRS to maintain the current infrastructure environment and adapt to new changes and upgrades that are inherent in information technology components.  Major components and special projects have been detailed within the task order and are being executed by the PRIME contractor.[7]  The FY 2004/2005 ISS task order has a period of performance from February 2004 through November 2005 with an estimated cost of $36.9 million.[8]

This audit focused on the value received from products and services provided through the FY 2004/2005 ISS task order.  The Treasury Inspector General for Tax Administration is conducting an additional audit focusing on whether vouchers submitted and paid under the FY 2004/2005 ISS task order were appropriate and in accordance with the task order’s terms and conditions.

We discussed our audit results with the BSMO in September 2005.  At that time, changes were being made at both the BSM program level and the ISS program level.  Therefore, any changes that occurred after we completed our analyses are not reflected in this report.

This review was performed at the BSMO facility in New Carrollton, Maryland, during the period April through November 2005.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

 

Results of Review

 

The Modernized Infrastructure Is Being Maintained by the Internal Revenue Service Infrastructure Shared Services Team

The ISS team, through execution of the FY 2004/2005 ISS task order, is maintaining the modernized infrastructure to support new and current modernized projects such as the Modernized e-File[9] and the e-Services projects.[10]  The BSMO and the PRIME contractor have been working together to continue improving on the products and services received from the ISS program and to ensure the modernized infrastructure is available for other modernized projects and systems.

The IRS has improved its ability to perform detailed reviews of the contractor’s invoices

The PRIME contractor submits monthly invoices with supporting vouchers[11] for the FY 2004/2005 ISS task order.  Upon review and substantiation, the IRS will pay the invoice.  Between February 2004 and March 2005, the IRS worked with the PRIME contractor to revise the voucher format.  In April 2005, the IRS began receiving from the PRIME contractor a revised voucher format that included data the IRS had not previously received (e.g., specific contractor employees with the hours and travel costs they charged).  By obtaining this additional information, the IRS has improved its ability to perform detailed reviews of the PRIME contractor’s invoices.

The PRIME contractor is making corrections to work items as requested by the IRS

Issues and comments may arise during the IRS’ review of work items produced by the PRIME contractor.  We determined the ISS Project Account Managers[12] and the Office of Procurement Contracting Officer’s Technical Representative are taking steps to ensure the PRIME contractor is making corrections to work items by addressing those comments and issues.

Significant work items and special projects provided through the ISS task order are providing value to the IRS

We reviewed a judgmental sample of 27 (14 percent) of 195 work items produced by the PRIME contractor and received by the IRS between February 2004 and May 2005 and determined the IRS received the anticipated value for 12 work items and received value for an additional 3 work items as part of different projects.  For example, the PRIME contractor delivered a system security plan and an outline for disaster recovery.[13]  We also determined the IRS received the anticipated value from 9 (82 percent) of 11 special projects within the FY 2004/2005 ISS task order.[14]  Many of these special projects were necessary to provide enhancements, upgrades, and improvements to current infrastructure systems.  While value is being received for many of the work items and special projects in the FY 2004/2005 ISS task order, we determined the IRS is not obtaining the anticipated value for other work items and special projects, and it is difficult to determine the value being received for other work items and special projects.

The Internal Revenue Service Did Not Receive the Anticipated Value for All Work Items It Received

As mentioned previously, we determined 15 of 27 sampled work items have provided or are providing value to the IRS either for their original purpose or as part of different projects.  However, the value received for the remaining 12 (44 percent) work items was more difficult to determine.  Initially, we concluded the IRS did not receive full value for these work items.  Upon meeting with IRS officials in September 2005, December 2005, and January 2006, the IRS provided additional information concerning whether these remaining work items provided value.  The IRS agreed 7 (26 percent) work items did not provide the IRS with the anticipated value of the work items.  However, the IRS stated it believed that, while the original anticipated value may not have been obtained, the IRS did receive value from the remaining five work items.  In two instances, the IRS provided additional information after the conclusion of our fieldwork that we were unable to verify.  In the other three cases, the IRS believed the value of the items had either yet to be realized or was obtained by allowing the organization to determine an alternate, less costly path.  Figure 1 summarizes the results of the 27 sampled items.

Figure 1:  Selected Sample Work Items and Results[15]

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

In determining value for the sampled items, we performed a detailed review to identify each item’s purpose, results, and actual IRS users/owners.  We also interviewed users/owners to determine if the sampled items met their stated purposes and needs.  Our conclusions that some work items did not provide the anticipated value were based on one or more of the following results found during the review:

  • The IRS could not identify users/owners for the sampled items.
  • Actual IRS users/owners stated the sampled items were of no value.
  • Sampled items were not being used by the IRS and/or did not have a documented purpose.

We attempted to calculate and report the costs (dollar amounts) for the 195 total work items, the 27 sampled work items, and each of the 7 sampled work items that did not provide the anticipated value to the IRS.  However, the IRS could not provide cost data on individual work items paid to the PRIME contractor because multiple work items and services were acquired together.  For each set of work items and services, the PRIME contractor provides a total estimate (number of hours) of what it would take to complete all work items and services for the IRS; however, the IRS does not request a breakdown by work item (see The IRS does not have cost estimates of individual work items section in this report).

The IRS has spent Federal Government resources for Infrastructure Shared Services work items that are not being used.

In addition to the sampled items, we reviewed all 11 special projects performed under the FY 2004/2005 ISS task order.[16]  These special projects varied in dollar amounts, with the largest coming from the Development, Integration, and Test Environment General Lab Support special project that cost the IRS $6.7 million.[17]  We determined 2 (18 percent) of the 11 special projects did not provide the anticipated value to the IRS.  The IRS spent $2,995,676 on these 2 special projects.[18]  According to the IRS, the PRIME contractor’s performance on the Electronic File Transfer Utility special project was not worth the cost during the last period we reviewed, and funding priorities made it wise to take what the PRIME contractor had produced to date and transition it to the IRS for completion.  Regarding the Portal Upgrade and Infrastructure Development Sandbox,[19] the IRS stated the special project was never completed and was ultimately cancelled by the IRS in December 2004.  According to the IRS, it estimates it spent $310,863 on the portions of the 2 special projects that did not provide anticipated value.[20] 

In both cases, the IRS stated that, while it did not receive the anticipated value from these special projects, it did receive value by either completing work internally or as part of another project.

The IRS could do more to proactively assess whether value is being received from the PRIME contractor.

The IRS does not individually evaluate the value obtained from specific work items or special projects during the execution of activities within the FY 2004/2005 ISS task order.  We determined the IRS could do more to proactively assess whether value is being received from the PRIME contractor.  The IRS needs to enhance awareness and provide the information necessary to incorporate value within task order monitoring activities.  Currently, the IRS does not have all the information necessary to perform value measurements because 1) users/owners are not documenting the purposes and planned results for work items, 2) the IRS does not have cost estimates of individual work items, and 3) the IRS is not performing quality reviews of deliverables or making overall performance assessments of the PRIME contractor.

Users/owners are not documenting the purposes and planned results for work items

According to the American Society for QualityTM, one way of achieving performance measures is through the implementation of a continuous quality management process called the Plan-Do-Check-Act cycle, which is a continuous cycle of planning, implementing, reviewing, and improving the processes and actions that an organization undertakes to meet its business goals.

Figure 2:  Plan-Do-Check-Act Cycle

Figure 2 was removed due to its size.  To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

Work items and special projects from the FY 2004/2005 ISS task order did not always list the specific purpose, criteria, or details as to what the end results should contain.  Furthermore, acceptance and success criteria for those work items and special projects were not always defined.  This made it difficult to evaluate the work items and does not provide the IRS information necessary to determine if value is being received or to hold the contractor accountable for the work items and services.  In addition, adequate user involvement helps ensure user requirements are met.  However, users/owners for work items and special projects in the FY 2004/2005 ISS task order were not always identified or known by the IRS.

Due to the complexity and size of the ISS program, multiple Acquisition Project Managers[21] and Project Account Managers are using different techniques to execute their portions of the task order.  As a result of the differing techniques, users/owners are not always identified, acceptance or success criteria are not always defined, and users/owners do not always participate in measuring the success of work items and services received from the PRIME contractor.

Recommendations

Recommendation 1:  To ensure users/owners are appropriately identified, the Chief Information Officer (CIO) should implement a consistent process to formally document users/owners for all task order activities (work products, deliverables, and management and support services).

Management’s Response:  The CIO agreed with this recommendation.  To ensure users/owners are appropriately identified, the IRS will modify templates to ensure users/owners are identified and documented.

Recommendation 2:  To ensure users/owners actively participate in measuring the value received from work items delivered by the PRIME contractor, the CIO should require users/owners to 1) document the purpose, 2) create success criteria, and 3) document whether each work item met the stated purpose and defined success criteria.

Management’s Response:  The CIO partially agreed with this recommendation.  To ensure users/owners actively participate in measuring the value received from work items delivered by the PRIME contractor, the IRS will modify templates to define expected outcomes.  This will allow the IRS to determine whether stated objectives were met.

Office of Audit Comment:  While the IRS plans to modify templates to define expected outcomes, it does not plan to implement our recommendation at the work item level.  Based on a preliminary version of this report, IRS officials explained they planned to evaluate logical groupings of work items against expected outcomes instead of defining expected outcomes for every work item.  During our preliminary discussions, we agreed the IRS’ planned corrective action would meet the intent of our recommendation.

The IRS does not have cost estimates of individual work items

The Office of Federal Procurement Policy document entitled A Guide to Best Practices for Performance-Based Service Contracting[22] states costs must be computed for each service or output based on available data.  These costs are used in preparing the Federal Government estimate, evaluating proposals, and determining positive and negative incentives.  In addition, the Enterprise Life Cycle[23] states the IRS should assess the contractor’s basis of estimate in response to an acquisition document issued to establish a task order or in response to tasking from an existing task order.  While complying with the full extent of these practices may not be appropriate in all instances for the ISS task order, we believe the IRS should know the relative cost of individual special projects or work items to be able to determine whether it is cost beneficial to acquire individual special projects or work items and whether value is being received.

The IRS process for initiating work items and special projects within the contract does not require estimates of hours or dollar amounts for specific work items.  For example, one group of products and services requested from the PRIME contractor was support for project management, business management, risk management, and quality management, as well as the creation of a Project Management Plan.  The request included an estimate of over 3,500 hours.  Since the estimate is for all products and services, it is not possible to determine whether the PRIME contractor planned to spend a relatively small or large amount of time creating the Project Management Plan.  As a result, the IRS can not determine the relative cost of each work item or service, which makes it difficult to judge the value being received for any specific item within the contract.  Without this information, the IRS increases the chances that Federal Government resources are not used effectively.

Recommendation

Recommendation 3:  To ensure adequate cost estimates are available, the CIO should require the PRIME contractor to provide original and revised (if applicable) cost estimates for all task order activities (work products, deliverables, and management and support services).

Management’s Response:  The CIO partially agreed with this recommendation.  To ensure adequate cost estimates are available, the IRS will ensure certain cost estimate revisions are documented as separate estimates.

Office of Audit Comment:  While the IRS plans to ensure certain cost estimate revisions are documented as separate estimates, it does not believe a separate accounting of each task order activity is needed.  As stated previously, the IRS process for initiating work items and special projects within the contract does not require estimates of hours or dollar amounts for specific work items.  As a result, the IRS can not determine the relative cost of each work item or service, which makes it difficult to judge the value being received for any specific item within the contract.

The IRS is not performing quality reviews of work items or making overall performance assessments of the PRIME contractor

The Office of Federal Procurement Policy document entitled Best Practices for Collecting and Using Current and Past Performance Information[24] states recording contractors’ current performance periodically during contract performance and discussing the results with contractors is a powerful motivator for contractors to maintain high-quality performance or improve inadequate performance.  It also states current performance assessment is a basic “best practice” for good contract administration and is one of the most important tools available for ensuring good contractor performance.  The IRS uses the Office of Procurement Surveillance Reports to report on the quality of deliverables.  The ISS task order monitoring plan defines specific quality ratings for the Surveillance Report.  These criteria include evaluating cost control, timeliness of performance, and quality of product or service.

We determined the ISS Surveillance Reports do not provide measurements on the quality of work items as required by the ISS task order monitoring plan.  In addition, the PRIME contractor’s overall performance for the FY 2004/2005 ISS task order is not measured or reported.  As a result, the IRS may not know how well the PRIME contractor is performing or if the PRIME contractor is meeting the needs and requirements of users/owners to adequately measure the value received.  Additionally, the risk of wasted resources is increased.

The Acquisition Project Managers and Project Account Managers are not required to follow the quality ratings process, and they do not have a process which details specific steps for measuring and reporting the contractor’s specific performance on the task order or to provide input for the PRIME contractor’s overall performance on all modernization task orders.  In addition, the Associate CIO, BSM, stated the IRS stopped reporting on the overall performance of the PRIME contractor due to inconsistent application of the prior contractor performance evaluation process.

Management Action:  The Associate CIO, BSM, stated the BSMO is currently testing a revised contractor performance evaluation process over a 6-month period.  The revised process will create a rating for the entire PRIME contract, instead of providing individual ratings for each task order.  According to the Office of Procurement, the IRS will complete the contractor performance evaluation after January 2006.

Recommendations

Recommendation 4:  To ensure quality ratings and measurements are performed, the CIO should require all Acquisition Project Managers and Project Account Managers to follow the ISS task order monitoring plan.

Management’s Response:  The CIO agreed with this recommendation.  To ensure quality ratings and measurements are performed, the IRS will perform quarterly surveillance of expected outcomes.

Recommendation 5:  To ensure an overall contractor performance evaluation process is available to create a rating for the entire PRIME contract, the BSMO should complete activities to implement an overall contractor performance evaluation process.

Management’s Response:  The CIO agreed with this recommendation.  To ensure an overall contractor performance evaluation process is available to create a rating for the entire PRIME contract, the IRS has implemented a contractor performance evaluation process and is completing an evaluation for the PRIME contractor.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of the review was to determine whether the Internal Revenue Service (IRS) is receiving full value from the PRIME contractor[25] under the terms and conditions of the Fiscal Year 2004/2005 Infrastructure[26] Shared Services task order.[27]  To accomplish this objective, we:

I.                   Determined whether work items were clearly identified and adequate acceptance criteria had been developed.

II.                Determined whether a judgmental sample of work items provided value to the IRS by:[28]

A.    Comparing work items to product descriptions.

B.     Interviewing users/owners.

C.     Determining whether requested corrections were made by the PRIME contractor.

III.             Determined whether the IRS evaluated the PRIME contractor’s overall performance on the task order by regularly monitoring and measuring performance for specific work items and special projects.

A.    Determined whether the IRS evaluated the PRIME contractor’s performance on specific work items.

B.     Reviewed the 11 special projects within the Fiscal Year 2004/2005 Infrastructure Shared Services task order and determined the IRS’ process for managing, measuring, and reporting performance and results.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Program)

Gary Hinkle, Director

Troy Paterson, Audit Manager

Phung Nguyen, Lead Auditor

Tina Wong, Senior Auditor

Charlene Elliston, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Associate Chief Information Officer, Enterprise Services  OS:CIO:ES

Director, Procurement  OS:A:P

Director, Stakeholder Management  OS:CIO:SM

Deputy Associate Chief Information Officer, Systems Integration  OS:CIO:B:SI

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaisons:

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Director, Procurement  OS:A:P

Manager, Program Oversight Office  OS:CIO:SM:PO

 

Appendix IV

 

Outcome Measure

 

This appendix presents detailed information on the measurable impact our recommended corrective actions will have on tax administration.  This benefit will be incorporated into our Semiannual Report to Congress.

Type and Value of Outcome Measure:

·         Inefficient Use of Resources – Actual; $310,863 (see page 4).

Methodology Used to Measure the Reported Benefit:

We interviewed the Acquisition Project Manager and Project Account Managers[29] assigned to implement and monitor special projects from the Fiscal Year 2004/2005 Infrastructure[30] Shared Services task order.[31]  According to the Internal Revenue Service (IRS), the PRIME contractor’s performance was not worth the cost near the conclusion of its involvement in the project, and funding priorities made it wise to take what the PRIME contractor had produced and transition it to the IRS for completion.  Regarding the Portal Upgrade and Infrastructure Development Sandbox,[32] the IRS stated the special project was never completed and was ultimately cancelled by the IRS in December 2004.

According to the IRS, it estimates it spent approximately $310,863 ($83,613 for the Enterprise File Transfer Utility Support Services special project and $227,250 for the Portal Upgrade and Infrastructure Development Sandbox special project) on the portions of the 2 special projects that did not provide anticipated value.  In both cases, the IRS stated that, while it did not receive the anticipated value from these special projects, it did receive value by either completing work internally or as part of another project.

 

Appendix V

 

Major Components and Special Projects for the Fiscal Year 2004/2005 Infrastructure Shared Services Task Order

 

The Fiscal Year (FY) 2004/2005 Infrastructure[33] Shared Services (ISS) task order[34] contains many different components and special projects for which the PRIME contractor[35] is tasked to perform activities and provide work items and services.  Each component and special project has a total estimated cost associated for each contracted period.  Table 1 provides a snapshot of the major components and Table 2 provides a snapshot of the special projects from the FY 2004/2005 ISS task order dated May 17, 2005.  The total estimated costs shown are provided for the contracted base period of February 21 through November 30, 2004, and the option period of December 1, 2004, through November 30, 2005.

Table 1:  Estimated Costs for FY 2004/2005 ISS Task Order Major Components

Major Components

February 2004 -

November 2004

December 2004 -

November 2005

Program Management Support

$3,708,595

$4,453,481

Consulting Services

Not Available

$0

Performance

$487,174

$430,025

Capacity

$1,023,236

$858,987

Enhancements

$1,666,037

$2,312,884

Requirements

$402,887

$676,965

Customer Support Services

$4,192,271

$5,434,676

Engineering

$95,356

$44,966

Development, Integration, and Test Environment[36]

$3,179,833

$10,795,704

TOTALS

$14,755,389

$25,007,688

Source:  Treasury Inspector General for Tax Administration analysis of the FY 2004/2005 ISS task order (modification 40, dated May 17, 2005).

Table 2:  Estimated Costs for FY 2004/2005 ISS Task Order Special Projects[37]

Special Projects

February 2004 -

November 2004

December 2004 - November 2005

Initial Cost Estimate for Modernization Disaster Recovery

$47,483

None in FY 2005

Enterprise File Transfer Utility Support Services[38]

Not Available

Not Listed

Infrastructure Enhancements Plan and Technology Refresh

$3,099,745

None in FY 2005

Infrastructure Enhancements – Employee
Registration 1.1

$1,026,227

None in FY 2005

Portal Upgrade and Infrastructure Development Sandbox[39]

$967,995

None in FY 2005

Support and Maintenance for Production Systems

$1,175,760

$1,199,989

General Lab Support

$7,145,227

None in FY 2005

Modernization Security System

None in FY 2004

$554,041

Security Tiger Team

None in FY 2004

$2,376,775

Disaster Recovery for Tier 2[40]

None in FY 2004

$1,744,066

Ongoing Development, Integration, and Test Environment Operations for Security and Systems Administration

None in FY 2004

$381,740

TOTALS

$13,462,437

$6,256,611

Source:  The FY 2004/2005 ISS task order (modification 40, dated May 17, 2005).

 

Appendix VI

 

Enterprise Life Cycle Overview

 

The Enterprise Life Cycle (ELC) defines the processes, products, techniques, roles, responsibilities, policies, procedures, and standards associated with planning, executing, and managing business change.  It includes redesign of business processes; transformation of the organization; and development, integration, deployment, and maintenance of the related information technology applications and infrastructure.  Its immediate focus is the Internal Revenue Service (IRS) Business Systems Modernization (BSM) program.  Both the IRS and the PRIME contractor[41] must follow the ELC in developing/acquiring business solutions for modernization projects.

The ELC framework is a flexible and adaptable structure within which one plans, executes, and integrates business change.  The ELC process layer was created principally from the Computer Sciences Corporation’s Catalyst® methodology.[42]  It is intended to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision making and information sharing.  Other components and extensions were added as needed to meet the specific needs of the IRS BSM program.

ELC Processes

A process is an ordered, interdependent set of activities established to accomplish a specific purpose.  Processes help to define what work needs to be performed.  The ELC methodology includes two major groups of processes:

Life-Cycle Processes, which are organized into phases and subphases and address all domains of business change.

Management Processes, which are organized into management areas and operate across the entire life cycle.

Enterprise Life-Cycle Processes

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Life-Cycle Processes

The life-cycle processes of the ELC are divided into six phases, as described below:

·         Vision and Strategy - This phase establishes the overall direction and priorities for business change for the enterprise.  It also identifies and prioritizes the business or system areas for further analysis.

·         Architecture - This phase establishes the concept/vision, requirements, and design for a particular business area or target system.  It also defines the releases for the business area or system.

·         Development - This phase includes the analysis, design, acquisition, modification, construction, and testing of the components of a business solution.  This phase also includes routine planned maintenance of applications.

·         Integration - This phase includes the integration, testing, piloting, and acceptance of a release.  In this phase, the integration team brings together individual work packages of solution components developed or acquired separately during the Development phase.  Application and technical infrastructure components are tested to determine whether they interact properly.  If appropriate, the team conducts a pilot to ensure all elements of the business solution work together.

·         Deployment - This phase includes preparation of a release for deployment and actual deployment of the release to the deployment sites.  During this phase, the deployment team puts the solution release into operation at target sites.

·         Operations and Support - This phase addresses the ongoing operations and support of the system.  It begins after the business processes and system(s) have been installed and have begun performing business functions.  It encompasses all of the operations and support processes necessary to deliver the services associated with managing all or part of a computing environment.

The Operations and Support phase includes the scheduled activities, such as planned maintenance, systems backup, and production output, as well as the nonscheduled activities, such as problem resolution and service request delivery, including emergency unplanned maintenance of applications.  It also includes the support processes required to keep the system up and running at the contractually specified level.

Management Processes

Besides the life-cycle processes, the ELC also addresses the various management areas at the process level.  The management areas include:

·         IRS Governance and Investment Decision Management - This area is responsible for managing the overall direction of the IRS, determining where to invest, and managing the investments over time.

·         Program Management and Project Management - This area is responsible for organizing, planning, directing, and controlling the activities within the program and its subordinate projects to achieve the objectives of the program and deliver the expected business results.

·         Architectural Engineering/Development Coordination - This area is responsible for managing the technical aspects of coordination across projects and disciplines, such as managing interfaces, controlling architectural changes, ensuring architectural compliance, maintaining standards, and resolving issues.

·         Management Support Processes - This area includes common management processes, such as quality management and configuration management that operate across multiple levels of management.

Milestones

The ELC establishes a set of repeatable processes and a system of milestones, checkpoints, and reviews that reduce the risks of systems development, accelerate the delivery of business solutions, and ensure alignment with the overall business strategy.  The ELC defines a series of milestones in the life-cycle processes.  Milestones provide for “go/no-go” decision points in the project and are sometimes associated with funding approval to proceed.  They occur at natural breaks in the process where there is new information regarding costs, benefits, and risks and where executive authority is necessary for next phase expenditures.

There are five milestones during the project life cycle: 

·         Milestone 1 - Business Vision and Case for Action.  In the activities leading up to Milestone 1, executive leadership identifies the direction and priorities for IRS business change.  These guide which business areas and systems development projects are funded for further analysis.  The primary decision at Milestone 1 is to select BSM projects based on both the enterprise-level Vision and Strategy and the Enterprise Architecture.

·         Milestone 2 - Business Systems Concept and Preliminary Business Case.  The activities leading up to Milestone 2 establish the project concept, including requirements and design elements, as a solution for a specific business area or business system.  A preliminary business case is also produced.  The primary decision at Milestone 2 is to approve the solution/system concept and associated plans for a modernization initiative and to authorize funding for that solution.

·         Milestone 3 - Business Systems Design and Baseline Business Case.  In the activities leading up to Milestone 3, the major components of the business solution are analyzed and designed.  A baseline business case is also produced.  The primary decision at Milestone 3 is to accept the logical system design and associated plans and to authorize funding for development, test, and (if chosen) pilot of that solution.

·         Milestone 4 - Business Systems Development and Enterprise Deployment Decision.  In the activities leading up to Milestone 4, the business solution is built.  The system is integrated with other business systems and tested, piloted (usually), and prepared for deployment.  The primary decision at Milestone 4 is to authorize the release for enterprise-wide deployment and commit the necessary resources.

·         Milestone 5 - Business Systems Deployment and Postdeployment Evaluation.  In the activities leading up to Milestone 5, the business solution is fully deployed, including delivery of training on use and maintenance.  The primary decision at Milestone 5 is to authorize the release of performance-based compensation based on actual, measured performance of the business system.

 

Appendix VII

 

Review of Value Received From Sample Work Items

 

The Treasury Inspector General for Tax Administration (TIGTA) reviewed a judgmental sample of 27 work items from a population of 195 work items produced by the PRIME contractor[43] and received by the Internal Revenue Service (IRS) between February 2004 and May 2005.  Table 1 provides an analysis of the value received by the IRS for each of the sample work items.

Table 1:  Independent Assessment of Value Received From the 27 Sample Work Items

Number

Sample Work Items

Value Analysis

1

Monthly Technical Project Status Review Package

Anticipated value was not obtained – We determined IRS officials no longer attend the meetings, IRS officials no longer use the items from the Package, and the PRIME contractor questioned the purpose of the meetings associated with the Package.  In response to preliminary results from this audit, the IRS stated it believed the initial Technical Project Status Review meetings served a specific function of coordinating technical issues within the PRIME contractor organization; however, it believed, as time has passed, the Technical Project Status Review has had limited value and can be eliminated.

2

Project Management Plan (Draft)

Anticipated value was not obtained – We determined Acquisition Project Managers and Project Account Managers[44] do not use the Project Management Plan and not all procedures outlined in the Project Management Plan are being followed.  This item was discussed with the IRS in August 2005.  We discussed our value determination with the IRS in
September 2005, at which time the IRS provided verbal comments.  In December 2005, based on a preliminary version of this report, the IRS clarified it chose to align staff and processes with program-level requirements instead of implementing a final version of the Project Management Plan.

3

Performance Problem Reporting Procedures (Final)

Deferred or other value received – We determined performance measures from the work item were never implemented.  In addition, the IRS Project Account Manager stated no value was obtained from the work item.  This item was discussed with the IRS in July and August 2005.  We discussed our value determination with the IRS in September 2005, at which time the IRS provided verbal comments.  In December 2005, based on a preliminary version of this report, the IRS stated that, due to a change in modernization roles and the Modernization and Information Technology Services (MITS) organization’s reorganization, the value of this product was deferred, not lost.

4

CISCOWorks Deployment Plan

Deferred or other value received – We determined IRS users/owners stated no value was obtained from the Plan.  This item was discussed with the IRS in July and August 2005.  We discussed our value determination with the IRS in September 2005, at which time the IRS provided verbal comments.  In December 2005, based on a preliminary version of this report, the IRS stated it determined it could implement the CISCOWorks upgrade at a lower cost internally.

5

Firewall Coordination Process Document

Anticipated value was not obtained – We determined the IRS users/owners have not used the Document.

6

Cost-Benefit Analysis Approach - Consolidating Master File[45] Datastore

Anticipated value was obtained – We determined the IRS used the Analysis to determine whether to consolidate the Master File datastore.

7

Enterprise Systems Management Load/Balancer Reports

Deferred or other value received – The IRS could not locate any project personnel who were using these Reports.  We initially requested to discuss these Reports with a user in June 2005.  In December 2005, after our audit work was completed, the IRS informed us these Reports are being used by another contractor who uses them to create additional reports for use by the IRS.

8

System Security Plan for Web Hosting

Anticipated value was obtained – We determined IRS officials used the Plan for security certification purposes.

9

Evidence of Customer Sign-off

Anticipated value was obtained – We determined IRS officials used this product during the requirements testing process.

10

Traceability Matrix for Employee
Registration 1.1 Requirements

Anticipated value was obtained – We determined IRS officials used this product during the requirements testing process.

11

Revised Update to System Administrator Guide

Anticipated value was obtained – We determined IRS system administrators are using the Guide.

12

Revision 3 to the Disaster Recovery Plan (Tier 2)[46]

Anticipated value was not obtained – The IRS Project Account Manager stated no value was obtained from Revision 3.

13

Gap Analysis With Risk Impact Assessment

Deferred or other value received – We determined the Gap Analysis was not used as originally intended by the IRS.  However, the IRS is using this work item and receiving value as part of a current infrastructure[47] project.

14

Lab Build-out Technology Model View

Anticipated value was obtained – We determined the IRS is currently using this work item.

15

Lab Build-out Physical Technology Model

Anticipated value was obtained – We determined the IRS is currently using this work item.

16

Interconnection Security Agreement

Anticipated value was obtained – We determined the Agreement was needed and was used between IRS organizations.

17

System Integration and Test Report

Anticipated value was obtained – We determined the Report was a required product of an IRS testing process.

18

Infrastructure Enhancements Plan – New Technology Migration and Tivoli®/Cognos®

Deferred or other value received – We determined the current IRS Project Account Manager was not using the Plan.  This item was discussed with the IRS in July 2005.  We discussed our value determination with the IRS in September 2005, at which time the IRS provided verbal comments.  In December 2005 and January 2006, based on a preliminary version of this report, the IRS stated this “Plan” was a high-level product documenting an approach that could be taken.  As such, it would not be used by the current implementation Project Account Manager.  After assessing the associated costs, the IRS decided to implement the solution internally.

19

Interface Control Document

Anticipated value was obtained – We determined the Document was needed and was used by the IRS.

20

Component-Level Inventory Reports

Anticipated value was not obtained – We determined IRS users/owners received the Reports but did not use the items.  The IRS eliminated the requirement for this Report in April 2005.

21

Information Technology and Asset Management Systems Inventory Reports

Anticipated value was not obtained – We determined IRS users/owners received the Reports but did not use the items.  The IRS eliminated the requirement for this Report in April 2005.

22

Draft Requirements Report

Deferred or other value received – The IRS Acquisition Project Manager and Project Account Manager could not determine the status of the Report or the users/owners of the Report.  We requested a meeting with the user of this item in June 2005.  By September 2005, the IRS could not provide us with a contact point.  After our audit work was completed, the IRS informed us it used the requirements in another work item.  In December 2005, based on a preliminary version of this report, the IRS stated that, due to a change in modernization roles and the MITS organization’s reorganization, the value of this product was deferred, not lost.

23

Draft Modernization Network Requirements Forecast

Anticipated value was not obtained – We determined IRS users/owners needed the Forecast; however, the work item was never provided to them.  This item was discussed with the IRS in July and August 2005.  We discussed our value determination with the IRS in September 2005, at which time the IRS provided verbal comments.  In December 2005, based on a preliminary version of this report, the IRS clarified anticipated value was not obtained because the Forecast was not available in time to meet the intended purpose.

24

Modernized Shared Services Logical Diagram

Deferred or other value received – We determined the Diagram was not used as originally intended by the IRS.  However, the IRS is using this work item and receiving value as part of a current infrastructure project.

25

Infrastructure Customer Requirements for Secure Web

Deferred or other value received – We determined requirements were not used as originally intended by the IRS.  However, the IRS is using this work item and receiving value as part of a current infrastructure project.

26

Development, Integration, and Test Environment Build-out Technology Model View Update

Anticipated value was obtained – We determined IRS project managers are currently using this work item.

27

Outline for Disaster Recovery Strategy for Tier 2, Version 4.0

Anticipated value was obtained – We determined IRS officials are currently using this work item.

Source:  TIGTA analysis of the selected sample work items and IRS input on a preliminary version of this report.

 

Appendix VIII

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.



[1] Infrastructure refers to the hardware, software, and security systems that computer systems use to communicate and share information.

[2] A task order is an order for services placed against an established contract.

[3] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[4] See Appendix V (Table 2) for a list of the 11 special projects.

[5] Acquisition Project Managers provide detailed acquisition/technical management of the contractor’s performance.  Project Account Managers provide the day-to-day technical oversight on all activities outlined in the task order and related documents.

[6] A task order is an order for services placed against an established contract.

[7] Major components and special projects are listed in Tables 1 and 2 of Appendix V.

[8] The $36.9 million is based on the FY 2004/2005 ISS task order (modification 31, dated February 17, 2005).

[9] This project is developing the modernized, web-based platform for electronically filing approximately 330 IRS forms.

[10] This project allows third parties to interact with the IRS via the Internet.

[11] Vouchers are detailed listings of charges made by the contractor on a month-to-month basis.

[12]  Project Account Managers provide the day-to-day technical oversight on all activities outlined in the task order and related documents.

[13] See Appendix VII for a complete list of the sampled work items.

[14] See Appendix V (Table 2) for details on the 11 special projects.

[15] See Appendix VII for a full list of the sampled items and their results.

[16] See Appendix V (Table 2) for a list of the 11 special projects.

[17] The $6.7 million amount is based upon the PRIME contractor’s voucher for the FY 2004/2005 ISS task order sent to the IRS for March 2005.  The amount shown here is actual, while the amounts in Appendix V are estimates.

[18] See Appendix IV for additional details.

[19] The Infrastructure Development Sandbox provides a development and testing environment for IRS shared infrastructure components.

[20] See Appendix IV for additional details.

[21]  Acquisition Project Managers provide detailed acquisition/technical management of the contractor’s performance.

[22] Dated October 1998.

[23] See Appendix VI for an overview of the ELC.

[24] Dated May 2000.

[25] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[26] Infrastructure refers to the hardware, software, and security systems that computer systems use to communicate and share information.

[27] A task order is an order for services placed against an established contract.

[28] We accomplished this subobjective by reviewing a judgmental sample of 27 (14 percent) of 195 work items produced by the PRIME contractor and formally accepted by the IRS between February 2004 and May 2005.  We selected the sample to include a cross-section of work items from the task order, with more work items being selected in areas where more money was being expended.  We used a judgmental sample because we did not plan to project the results.  See Appendix VII for more information related to the sample results.

[29] Acquisition Project Managers provide detailed acquisition/technical management of the contractor’s performance.  Project Account Managers provide the day-to-day technical oversight on all activities outlined in the task order and related documents.

[30] Infrastructure refers to the hardware, software, and security systems that computer systems use to communicate and share information.

[31] A task order is an order for services placed against an established contract.

[32] The Infrastructure Development Sandbox provides a development and testing environment for IRS shared infrastructure components.

[33] Infrastructure refers to the hardware, software, and security systems that computer systems use to communicate and share information.

[34] A task order is an order for services placed against an established contract.

[35] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the Internal Revenue Service’s (IRS) efforts to modernize its computer systems and related information technology.

[36] The Development, Integration, and Test Environment is a consolidated administrative and oversight function for three related Business Systems Modernization infrastructure support environments:  the Solutions Development Laboratory, Virtual Development Environment, and Enterprise Integration and Test Environment.

[37] The IRS received value from the 9 of 11 unshaded special projects listed in Table 2.  The two shaded special projects did not provide full value to the IRS.

[38] Within the FY 2004/2005 ISS task order (modification 40), no total estimated costs were provided for this special project.  See Appendix IV for a discussion of actual funds spent on this special project.

[39] The Infrastructure Development Sandbox provides a development and testing environment for IRS shared infrastructure components.

[40] Tier 2 refers to mid-level computer systems.

[41] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[42] The IRS has acquired a perpetual license to Catalyst® as part of the PRIME contract, subject to certain restrictions.  The license includes rights to all enhancements made to Catalyst® by the Computer Sciences Corporation during the contract period.

[43] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the Internal Revenue Service’s efforts to modernize its computer systems and related information technology.

[44] Acquisition Project Managers provide detailed acquisition/technical management of the contractor’s performance.  Project Account Managers provide the day-to-day technical oversight on all activities outlined in the task order and related documents.

[45] The Master File is the IRS database that stores various types of taxpayer account information.  This database includes individual, business, and employee plans and exempt organizations data.

[46] Tier 2 refers to mid-level computer systems.

[47] Infrastructure refers to the hardware, software, and security systems that computer systems use to communicate and share information.