TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

The Business Systems Modernization Program Is Receiving Value From Contract Work; However, Monitoring Can Be Strengthened

 

 

 

May 2006

 

Reference Number:  2006-20-079

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

Redaction Legend:

10 = Trade Secrets or Commercial/Financial Information

Phone Number   | 202-927-7037

Email Address   |  Bonnie.Heald@tigta.treas.gov

Web Site           | http://www.tigta.gov

 

 

May 12, 2006

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:                            Michael R. Phillips /s/ Michael R. Phillips

                                         Deputy Inspector General for Audit

 

SUBJECT:                    Final Audit Report – The Business Systems Modernization Program Is Receiving Value From Contract Work; However, Monitoring Can Be Strengthened (Audit # 200520017)

 

This report presents our review of the PRIME[1] Program Management Office (PMO) task order. The overall objective of this review was to determine whether the Internal Revenue Service (IRS) received full value from the PRIME contractor under the terms and conditions of the PRIME PMO task orders for Fiscal Years (FY) 2004 and 2005.  For purposes of this review, “full value” is defined as a good or service that met a defined purpose and acceptance/success criteria (when appropriate) and was judged to be of high quality by the IRS owners and users.

Under the PRIME contract with the Computer Sciences Corporation, the IRS negotiates a task order for each fiscal year to accomplish various program management tasks.  In FYs 2004 and 2005, the IRS issued task orders in the amounts of $23,582,944 and $4,935,230, respectively, to provide program management support for the Business Systems Modernization program.

Synopsis

The PRIME PMO task orders are intended to provide program management support for the Business Systems Modernization program.  To provide such support effectively, the task orders, as well as provisions of the Enterprise Life Cycle (ELC),[2] establish processes and procedures to accomplish that goal.  We determined the IRS generally followed task order and ELC procedures.  For example, the ELC requires task orders to include acceptance criteria for deliverables;[3] in most instances, the IRS documented acceptance criteria in the FY 2004 and FY 2005 PRIME PMO task orders.  We also determined the IRS received significant value from the majority of the products and services we reviewed.  For example, the task orders require the contractor to provide project schedules twice each month.  We determined the IRS received significant value from the project schedules because it has developed a tool to analyze schedules and document key issues, such as tasks that are starting late and key dates that have been changed.

The IRS could strengthen task order monitoring to ensure it is spending Federal Government funds wisely.

While the majority of sample items we reviewed provided significant value to the IRS, two sample items did not provide full value to the IRS.  For example, we determined the IRS is not currently receiving the Baseline Control Logs, as required by the FY 2005 PRIME PMO task order, and has negotiated for more staff than necessary to support the number of documented help desk[4] requests the IRS is submitting.  In addition, the IRS did not prepare Quarterly Status Reports, as required by the task order monitoring plan to document monitoring results, and inconsistently implemented some monitoring processes.

Recommendations

To ensure the IRS receives full value for the products and services paid for under future PRIME PMO task orders, we recommended the Associate Chief Information Officer, Enterprise Services, ensure 1) Baseline Control Logs are prepared or the PRIME PMO task order is updated, 2) help desk support hours are reduced in the FY 2006 PRIME PMO task order or help desk requests are documented, 3) the IRS has a plan to use each of the products negotiated and paid for under future task orders, 4) the Acquisition Project Manager documents the quality of products and services as required by the task order monitoring plan, and 5) the application of task order development and monitoring activities are consistent.

Response

The Chief Information Officer agreed with our recommendations.  To ensure the IRS receives full value for the products and services paid for under future PRIME PMO task orders, the IRS is updating the task order, ensuring task order requirements remain current and complete, assessing the overall quality of work products, and ensuring acceptance criteria are developed.  Management’s complete response to the draft report is included as Appendix VII.

Office of Audit Comment

The Chief Information Officer agreed with our recommendation to ensure consistent application of task order development and monitoring activities for the PRIME PMO task order.  However, the IRS response only addresses one of five examples provided in our report.  The IRS’ corrective action does not address the other examples or the underlying cause of the issues identified.  If the IRS does not address the cause of these issues, it may be unable to evaluate or provide input on the quality of products received from the contractor.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

 

Table of Contents

 

Background

Results of Review

The Business Systems Modernization Program Has Received Value From the Fiscal Year 2004 and Fiscal Year 2005 PRIME Program Management Office Task Orders

The Business Systems Modernization Program Received Partial Value for Two Services

Recommendation 1:

Recommendations 2 and 3:

Task Order Monitoring Can Be Strengthened

Recommendations 4 and 5:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measure

Appendix V – Enterprise Life Cycle Overview

Appendix VI – Fiscal Year 2004 and Fiscal Year 2005 Task Order Comparison

Appendix VII – Management’s Response to the Draft Report

 

 

Background

 

The Internal Revenue Service (IRS) is currently engaged in an effort, known as Business Systems Modernization (BSM), to modernize its systems and associated processes.  To facilitate the success of its modernization efforts, the IRS hired the Computer Sciences Corporation as the PRIME contractor.[5]

Under the PRIME contract, the IRS negotiates a task order[6] for each fiscal year to accomplish various program management tasks, including financial planning, risk management, and Enterprise Life Cycle (ELC)[7] updates.  In Fiscal Years (FY) 2004 and 2005, the IRS issued task orders in the amounts of $23,582,944 and $4,935,230, respectively, to provide program management support for the BSM program.[8]  The IRS reduced the scope and value of the FY 2005 task order to reflect the transition of primary responsibility for program management operations from the PRIME contractor to the IRS.

This audit was conducted while changes were being made at the BSM program level, including a significant change in the relationship between the IRS and the PRIME contractor. For example, the IRS stated it was examining alternatives for making future PRIME Program Management Office (PMO) task orders more economical based on the change in roles between the IRS and the PRIME contractor.  Any changes that have occurred since we concluded our analyses in December 2005 are not reflected in this report.

This review was performed at the Modernization and Information Technology Services organization facilities in New Carrollton, Maryland, during the period August 2005 through January 2006.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

 

 

Results of Review

 

The Business Systems Modernization Program Has Received Value From the Fiscal Year 2004 and Fiscal Year 2005 PRIME Program Management Office Task Orders

The PRIME PMO task orders are intended to provide management support for the BSM program.  To provide such support effectively, the task orders, as well as provisions of the ELC, establish processes and procedures to accomplish that goal.  We determined the IRS generally followed task order and ELC procedures and received significant value from the majority of the products and services we reviewed.

Task order and ELC procedures were generally being followed

We determined the IRS generally adhered to the following task order and ELC procedures:

Acceptance Criteria – The ELC requires task orders to include acceptance criteria for deliverables;[9] in most instances, the IRS documented acceptance criteria in the FY 2004 and FY 2005 PRIME PMO task orders.  This documentation enables the IRS to determine if the products being delivered meet the purpose(s) stated in the task orders.

Monitoring Methods – The ELC requires the IRS to monitor contractor work progress by documenting review findings in accordance with a task order monitoring plan.  We determined the IRS documented a “monitoring method” in several sections of the task orders providing details for how task order monitoring would be accomplished.  Documenting the monitoring methods provides the IRS with a means to determine contractor performance in providing specific products and services.

Quality Review Process – The IRS has a process for reviewing many of the products and services received from the PRIME contractor.  For example:

·         Deliverables – Both the task order and the task order monitoring plan require the IRS to comment on deliverables; the task order monitoring plan also requires the IRS to assign a quality rating to each deliverable. For the deliverables we reviewed, the IRS provided deliverable comments and received satisfactory responses from the PRIME contractor.  In addition, the IRS assigned a quality rating to all deliverables.  This deliverable review process enables the IRS to ensure each deliverable is of acceptable quality and provides input for determining whether the contractor’s performance is satisfactory.

·         Work Products[10] – The IRS acquisition team for the PRIME PMO task orders informed us it prepares technical comments upon receipt of work products and provides these comments to the contractor for resolution, if necessary.  We verified the IRS was preparing technical comments for several work products that are received on a recurring basis.  While work products are not subject to formal acceptance or rejection, this process enables the IRS to ensure work products are of a technically acceptable quality.

·         ELC Document Review – Whenever an ELC policy or procedure is proposed, changed, or removed, the IRS follows a defined process[11] that affords stakeholders the opportunity to provide feedback before making any change.  The IRS required one of the products in the FY 2005 task order, the ELC Guide,[12] to follow this process.  We reviewed all six IRS comment documents for the ELC Guide and determined the IRS followed the defined review process.

The IRS received significant value from the majority of items we reviewed

For purposes of this audit, “full value” is defined as a good or service that was received by the IRS that met each of the following three criteria:  1) the good or service met a defined purpose, 2) the good or service met defined acceptance/success criteria, and 3) the good or service was judged to be of high quality by the IRS owners and users.  We determined 7 (78 percent) of the 9 products and services we reviewed[13] provided full value to the IRS.  Figure 1 and the narrative following it present our analysis of the value received from the items we reviewed.

Figure 1: Analysis of Value Received for the Sample Items Reviewed

 

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

The IRS received significant value from the monthly status reports, Program Management Plan,[14] Contract Work Breakdown Structure,[15] and Process Asset Library[16] updates.  In addition, the IRS received significant value from the following work products:

·         Project Schedule – The FY 2004 and FY 2005 task orders require the contractor to provide project schedules twice each month.  The contractor uses the schedules to plan and control the project development process, ensure work elements are completed in order, and ensure the critical path is maintained.  The IRS uses the schedules to monitor contractor performance.  A key process in the Software Engineering Institute’s Software Acquisition Capability Maturity Model®[17] is contract tracking and oversight, which ensures activities under contract are being performed in accordance with contractual requirements.  To achieve a repeatable level under the Software Acquisition Capability Maturity Model®, agencies should review planned schedules against actual schedules and document any issues.  We determined the IRS received significant value from the project schedules because it has developed a tool to analyze schedules and document key issues, such as tasks that are starting late and key dates that have been changed.

  • Cost Performance Report – The FY 2004 and FY 2005 task orders require submission to the IRS of monthly Cost Performance Reports that include earned value[18] information as required by the Office of Management and Budget Circular A-11, Preparation, Submission and Execution of the Budget.[19]  The earned value information in the Cost Performance Reports provides significant value to the IRS because it is posted to a centralized management repository for monitoring and is discussed in various meetings established to oversee the BSM projects and the BSM program.

·         Life Cycle Review Procedures – In response to a previous Treasury Inspector General for Tax Administration report,[20] the IRS stated it had created processes to ensure life cycle reviews are conducted across all phases of the ELC.  The FY 2005 PRIME PMO task order requires several activities to ensure these processes are effectively implemented. These activities include updating training materials, scheduling training sessions, and updating processes and procedures concerning life cycle reviews.  To determine if the IRS received value for these activities, we interviewed a judgmental sample of attendees at the training sessions.[21]  We determined participants uniformly found the training to be valuable and a useful tool in their work.

While the majority of sample items we reviewed provided significant value to the IRS, two sample items did not provide full value to the IRS, and the IRS did not consistently implement some task order development and monitoring processes.

The Business Systems Modernization Program Received Partial Value for Two Services

While the size and scope of the FY 2005 PRIME PMO task order was reduced significantly to reflect the transition of primary responsibility for program management operations from the PRIME contractor to the IRS, the task order requirements still include many products and services to be provided by the PRIME contractor.  We determined the IRS received partial value for the Baseline Control service and the Risk Management and Item Tracking, Reporting, and Control System[22] maintenance and enhancement service.

Baseline Control service

The FY 2005 PRIME PMO task order requires the PRIME contractor to deliver Baseline Control Logs.[23] The task order also lists the Baseline Control Logs as a monitoring method for determining if other products required by the task order are of sufficient quality.  In addition, the IRS has proposed requiring Baseline Control Logs from all modernization contractors in the future and using the Logs as a monitoring tool.

The IRS could strengthen task order monitoring to ensure it is spending Federal Government funds wisely.

We determined the IRS is not currently receiving the Baseline Control Logs as required by the FY 2005 PRIME PMO task order.  While the IRS does receive other reports identifying some of the information needed to evaluate baseline control, the task order does not explicitly require the delivery of these other reports.  The IRS indicated the requirement for the Baseline Control Logs had been eliminated; however, the requirement for delivery of Baseline Control Logs and the use of Baseline Control Logs as a monitoring method still remains in the task order.  Without the Baseline Control Logs, the IRS is not receiving all of the information needed for baseline control as expressed in the task order and is unable to complete monitoring of other products being delivered under the task order.

Management Action:  After we completed our audit, the IRS provided feedback regarding the Baseline Control Logs.  IRS officials stated they had removed the requirement for Baseline Control Logs due to the cost and the fact they could get the information from two other products the IRS was receiving from the PRIME contractor.

Risk Management and Item Tracking, Reporting, and Control System maintenance and enhancement service

The FY 2005 PRIME PMO task order requires help desk[24] support for the Item Tracking, Reporting, and Control System.  While the IRS has indicated it considers the level of help desk support to be of high quality, the IRS could provide documentation of only one instance when help desk support was required.  The Basis of Estimate used to estimate costs of various products and services provided under the task order reflects the PRIME contractor was staffed to respond to up to ****10**** help desk support requests for the FY 2005 PMO task order.  The IRS stated it no longer needs the amount of support initially envisioned or it is not adequately documenting help desk requests.  As a result, available documentation indicates the IRS is paying for more staff than necessary to support the number of help desk requests the IRS is submitting.  We determined the cost of this unused support was approximately $20,000.[25]

The FY 2005 PRIME PMO task order also requires the delivery of a Software Configuration Report.  The IRS indicated the Report would enable the IRS to compare the Item Tracking, Reporting, and Control System software to other commercially available risk management tools and investigate possible interfaces with other software applications.  However, as a result of diminished resources, the IRS did not conduct this analysis.  We determined the IRS paid about $11,000 for a product it is not using.[26]

Management Action:  After we completed our audit, the IRS provided feedback on help desk usage and the Software Configuration Report.  IRS officials stated they were certain the PRIME contractor was receiving more Item Tracking, Reporting, and Control System help desk requests than are documented.  IRS officials stated they had already contacted the PRIME contractor to begin documenting these requests.  IRS officials also stated they agreed they did not use the Software Configuration Report as originally intended; however, the IRS needed the Report to document the components needed to recreate the Item, Tracking, Reporting, and Control System software and supporting functionality.  In addition, IRS officials stated they could use the Software Configuration Report as a baseline against which to measure future changes.

Recommendations

To ensure the IRS receives full value from the products and services paid for under future PRIME PMO task orders, the Associate Chief Information Officer, Enterprise Services, should ensure:

Recommendation 1:  Baseline Control Logs are prepared as required or the PRIME PMO task order is updated.

Management’s Response:  The Chief Information Officer agreed with this recommendation and will update the task order to reflect the current requirement.

Recommendation 2:  Help desk support hours are reduced in the FY 2006 PRIME PMO task order or help desk requests are documented.

Management’s Response:  The Chief Information Officer agreed with this recommendation and will update the task order to require the PRIME contractor to track all help desk requests.

Recommendation 3: The IRS has a plan to use each of the products negotiated and paid for under future task orders.

Management’s Response:  The Chief Information Officer agreed with this recommendation and will require quarterly reviews of task order requirements to ensure the requirements remain current and complete.

Task Order Monitoring Can Be Strengthened

The ELC states performance monitoring is the key to ensuring contractors are performing in accordance with stated outcomes and the Federal Government is receiving the desired results.  In addition, the ELC requires the IRS to monitor contractor work progress by documenting review findings according to a task order monitoring plan.  While we found several good monitoring processes being conducted, the IRS did not prepare Quarterly Status Reports, as required by the task order monitoring plan to document monitoring results, and inconsistently implemented some monitoring processes.

Quarterly Status Reports

The PRIME PMO task order monitoring plan requires the Acquisition Project Manager to document monitoring results and provide Quarterly Project Status Reports to the Contracting Officer’s Technical Representative and Program Director.  We determined the IRS did not prepare Quarterly Project Status Reports.  The Acquisition Project Manager stated the Quarterly Status Reports were not prepared because the PRIME contractor provided monthly reports and information.  In addition, the IRS evaluates and documents the quality of some products once they are delivered from the PRIME contractor.  However, as a result of not preparing the Quarterly Status Reports, there is no documentation of the IRS’ interim independent evaluation of the quality of all products and services being received from the PRIME contractor.

Some task order development and monitoring processes were inconsistently implemented

We determined some task order development and monitoring processes were inconsistently implemented.  When we discussed the following examples, IRS officials indicated the reason for the inconsistencies was that different IRS employees participate at different times in task order development and monitoring activities.

·         Acceptance Criteria The ELC requires task orders to include acceptance criteria for each deliverable.  While the IRS defined acceptance criteria for most deliverables, it did not define acceptance criteria for Process Asset Library updates in the FY 2004 and FY 2005 PRIME PMO task orders.  If the IRS does not document acceptance criteria, it may be unable to evaluate the quality of products received from the contractor.  If the IRS does not review products against acceptance criteria, it may not receive the full value desired.

·         Technical Comments on Work Products – The ELC states the Acquisition Project Manager or acquisition project team should review evolving products (e.g., ELC work products) to monitor their compliance with task order requirements and document the results of contractor performance.  Technical comments for the Cost Performance Reports are not documented.  If technical comments are not documented, some work product concerns may not be resolved by the PRIME contractor, and the Acquisition Project Manager will not have input on the quality of work products received.

·         Tracking of Work Products – We determined the IRS was tracking some work products required by the PRIME PMO task orders in the Deliverables Tracking Report.  The IRS recorded the receipt date for some work products but not others, particularly those that are scheduled to be received on a recurring basis.  If the IRS does not centrally compare receipt dates with due dates, it may be unable to easily determine the contractor’s compliance with scheduled due dates.

·         Monitoring Methods – The Federal Acquisition Regulations require the use of performance-based contracting to the maximum extent practicable.  These same Regulations also require a statement of work to enable assessment of work performance against measurable performance standards.[27]  We determined some sections of the task orders include monitoring methods, while other sections do not.  While monitoring methods may not be applicable to all sections within the task orders, it was not apparent why certain sections of the task order included monitoring methods and other sections did not.  If the IRS does not document monitoring methods, it may be unable to evaluate the quality of products received from the contractor.

·         Data Item Description[28] – The task orders include a requirement that the Contract Work Breakdown Structure and the Cost Performance Report conform to the Data Item Description.  While the data provided in both the Contract Work Breakdown Structure and the Cost Performance Reports met the information requirements of the task order, the information format did not conform to the Data Item Description.  If the IRS does not review products against stated criteria, it may not receive products in the correct format.

Recommendations

To ensure the IRS receives full value from the products and services paid for under future PRIME PMO task orders, the Associate Chief Information Officer, Enterprise Services, should ensure:

Recommendation 4:  The Acquisition Project Manager documents the quality of products and services received through the PRIME PMO task orders as required by the task order monitoring plan.

Management’s Response:  The Chief Information Officer agreed with this recommendation and will update the task order monitoring plan to include tracking the timeliness of receipt of the work products.  In addition, the IRS will assess the overall quality of work products on a quarterly basis and provide input into the annual assessment of contractor performance.

Recommendation 5: The application of task order development and monitoring activities for the PRIME PMO task order are consistent.

Management’s Response:  The Chief Information Officer agreed with this recommendation and will ensure acceptance criteria are developed.

Office of Audit Comment:  The Chief Information Officer agreed with our recommendation to ensure consistent application of task order development and monitoring activities for the PRIME PMO task order.  However, the IRS response only addresses one of five examples provided in our report.  The IRS’ corrective action does not address the other examples or the underlying cause of the issues identified. If the IRS does not address the cause of these issues, it may be unable to evaluate or provide input on the quality of products received from the contractor.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this audit was to determine whether the Internal Revenue Service (IRS) received full value[29] from the PRIME contractor[30] under the terms and conditions of the Fiscal Year (FY) 2004 and FY 2005 PRIME Program Management Office (PMO) task orders.  To accomplish our objective, we performed the following steps using a judgmental sample of products and services from the FY 2004 and FY 2005 PRIME PMO task orders (see Sample Selection Methodology):

I.                   Determined whether the IRS obtained full value from the Program Operations and Document Delivery subtask.

II.                Determined whether the IRS obtained full value from the Financial Planning and Management subtask.

III.             Determined whether the IRS obtained full value from the Enterprise Life Cycle[31] and Process Asset Library Maintenance and Enhancement subtask.

IV.             Determined whether the IRS obtained full value from the Risk Management and Item Tracking, Reporting, and Control System[32] subtask.

Sample Selection Methodology

We did not use statistical sampling during the audit because we did not intend to project our results. Each task order contains three program-specific activity areas that classify and define the primary required services to be delivered to the IRS by the PRIME contractor.  Two of these areas, Program Management and Management Processes, include detailed subtasks that more closely define each required service.  The IRS included 14 subtasks in the FY 2004 PRIME PMO task order.  The IRS reduced the number of subtasks in the FY 2005 PRIME PMO task order to seven subtasks.[33]  The 7 subtasks include 40 required services.

We judgmentally selected four of the seven subtasks (Program Operations and Document Delivery; Financial Planning and Management; Enterprise Life Cycle and Process Asset Library Maintenance and Enhancement; and Risk Management and Item Tracking, Reporting, and Control System Maintenance and Enhancement) in the FY 2005 PRIME PMO task order for review.  These subtasks represent the majority of the task order expenditures and contain more specific required services that were easier to measure regarding whether the IRS received value for the services.

Within the 4 subtasks selected for review, there are 23 required services.  We judgmentally selected 12 (52 percent) of the 23 required services for review.  If these required services were also included in the FY 2004 PRIME PMO task order, we reviewed both FY 2004 and FY 2005 activities.

Table 1 includes a list of all 7 subtasks from the FY 2005 PRIME PMO task order and all 40 required services.  The nonsampled subtasks are shown in gray.  Within the sampled subtasks, the 12 required services selected for further review are shown in bold.  If a required service appears in both the FY 2004 and FY 2005 task orders, “2004/2005” is shown in front of the required service.

Table 1: FY 2005 PRIME PMO Task Order Sample Selection

Sub-Task 1:  Executive Office

1.      2004/2005:  Cultivate a [Federal] Government-contractor partnership.

2.      2004/2005:  Participate in meetings.

3.      2004/2005:  Provide current status of all aspects of the task orders.

4.      2004/2005:  Provide overall management of the program/projects.

5.      2004/2005:  Staff the PRIME [contractor] with highly qualified, motivated personnel.

6.      2004/2005:  Perform quality assurance activities.

Sub-Task 2:  Program Operations and Document Delivery

7.      2004/2005:  Monitor/report on [PRIME] PMO task order (sample item #1).

8.      2004/2005:  Deliver/maintain the Program Management Plan (sample item #2).[34]

9.      2004/2005:  Maintain policies and procedures.

10.  2004/2005:  Support communication.

11.  2004/2005:  Support task order management.

12.  2004/2005:  Deliverables and work products should be timely and in specified format.

Sub-Task 3:  Contracts and Procurement

13.  2004/2005:  Contract administration.

14.  2004/2005:  Subcontract administration.

15.  2004/2005:  Compliance and analysis activities.

16.  2004/2005:  Small business activities.

17.  2004/2005:  Proposal development.

18.  2004/2005:  Support past performance evaluation process.

Sub-Task 4:  Facilities and Security Services

19.  2004/2005:  Update IRS PRIME Facility Security Plan.

20.  2004/2005:  Ensure compliance to security requirements.

21.  2005:  Serve as point of contact for Form 5081[35] requests.

22.  2004/2005:  Facilities services support.

23.  2004/2005:  Desktop services support.

Sub-Task 5:  Financial Planning and Management

24.  2004/2005:  Contract Work Breakdown Structure (sample item #3).[36]

25.  2004/2005:  Project Schedule (sample item #4).

26.  2004/2005:  Cost Performance Report (sample item #5). 

27.  2004/2005:  Integrated Baseline Review (sample item #6).

28.  2005:  Budget versus Actual Report.

29.  2004/2005:  Pricing.

30.  2004/2005:  Invoicing.

31.  2005:  Baseline Control (sample item #7).

32.  2004/2005:  Reporting (sample item # 8).[37]

Sub-Task 6:  Enterprise Life Cycle and Process Asset Library Maintenance and Enhancement

33.  2004/2005:  Enterprise Life Cycle overview training.

34.  2005:  Support rollout of Customer Technical Review, Life Cycle Stage Review, and Milestone Readiness Review procedures (sample item #9).

35.  2005:  Maintain currency and consistency in the Enterprise Life Cycle.

36.  2005:  Finalize Enterprise Life Cycle Guide (sample item # 10).

37.  2005:  Support assessment of rapid application development/joint application development.

38.  2005:  Perform Process Asset Library[38] maintenance and update (sample item #11).

39.  2005:  Support Enterprise Life Cycle Center of Excellence.

Sub-Task 7:  Risk Management and Item Tracking, Reporting, and Control System Maintenance and Enhancement

40.  2004/2005:  Provide Item Tracking, Reporting, and Control System maintenance and enhancements (sample item #12).

Source: FY 2004 and FY 2005 PRIME PMO task orders.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary V. Hinkle, Director

Troy D. Paterson, Audit Manager

Paul M. Mitchell, Lead Auditor

Perrin T. Gleaton, Auditor

Kim M. McManis, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn: Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Applications Development  OS:CIO:B

Associate Chief Information Officer, Enterprise Services  OS:CIO:ES

Deputy Associate Chief Information Officer, Business Integration OS:CIO:ES:BI

Director, Procurement  OS:A:P

Director, Stakeholder Management OS:CIO:SM

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaisons:

Associate Chief Information Officer, Applications Development  OS:CIO:B

Director, Procurement  OS:A:P

Director, Program Oversight Office  OS:CIO:SM:PO

 

Appendix IV

 

Outcome Measure

 

This appendix presents detailed information on the measurable impact our recommended corrective actions will have on tax administration.  This benefit will be incorporated into our Semiannual Report to Congress.

Type and Value of Outcome Measure:

·         Inefficient Use of Resources – Potential; $31,000 (see page 5).

Methodology Used to Measure the Reported Benefit:

The Fiscal Year (FY) 2005 PRIME[39] Program Management Office task order requires help desk support for the Item Tracking, Reporting, and Control System.[40]  While the Internal Revenue Service (IRS) has indicated it considers the level of help desk support to be of high quality, the IRS could provide documentation of only one instance when help desk support was required.  The Basis of Estimate used to estimate costs of various products and services provided under the task order reflects the PRIME contractor was staffed to respond to up to ****10**** help desk support requests. The IRS stated the reason for this is either it no longer needs the amount of support initially envisioned or it is not adequately documenting help desk requests.  As a result, available documentation indicates the IRS is paying for more staff than is necessary to support the number of help desk requests the IRS is submitting.

We determined the cost of this unused support by reviewing the FY 2005 PRIME Program Management Office task order. The task order provides a cost of ****10****.  The contractor’s Basis of Estimate provides for ****10**** per request.  Since only ****10****, we determined the IRS paid for staffing to handle ****10****.  The cost of ****10**** per request is approximately $20,000 ****10****.

The FY 2005 PRIME Program Management Office task order also requires the delivery of a Software Configuration Report.  The IRS indicated the Report would enable the IRS to compare the Item Tracking, Reporting, and Control System software to other commercially available risk management tools and investigate possible interfaces with other software applications.  However, as a result of diminished resources, the IRS explained this analysis had not been conducted and would probably not be initiated until sometime in mid-FY 2006.

The PRIME contractor’s Basis of Estimate for the FY 2005 PRIME Program Management Office task order did not include an estimate for the Software Configuration Report. Therefore, we asked the IRS to contact the PRIME contractor and ask how much time was spent preparing the Report. The PRIME contractor estimated a total of ****10**** to produce the Report.  As stated previously, the contractor’s cost per hour in the task order is ****10****. Therefore, we determined the cost of the Software Configuration Report to the IRS to be approximately $11,000 ****10****.

The total estimated cost for the unused help desk support and the Software Configuration Report is $31,074.12 ****10****.

Management Action:  After our audit was complete, the IRS provided feedback on help desk usage and the Software Configuration Report.  IRS officials stated they were certain the PRIME contractor was receiving more Item Tracking, Reporting, and Control System help desk requests than are documented.  IRS officials stated they had already contacted the PRIME contractor to begin documenting these requests.  IRS officials also stated they agreed they did not use the Software Configuration Report as originally intended; however, the IRS needed the Report to document the components needed to recreate the Item, Tracking, Reporting, and Control System software and supporting functionality.  In addition, IRS officials stated they could use the Software Configuration Report as a baseline against which to measure future changes.

 

Appendix V

 

Enterprise Life Cycle Overview

 

The Enterprise Life Cycle (ELC) defines the processes, products, techniques, roles, responsibilities, policies, procedures, and standards associated with planning, executing, and managing business change. It includes redesign of business processes; transformation of the organization; and development, integration, deployment, and maintenance of the related information technology applications and infrastructure.  Its immediate focus is the Internal Revenue Service (IRS) Business Systems Modernization (BSM) program.  Both the IRS and the PRIME contractor[41] must follow the ELC in developing/acquiring business solutions for modernization projects.

The ELC framework is a flexible and adaptable structure within which one plans, executes, and integrates business change. The ELC process layer was created principally from the Computer Sciences Corporation’s Catalyst® methodology.[42]  It is intended to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision making and information sharing.  Other components and extensions were added as needed to meet the specific needs of the IRS BSM program.

ELC Processes

A process is an ordered, interdependent set of activities established to accomplish a specific purpose.  Processes help to define what work needs to be performed. The ELC methodology includes two major groups of processes:

            Life-Cycle Processes, which are organized into phases and subphases and address all domains of business change.

            Management Processes, which are organized into management areas and operate across the entire life cycle.

Enterprise Life-Cycle Processes

 

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Life-Cycle Processes

The life-cycle processes of the ELC are divided into six phases, as described below:

·         Vision and Strategy - This phase establishes the overall direction and priorities for business change for the enterprise.  It also identifies and prioritizes the business or system areas for further analysis.

·         Architecture - This phase establishes the concept/vision, requirements, and design for a particular business area or target system.  It also defines the releases for the business area or system.

·         Development - This phase includes the analysis, design, acquisition, modification, construction, and testing of the components of a business solution.  This phase also includes routine planned maintenance of applications.

·         Integration - This phase includes the integration, testing, piloting, and acceptance of a release.  In this phase, the integration team brings together individual work packages of solution components developed or acquired separately during the Development phase. Application and technical infrastructure components are tested to determine whether they interact properly. If appropriate, the team conducts a pilot to ensure all elements of the business solution work together.

·         Deployment - This phase includes preparation of a release for deployment and actual deployment of the release to the deployment sites.  During this phase, the deployment team puts the solution release into operation at target sites.

·         Operations and Support - This phase addresses the ongoing operations and support of the system. It begins after the business processes and system(s) have been installed and have begun performing business functions. It encompasses all of the operations and support processes necessary to deliver the services associated with managing all or part of a computing environment.

The Operations and Support phase includes the scheduled activities, such as planned maintenance, systems backup, and production output, as well as the nonscheduled activities, such as problem resolution and service request delivery, including emergency unplanned maintenance of applications.  It also includes the support processes required to keep the system up and running at the contractually specified level.

Management Processes

Besides the life-cycle processes, the ELC also addresses the various management areas at the process level.  The management areas include:

·         IRS Governance and Investment Decision Management - This area is responsible for managing the overall direction of the IRS, determining where to invest, and managing the investments over time.

·         Program Management and Project Management - This area is responsible for organizing, planning, directing, and controlling the activities within the program and its subordinate projects to achieve the objectives of the program and deliver the expected business results.

·         Architectural Engineering/Development Coordination - This area is responsible for managing the technical aspects of coordination across projects and disciplines, such as managing interfaces, controlling architectural changes, ensuring architectural compliance, maintaining standards, and resolving issues.

·         Management Support Processes - This area includes common management processes, such as quality management and configuration management that operate across multiple levels of management.

Milestones

The ELC establishes a set of repeatable processes and a system of milestones, checkpoints, and reviews that reduce the risks of systems development, accelerate the delivery of business solutions, and ensure alignment with the overall business strategy. The ELC defines a series of milestones in the life-cycle processes.  Milestones provide for “go/no-go” decision points in the project and are sometimes associated with funding approval to proceed.  They occur at natural breaks in the process where there is new information regarding costs, benefits, and risks and where executive authority is necessary for next phase expenditures.

There are five milestones during the project life cycle: 

·         Milestone 1 - Business Vision and Case for Action.  In the activities leading up to Milestone 1, executive leadership identifies the direction and priorities for IRS business change.  These guide which business areas and systems development projects are funded for further analysis.  The primary decision at Milestone 1 is to select BSM projects based on both the enterprise-level Vision and Strategy and the Enterprise Architecture.

·         Milestone 2 - Business Systems Concept and Preliminary Business Case.  The activities leading up to Milestone 2 establish the project concept, including requirements and design elements, as a solution for a specific business area or business system.  A preliminary business case is also produced.  The primary decision at Milestone 2 is to approve the solution/system concept and associated plans for a modernization initiative and to authorize funding for that solution.

·         Milestone 3 - Business Systems Design and Baseline Business Case.  In the activities leading up to Milestone 3, the major components of the business solution are analyzed and designed.  A baseline business case is also produced.  The primary decision at Milestone 3 is to accept the logical system design and associated plans and to authorize funding for development, test, and (if chosen) pilot of that solution.

·         Milestone 4 - Business Systems Development and Enterprise Deployment Decision. In the activities leading up to Milestone 4, the business solution is built.  The system is integrated with other business systems and tested, piloted (usually), and prepared for deployment.  The primary decision at Milestone 4 is to authorize the release for enterprise-wide deployment and commit the necessary resources.

·         Milestone 5 - Business Systems Deployment and Post Deployment Evaluation.  In the activities leading up to Milestone 5, the business solution is fully deployed, including delivery of training on use and maintenance.  The primary decision at Milestone 5 is to authorize the release of performance-based compensation based on actual, measured performance of the business system.

 

Appendix VI

 

Fiscal Year 2004 and Fiscal Year 2005 Task Order Comparison

 

Figures 1 and 2 provide a list of subtasks within the Fiscal Year 2004 and Fiscal Year 2005 PRIME[43] Program Management Office task orders, respectively.  The subtasks shown in bold are consistent between the two task orders.

Figure 1:  Fiscal Year 2004 Task Order

Program Management

Management Processes

Executive Office

Financial Planning and Management

Program Office Integration and Document Management

Enterprise Life Cycle Maintenance and Enhancement

Business Systems Modernization Planning

Process Management

Contracts and Procurement

Estimation and Measurement

Facilities Management/Business Services

Training

Communication

Risk Management

Human Resources

Quality Assurance

Source:  Fiscal Year 2004 PRIME Program Management Office task order.

Figure 2:  Fiscal Year 2005 Task Order

Program Management

Management Processes

Executive Office

Enterprise Life Cycle and Process Asset Library Maintenance and Enhancement

Program Operations and Document Delivery

Risk Management and Item Tracking, Reporting, and Control System[44]  Maintenance and Enhancement

Contracts and Procurement

 

Facilities and Security Services

 

Financial Planning and Management

 

Source:  Fiscal Year 2005 PRIME Program Management Office task order.

 

Appendix VII

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.



[1] The Internal Revenue Service hired the Computer Sciences Corporation as its PRIME contractor to head an alliance of leading technology companies in assisting with the Business Systems Modernization program.

[2] See Appendix V for an overview of the ELC.

[3] Deliverables are items identified in the task order for delivery by the contractor for acceptance or rejection.

[4] The help desk is a service that provides information and assistance to the users of the IRS Item Tracking, Reporting, and Control System.  The Business Systems Modernization Risk Management Plan requires all modernization organizations to use the Item Tracking, Reporting, and Control System database to record and update the status of risks and issues.

[5] The PRIME contractor heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[6] A task order is an order for goods and services placed against an established contract.

[7] See Appendix V for an overview of the ELC.

[8] The $23,582,944 is based on the FY 2004 PRIME PMO task order (modification 29, dated May 5, 2005).  The FY 2004 task order covered 19 months (November 2003 through May 2005).  The $4,935,230 is based on the FY 2005 PRIME PMO task order (modification 3, dated July 6, 2005).  The FY 2005 task order covered 7 months (May 2005 through November 2005).

[9] Deliverables are items identified in the task order for delivery by the contractor for acceptance or rejection.

[10] Work products are items developed during performance of a contract and delivered by the contractor to the IRS.  However, the IRS does not formally accept or reject work products.

[11] The Process Asset Integration Group process is a process whereby affected BSM program employees are allowed to review and comment on draft processes and procedures before they are finalized.

[12] The purpose of the ELC Guide is to provide a central source of information on the ELC.  The ELC Guide defines the ELC and relates the ELC methodology to the IRS organization by discussing roles and responsibilities.

[13] We initially selected 12 products and services for review.  However, there were three products we could not measure:  1) Integrated Baseline Reviews have not been conducted; 2) the IRS determined two sample items (Cost Performance Reports and Earned Value Cost Performance Reports), while described separately within the task order, were the same item; and 3) the updated ELC Guide had not been finalized at the conclusion of our audit work.  See Appendix I for further details about our sample selection methodology.

[14] The Program Management Plan describes the current PRIME PMO organization and reflects the most current program management practices being employed at the time the Plan is submitted.

[15] The Contract Work Breakdown Structure is used as the foundation for all project work scope planning, scheduling, project cost estimates, life cycle cost estimates, resource allocations, performance measurement (including Earned Value metrics), configuration management, and status reporting.

[16] The Process Asset Library is a web-based repository that contains ELC process documentation.

[17] The Software Engineering Institute is a Federally Funded Research and Development Center whose purpose is to help others make measured improvements in their software engineering capabilities.  The Capability Maturity Model® is a widely recognized industry standard for effectively and efficiently developing computer projects and systems.

[18] Earned value management involves measuring actual cost and work accomplished against the budgeted cost and planned work scheduled. Variances are analyzed for decision making.

[19] Dated November 2005.

[20] The Business Systems Modernization Program Has Achieved Mixed Success in Addressing Weaknesses Identified in Internal and External Studies (Reference Number 2006-20-003, dated November 2005).

[21] We interviewed 8 (5 percent) of 152 employees who attended the life cycle review procedure training.

[22] The BSM Risk Management Plan requires all modernization organizations to use the Item Tracking, Reporting, and Control System database to record and update the status of risks and issues.

[23] Baseline Control Logs document various baseline modifications to current projects.

[24] The help desk is a service that provides information and assistance to the users of the IRS Item Tracking, Reporting, and Control System.

[25] See Appendix IV for further details concerning this calculation.

[26] See Appendix IV for further details concerning this calculation.

[27] 48 C.F.R. §§ 7.105 and 7.304 (2001).

[28] Data Item Descriptions contain the format and content preparation instructions for required documents.

[29] For purposes of this audit, “full value” is defined as a good or service that was received by the IRS that met each of the following three criteria:  1) the good or service met a defined purpose, 2) the good or service met defined acceptance/success criteria, and 3) the good or service was judged to be of high quality by the IRS owners and users.

[30] The PRIME contractor heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[31] See Appendix V for an overview of the Enterprise Life Cycle.

[32] The Business Systems Modernization Risk Management Plan requires all modernization organizations to use the Item Tracking, Reporting, and Control System database to record and update the status of risks and issues.

[33] See Appendix VI for a comparison of the FY 2004 and FY 2005 PRIME PMO task orders.

[34] The Program Management Plan describes the current PRIME PMO organization and reflects the most current program management practices being employed at the time the Plan is submitted.

[35] The IRS uses an Information System User Registration/Change Request (Form 5081) to request and authorize user accounts for employees on all systems.

[36] The Contract Work Breakdown Structure is used as the foundation for all project work scope planning, scheduling, project cost estimates, life cycle cost estimates, resource allocations, performance measurement (including Earned Value metrics), configuration management, and status reporting.

[37] Earned Value Cost Performance Reports are included under the Reporting section of the task order.

[38] The Process Asset Library is a web-based repository that contains Enterprise Life Cycle process documentation.

[39] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[40] The Business Systems Modernization Risk Management Plan requires all modernization organizations to use the Item Tracking, Reporting, and Control System database to record and update the status of risks and issues.

[41] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the IRS’ efforts to modernize its computer systems and related information technology.

[42] The IRS has acquired a perpetual license to Catalyst® as part of the PRIME contract, subject to certain restrictions.  The license includes rights to all enhancements made to Catalyst® by the Computer Sciences Corporation during the contract period.

[43] The PRIME contractor is the Computer Sciences Corporation, which heads an alliance of leading technology companies brought together to assist with the Internal Revenue Service’s efforts to modernize its computer systems and related information technology.

[44] The Business Systems Modernization Risk Management Plan requires all modernization organizations to use the Item Tracking, Reporting, and Control System database to record and update the status of risks and issues.