While Many Improvements Have Been Made, Continued Focus Is Needed to Improve Contract Negotiations and Fully Realize the Potential of Performance-Based Contracting

 

May 2005

 

Reference Number:  2005-20-083

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

May 26, 2005

 

 

MEMORANDUM FOR CHIEF, AGENCY-WIDE SHARED SERVICES

                                         CHIEF INFORMATION OFFICER

 

FROM:     Pamela J. Gardiner /s/ Pamela J. Gardiner

                 Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - While Many Improvements Have Been Made, Continued Focus Is Needed to Improve Contract Negotiations and Fully Realize the Potential of Performance-Based Contracting (Audit # 200420002)

 

This report represents the results of our review of the Internal Revenue Service’s (IRS) contract negotiation and management practices for the Business Systems Modernization (BSM) program.  The overall objective of this review was to determine whether the IRS had established and was following adequate contract negotiation and contract management practices for the BSM program.  As part of this objective, we also determined the status of the IRS’ corrective actions to a prior audit.

In summary, the BSM Office (BSMO) and the IRS Office of Procurement have long recognized the need to improve management of task orders for the BSM program and have been emphasizing the increased use of performance-based contracting (PBC).  Use of PBC means structuring all aspects of an acquisition around the purpose of the work to be performed, with the contract requirements set forth in clear, specific, and objective terms with measurable outcomes.

Based on management’s actions to our prior recommendations and the BSM Challenges Plan, the BSMO and Office of Procurement have made improvements to contract management practices.  For example, the BSM program has created a PBC lessons-learned report and increased use of combined full and partial firm fixed-price (FFP) task orders.  While the BSMO and Office of Procurement continue to make improvements to fully implement PBC for the BSM program, further improvements can be made to increase the use of FFP task orders, provide for more successful contract negotiations, and ensure improved PBC processes are followed.

We determined the use of FFP task orders within the BSM program has increased since Fiscal Year 2001; however, further increases in the use of FFP task orders and sharing of risk between the IRS and contractors are possible by using hybrid FFP task orders when appropriate, gaining insight into the barriers blocking the increased use of FFP task orders, removing barriers, and using other contracting provisions to share risk with the contractors when use of an FFP task order is not possible.  We also determined the BSMO and Office of Procurement were not always following best practices during contract negotiations.  Lastly, we determined PBC improvements have produced mixed results as the BSM program has struggled to institute revised PBC processes.

To further balance risk between the IRS and the modernization contractors, we recommended the Chief Information Officer (CIO) continue to increase the use of FFP task orders and other risk-balancing provisions.  As part of ongoing improvement efforts, we recommended the CIO and the Chief, Agency-Wide Shared Services, implement contract negotiation procedures to promote consistent application of best practices and ensure the BSM program more fully realizes the potential of PBC by developing a reasonable number of measurable performance standards; ensuring performance standards and monitoring plans are linked to awards and incentives, when appropriate; documenting the results of the contractor’s performance; and allowing adequate time for negotiation activities.

Management’s Response:  The CIO agreed with our recommendations and has completed corrective actions on two of our three recommendations.  To further balance risk between the IRS and the modernization contractors, the CIO stated the IRS has issued FFP guidance.  In addition, the IRS will continue to consider FFP contracting, address stabilizing requirements, and explore options to balance risk when FFP is not appropriate.  To improve preaward planning and processes, the IRS will assess alpha contracting to determine feasibility for broader application.  The CIO also stated the IRS has implemented a negotiation process to include all stakeholders in negotiations, drafted an independent Government cost estimate template and procedure, issued a procedure to ensure memorandums of understanding are negotiated, and will continue to determine negotiation priorities and strategies.

To improve compliance with revised PBC guidance, the CIO stated the IRS has issued statements of work templates containing measurable performance standards and distributed historical data for all preaward activities.  Also, the Office of Procurement will continue to perform quality reviews of statements of work and monitoring plans, and an independent review organization will conduct formal reviews of the statements of work, monitoring plans, and monitoring artifacts twice yearly.  Lastly, PBC training will be scheduled and lessons learned will be posted.

In addition to responding to our recommendations, the CIO stated the IRS had implemented additional improvement initiatives, such as issuing a procurement negotiation procedure and conducting a PBC compliance review.  Further, the CIO stated the IRS had enhanced its contracting practices by using FFP and incentive arrangements on four key projects.  Management’s complete response to the draft report is included in Appendix IX.

Office of Audit Comment:  While the CIO agreed with all of our recommendations, we are concerned with the stated corrective actions and associated implementation dates.  In several instances, we do not believe the stated corrective actions have corrected the issues identified.  In other instances, the corrective actions state there are ongoing activities that are not yet completed; however, the implementation dates show the corrective actions are closed.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

Table of Contents

Background

Improvements Have Been Made to Contract Management Practices

Barriers Exist to the Increased Use of Firm Fixed-Price Task Orders

Recommendation 1:

Contract Negotiations Are Not Consistently Following Best Practices

Recommendation 2:

Performance-Based Contracting Improvements Have Produced Mixed Results

Recommendation 3:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measures

Appendix V – Memorandum #1:  Business Systems Modernization Contracting and Cost/Schedule Estimation Recommendations

Appendix VI – Prior Treasury Inspector General for Tax Administration Recommendations

Appendix VII – Enterprise Life Cycle Overview

Appendix VIII – Application of Contract Negotiation Best Practices

Appendix IX – Management’s Response to the Draft Report

 

Background

The Internal Revenue Service (IRS) is currently in the midst of a multibillion dollar, multiyear Business Systems Modernization (BSM) effort.  The IRS created the BSM Office (BSMO) to manage the effort and selected the Computer Sciences Corporation to be the PRIME contractor.  Additional contractors have been hired to supplement the design and development of modernization projects.

The BSMO and the IRS Office of Procurement have long recognized the need to improve management of task orders for the BSM program and have been emphasizing the increased use of performance-based contracting (PBC) as one road toward this improvement.  Use of PBC means structuring all aspects of an acquisition around the purpose of the work to be performed, with the contract requirements set forth in clear, specific, and objective terms with measurable outcomes.

In March 2000, the IRS tasked Jefferson Solutions to teach IRS and PRIME contractor personnel how to properly implement PBC concepts.  Since that time, the Treasury Inspector General for Tax Administration (TIGTA) has provided recommendations to improve contract management for the BSM program.  See Appendix V for a memorandum we provided to the Chief Information Officer (CIO) summarizing our prior recommendations concerning contract management.

We issued a report in September 2002 concerning the use of PBC techniques to manage task orders for the BSM program and whether contract terms and requirements were being met.  Throughout the remainder of this report, when we refer to our prior audit, we are referring to that report.  We determined the IRS had made significant improvements in getting agreements on task order requirements and reducing modifications.  However, we also determined continuing improvements in the application of PBC techniques in the following areas would improve the BSMO’s ability to manage contractor performance:

·         Performance standards in project work statements were sometimes too numerous to be monitored, were not always measurable, and often focused on the production of documentation rather than the development of the business systems.

·         Plans for monitoring contractor performance were not consistent among projects, subjective monitoring methods were often used, and determinations of whether the PRIME contractor met the performance standard were often not made and documented.

·         Incentives were not consistently tied to specific levels of performance or balanced among the key areas of cost, schedule, and timeliness.

·         The BSMO had reduced the use of firm fixed-price (FFP) task orders, the type most recommended for PBC.

In mid-2003, the IRS and PRIME contractor initiated four studies to help identify the root causes of the problems hindering the BSM effort and make recommendations for remedying the problems identified.  The BSM Challenges Plan was created to address the studies’ recommendations and resolve longstanding BSM issues.  The BSM Challenges Plan includes several actions to focus on PBC, including the establishment of a BSM Acquisitions Officer and the implementation of an FFP policy.

In June 2004, the CIO met with us to express his concerns regarding contracting for the BSM program.  The CIO stated he was mainly concerned with FFP policy implementation and whether contract negotiation sessions were being conducted following best practices.

This review was performed at the BSMO facilities in New Carrollton, Maryland, during the period July 2004 through January 2005.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

This audit was conducted while changes were being made at both the BSM program level and the project level.  We communicated the interim results of our review, as well as suggestions for improvement, to the BSMO in October 2004 and the CIO in November 2004 and completed the majority of our fieldwork in December 2004.  Any changes that have occurred since we concluded our analyses are not reflected in this report.  As a result, this report may not reflect the most current status.

Improvements Have Been Made to Contract Management Practices

Based on management’s actions to recommendations from our prior audit and the BSM Challenges Plan, the BSMO and Office of Procurement have made the following improvements to contract management practices:

·         Issued guidance on developing measurable performance standards, performance incentives, and monitoring plans.

·         Developed monitoring plans for cost, schedule, and quality.

·         Created templates for PBC matrices and performance-based work statements.

·         Issued guidance on implementing an FFP policy.

·         Created a PBC lessons-learned report.

·         Developed a preaward timeline to capture the time it takes to develop, negotiate, and award PBC task orders.

·         Trained IRS and PRIME contractor personnel in the implementation of the FFP policy.

·         Established a BSM Acquisition Executive, now known as the Procurement Modernization Executive.

·         Established plans to hire 13 additional staff years to support BSM acquisitions.

·         Increased use of combined full and partial FFP task orders from 31 percent in Fiscal Year (FY) 2003 to 45 percent in FY 2004 (see Figure 1).

Figure 1:  FFP Task Orders by Fiscal Year

 

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

While the BSMO and Office of Procurement continue to make improvements to fully implement PBC for the BSM program, further improvements can be made to increase the use of FFP task orders, provide for more successful contract negotiations, and ensure improved PBC processes are followed.

Barriers Exist to the Increased Use of Firm Fixed-Price Task Orders

 In our prior audit, we recommended the BSMO require the use of FFP task orders whenever possible and appropriate for projects in development and deployment and for any other task orders where requirements are clearly identified.  Office of Management and Budget (OMB) guidance also indicates PBC encourages and enables an increased use of FFP contracts and incentives to encourage optimal performance.  In addition, the IRS released FFP guidance in September 2003 and April 2004.  According to the CIO’s FFP policy memorandum, the fixed-price policy is necessary to ensure delivery and performance in the BSM program are increased and the financial risks to the IRS are reduced.

While we determined the use of FFP task orders within the BSM program has increased since FY 2001, further increases in their use and sharing of risk between the IRS and the contractors are possible by using hybrid FFP task orders when appropriate, gaining insight into the barriers blocking the increased use of FFP task orders, removing barriers, and using other contracting provisions to share risk with the contractors when use of an FFP task order is not possible.

Hybrid FFP task orders can be issued when requirements are clear

The Office of Federal Procurement Policy A Guide to Best Practices for Performance-Based Service Contracting states, “When acquiring services that previously have been acquired by contract, agencies should rely on experience, knowledge, and historical data gained from the prior contract to incorporate performance-based service contracting methods.  Where appropriate, conversion from cost-reimbursement to fixed-price arrangements should be accomplished.”  In addition, the Office of Federal Procurement Policy states PBC should be used to the maximum extent possible, even when the entire contract is not conducive to PBC. 

In our prior audit, we recommended using FFP for any task orders in which requirements are clearly defined.  According to the IRS FFP guidance, “Fixed-price contracting is not necessarily an ‘all-or-nothing’ proposition.  If specific task requirements suitable for fixed-pricing can be clearly segmented from those that are less suitable, then various hybrid arrangements can be negotiated.”  While program-level task orders were not a part of the IRS’ FFP policy, we determined program-level task orders should be considered for hybrid FFP task orders, when appropriate.

During our review of a sample of program-level task orders, we determined the IRS had contracted for several years for similar services (program management, systems integration, and infrastructure shared services).  All of these task orders were cost-reimbursable.  When cost-reimbursable rather than FFP task orders are used, Office of Procurement and BSMO personnel must focus significant levels of effort tracking cost information.  Acquisition personnel explained not all services, while under the same category, are similar from year to year.  Another acquisition team stated segments of their task order could not be written as FFP since the BSMO needed the flexibility to move funds on and off the task order due to budget problems.

Management Action:  In October 2004, we presented our interim results to BSMO officials and suggested considering hybrid, or partial, FFP task orders in the future for recurring services.  The Deputy Associate CIO, Systems Integration, replied his team was working on this for a future infrastructure task order.  In January 2005, we were informed the IRS would be taking on some of the program-level tasks that have to date been performed by the PRIME contractor.  This could have an impact on future plans to create hybrid FFP task orders for program-level activities.

Barriers are preventing increased use of FFP task orders

The Federal Acquisition Regulation (FAR) section on service contracting indicates an FFP performance-based contract is the preferred contract type.  In addition, the CIO’s FFP policy recommends BSM task orders for system development projects in Milestones 4 and 5 be fixed-price.  We determined barriers were preventing the BSMO and Office of Procurement from negotiating more FFP task orders for the BSM program.

The FAR requires a justification to be documented when a contract type, other than FFP, is chosen.  Because none of the seven task orders in our sample were completely FFP, we reviewed all seven justifications.  We determined the information included in the justifications did not always provide insight as to why FFP was not chosen.  Since adequate details could not always be found in the justifications, we reviewed the task orders and interviewed acquisition officials.  We determined FFP was not always being used because of the following barriers:

·         System requirements and business processes were not stable.

·         Negotiations were schedule driven.

·         Negotiations and contract performance relied on multiple contract vehicles or were dependent on multiple projects.

·         Funding was not stable.

As mentioned previously, when cost-reimbursable rather than FFP task orders are used, Office of Procurement and BSMO personnel must focus significant levels of effort tracking cost information. 

Other task order provisions can be used to balance risk with the contractor

The FAR states the contractor assumes the most risk when operating under an FFP contract with great uncertainty and assumes the least risk when being reimbursed those costs determined to be allocable and allowable.  When use of an FFP task order was not possible due to some of the barriers mentioned previously, we found the IRS had begun using provisions that could better share risk between it and the contractor.  However, we determined these provisions were not used consistently.

We reviewed a sample of seven task orders and found several task orders included positive incentives for improved contractor performance.  We identified little use of any negative incentives for poor performance, other than not receiving positive incentives for good performance.  In addition to rewards for good performance, the Office of Federal Procurement Policy recommends the use of quality assurance deduction schedules.

We also identified some contracting provisions that attempt to balance risk between the IRS and the contractor, even when an FFP task order is not being used.  For example, the IRS and the PRIME contractor negotiated a modification to the contract for the Integrated Financial System (IFS) in July 2004.  Due to poor performance by the contractor, the task order was modified to allow for cost sharing (additional costs are only partially reimbursed by the IRS).  The contract was also modified to a cost-plus-fixed-fee completion type for the segment of work requiring the delivery of the system.  According to the FAR, the completion type describes the scope of work by stating a definite goal or target and specifying an end product.  This means the contractor must make every effort to deliver the end product within the estimated cost.

When the contractor does not bear enough risk under a task order, the contractor does not have an incentive to perform well.  One of the reasons for the inconsistent use of risk-balancing provisions is the IRS has stated in the past it did not believe the use of negative incentives would necessarily ensure more timely delivery of projects and further believed the use of negative incentives would lead to an adversarial relationship between the IRS and PRIME contractor.

Management Action:  The Associate CIO, BSM, has established 7 high-level key focus areas and developed 16 detailed high-priority issues to improve the BSM program.  One of the key focus areas is contract management.  The Associate CIO, BSM, agreed risk needed to be better balanced with the PRIME contractor and believed the use of FFP was not always the answer.

Recommendation

To further balance risk between the IRS and the modernization contractors, the CIO should:

1.      Continue to increase the use of FFP task orders and other risk-sharing provisions for both project- and program-level activities by:

a.       Using hybrid FFP task orders, when appropriate.

b.      Providing additional details to support the selection of non-FFP contract types.

c.       Removing barriers to FFP task orders (e.g., defining and stabilizing system requirements and business processes earlier in the life cycle).

d.      Determining if there are other provisions, such as use of completion type contracts and quality assurance deduction schedules, which can be used to balance the risk appropriately between all parties, when use of FFP task orders is not possible.

Management’s Response:  To further balance risk between the IRS and the modernization contractors, the CIO stated the IRS has issued FFP guidance.  In addition, the IRS will continue to consider FFP contracting, address stabilizing requirements, and explore options to balance risk when FFP is not appropriate.

Office of Audit Comment:  While the CIO agreed with this recommendation, we are concerned three of the stated corrective actions have not corrected the issues and one of the corrective actions indicates the action is complete although the corrective action has not yet been implemented.  Our specific comments on the four corrective actions labeled 1A through 1D are itemized below.

·         1A – The IRS states it completed the corrective action to use hybrid FFP task orders when appropriate by issuing FFP guidance on April 30, 2004.  As stated in this report, program-level task orders are not covered by the FFP guidance and program-level task orders should be considered for hybrid FFP task orders, when appropriate.  Therefore, we do not believe the stated corrective action has corrected the identified issue.

·         1B – The IRS states it completed the corrective action for providing additional details to support the selection of non-FFP contract types on April 30, 2004, by issuing FFP guidance and continuing to document contract type determinations in accordance with the FAR.  Our concern was not with a lack of documenting contract type determinations, but with a lack of detailed documentation supporting the determinations.  Therefore, we do not believe the stated corrective action has corrected the identified issue. 

·         1C – The IRS states it addressed stabilizing requirements by issuing FFP guidance on April 30, 2004.  Our concern is the Associate CIO, BSM, has identified a highest priority initiative of “Requirements and Demand Management” for the period March through September 2005 to control the scope of modernization requirements.  Since corrective actions are ongoing, we do not believe this issue has been corrected as indicated by the stated implementation date.

·         1D – The IRS states it will address balancing risk between all parties when FFP is not possible by continuing work to explore options to balance risk and continuing to perform PBC compliance reviews.  In addition, the IRS commented the IFS task order included a cost-sharing approach with the PRIME contractor beginning on May 1, 2004.  As stated in this report, we also identified the modification of the IFS task order as an example where the IRS balanced risk with the PRIME contractor.  However, we also determined provisions that could better share risk between the IRS and the contractor were not used consistently.  For example, we identified little use of any negative incentives for poor performance, other than not receiving positive incentives for good performance.  In addition to rewards for good performance, the Office of Federal Procurement Policy recommends the use of quality assurance deduction schedules.  Since the corrective action does not provide a specific action being taken, other than continuing actions and the IFS cost sharing approach, we are concerned the corrective action may not correct the identified issue.

Contract Negotiations Are Not Consistently Following Best Practices

We determined the BSMO and Office of Procurement were not always following best practices during contract negotiations.  Using a sample of seven task orders, we reviewed available contract negotiation documentation and interviewed acquisition personnel associated with each task order.  We compared the results of our interviews and documentation reviews with 10 contract negotiations best practices.  Figure 2 provides a summary of our results (see Appendix VIII for additional details).

Figure 2:  Application of Contract Negotiation Best Practices

Number

Best Practice

Followed

Partially Followed

1

Alpha contracting methods should be considered.

 

X

2

All stakeholders should be present for negotiations.

 

X

3

Memoranda of understanding should be prepared.

 

X

4

Independent Federal Government cost estimates should be obtained.

 

X

5

Cost and price analyses should be conducted.

X

 

6

Technical and cost evaluations should be conducted.

X

 

7

Cost and schedule estimates should be reviewed.

X

 

8

Issues matrices should be prepared.

 

X

9

Negotiation priorities should be documented.

 

X

10

Negotiation strategies should be documented.

 

X

Source:  TIGTA comparison for seven sampled task orders against best practices (IRS’ PBC Lessons Learned and Best Practices Summary and Analysis, Defense Procurement and Acquisition Policy Best Practices, World-Class Contracting 100+ Best Practices for Building Successful Business Relationships, and Office of Federal Procurement Policy A Guide to Best Practices for Performance-Based Service Contracting).

The following sections present details concerning best practices that were not consistently adhered to.

Alpha contracting methods should be considered

Alpha contracting is a proven approach to reducing administrative lead time, reducing costs, and improving both the negotiated agreement and the probability of success of the resulting contract.  Five of the seven sampled task orders did not use alpha contracting methods.  Two task orders used a modified alpha contracting approach.  One team that used the approach more fully stated the approach had led to a better task order for both the IRS and the contractor.  This team also stated it was able to use alpha contracting methods because the Contracting Officer had been trained on using alpha contracting methods at the Department of Defense.  An Office of Procurement official stated other teams may not be trained in alpha contracting methods and therefore are unable to use them.  Without considering alpha contracting methods, the IRS may take longer to negotiate task orders and negotiate less than optimal task orders.

Management Action:  In January 2005, the Procurement Modernization Executive informed us alpha contracting methods had mixed results over the past year.  We were also provided with a list of FY 2005 commitments.  One of the commitments was to “assess ‘modified alpha’ lessons learned to determine feasibility of broader application throughout BSM project acquisitions.”

All stakeholders should be present for negotiations

The IRS’ PBC Lessons Learned and Best Practices Summary and Analysis report states integrated project teams should be established to ensure multidisciplinary participation and involvement of stakeholders.  One of the seven sampled task orders did not have all stakeholders present during negotiation sessions.  The Associate CIO, BSM, stated this was happening because an integrated approach to negotiations was not always being followed.  If key stakeholders are not present, negotiation objectives may not be achieved or negotiations may take longer due to all disciplines not being in attendance.

Management Action:  The Associate CIO, BSM, has established 7 high-level key focus areas and developed 16 detailed high-priority issues to improve the BSM program.  One of the high-priority issues is to “strengthen [the] negotiation process through establishment of negotiating teams and use of performance-based contracting techniques.”

Memoranda of understanding should be prepared

Best practices require the contract agreement to be documented throughout the process.  The IRS requires the creation of a memorandum of understanding (MOU) to document the work to be performed and the constraints imposed on each task order prior to the start of proposal development.  In two of the seven sampled task orders, the teams did not prepare MOUs for their projects.  One acquisition team stated it did not prepare an MOU because its project was not part of the PRIME contract.  The other acquisition team had contracted for the same activities for several years.  By not preparing an MOU, parties may not begin the task order process with a mutual understanding of the work to be performed and any associated constraints.

Independent Federal Government cost estimates should be obtained

The FAR requires the initial negotiation position to be based on the results of the contracting officer’s analysis of the offeror’s proposal, taking into consideration technical analysis, fact-finding results, and independent Federal Government cost estimates (IGCE).  The Defense Procurement and Acquisition Policy Best Practices report states, “well-documented IGCEs provide contracting officers with essential program knowledge needed to evaluate and negotiate contract proposals.  Poor IGCEs can sub-optimize a program, waste resources, and may lead to contractor failure or default.”

Six of the seven sampled task orders did not obtain an IGCE.  As a result, the IRS may not be obtaining requested services at a fair and reasonable cost.  IGCEs were not developed because they were not required by BSM processes.

Management Action:  In January 2005, the Office of Procurement provided us with a list of FY 2005 commitments.  Two of the commitments are to “develop and issue IGCE template and guidance” and “revise BSMO processes to include [an] IGCE requirement.”

Issues matrices should be prepared

The Office of Federal Procurement Policy A Guide to Best Practices for Performance-Based Service Contracting suggests issue resolution be included as part of the contract negotiation process.  Also, the IRS has developed procedures to fully use verbal and written communications for all issues related to the contract.  We determined these communications often take the form of an issues matrix.

One of the seven sampled task orders did not include a prepared issues matrix.  Acquisition personnel stated they did not prepare an issues matrix because they were not part of the PRIME program.  If issues are not tracked to resolution, issues may not be resolved.

Negotiation priorities should be documented

The Defense Procurement and Acquisition Policy Best Practices report requires priorities to be documented prior to negotiations.  One of the seven sampled task orders did not include documentation of their negotiation priorities.  If negotiation priorities are not established and documented, it is difficult to determine what negotiation items to focus upon.  Negotiation priorities were not documented because this action was not required by BSM processes.

Management Action:  In January 2005, the Office of Procurement provided us with a list of FY 2005 commitments.  One commitment is to “enhance pre-award planning tools and processes based on FY 04 [2004] lessons learned.”

Negotiation strategies should be documented

Best practices require selecting a negotiation strategy prior to contract negotiations.  Six of the seven sampled task orders did not include documentation of a negotiation strategy.  Without establishing a negotiation strategy, the acquisition team may not have an overall framework to guide negotiations and may not achieve desired contracting results.  Negotiation strategies were not documented because this action was not required by BSM processes.

Management Action:  In January 2005, the Office of Procurement provided us with a list of FY 2005 commitments.  One commitment is to “enhance pre-award planning tools and processes based on FY 04 [2004] lessons learned.”

Recommendation

As part of the ongoing effort to improve preaward planning and processes, the CIO and the Chief, Agency-Wide Shared Services (AWSS), should:

2.      Implement contract negotiation procedures to promote consistent application of best practices, such as:

a.       Considering alpha contracting, when applicable.

b.      Ensuring all key stakeholders are present at contract negotiation meetings.

c.       Obtaining IGCEs.

d.      Consistently preparing MOUs and issues matrices.

e.       Documenting negotiation priorities.

f.       Documenting negotiation strategies.

Management’s Response:  To improve preaward planning and processes, the IRS will assess alpha contracting to determine feasibility for broader application.  The CIO also stated the IRS has implemented a negotiation process to include all stakeholders in negotiations, drafted an IGCE template and procedure, issued a procedure to ensure MOUs are negotiated, contracted with an independent firm to prepare an IGCE for the Modernized e-File project, and will continue to determine negotiation priorities and strategies.

Office of Audit Comment:  While the CIO agreed with this recommendation, we could not confirm whether two of the stated corrective actions have corrected the issues.  Our specific comments on the two corrective actions labeled 2E and 2F are itemized below.

·         2E – The IRS states it completed the corrective action to document negotiation priorities on November 3, 2004, by documenting negotiation issues and determining negotiation priorities in the Price Negotiation Memorandum in accordance with the operating instructions entitled “Documentation.”  As stated in this report, we communicated the interim results of our review, as well as suggestions for improvement, to the BSMO in October 2004.  We were not provided, and did not consider, instructions entitled “Documentation.”  Therefore, we were unable to confirm during the audit whether the stated corrective action has corrected the issue.

·         2F – The IRS states it completed the corrective action to document negotiation strategies on November 3, 2004, by documenting negotiation issues and determining negotiation strategies in the Price Negotiation Memorandum in accordance with the operating instructions entitled “Documentation.”  As stated in this report, we communicated the interim results of our review, as well as suggestions for improvement, to the BSMO in October 2004.  We were not provided, and did not consider, instructions entitled “Documentation.”  Therefore, we were unable to confirm during the audit whether the stated corrective action has corrected the issue.

Performance-Based Contracting Improvements Have Produced Mixed Results

In FY 2003, the IRS revised and improved PBC guidance.  In FY 2004, the IRS focused on monitoring implementation of the PBC guidance.  However, PBC improvements have produced mixed results as the BSM program has struggled to institute revised PBC processes.

Jefferson Solutions performed a review of PBC guidance implementation and assigned green, yellow, and red codes according to how well acquisition personnel were doing in complying with PBC guidance.  Figure 3 shows the results of Jefferson Solutions’ FY 2004 review.

Figure 3:  Compliance With PBC Guidance

Measurement

Compliance Percentage

Color Codes

Conformance of statements of work to applicable template and guidance.

80%

Yellow

Linkage of PBC performance standards with incentives and disincentives.

0%

Red

Conformance of monitoring plans to applicable template and guidance.

51.8%

Red

Monitoring conforms to approved plan.

74%

Yellow

Progress in moving to
fixed-price task orders.

37.5%

Yellow

Source:  Jefferson Solutions document entitled, Assessment of IRS Efforts to Enhance the Use of Performance-Based Contracting Techniques in the Modernization Program, dated September 30, 2004.

We also determined PBC improvements have not been completely effective.

Performance standards are numerous and not always measurable

Guidance from the OMB on implementing PBC indicates a measurable performance standard and an acceptable quality level are needed for each output in the Performance-Based Work Statement.  In our prior audit, we determined there was a great disparity among the number of performance standards in task orders and determined attempting to monitor large numbers of performance standards was not an efficient use of the BSMO’s limited resources.  We also found the performance standards and acceptable quality levels documented in the PBC work statements were not consistently measurable.  We recommended the BSMO provide additional guidance and useful examples to procurement and project personnel on developing measurable performance standards. 

On September 30, 2003, the BSMO and Office of Procurement issued revised guidance for procurement and project personnel in developing measurable performance standards.  To determine if the issuance of guidance had resulted in improvements, we reviewed three project task orders in our sample.  We determined the BSMO is still struggling to prepare a manageable number of performance measures.  Figure 4 shows the number of performance standards identified in our prior audit is comparable to the number of performance standards found in our current sample.

Figure 4:  Number of Performance Standards

Figure 4 was removed due to its size.  To see Figure 4, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

We also determined the performance standards documented in the PBC work statements were not always measurable.  For example, some performance measures could not be quantified, such as “PRIME personnel are knowledgeable about the TMR [transition management repository]. . . .”  Other performance measures were vague and could be improved, such as “achieve Release 1.1 milestone 4 LCSR [life cycle stage review] life cycle gate.”

As indicated in our prior audit, project personnel have struggled to develop measurable performance standards in certain areas, and developing these standards is a time-consuming process.  Without a reasonable number of measurable performance standards, it is more difficult to monitor and measure the contractor’s performance.

Incentives were not directly correlated to performance standards or monitoring plans

In our prior audit, we determined monitoring results did not always address the contractor’s achievement of or failure to meet performance standards.  This information is not only crucial in holding contractors accountable for meeting performance standards but also for providing necessary support for the payment or nonpayment of types of incentive or award fees.  As a result, we recommended monitoring results be documented and compiled for each task order.  We also recommended the BSMO require the development of balanced quality, schedule, and cost incentives.

In response to our recommendations, the BSMO and Office of Procurement revised task order development and monitoring guidance.  To determine if the issuance of guidance had resulted in the intended improvements, we reviewed incentives within task orders to determine if they correlated with what was being measured in the performance standards and monitoring plans.  We determined two of three sampled task orders with award or incentive fees did not have a direct correlation between what was being measured and the award/incentive fee.  Jefferson Solutions also concluded there was no linkage between incentive fees and performance standards for one of the task orders in our sample.

If the IRS does not correlate what is being measured with award and incentive fees, it may use subjective methods to award the contractor, rather than documented measurable results.  The one task order that did correlate an incentive fee with what was being measured had previously been involved in a pilot PBC improvement pilot.  Therefore, we determined lack of experience with new procedures caused the other two acquisition teams to not directly correlate measurements with awards/incentives.

Performance monitoring results were not always documented

In our prior audit, we determined performance monitoring results were not always adequately documented.  For example, one task order did not provide any monitoring results, and three others provided documentation that was either not complete or did not include the dates of monitoring.  As a result, we recommended the BSMO ensure monitoring results are documented and PRIME contractor performance data are compiled for each task order.

We requested key documentation being used to document monitoring of cost, schedule, and quality for our sampled task orders.  The documentation did not provide adequate support of consistent monitoring of the contractor.  In addition, we noted that, as part of its assessment, Jefferson Solutions reviewed six task orders, which included four of our seven sampled task orders.  Jefferson Solutions determined 74 percent of monitoring documentation was in full or partial compliance with monitoring plans.  While this is an improvement from our prior audit results, more work is needed to reach a green rating signifying 90 percent of monitoring documentation conforms to the approved monitoring plan.  Jefferson Solutions concluded additional monitoring and just-in-time training was needed to increase compliance with PBC practices.  Without sufficient methods to monitor contractor performance, monitoring efforts may not result in improvements in cost effectiveness and quality of performance. 

Sufficient time is not always allowed for developing, negotiating, and awarding PBC task orders

In our prior audit, we recommended the BSMO analyze the time required to develop and negotiate strong PBC task orders and monitoring plans and build that time into the project schedules and processes.  The Office of Procurement used a sample of task orders to determine the mean amount of time to develop, negotiate, and award both program- and project-level task orders, as well as an adjustment factor to account for differences between task orders.

To determine if acquisition personnel were using these data to build time into their project schedules, we reviewed the project schedules for five of our seven sampled task orders.  Figure 5 shows the five task orders reviewed; the mean amount of time needed to develop, negotiate, and award a PBC task order according to Office of Procurement data; and the amount of time allotted to develop, negotiate, and award task orders in the project schedules.

Figure 5:  Amount of Time Allotted to Develop, Negotiate, and Award Task Orders

Task Order

Mean Amount of Time Needed to Develop,  Negotiate, and Award Task Orders

Time Allotted to Develop, Negotiate, and Award Task Orders in the Project Schedule

Program Management Office

127 days

65 days

Systems Integration

127 days

34 days

Customer Account Data Engine
Release 1.2

173 days

128 days

Integrated Financial System

173 days

125 days

Modernized e-File

173 days

63 days

Source:  TIGTA comparison of Preaward Timeline – What We’ve Learned From The Program Level Task Orders and Acquisition Planning Improvement Preaward Timeline data with project schedules.

We determined all five task orders did not budget an amount of time to develop, negotiate, and award task orders consistent with the mean amount of time needed to accomplish these tasks.

For the two program-level task orders, the respective Contracting Officers explained the acquisition teams were planning to negotiate an extension to the current task order and believed a less than normal amount of time would be needed for negotiation.

For the three project task orders, the Office of Procurement data were not available to the acquisition teams prior to estimating the time needed to develop, negotiate, and award the respective PBC task orders.  As a result, we estimate the IRS may spend approximately $268,000 on the additional staff time needed to develop, negotiate, and award task orders over the amount of staff time allocated to complete these tasks (see Appendix IV).

Management Action:  In response to Jefferson Solutions’ findings regarding the need to improve compliance with PBC guidance, the IRS plans in FY 2005 to provide “just in time” training to integrated project teams that are developing task orders, perform semiannual assessments of IRS compliance with PBC guidance, and perform other activities needed to increase compliance with PBC guidelines.

Most task orders continue to be defined in a shorter period of time; however, one task order was not timely defined

The FAR requires the definitization of a letter task order to be within 180 days of task order issuance or before completion of 40 percent of the work to be performed, whichever occurs first.  If the task order can not be definitized within 180 days, the contracting officer should obtain approval from the head of the contracting office to continue negotiations.

In our prior audit, we determined the BSMO and the PRIME contractor had worked diligently to shorten the length of time it takes to determine the specific requirements and associated costs and to definitize them in a task order.  The average time to definitize had been reduced from 384 days to 90 days.

We reviewed the three PRIME contract project task orders in our sample and determined one was definitized before work began on the current milestone and another was definitized within 69 days.  In October 2004, we determined the third task order had exceeded the 180-day threshold and recommended the task order be definitized as soon as possible.  We later determined an extension had not been requested to continue negotiations beyond 180 days.

When task orders are not definitized quickly, the IRS’ negotiating position is diminished as work is completed under an undefinitized task order, since the contractor must be reimbursed for allowable costs incurred.  Therefore, the contractor has little incentive to quickly negotiate the terms and conditions.  The Contracting Officer responded definitization was not achieved timely due to scope issues, and an extension had not been requested due to lack of staffing.

Management Action:  The Modernized e-File task order was definitized in November 2004, resulting in a cost savings of $825,164 (see Appendix IV).  Since the IRS definitized the task order as recommended, we are making no additional recommendations in this area.

Recommendation

As part of the ongoing effort to improve compliance with revised PBC guidance, the CIO and the Chief, AWSS, should:

3.      Ensure the BSM program more fully realizes the potential of PBC by:

a.       Developing a reasonable number of measurable performance standards.

b.      Ensuring performance standards and monitoring plans are linked to awards and incentives, when appropriate.

c.       Documenting the results of the contractor’s performance.

d.      Allowing adequate time for negotiation activities.

Management’s Response:  To improve compliance with revised PBC guidance, the CIO stated the IRS has issued statements of work templates containing measurable performance standards and distributed historical data for all preaward activities.  Also, the Office of Procurement will continue to perform quality reviews of statements of work and monitoring plans, and an independent review organization will conduct formal reviews of the statements of work, monitoring plans, and monitoring artifacts twice yearly.  Lastly, PBC training will be scheduled and lessons learned will be posted.

Office of Audit Comment:  While the CIO agreed with this recommendation, we are concerned three of the stated corrective actions have not corrected the problems and one corrective action includes an incorrect implementation date.  Our specific comments on the four corrective actions labeled 3A through 3D are itemized below.

·         3A – The IRS states it completed the corrective action to develop a reasonable number of measurable performance standards on August 26, 2003, by issuing statement of work templates containing measurable performance standards.  The IRS also states it will continue to perform quality reviews and document lessons learned.  As stated in this report, we determined the BSMO was still struggling to prepare a manageable number of performance measures.  Also, we provided examples of PBC work statements that were not always measurable.  These issues were developed after August 2003.  Therefore, we do not believe the stated corrective action has corrected the identified issue.

·         3B – The IRS states it completed the corrective action to ensure performance standards and monitoring plans are linked to awards and incentives on August 23, 2003, by issuing statement of work and monitoring plan templates.  The IRS also states it will continue to perform quality reviews and document lessons learned.  As stated in this report, Jefferson Solutions (an independent reviewer) reported on September 30, 2004, that none of the task orders it reviewed linked performance standards with incentives.  Similarly, we determined two of three sampled task orders with award or incentive fees did not have a direct correlation between what was being measured and the award/incentive fee.  These issues were developed after August 2003.  Therefore, we do not believe the stated corrective action has corrected the identified issue.

·         3C – The IRS states it completed the corrective action to document the results of the contractor’s performance on October 1, 2004, by having an independent contractor continue to perform semi-annual reviews of task order monitoring plans and monitoring artifacts.  The IRS also states it will continue to document lessons learned.  As stated in this report, the documentation we reviewed did not provide adequate support of consistent monitoring of the contractor.  In addition, we noted that, as part of its assessment, Jefferson Solutions reviewed six task orders, which included four of our seven sampled task orders.  Jefferson Solutions determined 74 percent of monitoring documentation was in full or partial compliance with monitoring plans.  Jefferson Solutions concluded, in a report dated September 30, 2004, additional monitoring and just-in-time training was needed to increase compliance with PBC practices.  The CIO stated earlier in the response that the IRS had contracted for additional PBC overview and performance monitoring training in March 2005; however, this action was not specifically reported as a corrective action to this issue.  Since the IRS’ stated corrective action was 1 day after the September 2004 Jefferson Solutions report, we do not believe the stated corrective action has corrected the identified issue.

·         3D – The IRS states it completed the corrective action to allow adequate time for negotiation activities on March 22, 2004, by publishing historical data regarding scheduled and actual time for all preaward activities, including negotiations.  While program level historical data regarding scheduled and actual time for all preaward activities was published in early 2004, project level data was not published until August 2004.  Therefore, we do not believe the stated implementation date is correct.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to determine whether the Internal Revenue Service (IRS) had established and was following adequate contract negotiation and contract management practices for the Business Systems Modernization (BSM) program.  As part of this objective, we also determined the status of the IRS’ corrective actions to a prior audit.

To accomplish our objective, we:

 I.                  Determined how the contract negotiation, or strategy, sessions were conducted and whether they were documented in accordance with best practices.  

II.                  Determined whether contract management practices the IRS had in place (e.g., performance-based contracting (PBC) and firm fixed-price (FFP) policies) were being followed by selecting a judgmental sample of task orders.

A.       Determined if appropriate corrective actions were being taken to increase the use of FFP task orders within the BSM program.

B.        Obtained and reviewed Enterprise Life Cycle (ELC) and IRS contracting guidance to determine if corrective actions were taken based on recommendations from our prior audit.

C.        Used the judgmental sample of task orders to determine if the revised procedures for developing measurable performance standards were being followed.

D.       Used a sample of project schedules (related to the judgmental sample of task orders) to determine if time for building PBC task orders was built into each schedule.

E.        Used the judgmental sample of task orders to determine if the revised procedures on developing balanced quality, schedule, and cost incentives were being followed.

F.         Used the judgmental sample of task orders (project task orders only) to determine if the FFP contract type was chosen, if appropriate.

III.               Determined the effectiveness of the process the IRS had in place to properly monitor the PRIME contractor’s performance.

A.       Used the judgmental sample of task orders to determine if the IRS was following revised contract monitoring policies and procedures for developing monitoring plans.

B.        Used the judgmental sample of task orders to determine if monitoring results were documented.

C.        Used the judgmental sample of task orders to determine if monitoring plans allowed for monitoring of balanced incentives.

D.       Determined if an executive-level BSM acquisitions officer position had been appointed and selected.

Sample Selection Methodology

We selected a sample of 7 (47 percent) current program- and project-level task orders from the total population of 15 BSM 2004 active task orders.  We judgmentally selected the seven task orders because we wanted to focus on a combination of program- and project-level task orders, and we were not going to project the results to the entire population.

The seven task orders selected included four project-level task orders (Customer Account Data Engine Release 1.2 (CADE), Custodial Accounting Project (CAP), Integrated Financial System (IFS), and Modernized e-File (MeF)) and three program-level task orders (Infrastructure Shared Services (ISS), Program Management Office (PMO), and Systems Integration (SI)).  Descriptions of the projects and programs follow. 

·            CADE – The CADE is the foundation for managing taxpayer accounts in the IRS modernization plan.  It will consist of databases and related applications that will replace the IRS’ existing Master File processing systems and will include applications for daily posting, settlement, maintenance, refund processing, and issue detection for taxpayer tax account and return data.

·            CAP – The CAP will be a single, integrated data repository of taxpayer account information, integrated with the general ledger and accessible for management analysis and reporting.  The first release of the CAP will extract taxpayer account data from the Individual Master File (IMF) for the Taxpayer Account Subledger.

·            IFS – The IFS is intended to address administrative financial management weaknesses.  The first release of the IFS will include the Accounts Payable, Accounts Receivable, General Ledger, Budget Execution, Cost Management, and Financial Reporting activities.  A future IFS release will be needed to fully resolve all administrative financial management weaknesses.

·            MeFThe MeF project develops the modernized, web-based platform for filing approximately 330 IRS forms electronically, beginning with the U.S. Corporation Income Tax Return (Form 1120), U.S. Income Tax Return for an S Corporation (Form 1120S), and Return of Organization Exempt From Income Tax (Form 990).  The project serves to streamline filing processes and reduce the costs associated with a paper-based process.

·            ISS – The IRS established the ISS program to build and deliver an infrastructure that is scalable, interoperable, flexible, and manageable.  This infrastructure provides standardized operations and a single security and enterprise systems management framework, as well as enables the IRS to deploy modernized business systems and fully integrate them with the current processing environment.

·            PMO – The PMO task order requires the PRIME contractor to 1) maintain a comprehensive program focused on providing overall program management for BSM projects and 2) maintain program direction by continually aligning the program with the IRS business strategy.

·            SI – The SI task order requires the PRIME contractor to provide effective and efficient systems integration of business solutions from the current processing environment into the evolving modernized environment.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary V. Hinkle, Director

Troy D. Paterson, Audit Manager

Charlene L. Elliston, Auditor

Kim M. McManis, Auditor

Suzanne M. Noland, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn: Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Associate Chief Information Officer, Enterprise Services  OS:CIO:ES

Deputy Associate Chief Information Officer, Business Integration  OS:CIO:B:BI

Deputy Associate Chief Information Officer, Program Management  OS:CIO:B:PM

Deputy Associate Chief Information Officer, Systems Integration  OS:CIO:B:SI

Director, Procurement  OS:A:P

Director, Stakeholder Management  OS:CIO:SM

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaisons:

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Director, Procurement  OS:A:P

Manager, Program Oversight Office  OS:CIO:SM:PO

 

Appendix IV

 

Outcome Measures

 

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration.  These benefits will be incorporated into our Semiannual Report to the Congress.

Type and Value of Outcome Measure:

·         Reliability of Information – Potential; $268,000 (see page 17).

Methodology Used to Measure the Reported Benefit:

In a prior audit report, we recommended the Business Systems Modernization Office analyze the time required to develop and negotiate strong performance-based contracting (PBC) task orders and monitoring plans and build that time into the project schedules and processes.  The Office of Procurement used a sample of task orders to determine the mean amount of time to develop, negotiate, and award both program- and project-level task orders, as well as an adjustment factor to account for differences between task orders.  The mean time to develop, negotiate, and award a project task order was determined to be 173 days.

For the three project task orders we reviewed, Office of Procurement data were not available to the acquisition teams prior to estimating the time needed to develop, negotiate, and award the respective PBC task orders.  As a result, we estimate the Internal Revenue Service (IRS) may spend approximately $268,000 on the additional staff time needed to develop, negotiate, and award task orders over the amount of staff time allocated to complete these tasks.  Figure 1 provides details on how this amount was calculated.

Figure 1:  Computation of Outcome Measure

Row

Calculation

Integrated Financial System (IFS)

Customer Account Data Engine (CADE)

Modernized
e-File (MeF)

Total

A

Actual or Estimated Cost to Develop, Negotiate, and Award the Task Order

$190,670.98

$645,471.00

$72,500.00

 

B

Actual Number of Days to Develop, Negotiate, and Award the Task Order

n/a

180 days

238 days

 

C

Number of Days Allotted to Develop, Negotiate, and Award the Task Order in the Project Schedule

125 days

128 days

63 days

 

D

Difference Between the Mean Time to Develop, Negotiate, and Award a Project Task and the Number of Days Allotted to These Activities in the Project Schedule.  This is computed by subtracting row C from the mean time allotted to develop, negotiate, and award project task orders (173 days).

48 days

45 days

110 days

 

E

Actual or Estimated Cost to Develop, Negotiate, and Award the Task Order.  For the IFS project, row E is computed by dividing row A by row C to obtain the estimated cost per day to develop, negotiate, and award the task order.  We could only estimate the cost per day because the IFS task order for milestone 5 had yet to be awarded.  For the CADE and MeF projects, the actual cost to develop, negotiate, and award the task orders was computed by dividing row A by row B.

$1,525.37

$3,585.95

$304.62

 

F

Estimated Cost for the Difference Between the Amount of Time Allotted to Develop, Negotiate, and Award the Task Order and the Mean Time to Develop, Negotiate, and Award Project Task Orders.  This is computed by multiplying row D by row E.

$73,217.76

$161,367.75

$33,508.20

$268,093.71

Source:  Treasury Inspector General for Tax Administration analysis of Acquisition Planning Improvement Preaward Timeline data, project schedules, and cost information provided by Contracting Officers.

Type and Value of Outcome Measure:

·         Cost Savings – Actual; $825,164 (see page 17).

Methodology Used to Measure the Reported Benefit:

The Federal Acquisition Regulation requires the definitization of a task order to be within 180 days of task order issuance or before completion of 40 percent of the work to be performed, whichever occurs first.  In October 2004, we determined MeF task order 120 had exceeded the 180-day threshold and recommended the task order be definitized as soon as possible.  The Contracting Officer notified us on December 21, 2004, that task order 120 was definitized on November 19, 2004, with $825,164 in negotiated cost savings.

 

Appendix V

 

Memorandum #1:  Business Systems Modernization Contracting and Cost/Schedule Estimation Recommendations

 

July 2, 2004

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

                                     

FROM:                            Margaret E. Begg /s/ Margaret E. Begg

Assistant Inspector General for Audit (Information
Systems Programs)

 

SUBJECT:                     Business Systems Modernization Contracting and Cost/Schedule Estimation Issues

 

Based on our recent discussion, I’m providing you with some information on work that the Treasury Inspector General for Tax Administration (TIGTA) has performed in the subject areas since you have indicated the Internal Revenue Service (IRS) is planning to increase the use of firm-fixed price contracts and increase contractor accountability for meeting cost and schedule estimates. 

Attachment I contains a high-level summary of contracting and cost/schedule estimation issues and recommendations that we have reported over the past 5 years.  In case you need further information, I am also providing several attachments that provide further detail about our previous recommendations as well as the related IRS management responses.  The issues and recommendations are separated into the following categories: 

·         Contract administration (Attachment II).

·         Contractor accountability (including firm-fixed price task orders and incentives) (Attachment III).

·         Cost and schedule estimation (Attachment IV).

·         Deliverable acceptance (Attachment V).

·         Performance based contracting (Attachment VI).

To keep the information focused, recommendations from the listed reports that related to other areas, such as requirements management, are not included.  I hope that the attached information provides you with some historical perspective as the Business Systems Modernization (BSM) program moves forward.

We are currently initiating an audit in the area of contract management and will include the concerns you discussed with us in the scope of the review, as well as the previous contracting issues we have reported.  The engagement letter for this review should be issued soon. 

Please contact me at (202) 622-8510 if you have any questions.

 

 

Attachments

 

Attachment I

 

High-Level Summary of Contracting and Cost/Schedule Estimation Issues and Recommendations

 

Contract Administration (Attachment II contains additional detail)

 

Monitoring results should be documented and PRIME contractor performance data should be compiled for each task order.

 

Task orders should include the requirement that the PRIME contractor provide written assurance that it performed adequate diligence in defining all significant business requirements and that the proposed new systems will deliver all of the essential capabilities needed by users.

 

Contractor Accountability (Attachment III contains additional detail)

 

Require firm fixed-price task orders when appropriate.

 

Hold the PRIME contractor accountable, within a reasonable percentage, to cost and schedule estimates developed at Milestone (MS) 3.

 

The Business Systems Modernization Office (BSMO) must ensure PRIME contractor performance and accountability are effectively managed.

 

The BSMO should incorporate key personnel and qualifications into task orders.

 

The BSMO should develop incentives to increase PRIME contractor accountability for software quality and testing.

 

Cost increases are mounting for the Custodial Accounting Project (CAP) because the CAP contractor is not operating under a firm fixed-price contract.

 

Cost and Schedule Estimation (Attachment IV contains additional detail)

 

Project managers should build reserve and recovery time into work schedules.

 

Project managers should schedule adequate time for security testing and certification.

 

The Chief Information Officer (CIO) should establish an interim method of developing reliable estimates of IRS costs associated with modernization projects.

 

Lessons learned should be used in developing time estimates for critical tasks.

 

All BSM contractors should follow cost and schedule estimation policies and procedures.

 

The cost and schedule estimation model calibration process should be documented.

 

Cost and schedule estimation procedures should provide details of what is needed to support contractor estimates.

 

The Software Engineering Institute (SEI) should conduct an independent review of the cost and schedule estimation system when ready.

 

Issues surfaced during cost and schedule estimation reviews should be entered into the Item Tracking Reporting and Control (ITRAC) database.

 

The results of cost and schedule estimation reviews should be provided to the Office of Procurement.

 

Cost and schedule estimation review results should be trended.

 

Cost and schedule estimation review findings should be shared with the estimators.

 

Deliverable Acceptance (Attachment V contains additional detail)

 

Architecture deliverables should be thoroughly evaluated by the BSMO Architecture and Engineering Division.

 

Payment for architecture deliverables should not be made until the Director of the Architecture and Engineering Division ensures the deliverable meets the IRS’ needs.

 

The PRIME contractor should review draft work products to ensure the products meet task order requirements prior to delivery to the IRS.

 

The PRIME contractor quality review function should sign off on products prior to final delivery to the IRS.

 

Tests that are normally run sequentially are being run concurrently.  This raises the risk of accepting a system that does not work as intended.

 

Performance-Based Contracting (Attachment VI contains additional detail)

 

Task a contractor to review incentives and provide recommendations for improvement.

 

For the Customer Relationship Management Examination project, the contractor was being paid for hours expended and the contractor had not agreed to responsibilities for the upcoming project phase.

 

For the Telecommunications Modernization project, the IRS paid the contractor for hours worked versus an end result.  When the project was cancelled, the IRS had paid for incomplete products and had paid $300,000 more than the original contractor proposal.

 

For the e-Services project, the task orders did not include performance-based incentives for quality or timeliness.  In addition, task order negotiations were inconsistent.

 

The IRS should formalize the performance-based contract with the PRIME contractor prior to exiting the design phase.

 

The IRS should implement procedures to ensure the completion of fully defined and negotiated task orders for the next project phase prior to exiting the current project phase.

 

Task orders for system design, development, and implementation should be performance-based whenever appropriate.

 

The BSMO should provide guidance on developing measurable performance standards, performance incentives, and monitoring plans.

 

The BSMO should determine if performance-based contracting (PBC) is appropriate for all task orders.

 

The BSMO should develop balanced quality, schedule, and cost incentives for performance based task orders.

 

The BSMO should determine the time needed to negotiate strong performance based task orders and build this time into schedules.

 

Attachment II

 

Contract Administration Recommendations and Responses

 Additional Improvements Are Needed in the Application of Performance-Based Contracting to Business Systems Modernization Projects (Reference Number 2002-20-170, dated September 2002)

 Recommendation:  Ensure that monitoring results are documented and PRIME contractor performance data are compiled for each task order.

IRS Response:  As part of the PBC improvement effort, we will revise the monitoring guidance to ensure that BSM staff document results and compile contractor performance data for each PRIME task order.

Oversight of the Business Systems Modernization Contractor Needs Improvement (Reference Number 2004-20-034, dated January 2004)

 

Recommendation:  Ensure future task orders include the requirement that the [Computer Sciences Corporation] (CSC) provide written assurance that it and its major subcontractors performed adequate diligence in defining all significant business requirements and that the proposed new systems will deliver all of the essential functional and operational capabilities needed by the systems’ users.

IRS Response:  The CIO has issued a directive that requires fixed-price contracting for all systems development and implementation projects.  Fixed-price contracting requires mutually agreed detailed specifications, which are not created until part way toward MS 4.  To address this we are establishing a new formal checkpoint, MS 4A, at which point these specifications will be developed.  We will require CSC to provide their written assurance at this checkpoint.

 

Attachment III

 

Contractor Accountability Recommendations and Responses

 Additional Improvements Are Needed in the Application of Performance-Based Contracting to Business Systems Modernization Projects (Reference Number 2002-20-170, dated September 2002)

 Recommendation:  Require the use of firm fixed-price task orders whenever possible and appropriate for projects in development and deployment and for any other task orders where requirements are clearly identified.

IRS Response:  The Contract Executive Council (CEC) will assess various contracting strategies to ensure firm fixed-price contracts are used in accordance with this recommendation.

Improvements to the Modernized Infrastructure Are Needed to Support the Deployment of Business Systems Modernization Projects (Reference Number 2003-20-161, dated August 2003)

 Recommendation:  Hold the PRIME contractor accountable, within a reasonable percentage, to cost and schedule estimates developed at the end of the design phase [Baseline Business Case] (BBC).  This would help force the PRIME contractor to improve the estimates provided to the IRS.

IRS Response:  New PBC guidelines and templates for MS 1/2/3, 4/5, and support activities (program management, systems integration) have been developed and are being implemented.  We are in the process of developing a management directive for the use of firm-fixed price contracts on the BSM program, with an initial emphasis on firm-fixed price contracts for MS 4 and 5 activities, where we have experienced the greatest variance against cost and schedule baselines.  The directive will be in place by October 30, 2003.

At the present time, we are also entering on our tracking system a recommendation stemming from the June 2003 [Government Accountability Office] GAO report 03-768 to implement effective procedures for validating contractor developed cost and schedule estimates.  We are working with the contractor to develop methods for risk-adjusting baseline estimates in the baseline business case, which is a prerequisite for entering into more aggressive contract types.  All Office of Management and Budget (OMB) Exhibit 300s for Fiscal Year (FY) 2005 include these risk adjusted cost estimates.  We believe that selecting contract types that transfer consequences of inadequate estimates to contractors is the best motivation for improving estimation reliability, and hold them accountable.

We have established the CEC, which is pursuing improvements across the spectrum of contract management for the modernization program.  …In addition, contract management and cost schedule estimation processes are monitored monthly as part of our management process improvement.

 

Annual Assessment of the Business Systems Modernization Program (Reference Number 2003-20-208, dated September 2003)

 Issue:  In addition, we believe the BSMO is facing a fourth significant challenge…Specifically, the BSMO must ensure PRIME contractor performance and accountability are effectively managed.

 IRS Response:  We established a CEC over a year ago that has overseen the streamlining of the contracting process and eliminated all past undefinitized contracts.  We are also implementing a policy directing the use of fixed-price as the standard contracting convention for all contracts and task orders for BSM development projects following system specification.

 

Oversight of the Business Systems Modernization Contractor Needs Improvement (Reference Number 2004-20-034, dated January 2004)

 Recommendation:  Ensure future task orders include the requirement that the CSC provide written assurance that it and its major subcontractors performed adequate diligence in defining all significant business requirements and that the proposed new systems will deliver all of the essential functional and operational capabilities needed by the systems’ users.

 

IRS Response:  The CIO has issued a directive that requires fixed-price contracting for all systems development and implementation projects.  Fixed-price contracting requires mutually agreed detailed specifications, which are not created until part way toward MS 4.  To address this we are establishing a new formal checkpoint, MS 4A, at which point these specifications will be developed.  We will require CSC to provide their written assurance at this checkpoint.

 

Oversight of the Business Systems Modernization Contractor Needs Improvement (Reference Number 2004-20-034, dated January 2004)

 Recommendation:  Require the BSMO to incorporate all key personnel and the three qualifications required in the IRS’ April 2002 letter into the key personnel section of the task orders.

 IRS Response:  Agree with this recommendation.  We will work with the Office of Procurement to establish an Education and Experience Matrix that identifies and provides a correlation of the minimum education, expected experience (in years), and responsibilities of the labor categories identified as key personnel (see Corrective Action 3).  Procurement will in turn develop template language to be included in each task order that will require for other than fixed-price or capped contracts that PRIME:  submit key personnel availability certification for each newly hired key person proposed under the task order; submit a key personnel matrix that maps the skills and experience of the proposed key personnel to the task/skill areas identified in the task order; and, use a standard key personnel resume for all key personnel proposed.

 

Requirements Changes and Testing Delays Have Further Increased the Costs and Delayed the Benefits of the e-Services Project (Reference Number 2004-20-036, dated February 2004)

 Recommendation:  To improve testing, the CIO should require the BSMO to develop incentives to increase PRIME contractor accountability in the areas of software quality and testing.

 IRS Response:  Partially agreed with this recommendation.  We have taken actions to address this recommendation.  We agree that contractor incentives should be applied to the integration and test area.  We have included items to increase contractor accountability in the new Systems Integration Task Order for this fiscal year.  However, we did not include the specific incentive for the contractor for systems that successfully pass government acceptance testing with a minimum level of software defects.

The CIO has issued a directive that requires fixed-price contracting for all systems development and implementation projects.  Since fixed-price contracting requires mutually agreed specifications, we are establishing a new checkpoint, MS 4A, at which point these specifications should be developed.  We will require CSC to provide their written assurance at this checkpoint that they performed due diligence in defining all significant business requirements.

As mentioned in Corrective Action 1, we will be implementing the recommendations from recent BSM program reviews to further improve this area.  Some of these recommendations are:  identify key productivity and quality metrics across the life cycle based on industry standards; ensure more effective integration of IRS and PRIME/subs test teams; and, strengthen development environment by expanding capacity greater automation testing, and separating out infrastructure development.

 

The Custodial Accounting Project Team Is Making Progress; However, Further Actions Should Be Taken to Increase the Likelihood of a Successful Implementation (Reference Number 2004-20-061, dated March 2004)

 Issue:  One of the reasons cost increases continue to mount for the Federal Government is that the CAP contractor is not operating under a firm fixed-price contract.  In the past, we have recommended firm fixed-price contracts be used whenever possible, especially for projects in the development and deployment stages such as the CAP.

IRS Response:  Since no recommendation was made, the IRS was not required to respond to this comment.

 

Attachment IV

 

Cost and Schedule Estimation Recommendations and Responses

 Progress in Developing the Customer Communications Project Has Been Made, But Risks to Timely Deployment in 2001 Still Exist (Reference Number 2001-20-055, dated March 2001)

 Recommendation:  Ensure project managers build sufficient reserves and recovery time into work schedules to allow for the impact of unplanned events on project delivery.

 IRS Response:  The IRS agrees with this recommendation.  The need for time reserves should be determined initially by the PRIME project manager and submitted as part of the PRIME's milestone proposal.  Proposing prudent time reserves is the responsibility of the PRIME and involves their professional judgment.  The Service then reviews the proposed schedule to see that it contains realistic timeframes.

 

Recommendation:  Ensure project managers schedule adequate time to allow for security testing and certification before project development.  Managers need to continue to work closely with the IRS' Office of Security and Privacy Oversight to develop and schedule the testing processes.

 

IRS Response:  We agree with this recommendation and believe we have already implemented a corrective action.  PRIME Project managers are already working closely with the IRS' Office of Security and Privacy Oversight (SPO).  In the case of CCP 2001, PRIME and SPO met prior to the start of PRIME's Application Qualification Testing (AQT) to discuss several scheduling options and document the agreed upon testing schedules and requirements.  It should also be understood that the three-month security process cited in the audit report is not an absolute; it is subject to negotiation with SPO.

 

Modernization Project Teams Need To Follow Key Systems Development Processes (Reference Number 2002-20-025, dated November 2001)

 Recommendation:  The CIO should establish an interim method of developing reliable estimates of IRS costs associated with modernization projects.

 IRS Response:  Management’s response was due on November 8, 2001.  As of November 14, 2001, management had not responded to the draft report.

 

The Business Systems Modernization Office Needs to Strengthen Its Processes for Overseeing the Work of the PRIME Contractor (Reference Number 2002-20-059, dated March 2002)

 Recommendation:  Project managers use “lessons learned” from previous BSM projects in developing time estimates for critical tasks.  Until those lessons are effectively implemented, project schedules should include reserve time to compensate for delays or unplanned events.

IRS Response:  The IRS will require the PRIME to develop or enhance its schedule estimating capability to improve the estimates provided for modernization projects.  The IRS does not agree that reserve time should be built in for delays or unplanned events, because doing so could inadvertently encourage inefficient performance.

 

The Cost and Schedule Estimation Process for the Business Systems Modernization Program Has Been Improved, but Additional Actions Should Be Taken (Reference Number 2003-20-219, dated September 2003)

 Recommendation:  Ensure that all contractors working on BSM projects follow the PRIME contractor’s policies and procedures for preparing cost and schedule estimates and provide data for inclusion in the historical database.

 

IRS Response:  Agree with this recommendation.  Over the last year and a half, BSM has been working with PRIME to improve policies and procedures as well as the tools used to execute better cost and schedule estimates.  We now will extend this approach to non-PRIME led BSM projects, taking advantage of the lessons learned in building the PRIME based approach.  We are also studying how to include non-PRIME historical data in the PRIME's database under terms and conditions that ensure protection of proprietary data.

 

Recommendation:  Ensure that the PRIME contractor documents the process for cost and schedule estimation model calibrations.

 

IRS Response:  Agree with this recommendation.  We are addressing this recommendation by documenting the calibration process for the Constructive Cost Model (COCOMO) II model and the draft validation process for selected PRIME developed models.

 

Recommendation:  Ensure that the PRIME contractor revises the cost and schedule estimation guidebook and applicable Enterprise Life Cycle (ELC) references to provide details of what specific documentation is required to support estimates.  The CIO should also require the BSMO to ensure that guidance is clarified regarding when a second method is required for preparing estimates.

 

IRS Response:  The PRIME contractor revises the cost and schedule estimation guidebook and applicable ELC references to provide details of what specific documentation is required to support estimates.  The CIO should also require the BSMO to ensure that guidance is clarified regarding when a second method is required for preparing estimates.

 

Recommendation:  Ensure that the SEI is requested to conduct an independent review of the cost and schedule estimation system once the initial validation is complete and policies and procedures are fully implemented.

 

IRS Response:  Agree with this recommendation.  Before we consider this recommended independent review, we need to complete the validation, develop and deploy a methodology for risk adjustments, and complete revisions for incorporating enhanced cost and schedule estimating capability in business case (e300) and expenditure plan estimates.

The use of multiple estimation techniques and methods is discussed in the estimation guidebook dated June 30, 2003.  It states, "Use at least two estimation approaches or techniques when estimating your project.  Estimates that make up a large segment of the overall estimate or that have an especially large uncertainly should be estimated by more than one method.”

 

Recommendation:  Ensure that the BSMO updates draft procedures to include guidance on including all issues identified during the cost and schedule estimation system validation and estimate reviews in the ITRAC system.

 

IRS Response:  Agree with this recommendation.  We are taking actions to address this condition.  We will include risks and issues identified in the validation report and action item list in the ITRAC system following guidance governing ITRAC.  We will also update the Estimate Review Procedure to reflect the change.

 

Recommendation:  Ensure that the BSMO updates draft procedures to include guidance on providing cost and schedule estimation system validation and estimate review reports to the IRS Procurement function.

 

IRS Response:  Agree with this recommendation.  We are revising the review procedure, which will address sharing information with the Office of Procurement.

 

Recommendation:  Ensure that the BSMO updates draft procedures to include trending estimate review results.

 

IRS Response:  Agree with this recommendation.  We need time to discern the effects of cost and schedule estimation capability enhancements.  We will begin implementing processes and procedures in BSMO and at PRIME, and we should see improvements in FY 2004.  As we adapt guidance and practices for unique applications of cost and schedule estimation enhancements across all types of estimates proposals will benefit and spend plans.  Beginning in the first quarter of FY2004, we will coordinate with BSMO and PRIME program control and performance measurement functions to trend estimate review results.

 

Recommendation:  Ensure that the BSMO updates draft procedures to include providing all estimate review findings to the cost and schedule estimators.

 

IRS Response:  Agree with this recommendation.  We will update the Estimate Review Procedure to require all review checklists and reports to be forwarded to a central overseer, as resources are available.

 

Attachment V

 

Deliverable Acceptance Recommendations and Responses

 Additional Actions Are Needed To Strengthen The Development And Enforcement Of The Enterprise Architecture (Reference Number 2000-20-158, dated September 2000)

 Recommendation:  To establish a process to ensure architecture deliverables meet the IRS’ needs, the CIO should take actions to implement processes necessary to ensure the architecture deliverables received from the PRIME contractor are thoroughly evaluated by the Architecture Systems and Engineering (AE) Division staff.  These validation controls should be established prior to the delivery of the enterprise architecture products that are scheduled for delivery by the PRIME contractor.  These processes should ensure that benefits claimed from architecture investments are fully supported by sufficient data.

 IRS Response:  The IRS will establish validation procedures to ensure the architecture deliverables are thoroughly evaluated.

 

Recommendation:  To establish a process to ensure architecture deliverables meet the IRS’ needs, the CIO should take actions to implement processes necessary to strengthen the deliverable acceptance process to ensure that payment for an architecture deliverable is not initiated until the Director of AE approves that the deliverable meets the IRS’ needs.

 

IRS Response:  As part of the Enterprise Architecture supplement, the IRS will strengthen the deliverable acceptance process, as defined in the ELC, to include approval from the Director of the Architectural Engineering Division before accepting an architecture deliverable.

 

Additional Improvements Are Needed in the Application of Performance-Based Contracting to Business Systems Modernization Projects (Reference Number 2002-20-170, dated September 2002)

 Recommendation:  Require the PRIME contractor to conduct a review of draft work products with the IRS to ensure the products meet task order requirements prior to their delivery to the BSMO.

 IRS Response:  No corrective action is necessary because the IRS does not accept or reject draft work products from our PRIME contractor.  Nevertheless, as part of our partnership approach in the modernization effort, the IRS and PRIME play key roles in developing solutions.  The strategy of receiving draft work products facilitates the communication of IRS requirements up front and through the entire process.  We believe we can work closer on some draft work products, whether by co-developing them in an Integrated Product Team (IPT) environment or by using informal reviews with the IRS to better ensure eventual acceptance of task order deliverables.  To mandate a review of all draft work products may add cost without substantive value in some if not most cases.

 

Recommendation:  Ensure the final PRIME contractor quality review signoff occurs prior to the final delivery of contractor products.

 

IRS Response:  No additional corrective action is necessary.  A corrective action currently exists in the Treasury Corrective Action Database for Audit Report Number 2001-20-039.  Corrective Action 4-1-1, Revising the Milestone Exit Review (MER) Process, includes revised requirements regarding PRIME review before IRS acceptance of final deliverables.

We are modifying the MER process/procedure to reflect the following changes:  the documents reflect IRS stakeholder review and the Program Director Offices (PDOs) and their government task managers (GTM) have the authority and accountability for the approval of all project deliverables and work products; the MER documents provide definition and guidance for all milestone reviews to ensure consistency across all projects; a documented process will ensure project integration across all projects, identifying dependencies, critical paths, risks, and issues that impact more than one project, and ensuring that current (legacy) systems are considered throughout the process; and, the MER will provide a comprehensive checklist of products the PRIME contractor will review and sign, including the PRIME quality organization and the IRS operational area that is the authorized reviewer of the specific product as well as the PRIME.

 

Testing Practices for Business Systems Modernization Projects Need Improvement (Reference Number 2003-20-178, dated September 2003)

 

Issue:  When tests that should be performed sequentially are run concurrently, such as integration and acceptance testing, the IRS runs the risks of incurring additional costs and schedule delays due to the need to re-perform some tests.  For example, changes made to a system to address a problem identified in integration testing may affect the validity of previously conducted acceptance tests, so those tests may need to be performed again to ensure system changes did not impact acceptance criteria.  If previously conducted tests are not re-performed, the IRS may risk accepting a system that does not work as intended or meet all contractual requirements.

 

IRS Response:  Our business customers are fully engaged in the management process in assuring that any tailoring of testing represents the right tradeoffs.

 

Attachment VI

 

Performance Based Contracting Recommendations and Responses

 The Business Systems Modernization Office Has Made Solid Progress and Can Take Additional Actions to Enhance the Chances of Long-Term Success (Reference Number 2001-20-039, dated February 2001)

 

Recommendation:  Task Jefferson Solutions to review performance-based incentives and provide recommendations for improving incentives in future task orders.  Accepted recommendations and suggestions received from Jefferson Solutions should be used to establish policies and procedures for creating performance-based task orders.

 

IRS Response:  The IRS contracted Jefferson Solutions to review many of the IRS task orders to ensure proper incentives are in place.  In addition, Jefferson Solutions is creating PBC matrix templates.  The Program Management Office (PMO) tasked the PRIME FY 01 Task Order to incorporate and formalize these templates into the ELC.  Additionally, Jefferson Solutions will review performance based incentives and provide recommendations for improving incentives in future task orders.

 

The Customer Relationship Management Examination Project Experienced Delays and Increased Costs, But Lessons Learned Should Improve Future Modernization Projects (Reference Number 2001-20-140, dated August 2001)

 Issue:  The BSMO has recently been focusing on issuing contracts where payments are based on contractor performance rather than simply on the hours expended by the contractor.  The BSMO has also improved its ability to ensure specific contract requirements are agreed to prior to tasking the contractor to begin work.  However, the most recent contract with the CSC that we reviewed did not apply PBC methods and did not properly define the requirements of the next phase of the project’s development.  As a result, the contractor was being paid based on hours expended, instead of results achieved, and the IRS and the CSC had not agreed to responsibilities for the next project phase.

 IRS Response:  No response was required because the issues in this report and several other project audits were summarized and reported in 2 separate audit reports:  1) The Business Systems Modernization Office Needs to Strengthen Its Processes for Overseeing the Work of the PRIME Contractor (2002-20-059), and 2) Modernization Project Teams Need To Follow Key Systems Development Processes (2002-20-025).

 

The Telecommunications Modernization Project Provided Some Benefits, But Process Improvements Are Needed for Future Projects (Reference Number 2001-20-143, dated August 2001)

 

Issue:  A preliminary contract had been signed to allow the CSC to begin work on the system concept phase (also called the Architecture phase) of the project, but neither the requirements nor the full dollar amount to complete the phase had been negotiated.  The IRS paid for hours worked by the CSC, rather than a specific amount for each completed product.  The payments for the hourly work increased over 5 months to nearly $3.9 million by the time the project was cancelled.

 

Because this project was cancelled, the tasks included in the proposal received from the CSC were only partially completed.  Approximately $1.1 million of the $3.9 million had been spent on these incomplete products at the time the decision was made to cancel the project.  In addition, documentation indicates that the IRS paid nearly $300,000 more to the CSC than would have been expected if the CSC’s initial contract proposal had been accepted as submitted.

 

IRS Response:  No response was required because the issues in this report and several other project audits were summarized and reported in 2 separate audit reports:  1) The Business Systems Modernization Office Needs to Strengthen Its Processes for Overseeing the Work of the PRIME Contractor (2002-20-059), and 2) Modernization Project Teams Need To Follow Key Systems Development Processes (2002-20-025).

 

Improvements Are Needed in the Management of the e-Services Project to Enable Timely Progress Towards Future Goals (Reference Number 2001-20-144, dated September 2001)

 

Issue:  The BSMO issues contracts (called task orders) to the CSC for specific products and services.  Although the BSMO is improving the task order process, the task orders we reviewed did not include performance-based incentives for quality or timeliness.  Also, the e-Services project team did not consistently negotiate task orders in a timely manner.

 

IRS Response:  No response was required because the issues in this report and several other project audits were summarized and reported in 2 separate audit reports:  1) The Business Systems Modernization Office Needs to Strengthen Its Processes for Overseeing the Work of the PRIME Contractor (2002-20-059), and 2) Modernization Project Teams Need To Follow Key Systems Development Processes (2002-20-025).

 

Uncertainties Facing the Customer Communications 2002 Project May Jeopardize Its Timely Deployment (Reference Number 2001-20-179, dated September 2001)

 

Recommendation:  The IRS needs to formalize the performance-based contract with CSC before exiting the design phase of Customer Communications (CC) 2002.

 

IRS Response:  BSMO management requested an extension to respond to our draft report from August 29, 2001, to September 7, 2001.  As of September 20, 2001, management had not responded to the draft report.

 

The Business Systems Modernization Office Needs to Strengthen Its Processes for Overseeing the Work of the PRIME Contractor (Reference Number 2002-20-059, dated March 2002)

 

Recommendation:  Finalize and implement procedures that require the completion of fully defined and negotiated task orders for the next project phase prior to exiting the current development phase or milestone.  The BSMO’s milestone exit review should ensure that task orders for the next phase or milestone contain measurable performance standards, specific deliverables, costs, and due dates.

 

IRS Response:  Our goal is to fully define and negotiate task orders before exiting any milestone.  This is critical for a successful exit from MS 3.  Thus, we made it part of the Task Order Issuance Process Description - applicable to all projects and milestones - on October 12, 2001.  However, with projects of the complexity and scope currently in BSM, we may proceed with a letter contract, as permitted by the Federal Acquisition Regulation, to continue progress.

 

Recommendation:  Require that task orders for system design, development, and implementation be performance-based whenever possible.  These task orders should include incentive provisions to reward contractors for good performance and quality assurance deduction schedules to discourage unsatisfactory performance.  The incentive and disincentive provisions should be based on measurement against predetermined performance standards and review plans.

 

IRS Response:  No additional corrective action is necessary.  BSMO uses PBC for system design, development, and implementation whenever appropriate.  If we determine that additional incentives will properly motivate the contractor, and the benefits of such incentives will likely outweigh the cost of contract administration, then we will use them.  We do not believe the use of disincentives - in the form of quality assurance deduction schedules - in modernization contracts will resolve schedule shortfalls experienced to date.  However, we believe that disincentives at this relatively early stage of modernization would adversely impact our ability to cultivate the seamless management partnership with the PRIME that is crucial to the program's long-term success.

 

Additional Improvements Are Needed in the Application of Performance-Based Contracting to Business Systems Modernization Projects (Reference Number 2002-20-170, dated September 2002)

 

Recommendation:  Provide additional guidance and useful examples to procurement and project personnel on developing measurable performance standards, performance incentives, and monitoring plans.  This guidance should include lessons learned from the task orders and monitoring plans issued thus far.

 

IRS Response:  In May 2002, the CEC chartered its working group, the Contract Process Action Team (CPAT), to retool/improve the PBC process.  We initiated this effort to facilitate effective requirements documentation, including clearly stated outcomes, required services, performance standards, acceptable quality levels, and to monitor methodologies, incentives, and disincentives.  The CPAT will use PBC lessons learned and PBC best practices to refine the PBC matrices and Performance Based Work Statements (PBWS) for milestone and non-milestone task orders.  After CEC approves the CPAT recommendations for PBC refinement, BSM will provide guidance to IRS Procurements and project personnel.

 

Recommendation:  Determine whether PBC is appropriate for all the BSMO task orders.

 

IRS Response:  The BSM CEC and the CPAT will review this area and consider issuing more specific guidance for the use of PBC with various task orders.

 

Recommendation:  Require the development of balanced quality, schedule, and cost incentives for performance-based task orders.

 

IRS Response:  As part of the PBC improvement effort addressed in Corrective Action No. 2, the IRS will revise task order development guidance, such as the PBC matrices and monitoring plans, to address the development of balanced quality, schedule, and cost incentives as appropriate for performance-based task orders.

 

Recommendation:  Analyze the time required to develop and negotiate strong PBC task orders and monitoring plans, and build that time into the project schedules and processes.

 

IRS Response:  As part of the PBC improvement effort addressed in Corrective Action No. 2, the CPAT will use lessons learned to determine the approximate time required to develop and negotiate various PBC task orders.  We will revise affected BSM processes and procedures (e.g., the acquisition project planning process) accordingly.

 

Appendix VI

 

Prior Treasury Inspector General for Tax Administration Recommendations

 

During the course of this audit, we followed up on six Treasury Inspector General for Tax Administration (TIGTA) recommendations contained in a prior audit report.  The following were our recommendations and the Internal Revenue Service’s (IRS) responses.

1.                  TIGTA Recommendation:  To improve management of contractor performance, the Business Systems Modernization (BSM) Office (BSMO) should determine whether performance-based contracting (PBC) is appropriate for all the BSMO task orders.

IRS Response:  The BSM Contracting Executive Council (CEC) and the Contract Process Action Team (CPAT) would review this area and consider issuing more specific guidance for the use of PBC with various task orders.

2.                  TIGTA Recommendation:  To improve management of contractor performance, the BSMO should provide additional guidance and useful examples to procurement and project personnel on developing measurable performance standards, performance incentives, and monitoring plans.  This guidance should include lessons learned from the task orders and monitoring plans issued thus far.

IRS Response:  In May 2002, the CEC chartered its working group, the CPAT, to retool/improve the PBC process.  The IRS initiated this effort to facilitate effective requirements documentation, including clearly stated outcomes, required services, performance standards, and acceptable quality levels, and to monitor methodologies, incentives, and disincentives.  The CPAT planned to use PBC lessons learned and PBC best practices to refine the PBC matrices and Performance-Based Work Statements for milestone and nonmilestone task orders.  After the CEC approves the CPAT recommendations for PBC refinement, the BSMO will provide guidance to IRS procurement and project personnel.

3.                  TIGTA Recommendation:  To improve management of contractor performance, the BSMO should analyze the time required to develop and negotiate strong PBC task orders and monitoring plans and build that time into the project schedules and processes.

IRS Response:  As part of the PBC improvement effort addressed in number 2 above, the CPAT would use lessons learned to determine the approximate time required to develop and negotiate various PBC task orders.  The IRS will revise affected BSM processes and procedures (e.g., the acquisition project planning process) accordingly.

4.                  TIGTA Recommendation:  To improve management of contractor performance, the BSMO should ensure monitoring results are documented and PRIME contractor performance data are compiled for each task order.

IRS Response:  As part of the PBC improvement effort addressed in number 2 above, the IRS would revise the monitoring guidance to ensure BSM staff document results and compile contractor performance data for each PRIME contract task order.

5.                  TIGTA Recommendation:  To improve management of contractor performance, the BSMO should require the development of balanced quality, schedule, and cost incentives for performance-based task orders.

IRS Response:  As part of the PBC improvement effort addressed in number 2 above, the IRS would revise task order development guidance, such as the PBC matrices and monitoring plans, to address the development of balanced quality, schedule, and cost incentives as appropriate for performance-based task orders.

6.                  TIGTA Recommendation:  To better control contractor costs, the BSMO should require the use of firm fixed-price task orders whenever possible and appropriate for projects in development and deployment and for any other task orders where requirements are clearly identified.

IRS Response:  The CEC would assess various contracting strategies to ensure it uses firm fixed-price contracts in accordance with this recommendation.

 

Appendix VII

 

Enterprise Life Cycle Overview

 

The Enterprise Life Cycle (ELC) defines the processes, products, techniques, roles, responsibilities, policies, procedures, and standards associated with planning, executing, and managing business change.  It includes redesign of business processes; transformation of the organization; and development, integration, deployment, and maintenance of the related information technology applications and infrastructure.  Its immediate focus is the Internal Revenue Service (IRS) Business Systems Modernization (BSM) program.  Both the IRS and the PRIME contractor must follow the ELC in developing/acquiring business solutions for modernization projects.

The ELC framework is a flexible and adaptable structure within which one plans, executes, and integrates business change.  The ELC process layer was created principally from the Computer Sciences Corporation’s Catalyst® methodology.  It is intended to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision making and information sharing.  Other components and extensions were added as needed to meet the specific needs of the IRS BSM program.

ELC Processes

A process is an ordered, interdependent set of activities established to accomplish a specific purpose.  Processes help to define what work needs to be performed.  The ELC methodology includes two major groups of processes:

1.      Life-Cycle Processes, which are organized into phases and subphases and address all domains of business change.

2.      Management Processes, which are organized into management areas and operate across the entire life cycle.

Enterprise Life-Cycle Processes

 

The chart was removed due to its size.  To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Life-Cycle Processes

The life-cycle processes of the ELC are divided into six phases, as described below:

·                     Vision and Strategy – This phase establishes the overall direction and priorities for business change for the enterprise.  It also identifies and prioritizes the business or system areas for further analysis.

·                     Architecture – This phase establishes the concept/vision, requirements, and design for a particular business area or target system.  It also defines the releases for the business area or system.

·                     Development – This phase includes the analysis, design, acquisition, modification, construction, and testing of the components of a business solution.  This phase also includes routine planned maintenance of applications.

·                     Integration – This phase includes the integration, testing, piloting, and acceptance of a release.  In this phase, the integration team brings together individual work packages of solution components developed or acquired separately during the Development phase.  Application and technical infrastructure components are tested to determine whether they interact properly.  If appropriate, the team conducts a pilot to ensure all elements of the business solution work together.

·                     Deployment – This phase includes preparation of a release for deployment and actual deployment of the release to the deployment sites.  During this phase, the deployment team puts the solution release into operation at target sites.

·                     Operations and Support – This phase addresses the ongoing operations and support of the system.  It begins after the business processes and system(s) have been installed and have begun performing business functions.  It encompasses all of the operations and support processes necessary to deliver the services associated with managing all or part of a computing environment.

The Operations and Support phase includes the scheduled activities, such as planned maintenance, systems backup, and production output, as well as the nonscheduled activities, such as problem resolution and service request delivery, including emergency unplanned maintenance of applications.  It also includes the support processes required to keep the system up and running at the contractually specified level.

Management Processes

Besides the life-cycle processes, the ELC also addresses the various management areas at the process level.  The management areas include:

·                     IRS Governance and Investment Decision Management – This area is responsible for managing the overall direction of the IRS, determining where to invest, and managing the investments over time.

·                     Program Management and Project Management – This area is responsible for organizing, planning, directing, and controlling the activities within the program and its subordinate projects to achieve the objectives of the program and deliver the expected business results.

·                     Architectural Engineering/Development Coordination – This area is responsible for managing the technical aspects of coordination across projects and disciplines, such as managing interfaces, controlling architectural changes, ensuring architectural compliance, maintaining standards, and resolving issues.

·                     Management Support Processes – This area includes common management processes, such as quality management and configuration management that operate across multiple levels of management.

Milestones

The ELC establishes a set of repeatable processes and a system of milestones, checkpoints, and reviews that reduce the risks of systems development, accelerate the delivery of business solutions, and ensure alignment with the overall business strategy.  The ELC defines a series of milestones in the life-cycle processes.  Milestones provide for “go/no-go” decision points in the project and are sometimes associated with funding approval to proceed.  They occur at natural breaks in the process where there is new information regarding costs, benefits, and risks and where executive authority is necessary for next phase expenditures.

There are five milestones during the project life cycle: 

·                     Milestone 1 – Business Vision and Case for Action.  In the activities leading up to Milestone 1, executive leadership identifies the direction and priorities for IRS business change.  These guide which business areas and systems development projects are funded for further analysis.  The primary decision at Milestone 1 is to select BSM projects based on both the enterprise-level Vision and Strategy and the Enterprise Architecture.

·                     Milestone 2 – Business Systems Concept and Preliminary Business Case.  The activities leading up to Milestone 2 establish the project concept, including requirements and design elements, as a solution for a specific business area or business system.  A preliminary business case is also produced.  The primary decision at Milestone 2 is to approve the solution/system concept and associated plans for a modernization initiative and to authorize funding for that solution.

·                     Milestone 3 – Business Systems Design and Baseline Business Case.  In the activities leading up to Milestone 3, the major components of the business solution are analyzed and designed.  A baseline business case is also produced.  The primary decision at Milestone 3 is to accept the logical system design and associated plans and to authorize funding for development, test, and (if chosen) pilot of that solution.

·                     Milestone 4 Business Systems Development and Enterprise Deployment Decision.  In the activities leading up to Milestone 4, the business solution is built.  The Milestone 4 activities are separated by two checkpoints.  Activities leading up to Milestone 4A involve further requirements definition, production of the system’s physical design, and determination of the applicability of fixed-price contracting to complete system development and deployment.  To achieve Milestone 4B, the system is integrated with other business systems and tested, piloted (usually), and prepared for deployment.  The primary decision at Milestone 4B is to authorize the release for enterprise-wide deployment and commit the necessary resources.

·                     Milestone 5 – Business Systems Deployment and Postdeployment Evaluation.  In the activities leading up to Milestone 5, the business solution is fully deployed, including delivery of training on use and maintenance.  The primary decision at Milestone 5 is to authorize the release of performance-based compensation based on actual, measured performance of the business system.

 

Appendix VIII

 

Application of Contract Negotiation Best Practices

 

Using a sample of seven task orders, we reviewed available contract negotiation documentation and interviewed acquisition personnel associated with each task order.  We compared the results of our interviews and documentation reviews with 10 contract negotiations best practices.  Table 1 presents the results of our analysis.

Table 1:  Application of Contract Negotiation Best Practices

Project- or Program-Level Task Orders

Best Practice

CADE Release 1.2

CAP

IFS

MeF

ISS

PMO

SI

Alpha Contracting Methods Should Be Considered.

No

No

Yes

No

No

No

Yes

All Stakeholders Should Be Present for Negotiations.

No

Yes

Yes

Yes

Yes

Yes

Yes

Memoranda of Understanding Should Be Prepared.

Yes

No

Yes

n/a

Yes

No

Yes

Independent Federal Government Cost Estimate Should Be Obtained.

Yes

No

No

No

No

No

No

Cost and Price Analyses Should Be Conducted.

Yes

Yes

Yes

n/a

n/a

n/a

n/a

Technical and Cost Evaluations Should Be Conducted.

Yes

Yes

Yes

n/a

Yes

Yes

Yes

Cost and Schedule Estimates Should Be Reviewed.

Yes

Yes

Yes

Yes

n/a

n/a

n/a

Issues Matrices Should Be Prepared.

Yes

No

Yes

n/a

Yes

Yes

Yes

Negotiation Priorities Should Be Documented.

Yes

No

Yes

n/a

Yes

Yes

Yes

Negotiation Strategies Should Be Documented.

No

No

Yes

No

No

No

No

Source:  Treasury Inspector General for Tax Administration comparison for seven sampled task orders against best practices (IRS’ [Internal Revenue Service] PBC [Performance-Based Contracting] Lessons Learned and Best Practices Summary and Analysis, Defense Procurement and Acquisition Policy Best Practices, World-Class Contracting 100+ Best Practices for Building Successful Business Relationships, and Office of Federal Procurement Policy A Guide to Best Practices for Performance-Based Service Contracting).

NOTE:  The presence of the letters “n/a” signifies the particular task order was not judged against the best practice for one of several reasons (e.g., the best practice may not apply to all task orders).  The presence of the word “Yes” signifies the particular task order is in compliance with the best practice.  The presence of the word “No” signifies the particular task order is not in compliance with the best practice.

 

Appendix IX

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.