Additional Improvements Are Needed in the Application of
Performance-Based Contracting to Business Systems Modernization Projects
September 2002
Reference Number:
2002-20-170
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
September
13, 2002
MEMORANDUM FOR
DEPUTY COMMISSIONER FOR MODERNIZATION &
CHIEF INFORMATION OFFICER
FROM: Pamela J. Gardiner /s/Pamela J. Gardiner
Acting Inspector General
SUBJECT: Final Audit Report - Additional
Improvements Are Needed in the Application of Performance-Based Contracting to
Business Systems Modernization Projects (Audit # 200120045)
This
report presents the results of our review of the use of Performance-Based
Contracting in the Internal Revenue Service's (IRS) Business Systems
Modernization Office (BSMO). The
overall objective of this review was to
determine whether the BSMO was properly using performance-based contracting
techniques to manage task orders under the
PRIME contract and to ensure contract terms and requirements are being
met. This contract is potentially a
multi-billion dollar contract, and conscientious management is critical to ensure
this money is spent appropriately.
In
summary, we found that improvements have been made in managing task orders
issued to the PRIME contractor. The
time required to obtain agreement between the BSMO and the PRIME contractor on
task order requirements has been reduced from 384 days to 90 days, and the
average number of modifications required for task orders has been lowered from
8 to less than 5.
While
there have been significant improvements in getting agreements on task order
requirements and reducing modifications, we believe that continuing
improvements in the application of performance-based contracting techniques in
the following areas would improve the BSMO’s ability to manage the PRIME
contractor’s performance:
·
Performance standards in project work statements were
sometimes too numerous to be monitored, were not always measurable, and often
focused on the production of documentation rather than the development of the
business systems.
·
Plans for monitoring contractor performance were not consistent
among projects, subjective monitoring methods were often used, and
determinations of whether or not the PRIME contractor met the performance
standard were often not made and documented.
·
Incentives were not consistently tied to specific levels of
performance or balanced among the key areas of cost, schedule, and timeliness.
·
The BSMO has reduced the use of firm fixed-price task
orders, the type most recommended for performance-based contracting.
·
Improvements are needed in the quality of draft documents provided
by the PRIME contractor to the BSMO for review.
We
believe that improvements in these areas should be emphasized as a key segment
of the BSMO’s effort to enhance the level of contract management and to improve
the PRIME contractor’s performance. To
address these areas, we recommended that BSMO management determine whether performance-based
contracting techniques are appropriate for all the BSMO task orders. Additionally, the BSMO should develop a
“lessons learned” document from the task orders and monitoring plans issued
thus far and provide additional guidance and examples on developing measurable
performance standards, performance incentives, and monitoring plans. This guidance should require that monitoring
results are documented and PRIME contractor performance data are compiled for
each task order. As part of this
process, the BSMO should analyze the time required to develop and negotiate
strong performance-based contracting task orders and monitoring plans and build
that time into the project schedules and processes.
To ensure good
performance is rewarded and poor performance is improved, the BSMO should
require the development of balanced quality, schedule, and cost incentives for
performance-based task orders and use firm fixed-price task orders whenever
possible and appropriate. In addition,
the BSMO should ensure that the PRIME contractor conducts a thorough review of
draft products with IRS project personnel prior to their delivery and ensure
that the PRIME contractor quality review signoff occurs prior to the final
delivery of work products.
Copies of this
report are also being sent to IRS managers who are affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Scott E. Wilson, Assistant Inspector
General for Audit (Information Systems Programs), at (202) 622-8510.
Improvements
Have Been Made in Managing Task Orders
Incentives
Were Not Consistently Tied to Performance Standards
Use of Firm
Fixed-Price Task Orders Has Decreased
Improvements Are Needed in the Quality of Draft Products Provided for Review
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
Appendix IV – Summary
of Fees for Task Orders Reviewed
Appendix V – Management’s Response to the Draft Report
The Internal Revenue Service (IRS) is currently in the midst
of a multi-billion dollar, multi-year business systems modernization effort. The IRS selected Computer Sciences
Corporation to be the PRIME contractor and created the Business Systems
Modernization Office (BSMO) to manage this effort. The BSMO has recognized the need to improve management of task
orders under the PRIME contract and has been emphasizing the increased use of performance-based
contracting (PBC) as one road towards this improvement. Use of PBC means structuring all aspects of
an acquisition around the purpose of the work to be performed, with the
contract requirements set forth in clear, specific, and objective terms with
measurable outcomes.
To introduce
basic PBC principles, the IRS hired Jefferson Solutions to train the BSMO and
IRS contracting personnel. Jefferson
Solutions summarized PBC procedures as:
· Work statements that define
desired results.
· Requirements that reflect needs
of all stakeholders.
· Performance standards that are
meaningful, measurable, and fair.
· Practical quality assurance or
monitoring plans.
·
Evaluation
factors and processes that emphasize performance.
Under the PRIME contract, various types of task orders may
be issued. Standards for PBC recommend
the use of firm fixed-price task orders whenever possible.
After a task order is
issued and the work progresses, modifications to the task order may become
necessary. Modifications to contracts
and task orders may be made for various reasons: to definitize, to make administrative changes, or to adjust to
changes in requirements, schedule, or funding.
Contracts and task orders that contain ambiguous requirements may have
numerous modifications to clarify the work to be conducted, which can cause
projects to go over schedule and cost estimates.
We issued a
report in September 2000, which
showed that 25 of the 29 task orders reviewed, totaling $62 million in
obligations, had been issued undefinitized.
Task orders had not been definitized because requirements were vague,
incomplete, or constantly changing and the rates for consultants, sub-contractors,
and profit percentages had not been agreed upon. BSMO management asserted that the practice of issuing
undefinitized task orders would decrease as the contract progressed.
In June 2001, the BSMO Quality
Assurance function reported similar deficiencies with the BSMO's acquisition
and contract management activities. For
instance, Quality Assurance reported that the BSMO was continuing to issue
letter type (undefinitized) contracts/task orders and had experienced
significant delays in completing requirements and final agreements on cost,
schedule, and performance negotiations.
In a May 2002 briefing, the BSMO
reported that it had made various commitments to improve contract management
activities in Fiscal Year (FY) 2002 but reported that these activities were
still at a relatively early stage.
Processes still needed to be developed to define the steps to be taken
to address risks associated with contract management. Therefore, the BSMO categorized this area as “red” status,
indicating that more than one current or future due date was late.
We conducted our audit from September 2001 through June 2002
at the BSMO facilities in New Carrollton, Maryland, and the IRS National
Headquarters in Washington, D.C. The
audit was conducted in accordance with Government
Auditing Standards. Detailed
information on our audit objective, scope, and methodology is presented in
Appendix I. Major contributors to the
report are listed in Appendix II.
The BSMO and the PRIME contractor have worked diligently to
shorten the length of time it takes to determine the specific requirements and
associated costs and to definitize them in the task orders for systems
modernization projects. New
requirements have been added to the Task Order Issuance Process to ensure a
definitized task order is in place for the subsequent project milestone before
a project team is allowed to exit the current milestone. These efforts have resulted in a significant
improvement in the average length of time to definitize a task order and, thus,
a reduction in the amount of time the PRIME contractor is being reimbursed for
time spent working without having specific, defined requirements.
In addition, the number of modifications to the task orders
has decreased. This is indicative of
better requirements definition up-front and a better understanding by the
parties involved of what the initial task order should accomplish. Although a large number of modifications is
not always negative, a trend of decreasing numbers of modifications combined
with improvements in the time required to definitize indicates that, as a
whole, the BSMO and the PRIME contractor are more quickly coming to a clearer
understanding of what is required to develop modernized business systems.
The following table shows the trends in time to definitize
and the number of task order modifications:
Table 1:
Improvements in Task Order Trends
Table 1 was
removed due to its size. To see the Table,
please go to the Adobe PDF version of the report on the TIGTA Public Web Page.
Additionally, in early 2001, we reported that the BSMO had
made strides in developing and implementing a framework for monitoring the
performance of the PRIME contractor, including the execution of
performance-based task orders.
In our current audit, we evaluated how well the BSMO was
implementing PBC and selected six specific task orders, five of which were for
projects in the development or deployment phase, to determine the consistency
of application of PBC principles. The
projects selected for evaluation were:
·
Customer Account Data Engine (CADE) — Builds a
modernized database for managing customer information.
·
e-Services — Creates an integrated, web-based
replacement for existing third-party tools and data collection processes.
·
Enterprise Systems Management (ESM) — Establishes an
IRS-wide Help Desk and management system to improve responsiveness to business
customers and replaces asset management functions.
·
Internet Refund/Fact of Filing (IRFOF) — Provides
taxpayers the ability to access refund and certain tax filing information via
the Internet.
·
Security and Technology Infrastructure Release (STIR) —
Provides a customer-focused technical infrastructure for secure electronic
interaction among employees, tax practitioners, and taxpayers.
·
Systems Engineering (SE) Support — Provides a detailed
Enterprise Architecture to define the IRS' future business objectives,
processes, requirements, products, and services to be offered and the basic
computer hardware and software that will be used to provide these services.
In general, the BSMO has made progress towards implementing
PBC processes. All six task orders we
evaluated had been written using PBC work statements, initial PBC guidance had
been developed, and good applications of certain PBC principles had been made
in some of the projects. For example,
direct and concise performance standards were developed for the IRFOF task
order, and a balanced schedule of fees targeting performance was developed for
the CADE task order.
The work statements that were prepared to guide the PRIME
contractor in performing the work were developed in accordance with the
standard template used for the BSMO projects.
However, beyond using the same template, the projects’ work statements
and the associated monitoring plans we evaluated did not consistently implement
PBC concepts.
Project work statements did not consistently implement
PBC concepts
We found great disparity in the number of performance
standards included in the six task orders reviewed. The number of performance standards ranged from 14 in the IRFOF
project task order to 101 in the portion of the STIR project task order that
addressed infrastructure development.
Although the number of performance standards can legitimately vary among
task orders because of the different activities covered, we believe the
projects we reviewed are similar enough that the variance in the number of
measurable performance standards would not be significant. In addition, the number of standards must be
low enough to allow for reasonable monitoring.
We believe that attempting to monitor 101 performance standards is not
an effective or efficient use of the BSMO’s limited resources.
Another inconsistency we observed was that performance
standards for certain task orders were focused primarily on how project
documentation should be produced rather than the actual delivery of systems or
implementation of infrastructure. This
focus on production of documentation was present even though all but one of the
projects we reviewed were in the process of developing systems or delivering
infrastructure to support business systems.
We believe the performance standards should have been focused primarily
on this development and delivery and secondarily on the documentation to
support this work.
Lastly, but probably most critically, the performance
standards and acceptable quality levels documented in the work statements were
not consistently measurable.
Specifically, we found that the performance standards in the business
project task orders (IRFOF, e-Services, and CADE) tended to be more measurable
than those in the infrastructure projects (STIR and ESM) and SE Support task
orders.
Guidance from the Office of Management and Budget (OMB) on
implementing PBC indicates that a measurable performance standard and an
acceptable quality level are needed for each output in the Performance Work
Statement. Guidance from Jefferson
Solutions gives examples of PBC standards as quality,
timeliness, accuracy, completeness, reliability, and cost. Using PBC should result in a focus on
results, which should enable prioritization of work.
When we discussed the measurability of the performance
standards with the project personnel, they indicated that it was more difficult
to develop meaningful performance standards for infrastructure projects and
system engineering task orders than the business projects because of the nature
of the development work. Consequently,
they have focused more on the documentation in those areas. In addition, the project personnel indicated
that they have struggled to develop measurable performance standards in certain
areas, and that coming up with these standards is a time-consuming
process. Another observation they made
was that developing these standards was even more difficult in the earlier
stages of a project than it was in the stages that we reviewed.
We understand that developing measurable performance
standards for projects is more difficult in the SE Support and infrastructure
areas and in the early stages of project design. However, focusing on measurable performance in specific key areas
of project delivery, such as business functionality, documentation, security
and privacy, and project management should result in clearer, more concise task
orders that would be more easily monitored and measured.
Monitoring plans, when used, did not effectively
implement PBC principles
The IRS’ contracting guidance
requires that, for all acquisitions using the PBC process, the task order will
identify the method that will be used to track and measure performance
for each standard. A corresponding
monitoring plan should be prepared concurrently with the task order. Our review of six task orders showed that
four (e-Services, STIR, ESM, and SE Support) had a corresponding monitoring
plan, although not all plans were developed concurrently with the task
order. The two projects (IRFOF and
CADE) that did not develop a monitoring plan used project management activities
to monitor the PRIME contractor’s performance but did not tie these activities
to standards in the task order.
Although the four monitoring plans that we evaluated
followed the required format, they did not
consistently apply PBC methods to monitor the contractor’s performance. For example, the plans did not consistently
describe how the PRIME contractor’s performance would be measured against the
performance standard, and most of the monitoring methods consisted of reviews
and meetings. Reviews and meetings are
subjective monitoring methods because they rely heavily on the evaluator’s
impressions of performance quality. The
following table identifies the disparity that we saw in evaluating the
monitoring plans:
Table 2:
Subjective vs. Measurable Monitoring Methods
|
Task Order (TO) Monitoring Plan |
PERFORMANCE STANDARDS: |
|||
|---|---|---|---|---|
|
Total Number of Performance
Standards in Monitoring Plan |
(Reviews,
Meetings, or Other*) |
(Test
Results, Reports, or Other*) |
Measurable Percentage |
|
|
e-Services TO 76 |
21 |
13 |
8 |
38% |
|
STIR TO 61 |
21 |
20 |
1 |
4% |
|
ESM TO 70 |
16 |
9 |
7 |
43% |
|
SE Support TO 56 |
36 |
32 |
4 |
11% |
Source: As indicated in table (* denotes a very small percentage).
We also found that results were not always adequately documented
in the monitoring plans or elsewhere.
For instance, one monitoring plan (e-Services)
did not provide any monitoring results, and the other three plans (SE Support,
STIR, and ESM) provided information on the monitoring activities performed and
the results, but the documentation was not complete or dates of monitoring
activities were not documented. Lack of
documentation made it difficult to determine if monitoring plans were being
followed on a continuous basis.
More significantly, monitoring results did not always
address the PRIME contractor’s achievement of or failure to meet performance
standards. This information is not only
crucial in holding the PRIME contractor accountable for meeting performance
standards but also for providing the necessary support for the payment or
nonpayment of types of incentive or award fees and for the annual evaluation of the PRIME contractor’s performance.
PBC techniques provide that the
monitoring plan describe how the PRIME contractor’s performance will be measured
against the performance standards. The
plan also assists with the compiling of performance
data used for establishing accountability for achievement of or failure to meet
performance standards. Surveillance
based on the plan should be comprehensive, systematic, and well
documented.
Government policy requires agencies
to develop quality assurance surveillance plans that contain measurable inspection and acceptance criteria
corresponding to the performance standards contained in the statement of work/performance
work statements. Best practices provide
that the quality assurance plan should be written concurrently with the task
order performance standards, because it helps assure that the standard is
measurable and that measurement is not unduly burdensome. The value of the information should be
compared against the effort or cost to collect it.
As indicated earlier in this
report, we believe the lack of measurable performance standards in the task
order makes it very difficult for the BSMO to develop an effective monitoring
plan. In addition, the contracting
guidance on monitoring and documenting the PRIME contractor’s performance lacks
the necessary clarity and detail needed by the project to develop effective
monitoring plans. Also, the guidance
had not been adequately communicated to all the BSMO and IRS acquisition
personnel.
We also believe that monitoring performed for contract
compliance was not always documented effectively because the project teams were
conducting numerous project management activities that they viewed to be
monitoring activities. The project team
members stated that these activities were being documented as part of their
project management processes. Although
these management activities were important for other purposes, they were not
tied to contract performance standards and, thus, were not effective contract
monitoring tools. Without sufficient methods to monitor actual PRIME
contractor performance against a standard, monitoring efforts may not result in
improvements in cost-effectiveness and quality of performance.
Management Actions:
BSMO management has established an Acquisition Streamlining Task Force
to identify ways to improve the acquisition process. In addition, Program Office Directives for the various processes
for issuance and status reporting of task orders have been published. In FY 2002, the BSMO has committed to
identify key improvements to the PBC process.
However, because of other priorities and budget constraints, plans for
improvement in contract management are still being developed.
To improve management of contractor performance, the
BSMO should:
1.
Determine whether PBC is appropriate for all the BSMO task
orders.
2.
Provide additional guidance and useful examples to
procurement and project personnel on developing measurable performance
standards, performance incentives, and monitoring plans. This guidance should include lessons learned
from the task orders and monitoring plans issued thus far.
3.
Analyze the time required to develop and negotiate strong
PBC task orders and monitoring plans, and build that time into the project
schedules and processes.
4.
Ensure that monitoring results are documented and PRIME
contractor performance data are compiled for each task order.
Management’s Response: The BSMO
Contracting Executive Council (CEC) and the Contract Process Action Team (CPAT)
will review this area. The CPAT will
use PBC lessons learned and best practices to refine the PBC matrices and
Performance-Based Work Statements for milestone and non-milestone task
orders. After the CEC approves the CPAT
recommendations for PBC refinement, the BSMO will provide guidance to IRS
Procurement and project personnel. As
part of the PBC improvement effort, the CPAT will use lessons learned to determine
the approximate time required to develop and negotiate various PBC task
orders. Affected processes and
procedures will be revised accordingly.
As part of the PBC improvement effort, the BSMO will revise the
monitoring guidance to ensure that staff document results and compile
contractor performance data for each PRIME contractor task order.
The six task orders we reviewed each included some sort of
fee associated with the completion of the task order (see table in Appendix
IV). However, these fees were not
consistently tied to measurable performance standards. In only one case, the CADE project, did we
identify significant fees specifically associated with meeting performance
standards in the schedule and quality components of the task order. On the STIR project, a small fee was
associated with successful completion of testing, but the fee was much smaller
than the overall target fee for completion of the task order work. On other task orders, fees were associated
with the general completion of the task order work statement, or cost
management, but not for exceeding identified performance standards.
In earlier reviews, we identified concerns with the lack of
strong incentives tied to performance.
Guidance from the OMB indicates that incentives should be used when they
will encourage higher quality performance and may be either positive, negative,
or a combination of both. They should
be applied selectively to motivate contractor efforts that might not otherwise
be emphasized and to discourage inefficiency.
Incentives should be applied to the most critical aspects of the work,
rather than every individual task.
Additionally, incentives are especially useful in efforts such as Business
Systems Modernization that are complex, have a high-dollar value, or have a
history of performance or cost overrun problems.
Agencies should avoid rewarding contractors for simply
meeting minimum standards of contract performance and should create a proper
balance among cost, performance, and schedule incentives. The incentive amount should correspond to
the difficulty of the task required but should not exceed the value of the
benefits the government receives.
When we discussed performance incentives with project
contracting personnel, they indicated one reason that the incentives are not
more directly tied to performance is because these types of balanced incentives
are difficult to negotiate with the PRIME contractor. In addition, when we discussed the possible use of disincentives
when performance was below acceptable quality levels, the BSMO personnel
indicated that PRIME contracting staff consider achieving less than 80 percent
of their negotiated award or fee a disincentive in their company. The BSMO personnel also indicated that they
believe use of incentives would promote an adversarial relationship with the
PRIME contractor, which they wish to avoid.
Lastly, they indicated that prior BSMO executives had made a conscious
decision not to apply incentives to specific performance standards in the task
order but to apply them more broadly across the program. However, we have not seen indications of the
use of these types of broader incentives.
We believe that consistently tying
incentives to specific areas of performance (balancing cost, schedule, and
performance) should increase the level of performance achieved and would enable
the BSMO to emphasize the criticality of quality performance. Clear performance standards and strong award
clauses on the CADE project have helped the BSMO avoid payment of increased
costs to the PRIME contractor due to performance issues on that project.
To improve management of contractor performance, the BSMO
should:
5.
Require the development of balanced quality, schedule, and
cost incentives for performance-based task orders.
Management’s Response: As part of the PBC improvement effort, the IRS will revise task
order development guidance, such as the PBC matrices and monitoring plans, to
address the development of balanced quality, schedule, and cost incentives as
appropriate for performance-based task orders.
We evaluated the BSMO's use of firm fixed-price task orders
and found that although there was an increased use of this contract type from
FY 1999 to FY 2000, the use decreased between FY 2000 and FY 2001 from nearly 57
percent of task orders issued to less than 21 percent. None of the six task orders we reviewed were
firm fixed-price task orders. All were
cost-reimbursable task orders with some type of additional fee or incentive
associated.
The Federal Acquisition Regulations section on service
contracting indicates that a firm fixed-price performance-based contract is the
preferred contract type. The guidance
Jefferson Solutions provided to the BSMO also suggests that firm fixed-price
task orders should be used whenever possible to motivate contractors to perform
at the highest level. OMB guidance also
indicates that PBC encourages and enables an increased use of firm fixed-price
contracts and incentives to encourage optimal performance. Additionally, the IRS Deputy Commissioner
for Modernization & Chief Information Officer recently indicated that one
way the BSMO is attempting to control costs is to use firm fixed-price task
orders more frequently.
The IRS contracting and BSMO executives indicated that they
were instructing their personnel to use firm fixed-price task orders when
appropriate. However, due to risks
associated with formalizing requirements in this large-scale modernization
effort, they indicated it was not always appropriate to use this type of task
order. Personnel from one of the
Business Systems Modernization project teams indicated that they believed the
PRIME contractor would not usually agree to the use of these task orders or
that it would take too long to negotiate their use. They stated that the PRIME contractor was somewhat averse to
these types of task orders because in at least one instance, when a firm fixed-price
task order was used in FY 2000, the contractor was left responsible for various
costs because the project did not meet the delivery date.
Because cost-reimbursable rather than firm fixed-price task
orders are used, IRS contracting and BSMO personnel must focus significant
levels of effort tracking cost information.
In addition, when a contract exceeds planned costs and delivery dates,
the BSMO is typically responsible for paying these additional contractor
costs. We believe, based on our review of
PBC and general contracting principles, that the use of firm fixed-price task
orders is more effective in shifting the risk of non-performance to the PRIME
contractor. In addition, use of these
types of task orders automatically builds in incentives for cost
management. Use of firm fixed-price
task orders does require more time and focus initially to ensure that task
order requirements are identified and negotiated, but the benefits of reduced
contract administration and reduced liability for contractor costs when time
frames or cost projections are exceeded should offset the time spent in
up-front planning. We also believe that
the efforts the PRIME contractor is making to improve its cost and schedule
estimation should make negotiations of firm fixed-price task orders more
reasonable.
To better control contractor costs, the BSMO should:
6.
Require the use of firm fixed-price task orders whenever
possible and appropriate for projects in development and deployment and for any
other task orders where requirements are clearly identified.
Management’s Response: The CEC will assess various contracting strategies to ensure firm
fixed-price contracts are used in accordance with this recommendation.
The PRIME contractor is responsible for providing a final
and independent assessment of work products prior to delivery to the BSMO. This assessment is made to ensure that the
products meet task order requirements.
In March 2001, the
BSMO Quality Assurance function reported that the signature of the PRIME
contractor’s quality review personnel on products did not necessarily imply
approval of the work products. To
determine whether this issue still existed, we reviewed available final
products from three projects to determine if they were in compliance with the
task order requirements. We found that
18 of the 21 products reviewed were in compliance with the task order
requirements. The issues with
the remaining three products did not significantly affect their usability, or
the BSMO personnel indicated that they had agreed to accept the products as
delivered, knowing that all information was not included.
While most of the final delivered products met quality
requirements, we found that draft products provided to the BSMO for review did
not consistently meet quality standards.
As a result, these products frequently had to be returned to the PRIME
contractor for rework to address non-compliance with original requirements.
For example, we reviewed comments made by reviewers of 5 ESM
project draft products and found that 30 percent of the comments related to
issues with the products not meeting requirements. On the e-Services project, the Project Director initially rejected
the draft Project Management Plan because it lacked specific details required
to show how the PRIME contractor would successfully manage the project. Also, it did not include the required
Quality Management Plan, Training Plan, or Risk Management Plan. In another case, quality issues with the
transition strategy portion of the Enterprise Architecture caused the BSMO to
question whether it should release this portion for review by key stakeholders.
We believe one reason these quality problems existed was
that the PRIME contractor did not always complete the deliverable review
process prior to the delivery of draft products to the BSMO. We found that the PRIME contractor’s signoff
indicating completion of its review often occurs after delivery of the products
to the BSMO. The PRIME contractor’s
document delivery process that describes the activities involved in the quality
review process does not clearly identify the sequence of quality processes or
what level of quality review is required prior to delivery of draft products to
the BSMO for its review.
When the PRIME contractor’s quality review process is
finalized after the BSMO review and acceptance of products, the benefits this
process could provide are minimized.
Additional time and expense can occur to address quality problems that
should have been identified and addressed prior to initial document
delivery. In addition, in some cases
significant additional costs could accrue to the BSMO to address quality
problems identified during this process even after the product has been
accepted by the BSMO. For example, STIR
project personnel indicated that they have questioned costs in one instance in
which the PRIME contractor requested payment for changes made based on its
quality review after the BSMO had approved the product.
Management Actions:
The BSMO plans to conduct reviews of the PRIME contractor’s quality
review process as a follow-up to its March 2001 review. In addition, a review of the Documentation
Review and Delivery process is currently underway. This review will determine if the PRIME contractor and the BSMO
are following a documented process and will also focus on the document
acceptance and rejection process.
In order to improve the quality of
draft products, the BSMO should:
7.
Require the PRIME contractor to conduct a review of draft
work products with the IRS to ensure the products meet task order requirements
prior to their delivery to the BSMO.
8.
Ensure the
final PRIME contractor quality review signoff occurs prior to the final
delivery of contractor products.
Management’s Response: No additional
corrective action is necessary because Business Systems Modernization
management believes that to mandate a review of all draft work products may add
cost without substantive value in some, if not all, cases. Nevertheless, the BSMO believes it can work
more closely with the PRIME contractor on some draft work products, whether by
co-developing them or by using informal reviews to ensure eventual acceptance
of task order deliverables.
In addition, a corrective action
to a separate report includes revised requirements regarding PRIME contractor
review before IRS acceptance of final deliverables. Included in that action is a provision that the Milestone Exit
Review will provide a comprehensive checklist of products the PRIME contractor
will review and sign.
Office of Audit Comment: We agree that if
management works more closely with the PRIME contractor on draft work products
and takes the detailed actions to improve the Milestone Exit Review Process,
they will have addressed the substance of the issues reported in this section.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine
whether the Business Systems Modernization Office (BSMO) is properly using
performance-based contracting (PBC) techniques to manage task orders under the PRIME contract and to
ensure contract terms and requirements are being met.
To complete our work on this review, we conducted the following
tests:
A.
Determined whether
the BSMO properly used PBC techniques to prepare task orders for the following
Fiscal Year 2002 release projects:
·
Customer Account Data Engine — Builds a modernized
database for managing customer information.
·
e-Services — Creates an integrated, web-based
replacement for existing third-party tools and data collection processes.
·
Enterprise Systems Management — Establishes an IRS-wide
Help Desk and management system to improve responsiveness to business customers
and replaces asset management functions.
·
Internet Refund/Fact of Filing — Provides taxpayers the
ability to access refund and certain tax filing information via the Internet.
·
Security and Technology Infrastructure Release —
Provides a customer-focused technical infrastructure for secure electronic
interaction among employees, tax practitioners, and taxpayers.
·
Systems Engineering Support — Provides a detailed
Enterprise Architecture to define the IRS’ future business objectives,
processes, requirements, products, and services to be offered and the basic
computer hardware and software that will be used to provide these services.
B.
Determined how the
BSMO was using PBC techniques for the Fiscal Year 2002 project releases to
monitor task orders and ensure contract terms and requirements are being met.
Appendix II
Major Contributors to This Report
Scott E.Wilson, Assistant
Inspector General for Audit (Information Systems Program)
Scott A. Macfarlane, Director
Tammy L. Whitcomb, Audit Manager
Michael A. Garcia, Senior Auditor
W. Allen Gray, Senior Auditor
Theresa A. Haley, Senior Auditor
Esther M. Wilson, Senior Auditor
Charles R. Winn, Senior Auditor
Albert C. Greer, Auditor
Appendix III
Commissioner N:C
Deputy
Commissioner N:DC
Associate Commissioner,
Business Systems Modernization M:B
Deputy Associate Commissioner,
Program Management M:B:PM
Deputy
Associate Commissioner, Systems Integration
M:B:SI
Director,
Procurement A:P
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative
Affairs CL:LA
Director,
Office of Program Evaluation and Risk Analysis
N:ADC:R:O
Office of
Management Controls N:CFO:F:M
Audit Liaison:
Associate Commissioner, Business
Systems Modernization M:B
Appendix IV
Summary of Fees for Task Orders Reviewed
|
Project / Task Order (TO) |
Contract Amount |
Fixed Fee |
Performance Fee |
Total Est. Contract Cost |
|---|---|---|---|---|
|
CADE TO 73
|
$22,884,800 |
$685,600 |
$1,600,000 (Allocated as follows: $784,000 for meeting schedule, $648,000 for
quality, and $168,000 for managing costs) |
$25,170,400 |
|
e-Services TO 76 |
$30,491,816 |
|
$2,581,346 (Allocated solely for managing costs and TO
completion) |
$33,073,162 |
|
ESM TO 70
|
$5,441,185 |
$421,200 |
|
$5,862,385 |
|
IRFOF TO 79 |
$7,004,555 |
|
$594,360 (Allocated solely for managing costs and TO
completion) |
$7,598,915 |
|
STIR TO 61
|
$22,962,958 |
$704,247 |
$1,446,820 (Allocated as follows: $1,246,820 for managing costs and TO completion, and
$200,000 for successful completion of testing) |
$25,114,025 |
|
Systems Engineering Support TO 56 |
$15,799,678 |
$1,311,198 |
|
$17,110,876 |
Appendix V
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.