TREASURY
INSPECTOR GENERAL FOR TAX ADMINISTRATION
Stronger Management Oversight Is Required to Ensure Valuable
Systems Modernization Expertise Is Received From the Federally Funded Research
and
October 20, 2006
Reference Number: 2007-20-002
This
report has cleared the Treasury Inspector General for Tax Administration
disclosure review process and information determined to be restricted from
public release has been redacted from this document.
Phone Number |
202-927-7037
Email Address | Bonnie.Heald@tigta.treas.gov
Web Site | http://www.tigta.gov
October 20, 2006
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy
Inspector General for Audit
SUBJECT: Final
Audit Report – Stronger Management
Oversight Is Required to Ensure Valuable Systems Modernization Expertise Is
Received From the Federally Funded Research and
This report presents the results of our review of the Federally Funded Research and Development Center (FFRDC) [1] contractor’s performance. The overall objective of the review was to determine whether the FFRDC contractor effectively delivered services, and the Internal Revenue Service (IRS) effectively and timely monitored the contractor’s performance.
The IRS is currently
conducting a multiyear, multibillion dollar effort to update its core business
systems, known as Business Systems Modernization. To facilitate the modernization effort, the
IRS hired a FFRDC contractor to provide strategic, technical, and program
management advice, guidance, and support services. The FFRDC uses private resources to
accomplish tasks that cannot be completed effectively by existing Federal Government
employees or contractors. Since the
inception of the FFRDC contract in 1999, task orders related to the Business Systems
Modernization program have totaled about $123 million.
Impact on the Taxpayer
Our review determined a sample[2] of work products delivered by the contractor contributed significantly to the IRS modernization effort, and IRS officials were satisfied with the contractor’s performance in several areas. However, required annual assessments are not being timely completed, performance criteria are undefined, and requirements to document the contractor’s performance are lacking. Without strengthened management oversight, the contractor could deliver products and services that do not meet cost, schedule, and quality goals leading to wasteful spending of taxpayer funds.
Synopsis
We determined a sample of 25 work products delivered by the
contractor contributed significantly to the IRS modernization effort. For example, the contractor identified improvements that would enhance
the IRS’ ability to maintain a computer system and identified deficiencies in a
proposed schedule that was delivered by another contractor.
In addition to
delivering valuable products, the contractor provides advice and counsel, which
IRS officials use to make critical decisions related to the modernization
program. We asked key IRS officials responsible for monitoring
the contractor’s performance to rate the quality and timeliness of advice and
counsel provided by the contractor and the overall expertise of contract
officials. IRS officials indicated they
were satisfied with the contractor’s performance in these areas.
While the contractor is currently providing valuable products and services, the business environment within and outside of the IRS is changing rapidly. Just because the contractor is meeting today’s needs does not mean it will meet the IRS’ future needs. To ensure the contractor continues to meet the IRS’ needs, the IRS should more effectively monitor the contractor’s performance in terms of timeliness, quality, and cost. For example, the IRS does not prepare a Task Order Monitoring Plan, a practice commonly employed for modernization task orders, for this contractor’s task orders. A Task Order Monitoring Plan describes the various Government roles and responsibilities for monitoring and documenting the contractor’s performance against timeliness, quality, and cost goals. This occurs throughout the task order period of performance and contributes to a yearly evaluation. While not required, we believe this is a very good practice for ensuring the continual monitoring of a contractor’s performance. Also, required annual assessments are not being completed timely, performance criteria are undefined, and requirements to document the contractor’s performance are lacking.
Recommendation
To ensure the value
received from contractor activities remains high and funds are spent wisely, we
recommended the Chief Information Officer develop a Task Order Monitoring Plan
for the FFRDC modernization task orders.
Response
The Chief Information Officer agreed with our recommendation and will develop monitoring plans for FFRDC modernization task orders. The monitoring plans will include a requirement to conduct and document periodic evaluations to assess the timeliness, quality, and costs of deliverables and services provided by the FFRDC contractor. Management’s complete response to the draft report is included as Appendix V.
Copies of
this report are also being sent to the IRS managers affected by the report
recommendation. Please contact me at
(202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector
General for Audit (Information Systems Programs), at (202) 622-8510.
The Contractor Helped Modernize Systems by Providing Valuable
Products and Expert Advice
The Process to Effectively
and Timely Monitor the Contractor’s Performance Should be Enhanced
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix
IV – Glossary of Terms
Appendix V
– Management’s Response to the Draft Report
Abbreviations
|
BSM |
Business Systems Modernization |
|
FFRDC |
Federally Funded Research and |
|
FY |
Fiscal Year |
|
IRS |
Internal Revenue Service |
The Internal Revenue Service (IRS) is currently conducting a multiyear,
multibillion dollar effort to update its core business systems, known as
Business Systems Modernization (BSM). To
facilitate the modernization effort, the IRS hired a Federally Funded Research
and Development Center (FFRDC)[3] contractor to provide strategic, technical, and
program management advice, guidance, and support services. The FFRDC uses private resources to accomplish
tasks that cannot be completed effectively by existing Federal Government
employees or contractors. Numerous
Government agencies use FFRDCs to conduct long-term research and development.
In accordance with the Federal Acquisition Regulation,[4] the FFRDC contractor has access to Government data, employees, and facilities that is beyond that of normal contractors. In fact, the IRS considers the FFRDC contractor a partner, instead of a contractor. FFRDC officials work closely with the IRS and function as an integral part of the IRS management team. In 1999, the IRS initiated a long-term contract with its FFRDC contractor, hereafter referred to as the contractor.
Services to be provided under the FFRDC contract
are divided into major areas, such as Program and Project Management. Each major area is subdivided into smaller
functions and a statement of work is prepared for each function to document the
objectives to be achieved by the contractor and specific tasks designed to
accomplish the objectives. Once a
statement of work is agreed upon, a task order for each function is prepared to
define at a high level the scope of the work to be performed by the contractor.
This review focused
on statements of work specifically related to the BSM program, which include
the following task orders:
Since the inception of the FFRDC contract in
1999, task orders related to the BSM program have totaled about $123
million. The modernization task order
amounts for Fiscal Years (FY) 1999 through 2006 are listed in Figure 1.
Figure
1: Modernization Task Orders by Fiscal Year
Figure 1 was removed
due to its size. To see Figure 1, please
go to the Adobe PDF version of the report on the TIGTA Public Web Page.
This audit focused on whether the contractor effectively delivered products and services, and the IRS effectively monitored the contractor’s performance. A separate audit was conducted by another Treasury Inspector General for Tax Administration business unit to determine whether selected vouchers submitted and paid under FFRDC task orders were appropriate and in accordance with the task orders’ terms and conditions.[5]
This
audit was performed at the Modernization and Information Technology Services
organization’s office in New Carrollton, Maryland, during the period November
2005 through June 2006. The audit was
conducted in accordance with Government
Auditing Standards. Detailed
information on our audit objective, scope, and methodology is presented in
Appendix I. Major contributors to the
report are listed in Appendix II. A
glossary of terms is included in Appendix IV.
The Contractor Helped Modernize Systems by Providing Valuable Products and Expert Advice
Products delivered by the contractor supported IRS systems modernization
We determined a sample of contractor-provided products significantly contributed to the modernization effort.
We determined a sample[6] of 25 work products delivered by the contractor contributed significantly to the IRS modernization effort. For example, the contractor delivered products addressing the following significant areas:
·
System Maintenance – The contractor identified improvements
that would enhance the IRS’ ability to maintain the Integrated Financial System
after implementation. For example, the
contractor determined the Integrated Financial System would be difficult to
maintain due to inconsistent documentation.
The contractor recommended improving procedures and existing
documentation, and performing compliance reviews to ensure compliance with new
procedures.
·
Project Schedules – The contractor identified deficiencies in
a proposed Customer Account Data Engine schedule that was delivered by another
contractor. The other contractor made
clarifications and corrected the deficiencies.
·
IRS Vision and Goals – The contractor helped the IRS prepare and
deliver the Enterprise Transition Strategy, which documents the strategy for
achieving the vision and goals of the IRS.
Advice and counsel provided by the contractor supported IRS systems
modernization
In addition to delivering valuable products, the contractor provides advice and counsel, which IRS officials use to make critical decisions related to the modernization program. Because the advice and counsel does not directly result in delivered products, we could not readily determine whether the advice and counsel met user expectations. Therefore, we developed a questionnaire to obtain information from key IRS officials responsible for monitoring the contractor’s performance. The information requested included rating the quality and timeliness of advice and counsel provided by the contractor and the overall expertise of contract officials. IRS officials indicated they were satisfied with the contractor’s performance in these areas. Figure 2 shows the IRS responses to our questionnaire.
Figure 2: Advice and Counsel – Quality, Timeliness, and Expertise
Figure 2 was removed due to its
size. To see Figure 2, please go to the
Adobe PDF version of the report on the TIGTA Public Web Page.
We also asked IRS officials to indicate their level of agreement to the following
statements:
IRS officials
generally agreed with these statements.
Figure 3 shows the IRS responses.
Figure 3: Advice
and Counsel – Appropriate Value and
Figure 3 was removed due to its
size. To see Figure 3, please go to the
Adobe PDF version of the report on the TIGTA Public Web Page.
While the contractor is currently providing valuable products and services, the business environment within and outside of the IRS is changing rapidly. Just because the contractor is meeting today’s needs does not mean it will meet the IRS’ future needs. To ensure the contractor continues to meet the IRS’ needs, the IRS should more effectively monitor the contractor’s performance in terms of timeliness, quality, and cost.
The Process to Effectively and Timely Monitor the Contractor’s Performance Should Be Enhanced
The Federal Acquisition Regulation[7] states acquisition policies and procedures
should ensure contractors provide products and services that meet the
expectations of users in terms of timeliness, quality, and cost. The Federal
Acquisition Regulation[8] also requires an agency to evaluate the
contractor’s performance at the end of the period of performance for the
contract. If the period of performance
is longer than 1 year, the agency must schedule and conduct interim evaluations. In accordance with this requirement, the IRS
requires annual assessments of the contractor’s performance. In addition to annual assessments, the Office
of Procurement requests monthly feedback from customers related to the quality
of products delivered by the contractor and documented in the monthly progress
reports provided by the contractor. If
customers do not respond to the requests for feedback, the IRS assumes all
services were received and acceptable.
The
IRS could strengthen management oversight of its contractor because currently the
IRS lacks a comprehensive structure for monitoring the contractor’s
performance. For example, the IRS does not prepare a Task Order
Monitoring Plan, a practice commonly employed for modernization task orders,
for this contractor’s task orders. A
Task Order Monitoring Plan describes the various Government roles and
responsibilities for monitoring and documenting the contractor’s performance
against timeliness, quality, and cost goals.
This occurs throughout the task order period of performance and
contributes to a yearly evaluation. While
not required, we believe this is a very good practice for ensuring the
continual monitoring of a contractor’s performance in the areas of timeliness,
quality, and cost. Due to inadequate
management oversight on monitoring the contractor’s performance, we determined
the following:
As of February 2006, the IRS had not initiated required annual assessments of the contractor’s performance for FYs 2004 or 2005.
·
Required
annual assessments are not being completed timely – In February 2006, we
requested the annual assessments for the last 2 fiscal years. The assessment period normally ends in
November and assessments are normally initiated in January. The IRS responded it did not complete the FY
2004 annual assessment and had not begun the FY 2005 annual assessment. IRS officials did not conduct an annual
assessment for FY 2004 because of an oversight. The IRS did not initiate the annual assessment
for FY 2005 as scheduled due to resource constraints.
Management Action: The IRS initiated the annual assessments for FYs
2004 and 2005 in March 2006. In October
2006, the IRS informed us it was in the process of compiling the assessments.
· Performance criteria are undefined – The contractor’s performance is difficult to measure due to a lack of documented performance measures. For example, the IRS does not document specific requirements, due dates, or estimated costs for most products and services. The IRS does not define performance criteria because most products and services are determined on an as-needed basis and are requested and developed over a short period of time. We agree that creating controls that cost more than the products and services being provided is not an efficient use of taxpayer funds; however, we believe performance criteria could be defined for significant[9] products and services.
· Requirements to document the contractor’s performance are lacking – The IRS does not have procedures that require customers to document monitoring activities they perform. In some cases, this is due to the fact the IRS works side by side with the contractor.
For multiple reasons, the IRS has not implemented strong internal controls to ensure
the contractor’s performance is adequately monitored. One major reason is because the FFRDC has a
unique relationship with the Government.
As an FFRDC contractor, the contractor is viewed as a partner, instead
of a contractor. In addition, IRS officials believe the contractor
provides services that meet user expectations in terms of quality and
timeliness. Some IRS officials also
indicated the contractor’s performance has improved over the past several years.
While our review
of products and questionnaire responses also shows the contractor is providing
valuable products and services, some IRS officials were not completely
satisfied with the contractor’s performance.
For example, the IRS informed us some contractor officials were removed
from the assignment due to poor performance.
In addition, one key official responded to our questionnaire with
“somewhat disagreed” when asked if the contractor provided an independent
viewpoint that could not be attained internally. Another key official stated most contractor
officials are excellent; however, in some cases the services provided by the
contractor could have been provided by IRS officials.
Without strengthened management oversight, the contractor could deliver products and services that do not meet cost, schedule, and quality goals leading to wasteful spending of taxpayer funds.
If
the IRS does not conduct timely performance assessments, it may not be able to
identify and correct problems that could affect the contractor’s overall
performance. Additionally, untimely
performance assessments may provide unreliable and incomplete information. For example, some IRS officials were not
available to comment on the products and services we reviewed because they no
longer worked for the IRS. Lastly, it is
important for the IRS to ensure the millions of dollars it spends on this
contractor are spent wisely. Without strengthened
management oversight, the contractor could deliver products and services that
do not meet cost, schedule, and quality goals leading to wasteful spending of
taxpayer funds.
Recommendation
Recommendation 1: To ensure the value received from contractor
activities remains high and funds are spent wisely, the Chief Information Officer
should develop a Task Order Monitoring Plan for the FFRDC modernization task
orders. The Monitoring Plan should
include:
1. A requirement to conduct and document periodic
evaluations throughout the year to assess quality, timeliness, and cost of contractor
products and services.
2. Criteria
for measuring quality and timeliness of significant contractor products and
services.
Management’s Response: The Chief Information Officer agreed with our recommendation and will develop monitoring plans for FFRDC modernization task orders. The monitoring plans will include a requirement to conduct and document periodic evaluations to assess the timeliness, quality, and costs of deliverables and services provided by the FFRDC contractor.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine whether the FFRDC contractor effectively delivered
services, and the IRS effectively and timely monitored the contractor’s
performance. To accomplish this objective, we:
I.
Determined
whether products delivered by the contractor contributed to the achievement of
objectives included in the statements of work.
A. Selected a judgmental sample of 15 (7
percent) of 208 subtasks from 3 FY 2005 task orders specifically related to the
IRS modernization effort.[10] For
the 15 subtasks included in our sample, we selected 29 products delivered by
the contractor for review.[11] We
selected a judgmental sample because we did not plan to project our results.
B.
Determined whether products delivered by the
contractor contributed to the achievement of objectives in the statements of
work.[12]
II.
Determined
whether the IRS has processes in place to effectively and timely monitor the
contractor’s performance.
A. Determined whether the IRS is conducting
annual assessments of the contractor’s performance as required by the Federal Acquisition Regulation[13] and the FFRDC contract.
B.
Identified best practices that could be
implemented by IRS officials to monitor the contractor’s performance.
III.
Determined
whether contractor services that did not directly result in products
significantly contributed to the achievement of stated task order objectives.
A. Developed a questionnaire to document the quality
and timeliness of contractor services (advice and expertise). We submitted the questionnaire to all 10 key
IRS officials responsible for monitoring the contractor’s performance, and we
received feedback from 8 (80 percent response rate).
B.
Summarized
responses to our questionnaire to provide the IRS perspective on the value of
services that did not directly result
in products.
Appendix II
Major Contributors to This Report
Margaret
E. Begg, Assistant Inspector General for Audit (Information Systems Programs)
Gary
V. Hinkle, Director
Troy
D. Paterson, Audit Manager
James
A. Douglas, Lead Auditor
Tina
Wong, Senior Auditor
Perrin
T. Gleaton, Auditor
Suzanne
M. Noland, Auditor
Appendix III
Commissioner C
Office of the Commissioner – Attn:
Chief of Staff C
Deputy Commissioner for Operations Support OS
Associate Chief Information
Officer, Applications Development
OS:CIO:B
Associate Chief Information
Officer,
Deputy Associate Chief
Information Officer, Applications Development
OS:CIO:AD
Director, Procurement
OS:A:P
Director,
Stakeholder Management OS:CIO:SM
Deputy Associate Chief
Information Officer, Business Integration
OS:CIO:ES:BI
Deputy Associate Chief
Information Officer, Systems Integration OS:CIO:ES:SI
Chief Counsel CC
National Taxpayer Advocate TA
Director,
Office of Legislative Affairs CL:LA
Director, Office of
Program Evaluation and Risk Analysis
RAS:O
Office of Internal
Control OS:CFO:CPIC:IC
Audit Liaisons:
Associate Chief Information Officer, Applications Development OS:CIO:B
Director,
Procurement OS:A:P
Director,
Program Oversight OS:CIO:SM:PO
Appendix IV
|
Term |
Definition |
|
Best Practice |
A technique or
methodology that, through experience and research, has proven to reliably
lead to a desired result. |
|
Customer Account Data Engine |
The foundation for managing taxpayer
accounts in the IRS modernization plan.
It will consist of databases and related applications that will
replace the existing IRS Master File processing systems and will include
applications for daily posting, settlement, maintenance, refund processing,
and issue detection for taxpayer tax account and return data. |
|
Federally Funded Research and |
An organization that uses private resources
to accomplish tasks that cannot be effectively completed by existing Federal Government
employees or contractors. |
|
Integrated Financial System |
Intended to address administrative
financial management weaknesses. The
first release of the Integrated Financial System will include the Accounts
Payable, Accounts Receivable, General Ledger, Budget Execution, Cost
Management, and Financial Reporting activities. A future release will be needed to fully
resolve all administrative financial management weaknesses. |
|
Performance Standards or Criteria |
Criteria used
to measure the quality, timeliness, and cost of delivered products or
services. |
|
Release |
A specific edition of software. |
|
Statement Of Work |
Documents the work to be performed by the contractor, period of performance, performance standards, and special requirements. |
|
Task Order |
An order for services planned against an established
contract. |
|
Task Order Monitoring Plan |
A plan that documents the responsibilities and criteria for collecting and documenting monitoring information (e.g., costs, timeliness, and quality of performance). |
Appendix V
Management’s Response to the Draft Report
The
response was removed due to its size. To
see the response, please go to the Adobe PDF version of the report on the TIGTA
Public Web Page.
[1] The MITRE Corporation is the IRS FFRDC contractor. See Appendix IV for a glossary of terms.
[2] See Appendix I for more information related to our
methodology for selecting our sample.
[3] The MITRE Corporation is the IRS FFRDC contractor. See Appendix IV for a glossary of terms.
[4] 48 C.F.R. pt 35.017 (a)(2) (2005).
[5] Voucher Audit of the Federally Funded Research and Development Contract – TIRNO-99-D-00005 (Reference Number 2006-10-140, dated August 31, 2006).
[6] See Appendix I for more information related to our
methodology for selecting our sample.
[7] 48 C.F.R. pt 1.102-2 (a) (2) (2005).
[8] 48 C.F.R. pt 42.1502 (2005).
[9] The term “significant” refers to the nature and duration of the products and services.
[10] Modernization task orders for the contract include the Program Director Offices’ Support, Program Management, and System Integration task orders.
[11] The 3 task orders included 208 subtasks; however, the contractor was not instructed to provide services for all subtasks. We initially selected a judgmental sample of 20 subtasks, about 10 percent of the population. We then met with IRS officials to discuss our initial sample selection. Based on discussions with IRS officials, we removed some subtasks initially included in our sample because the contractor did not provide services related to the subtasks. We also added other subtasks that were deemed significant by IRS officials. The adjustments reduced our sample to 15 subtasks.
[12] We did not evaluate 4 of the 29 products submitted by the contractor because the products were not submitted during FY 2005, which was the focus of our review. This reduced the number of subtasks for which we reviewed products delivered by the contractor to 13 subtasks.
[13] 48 C.F.R. pt 42.1502 (2005).