TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

Stronger Management Oversight Is Required to Ensure Valuable Systems Modernization Expertise Is Received From the Federally Funded Research and Development Center Contractor

 

 

 

October 20, 2006

 

Reference Number:  2007-20-002

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

Phone Number   |  202-927-7037

Email Address   |  Bonnie.Heald@tigta.treas.gov

Web Site           |  http://www.tigta.gov

 

October 20, 2006

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:                            Michael R. Phillips /s/ Michael R. Phillips

                                         Deputy Inspector General for Audit

 

SUBJECT:                    Final Audit Report – Stronger Management Oversight Is Required to Ensure Valuable Systems Modernization Expertise Is Received From the Federally Funded Research and Development Center Contractor (Audit # 200620038)

 

This report presents the results of our review of the Federally Funded Research and Development Center (FFRDC) [1] contractor’s performance.  The overall objective of the review was to determine whether the FFRDC contractor effectively delivered services, and the Internal Revenue Service (IRS) effectively and timely monitored the contractor’s performance.

The IRS is currently conducting a multiyear, multibillion dollar effort to update its core business systems, known as Business Systems Modernization.  To facilitate the modernization effort, the IRS hired a FFRDC contractor to provide strategic, technical, and program management advice, guidance, and support services.  The FFRDC uses private resources to accomplish tasks that cannot be completed effectively by existing Federal Government employees or contractors.  Since the inception of the FFRDC contract in 1999, task orders related to the Business Systems Modernization program have totaled about $123 million.

Impact on the Taxpayer

Our review determined a sample[2] of work products delivered by the contractor contributed significantly to the IRS modernization effort, and IRS officials were satisfied with the contractor’s performance in several areas.  However, required annual assessments are not being timely completed, performance criteria are undefined, and requirements to document the contractor’s performance are lacking.  Without strengthened management oversight, the contractor could deliver products and services that do not meet cost, schedule, and quality goals leading to wasteful spending of taxpayer funds.

Synopsis

We determined a sample of 25 work products delivered by the contractor contributed significantly to the IRS modernization effort.  For example, the contractor identified improvements that would enhance the IRS’ ability to maintain a computer system and identified deficiencies in a proposed schedule that was delivered by another contractor.

In addition to delivering valuable products, the contractor provides advice and counsel, which IRS officials use to make critical decisions related to the modernization program.  We asked key IRS officials responsible for monitoring the contractor’s performance to rate the quality and timeliness of advice and counsel provided by the contractor and the overall expertise of contract officials.  IRS officials indicated they were satisfied with the contractor’s performance in these areas.

While the contractor is currently providing valuable products and services, the business environment within and outside of the IRS is changing rapidly.  Just because the contractor is meeting today’s needs does not mean it will meet the IRS’ future needs.  To ensure the contractor continues to meet the IRS’ needs, the IRS should more effectively monitor the contractor’s performance in terms of timeliness, quality, and cost.  For example, the IRS does not prepare a Task Order Monitoring Plan, a practice commonly employed for modernization task orders, for this contractor’s task orders.  A Task Order Monitoring Plan describes the various Government roles and responsibilities for monitoring and documenting the contractor’s performance against timeliness, quality, and cost goals.  This occurs throughout the task order period of performance and contributes to a yearly evaluation.  While not required, we believe this is a very good practice for ensuring the continual monitoring of a contractor’s performance.  Also, required annual assessments are not being completed timely, performance criteria are undefined, and requirements to document the contractor’s performance are lacking.

Recommendation

To ensure the value received from contractor activities remains high and funds are spent wisely, we recommended the Chief Information Officer develop a Task Order Monitoring Plan for the FFRDC modernization task orders.

Response

The Chief Information Officer agreed with our recommendation and will develop monitoring plans for FFRDC modernization task orders.  The monitoring plans will include a requirement to conduct and document periodic evaluations to assess the timeliness, quality, and costs of deliverables and services provided by the FFRDC contractor.  Management’s complete response to the draft report is included as Appendix V.

Copies of this report are also being sent to the IRS managers affected by the report recommendation.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

 

Table of Contents

 

Background

Results of Review

The Contractor Helped Modernize Systems by Providing Valuable Products and Expert Advice

The Process to Effectively and Timely Monitor the Contractor’s Performance Should be Enhanced

Recommendation 1:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Glossary of Terms

Appendix V – Management’s Response to the Draft Report

 

 

Abbreviations

 

BSM

Business Systems Modernization

FFRDC

Federally Funded Research and Development Center

FY

Fiscal Year

IRS

Internal Revenue Service

  

Background

 

The Internal Revenue Service (IRS) is currently conducting a multiyear, multibillion dollar effort to update its core business systems, known as Business Systems Modernization (BSM).  To facilitate the modernization effort, the IRS hired a Federally Funded Research and Development Center (FFRDC)[3] contractor to provide strategic, technical, and program management advice, guidance, and support services.  The FFRDC uses private resources to accomplish tasks that cannot be completed effectively by existing Federal Government employees or contractors.  Numerous Government agencies use FFRDCs to conduct long-term research and development.

In accordance with the Federal Acquisition Regulation,[4] the FFRDC contractor has access to Government data, employees, and facilities that is beyond that of normal contractors.  In fact, the IRS considers the FFRDC contractor a partner, instead of a contractor.  FFRDC officials work closely with the IRS and function as an integral part of the IRS management team.  In 1999, the IRS initiated a long-term contract with its FFRDC contractor, hereafter referred to as the contractor.

Services to be provided under the FFRDC contract are divided into major areas, such as Program and Project Management.  Each major area is subdivided into smaller functions and a statement of work is prepared for each function to document the objectives to be achieved by the contractor and specific tasks designed to accomplish the objectives.  Once a statement of work is agreed upon, a task order for each function is prepared to define at a high level the scope of the work to be performed by the contractor.

This review focused on statements of work specifically related to the BSM program, which include the following task orders:

  • Program Director Offices’ Support – Provides support to the Deputy Associate Chief Information Officer, Program Management, and others responsible for the acquisition, management, and deployment of IRS modernization projects.
  • Program Management – Provides strategic and technical program management advice, guidance, and support services.
  • System Integration – Provides support to the Systems Integration and Infrastructure Program Director’s Office.

Since the inception of the FFRDC contract in 1999, task orders related to the BSM program have totaled about $123 million.  The modernization task order amounts for Fiscal Years (FY) 1999 through 2006 are listed in Figure 1.

Figure 1:  Modernization Task Orders by Fiscal Year

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

This audit focused on whether the contractor effectively delivered products and services, and the IRS effectively monitored the contractor’s performance.  A separate audit was conducted by another Treasury Inspector General for Tax Administration business unit to determine whether selected vouchers submitted and paid under FFRDC task orders were appropriate and in accordance with the task orders’ terms and conditions.[5]

This audit was performed at the Modernization and Information Technology Services organization’s office in New Carrollton, Maryland, during the period November 2005 through June 2006.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.  A glossary of terms is included in Appendix IV.

 

 

Results of Review

 

The Contractor Helped Modernize Systems by Providing Valuable Products and Expert Advice

Products delivered by the contractor supported IRS systems modernization

We determined a sample of contractor-provided products significantly contributed to the modernization effort.

We determined a sample[6] of 25 work products delivered by the contractor contributed significantly to the IRS modernization effort.  For example, the contractor delivered products addressing the following significant areas:

·         System Maintenance – The contractor identified improvements that would enhance the IRS’ ability to maintain the Integrated Financial System after implementation.  For example, the contractor determined the Integrated Financial System would be difficult to maintain due to inconsistent documentation.  The contractor recommended improving procedures and existing documentation, and performing compliance reviews to ensure compliance with new procedures.

·         Project Schedules – The contractor identified deficiencies in a proposed Customer Account Data Engine schedule that was delivered by another contractor.  The other contractor made clarifications and corrected the deficiencies.

·         IRS Vision and Goals – The contractor helped the IRS prepare and deliver the Enterprise Transition Strategy, which documents the strategy for achieving the vision and goals of the IRS.

Advice and counsel provided by the contractor supported IRS systems modernization

In addition to delivering valuable products, the contractor provides advice and counsel, which IRS officials use to make critical decisions related to the modernization program.  Because the advice and counsel does not directly result in delivered products, we could not readily determine whether the advice and counsel met user expectations.  Therefore, we developed a questionnaire to obtain information from key IRS officials responsible for monitoring the contractor’s performance.  The information requested included rating the quality and timeliness of advice and counsel provided by the contractor and the overall expertise of contract officials.  IRS officials indicated they were satisfied with the contractor’s performance in these areas.  Figure 2 shows the IRS responses to our questionnaire.

Figure 2:  Advice and Counsel – Quality, Timeliness, and Expertise

Figure 2 was removed due to its size.  To see Figure 2, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

We also asked IRS officials to indicate their level of agreement to the following statements:

  • Appropriate Value – The IRS is receiving appropriate value for the funds being spent for contractor advice and counsel.
  • Independence – The contractor provides an independent viewpoint with expertise that could not be obtained internally.

IRS officials generally agreed with these statements.  Figure 3 shows the IRS responses.

Figure 3:  Advice and Counsel – Appropriate Value and Independence

Figure 3 was removed due to its size.  To see Figure 3, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

While the contractor is currently providing valuable products and services, the business environment within and outside of the IRS is changing rapidly.  Just because the contractor is meeting today’s needs does not mean it will meet the IRS’ future needs.  To ensure the contractor continues to meet the IRS’ needs, the IRS should more effectively monitor the contractor’s performance in terms of timeliness, quality, and cost.

The Process to Effectively and Timely Monitor the Contractor’s Performance Should Be Enhanced

The Federal Acquisition Regulation[7] states acquisition policies and procedures should ensure contractors provide products and services that meet the expectations of users in terms of timeliness, quality, and cost.  The Federal Acquisition Regulation[8] also requires an agency to evaluate the contractor’s performance at the end of the period of performance for the contract.  If the period of performance is longer than 1 year, the agency must schedule and conduct interim evaluations.  In accordance with this requirement, the IRS requires annual assessments of the contractor’s performance.  In addition to annual assessments, the Office of Procurement requests monthly feedback from customers related to the quality of products delivered by the contractor and documented in the monthly progress reports provided by the contractor.  If customers do not respond to the requests for feedback, the IRS assumes all services were received and acceptable.

The IRS could strengthen management oversight of its contractor because currently the IRS lacks a comprehensive structure for monitoring the contractor’s performance.  For example, the IRS does not prepare a Task Order Monitoring Plan, a practice commonly employed for modernization task orders, for this contractor’s task orders.  A Task Order Monitoring Plan describes the various Government roles and responsibilities for monitoring and documenting the contractor’s performance against timeliness, quality, and cost goals.  This occurs throughout the task order period of performance and contributes to a yearly evaluation.  While not required, we believe this is a very good practice for ensuring the continual monitoring of a contractor’s performance in the areas of timeliness, quality, and cost.  Due to inadequate management oversight on monitoring the contractor’s performance, we determined the following:

As of February 2006, the IRS had not initiated required annual assessments of the contractor’s performance for FYs 2004 or 2005.

·         Required annual assessments are not being completed timely – In February 2006, we requested the annual assessments for the last 2 fiscal years.  The assessment period normally ends in November and assessments are normally initiated in January.  The IRS responded it did not complete the FY 2004 annual assessment and had not begun the FY 2005 annual assessment.  IRS officials did not conduct an annual assessment for FY 2004 because of an oversight.  The IRS did not initiate the annual assessment for FY 2005 as scheduled due to resource constraints.

Management Action:  The IRS initiated the annual assessments for FYs 2004 and 2005 in March 2006.  In October 2006, the IRS informed us it was in the process of compiling the assessments.

·         Performance criteria are undefinedThe contractor’s performance is difficult to measure due to a lack of documented performance measures.  For example, the IRS does not document specific requirements, due dates, or estimated costs for most products and services.  The IRS does not define performance criteria because most products and services are determined on an as-needed basis and are requested and developed over a short period of time.  We agree that creating controls that cost more than the products and services being provided is not an efficient use of taxpayer funds; however, we believe performance criteria could be defined for significant[9] products and services.

·         Requirements to document the contractor’s performance are lackingThe IRS does not have procedures that require customers to document monitoring activities they perform.  In some cases, this is due to the fact the IRS works side by side with the contractor.

  • Feedback is not requested on the timeliness of contractor products and services – The monthly request for customer feedback asks for comments from customers on the quality of products and services received.  If customers do not respond to the requests for feedback, the IRS assumes all services were received and acceptable.  However, the request does not address whether contractor products and services were provided timely.

For multiple reasons, the IRS has not implemented strong internal controls to ensure the contractor’s performance is adequately monitored.  One major reason is because the FFRDC has a unique relationship with the Government.  As an FFRDC contractor, the contractor is viewed as a partner, instead of a contractor.  In addition, IRS officials believe the contractor provides services that meet user expectations in terms of quality and timeliness.  Some IRS officials also indicated the contractor’s performance has improved over the past several years.

While our review of products and questionnaire responses also shows the contractor is providing valuable products and services, some IRS officials were not completely satisfied with the contractor’s performance.  For example, the IRS informed us some contractor officials were removed from the assignment due to poor performance.  In addition, one key official responded to our questionnaire with “somewhat disagreed” when asked if the contractor provided an independent viewpoint that could not be attained internally.  Another key official stated most contractor officials are excellent; however, in some cases the services provided by the contractor could have been provided by IRS officials.

Without strengthened management oversight, the contractor could deliver products and services that do not meet cost, schedule, and quality goals leading to wasteful spending of taxpayer funds.

If the IRS does not conduct timely performance assessments, it may not be able to identify and correct problems that could affect the contractor’s overall performance.  Additionally, untimely performance assessments may provide unreliable and incomplete information.  For example, some IRS officials were not available to comment on the products and services we reviewed because they no longer worked for the IRS.  Lastly, it is important for the IRS to ensure the millions of dollars it spends on this contractor are spent wisely.  Without strengthened management oversight, the contractor could deliver products and services that do not meet cost, schedule, and quality goals leading to wasteful spending of taxpayer funds.

Recommendation

Recommendation 1:  To ensure the value received from contractor activities remains high and funds are spent wisely, the Chief Information Officer should develop a Task Order Monitoring Plan for the FFRDC modernization task orders.  The Monitoring Plan should include:

1.   A requirement to conduct and document periodic evaluations throughout the year to assess quality, timeliness, and cost of contractor products and services.

2.   Criteria for measuring quality and timeliness of significant contractor products and services.

Management’s Response:  The Chief Information Officer agreed with our recommendation and will develop monitoring plans for FFRDC modernization task orders.  The monitoring plans will include a requirement to conduct and document periodic evaluations to assess the timeliness, quality, and costs of deliverables and services provided by the FFRDC contractor.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to determine whether the FFRDC contractor effectively delivered services, and the IRS effectively and timely monitored the contractor’s performance. To accomplish this objective, we:

I.                   Determined whether products delivered by the contractor contributed to the achievement of objectives included in the statements of work.

A.    Selected a judgmental sample of 15 (7 percent) of 208 subtasks from 3 FY 2005 task orders specifically related to the IRS modernization effort.[10]  For the 15 subtasks included in our sample, we selected 29 products delivered by the contractor for review.[11]  We selected a judgmental sample because we did not plan to project our results.

B.     Determined whether products delivered by the contractor contributed to the achievement of objectives in the statements of work.[12]

II.                Determined whether the IRS has processes in place to effectively and timely monitor the contractor’s performance.

A.    Determined whether the IRS is conducting annual assessments of the contractor’s performance as required by the Federal Acquisition Regulation[13] and the FFRDC contract.

B.     Identified best practices that could be implemented by IRS officials to monitor the contractor’s performance.

III.             Determined whether contractor services that did not directly result in products significantly contributed to the achievement of stated task order objectives.

A.    Developed a questionnaire to document the quality and timeliness of contractor services (advice and expertise).  We submitted the questionnaire to all 10 key IRS officials responsible for monitoring the contractor’s performance, and we received feedback from 8 (80 percent response rate).

B.     Summarized responses to our questionnaire to provide the IRS perspective on the value of services that did not directly result in products.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary V. Hinkle, Director

Troy D. Paterson, Audit Manager

James A. Douglas, Lead Auditor

Tina Wong, Senior Auditor

Perrin T. Gleaton, Auditor

Suzanne M. Noland, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Applications Development  OS:CIO:B

Associate Chief Information Officer, Enterprise Services  OS:CIO:ES

Deputy Associate Chief Information Officer, Applications Development  OS:CIO:AD

Director, Procurement  OS:A:P

Director, Stakeholder Management  OS:CIO:SM

Deputy Associate Chief Information Officer, Business Integration  OS:CIO:ES:BI

Deputy Associate Chief Information Officer, Systems Integration  OS:CIO:ES:SI

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Internal Control  OS:CFO:CPIC:IC

Audit Liaisons:

Associate Chief Information Officer, Applications Development  OS:CIO:B

Director, Procurement  OS:A:P

Director, Program Oversight  OS:CIO:SM:PO

 

Appendix IV

 

Glossary of Terms

 

Term

Definition

Best Practice

A technique or methodology that, through experience and research, has proven to reliably lead to a desired result.

Customer Account Data Engine

The foundation for managing taxpayer accounts in the IRS modernization plan.  It will consist of databases and related applications that will replace the existing IRS Master File processing systems and will include applications for daily posting, settlement, maintenance, refund processing, and issue detection for taxpayer tax account and return data.

Federally Funded Research and Development Center

An organization that uses private resources to accomplish tasks that cannot be effectively completed by existing Federal Government employees or contractors.

Integrated Financial System

Intended to address administrative financial management weaknesses.  The first release of the Integrated Financial System will include the Accounts Payable, Accounts Receivable, General Ledger, Budget Execution, Cost Management, and Financial Reporting activities.  A future release will be needed to fully resolve all administrative financial management weaknesses.

Performance Standards or Criteria

Criteria used to measure the quality, timeliness, and cost of delivered products or services.

Release

A specific edition of software.

Statement Of Work

Documents the work to be performed by the contractor, period of performance, performance standards, and special requirements.

Task Order

An order for services planned against an established contract.

Task Order Monitoring Plan

A plan that documents the responsibilities and criteria for collecting and documenting monitoring information (e.g., costs, timeliness, and quality of performance).

 

Appendix V

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.



[1] The MITRE Corporation is the IRS FFRDC contractor.  See Appendix IV for a glossary of terms.

[2] See Appendix I for more information related to our methodology for selecting our sample.

[3] The MITRE Corporation is the IRS FFRDC contractor.  See Appendix IV for a glossary of terms.

[4] 48 C.F.R. pt 35.017 (a)(2) (2005).

[5] Voucher Audit of the Federally Funded Research and Development Contract – TIRNO-99-D-00005 (Reference Number 2006-10-140, dated August 31, 2006).

[6] See Appendix I for more information related to our methodology for selecting our sample.

[7] 48 C.F.R. pt 1.102-2 (a) (2) (2005).

[8] 48 C.F.R. pt 42.1502 (2005).

[9] The term “significant” refers to the nature and duration of the products and services.

[10] Modernization task orders for the contract include the Program Director Offices’ Support, Program Management, and System Integration task orders.

[11] The 3 task orders included 208 subtasks; however, the contractor was not instructed to provide services for all subtasks.  We initially selected a judgmental sample of 20 subtasks, about 10 percent of the population.  We then met with IRS officials to discuss our initial sample selection.  Based on discussions with IRS officials, we removed some subtasks initially included in our sample because the contractor did not provide services related to the subtasks.  We also added other subtasks that were deemed significant by IRS officials.  The adjustments reduced our sample to 15 subtasks.

[12] We did not evaluate 4 of the 29 products submitted by the contractor because the products were not submitted during FY 2005, which was the focus of our review.  This reduced the number of subtasks for which we reviewed products delivered by the contractor to 13 subtasks.

[13] 48 C.F.R. pt 42.1502 (2005).