The Modernization Program Is Establishing a Requirements Management Office to Address Requirements Development and Management Problems

January 2005

Reference Number: 2005-20-023

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

January 19, 2005

MEMORANDUM FOR CHIEF INFORMATION OFFICER

FROM: Gordon C. Milbourn III /s/ Gordon C. Milbourn III

Assistant Inspector General for Audit

(Small Business and Corporate Programs)

SUBJECT: Final Audit Report - The Modernization Program Is Establishing a Requirements Management Office to Address Requirements Development and Management Problems (Audit # 200320026)

This report presents the results of our review of the Internal Revenue Service’s (IRS) Business Systems Modernization Office (BSMO) requirements development and management activities. The overall objective of this review was to determine whether the BSMO has established and is following adequate requirements management practices to assure the effective development of modernization projects that meet customer needs. This review was part of our Fiscal Year (FY) 2004 Annual Audit Plan for reviews of the IRS Business Systems Modernization (BSM) efforts.

In summary, from November 2001 through November 2004, the Treasury Inspector General for Tax Administration (TIGTA) issued 16 reports identifying BSM requirements development and management problems. Our analysis of these reports identified two recurring issues:

1) Project management did not adequately identify requirements to address customers’ needs and project development criteria (e.g., computer programming naming standards and financial reporting requirements) before initiating development activities.

2) Project management did not trace all requirements to test cases to ensure the projects’ operations met expectations.

As a result of these continuing problems, some modernization projects have experienced scheduling delays and increased costs to resolve the requirements development and management issues.

The proposal for establishing the Requirements Management Office includes three options. These options have varying degrees of depth in proposed management and control goals, each dependent on the FY 2005 budget allocation for this Office. All three options include using an independent requirements contractor to facilitate the requirements development and management of the modernization projects. The BSMO is in the process of defining and deciding upon the responsibilities for this Office. Consequently, detailed analyses estimating the costs and resources necessary for each of the proposed options have not been completed. The BSMO is currently developing a work breakdown structure which should aid in defining and clarifying the related tasks needed to establish and operate the Requirements Management Office.

Our review of the proposal determined the Requirements Management Office options can be enhanced to ensure effective development and management of requirements. The options as presented in the proposal do not include specific direction to adequately address issues we previously reported. In addition, the options can be enhanced by addressing all guidelines for developing and managing requirements that are included in the Enterprise Life Cycle (ELC) and the Carnegie Mellon Software Engineering Institute’s (SEI) Capability Maturity Model Integration (CMMI) models.

Further, the Requirements Management Office proposal has not yet thoroughly considered several key scope aspects that will allow it to begin operating effectively. These scope aspects include determining the Office’s specific responsibilities, the source and amount of funding, the staffing needs, the time period for establishing the Office, and a schedule of current or planned modernization projects that will need requirements development and management services.

As part of the ongoing development of the proposed Requirements Management Office, we recommended the CIO ensure the Office’s proposed oversight responsibilities address previous TIGTA report findings and consider incorporating ELC and CMMI guidance. In addition, the CIO should ensure the BSMO prepares detailed analyses of anticipated costs and resources to serve as justification for establishing and operating the Office.

Management’s Response: IRS management agreed with the report recommendations, and corrective actions are underway to address them. The CIO is reviewing the previous TIGTA reports for program-wide requirements management issues. Consideration is being given to adopting applicable CMMI capabilities in establishing the Requirements Management Office and in developing more effective requirements development and management processes. The CIO also plans to follow the required guidance and procedures for standing up a new office. To date, the CIO has a high-level plan for standing up the Requirements Management Office and has drafted mission and functional statements, built a framework for the concept of operations, and identified staffing needs for the proposed Office. The CIO is evaluating proposals for requirements development and management contractual services for developing a pilot plan and schedule. After evaluating stakeholder input to the Requirements Management Office’s concept of operations and the results of the pilot, the CIO will assess the cost and risk parameters to make changes and implement new capabilities. Management’s complete response to the draft report is included as Appendix IX.

Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

Table of Contents

Background

Continuing Requirements Development and Management Problems Encouraged the Modernization Program to Propose a Requirements Management Office

Industry Guidance and Prior Audit Results Will Assist the Proposed Requirements Management Office in Effectively Developing and Managing Project Requirements

Recommendation 1:

Detailed Cost and Resource Analyses Will Help the Proposed Requirements Management Office Determine the Extent of Services It Can Offer

Recommendation 2:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measures

Appendix V – Internal Revenue Service Modernization Projects

Appendix VI – Previously Reported Findings on Requirements Development and Management Activities

Appendix VII – Comparison of Audit Findings, Requirements Development and Management Guidelines, and Proposed Requirements Management Office Responsibility Options

Appendix VIII – Enterprise Life Cycle Overview

Appendix IX – Management’s Response to the Draft Report

Background

A requirement is a condition or capability that a user must have to solve a problem or achieve an objective. More specifically, a requirement is a formalization of a need and is the statement of a capability or condition that a system, subsystem, or system component must have or meet to satisfy a contract, standard, or specification.

Requirements management is the process that controls and documents all project requirements for the duration of the project. It involves establishing the requirements, controlling all subsequent requirements changes, and establishing and maintaining agreement among the customers and those who provide the requested products or services. Requirements management ensures requirements are unambiguous, traceable, verifiable, documented, and controlled. An effective requirements development and management process can prevent potential problems before they become serious problems that cause schedule delays and additional costs.

In the Internal Revenue Service’s (IRS) Business Systems Modernization (BSM) program, project managers work jointly with other IRS representatives and stakeholders to develop product requirements to ensure business needs are well understood. Typically a contractor, working with the IRS, is tasked to define the requirements.

A study by the Savant Institute reported poor communication between the user and analyst in defining requirements caused 56 percent of the errors in installed systems. These errors were the most expensive to correct, using 82 percent of available staff time. This study further noted over 30 percent of application development projects will be cancelled before they begin, over 50 percent of the projects will exceed their budget by 89 percent, and only 16 percent of the projects are completed on time. The primary reasons for the success and failure of projects center on requirements development and management.

The IRS is involved in a 15-year, $8 billion BSM effort and has started 13 modernization projects in the past 5 years. To date, we have reported nine of these projects have encountered varying degrees of problems associated with requirements development and management. In four projects, these problems have been severe enough to contribute to schedule delays and cost overruns. The cost of repairing requirement-related defects grows exponentially as the project progresses through its life cycle.

This audit was performed at the BSM Office (BSMO) facilities in New Carrollton, Maryland, during the period August through November 2004. The audit was conducted in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.

Continuing Requirements Development and Management Problems Encouraged the Modernization Program to Propose a Requirements Management Office

From November 2001 through November 2004, the Treasury Inspector General for Tax Administration (TIGTA) issued 16 reports identifying requirements development and management problems. We made 26 recommendations about requirements development and management in these reports. The BSMO completed corrective actions to implement 19 of these recommendations, 4 are in process, and 3 have not been acted on by management. The TIGTA disagreed with the adequacy of one of the completed corrective actions. See Appendix VI for details on the recommendations and corrective actions for the 16 reports.

The new Associate Chief Information Officer (CIO), BSM, recognized the significance of these issues and has developed a proposal for establishing a Requirements Management Office. Once established, this Office will work to resolve the requirements development and management problems affecting the BSM program. This includes providing oversight for all participants involved in modernization requirements development and management activities.

Our analysis of the 16 reports identified 2 recurring problems that will need to be addressed by the new Requirements Management Office. These problems recurred even though the BSMO addressed similar issues with corrective actions in other project activities. The recurring problems are:

· Project management did not adequately identify requirements to address customers’ needs and project development criteria (computer programming naming standards, financial reporting requirements, etc.) before initiating development activities.

· Project management did not trace all requirements to test cases to ensure the projects’ operations met expectations.

Identifying requirements prior to initiating development activities

In a November 2001 report about the BSM’s key system development processes, we reported business requirement identification was not completed by the modernization projects before entering the Development phase. We recommended the BSMO identify and incorporate business requirements meeting customer needs prior to entering the Development phase.

In July 2002, the BSMO responded that no additional corrective action was needed because the BSMO approved the Enterprise Architecture which established the high-level requirements and standards for the IRS’ business activities. Additionally, the Enterprise Life Cycle (ELC) already called for the identification of requirements early in the development of the projects.

Since November 2001, we have issued nine reports with findings about project activities not completely identifying requirements before entering the Development phase (see Appendix VI, Table 1 for details).

Tracing requirements to test cases

In March 2002, we reported testing processes did not ensure all requirements were working as intended and recommended the BSMO receive documentation from the contractor tracing the project requirements to test cases. The BSMO took no additional action to address this issue, citing a July 2001 procedure requiring the traceability of requirements to test cases.

Since March 2002, we have issued six reports with findings about problems in tracing the development and performance of modernization projects’ requirements (see Appendix VI, Table 2 for details).

The BSMO has taken corrective actions to address most of the requirements development and management findings we reported. However, the recurring issues have continued. As a result of these issues, some modernization projects have experienced scheduling delays and increased costs to resolve their requirements development and management issues.

Industry Guidance and Prior Audit Results Will Assist the Proposed Requirements Management Office in Effectively Developing and Managing Project Requirements

The BSMO’s July 2004 presentation proposing the Requirements Management Office includes three options. The three options have varying degrees of depth in proposed management and control goals, each dependent on the Fiscal Year 2005 budget allocation for this Office. All three options include using an independent requirements contractor to facilitate the requirements development and management of the modernization projects.

Table 1 presents the proposed Requirements Management Office’s responsibilities under each of the recommended options. Currently, BSMO management intends to implement Option 2.

Table 1: Proposed Requirements Management Office
Responsibility Options

Proposed Requirements Management Office Responsibilities	Option 1	Option 2	Option 3
Cofacilitate business requirements with the requirements contractor to ensure use of standard requirements development techniques.	·
Assist in the identification and integration of enterprise requirements.	·
Build the consolidated IRS requirements database.	·	·
Set up and manage a requirements development and management web site.	·	·
Provide requirements training and coaching to modernization projects.	·	·
Establish requirements standards and templates. Monitor and report on compliance and performance.	·	·	·
Establish requirements governance.	·	·	·
Establish requirements development and management responsibilities across the enterprise.	·	·	·
Manage the requirements contractor.	·	·	·
Define requirements metrics.	·	·	·

Source: BSMO Business Integration Office – BSMO Requirements Management Office – Recommendations, July 2004.

When developing its processes, the Requirements Management Office should consider addressing previous TIGTA report issues

To assess the adequacy of the plans for the Requirements Management Office, we identified ELC and Carnegie

Mellon Software Engineering Institute (SEI) guidance for requirements development and management. Both the ELC and SEI provide guidance that supports the need to address the two recurring issues we have reported.

As shown in the July 2004 presentation, the BSMO’s recommended options for the proposed Requirements Management Office do not include detailed direction to address the previously reported issues. Specifically, the options do not provide details to ensure modernization project requirements are complete prior to beginning development activities, nor do they specify processes to ensure test cases are traced to the original customer requirements. Appendix VII presents a comparison of the audit findings and related requirements development and management guidelines with the July 2004 presentation’s proposed Requirements Management Office responsibility options. We provided the BSMO an early version of this analysis to help it with start-up activities for the Office.

When developing its processes and responsibilities, the proposed Requirements Management Office should incorporate all appropriate industry guidance for developing and managing requirements

The SEI’s Capability Maturity Model Integration (CMMI) models provide guidance for an organization to use when developing its processes. These models help an organization appraise its capability, establish priorities for improvement, and implement these improvements. The CMMI models include the essential elements to guide an organization to improve its requirements development, management, and verification and validation processes.

The BSMO requested the SEI to perform a Standard CMMI Appraisal Method for Process Improvement (SCAMPI). The September 28, 2004, SCAMPI Final Findings Briefing identified acquisition program strengths of project team requirements development and management activities that included using a team approach with business representatives and other stakeholders throughout the project life cycle, developing and maintaining a requirements management plan, and using a change control board to assess the impact of changes to a system. The SCAMPI Final Findings Briefing also identified overall modernization program acquisition weaknesses affecting project requirements development and management activities, including problems in managing acquisition project schedules and resources, an absence of adequate senior management reviews, and an absence of the use of measures to assess project status.

We compared CMMI model guidelines to the three recommended options for the proposed Requirements Management Office. The July 2004 presentation proposing the Office’s responsibilities did not incorporate the following important CMMI requirements development and management guidelines:

· Develop an understanding with the requirements providers of the meaning of the requirements.

· Manage changes to the requirements as they evolve during the project.

· Identify inconsistencies between project work and requirements.

· Analyze requirements to ensure they are necessary and sufficient.

· Validate requirements to ensure the resulting product will perform appropriately in its intended-user environment.

· Place designated work products from the requirements management, development, and verification and validation processes under appropriate levels of configuration management.

· Review the activities, status, and results of the requirements management, development, and verification and validation processes with higher-level management and resolve any issues.

Incorporating all applicable requirements development and management guidelines into the oversight responsibilities for the proposed Requirements Management Office will help avoid the recurring issues that have negatively affected the BSM program and contributed to additional project costs and delays. For example, in March 2004 we reported ineffective project design coordination delayed the Modernized e-File project’s deployment. The delay occurred because the infrastructure requirements were not communicated to the application developer during the project design. During the development of the Modernized e-File application, the project team determined the modernized infrastructure could not accept the application. To deploy the application, the project team had to develop a solution to allow it to function on the infrastructure.

At the time of the report’s issuance in March 2004, a cost estimate was not available for the infrastructure changes needed to accept the Modernized e-File application. The PRIME contractor has now estimated $1.25 million was needed to pay for the changes to the infrastructure to accept the application. If the requirements had been identified prior to development activities, this rework would not have been needed and $1.25 million of BSM funding could have been put to better use. Appendix IV presents detailed information on the measurable impact that our recommended corrective action will have on tax administration.

The BSMO recognizes the need for improvements in its requirements development and management processes. It has taken steps to improve project development activities, which include requirements development and management. In August 2004, the IRS implemented an ELC directive which includes revised ELC Milestone Readiness Criteria. As of August 20, 2004, all BSMO modernization projects were expected to follow the revised ELC processes and procedures. This revised ELC places greater emphasis on agreement and ownership, establishes more rigorous milestone exit criteria, and requires completion of the physical design prior to initiation of project development.

Although the BSMO has established the above policy, the proposed Requirements Management Office has not yet completed the details of its processes to include the revised milestone readiness criteria as part of its oversight responsibilities. Further, it has not yet included all necessary requirements development and management guidance suggested by the CMMI to aid in ensuring the success of BSM project design and development.

Recommendation

As part of the ongoing development of the proposed Requirements Management Office, the CIO should ensure:

1. The Office’s proposed oversight responsibilities address the previous TIGTA report findings and consider incorporating the recently updated ELC milestone exit criteria and all appropriate CMMI guidance.

Management’s Response: The CIO is reviewing the previous TIGTA reports for program-wide requirements management issues. Consideration is being given to adopting applicable CMMI capabilities in establishing the Requirements Management Office and in developing more effective requirements development and management processes. After deciding on the approach and assessing the gaps, the CIO will assess the cost and risk parameters to make changes and implement new capabilities.

Detailed Cost and Resource Analyses Will Help the Proposed Requirements Management Office Determine the Extent of Services It Can Offer

The BSMO is in the process of defining and deciding upon the responsibilities for the Requirements Management Office. Consequently, detailed analyses estimating the costs and resources necessary for each of the proposed options have not been completed. The BSMO is currently developing a work breakdown structure which should aid in defining and clarifying the related tasks needed to establish and operate this Office.

Before the Requirements Management Office can be established and begin operating effectively, a number of key scope aspects need to be addressed, including:

· Specific responsibilities of the Office.

· Source and amount of funding for the Office.

· Staffing needs to carry out the Office’s responsibilities.

· Time period for establishing the Office.

· Schedule of which current or planned modernization projects will need requirements development and management services.

The Internal Revenue Manual (IRM) provides guidelines for establishing policy, procedures, and responsibility for organizing the mission, function and structure of the IRS. This guidance provides direction for planning and proposing the desired organizational component. The guidance suggests development of a written proposal describing and justifying the need, the proposed changes and the expected impact, and describing how the impact will be implemented and evaluated. The proposal provides the approving official all information required to make a sound, informed decision.

The IRM guidance also provides for a comprehensive reorganization proposal including elements such as the purpose of the proposed changes in terms of the business need, the mission and functional statements of the proposed organization, and current and proposed staffing. Also, the IRM suggests a list and analysis of costs, savings and benefits – particularly the costs and savings for personnel, support services (e.g., space, equipment, relocation, telecommunications, supplies), and any intangible costs and benefits.

In the development of the work breakdown structure, detailed cost and resource analyses would help the BSMO justify the funding needed to address the requirements development and management issues. In addition, these analyses would help identify when and how the proposed Requirements Management Office can perform requirements development and management activities to support the BSM projects.

Recommendation

The CIO should ensure the BSMO:

2. Prepares detailed analyses of anticipated costs and resources to serve as justification for establishing the Requirements Management Office. The analyses should include items such as the Office’s responsibilities, the source and amount of funding, the staffing needs, the time period for establishing the Office, and a schedule for providing its services to modernization project teams.

Management’s Response: The CIO will follow the required guidance and procedures for standing up a new office. To date, the CIO has a high-level plan for standing up the Requirements Management Office and has drafted mission and functional statements, built a framework for the concept of operations, and identified staffing needs for the proposed Office. The CIO is evaluating proposals for requirements development and management contractual services for developing a pilot plan and schedule. After evaluating stakeholder input to the Requirements Management Office’s concept of operations and the results of the pilot, the CIO will assess the cost and risk parameters to make changes and implement new capabilities.

Appendix I

Detailed Objective, Scope, and Methodology

The overall objective for this review was to determine whether the Business Systems Modernization Office (BSMO) has established and is following adequate requirements management practices to assure the effective development of modernization projects that meet customer needs. To accomplish this objective, we:

I. Researched the Internal Revenue Manual, the Enterprise Life Cycle, and the Carnegie Mellon Software Engineering Institute’s (SEI) Capability Maturity Model Integration (CMMI) guidance to identify applicable guidance to ensure project requirements are appropriately considered in managing modernization project development.

II. Researched prior Treasury Inspector General for Tax Administration (TIGTA) reports for previously reported requirements development and management issues.

A. Determined whether there were recurring issues identified even though there have been corrective actions reported as completed.

B. Identified potential outcomes based on the BSMO’s actions to ensure the requirements were developed and deployed.

III. Identified potential results of ongoing or recently completed audits related to requirements development and management and determined whether repeat issues were to be reported in these audits.

IV. Analyzed planned actions to improve requirements development and management to determine whether guidelines and previously reported TIGTA issues will be addressed.

A. Interviewed BSMO staff to determine the actions being taken to address requirements development and management guidelines and previously identified TIGTA issues not identified in the Requirements Management Office – Recommendations presentation, dated July 23, 2004.

B. Interviewed the Deputy Associate Chief Information Officer, Business Integration, to determine the current status for establishing the proposed Requirements Management Office including the time periods for initiating the Office and problems that might affect its implementation.

Appendix II

Major Contributors to This Report

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary Hinkle, Director

Edward A. Neuwirth, Audit Manager

Michael Garcia, Senior Auditor

Perrin Gleaton, Senior Auditor

Beverly Tamanaha, Senior Auditor

Appendix III

Report Distribution List

Commissioner C

Office of the Commissioner – Attn: Chief of Staff C

Deputy Commissioner for Operations Support OS

Associate Chief Information Officer, Business Systems Modernization OS:CIO:B

Director, Stakeholder Management Division OS:CIO:SM

Deputy Associate Chief Information Officer, Business Integration OS:CIO:B:BI

Deputy Associate Chief Information Officer, Program Management OS:CIO:B:PM
Deputy Associate Chief Information Officer, Systems Integration OS:CIO:B:SI

Chief Counsel CC

National Taxpayer Advocate TA

Director, Office of Legislative Affairs CL:LA

Director, Office of Program Evaluation and Risk Analysis RAS:O

Office of Management Controls OS:CFO:AR:M

Audit Liaisons:

Associate Chief Information Officer, Business Systems Modernization OS:CIO:B

Manager, Program Oversight Office OS:CIO:SM:PO

Appendix IV

Outcome Measures

This appendix presents detailed information on the measurable impact that our recommended corrective action will have on tax administration. This benefit will be incorporated into our Semiannual Report to the Congress.

Type and Value of Outcome Measure:

· Cost Savings, Funds Put to Better Use – Actual; $1,250,000 (see page 4).

Methodology Used to Measure the Reported Benefit:

In a prior audit report, we reported ineffective project design coordination delayed the Modernized e-File project’s deployment. The delay occurred because the infrastructure requirements were not communicated to the application developer during the project’s design. During the development of the Modernized e-File application, the project team determined the modernized infrastructure could not accept the application. To deploy the application, the project team had to develop a solution to allow it to function on the infrastructure. At the time of the report’s issuance in March 2004, a cost estimate was not available for the infrastructure changes needed to accept the Modernized e-File application. The PRIME contractor has now estimated $1.25 million was needed to pay for the changes to the infrastructure to accept the application based on actual charges during the period September through November 2003. If the requirements had been identified prior to development activities, this rework would not have been needed.

Appendix V

Internal Revenue Service Modernization Projects

Table 1 presents the modernization projects initiated by the Internal Revenue Service (IRS) in the past 5 years.

Table 1: IRS Modernization Projects

	Project	Year Initiated	Description
1.	Custodial Accounting Project/Planning, Analysis and Decision Support*	1999	Uses a data warehousing approach for storing, analyzing, and reporting taxpayer accounts and collections information.
2.	Customer Communications*	1999	Provides needed enhancements to the IRS telephone infrastructure.
3.	Customer Relationship Management Exam	1999	Provides a commercial-off-the-shelf software package that can be used by revenue agents to accurately and consistently compute complex corporate taxes.
4.	Security and Technology Infrastructure Release	1999	Provides a customer-focused technical infrastructure for secure telephony and electronic interaction among employees, tax practitioners, and taxpayers.
5.	Customer Account Data Engine*	2000	Provides an online, modernized data infrastructure which will house the authoritative taxpayer account and return data.
6.	e-Services*	2000	Focuses on revolutionizing the way taxpayers transact and communicate with the IRS.
7.	Enterprise Systems Management*	2000	Executes a strategy to provide network and systems management to improve the information technology infrastructure availability and performance.
8.	Integrated Financial System*	2001	Provides the IRS better financial budgeting, planning, tracking, reporting, and management.
9.	Internet Refund/Fact of Filing*	2001	Builds upon the improvements delivered in the Customer Communications project by providing initial Internet capability for refund information.
10.	Customer Account Management	2002	Creates the interface and redesigned business processes that will be used on a daily basis by IRS customer service representatives. Due to budget constraints, the project has not been funded since Fiscal Year 2003.
11.	Filing and Payment Compliance	2002	Improves the processes and technologies that support the IRS’ filing compliance and collection activities and managing the associated organizational change.
12.	Infrastructure Shared Services*	2002	Establishes a program to build and deliver an agile infrastructure that is scalable, interoperable, flexible, manageable, and features standardized operations and a single security and enterprise systems management framework.
13.	Modernized e-File*	2002	Develops the modernized web-based platform for filing IRS forms electronically.

Source: The IRS Business Systems Modernization online web site.

* We reported problems in developing and managing requirements for this project.

Appendix VI

Previously Reported Findings on Requirements Development and Management Activities

The following tables present the two major requirements management issues identified as continuing concerns in Treasury Inspector General for Tax Administration (TIGTA) reports issued since November 2001. Table 1 presents TIGTA report findings for which requirements identification was not complete before projects entered the development process. Table 2 presents TIGTA report findings for which project requirements were not traced throughout the project development life cycle. The tables identify the TIGTA report, the findings and recommendations, and the status and synopsis of management’s proposed corrective actions. Unless otherwise noted, the recommendations were directed to the Chief Information Officer (CIO) for corrective action.

Table 1: TIGTA Reports With Incomplete Requirements Issues

Report	Finding/Recommendation	Corrective Action
Modernization Project Teams Need to Follow Key Systems Development Processes (Reference Number 2002-20-025, dated November 2001).	Finding: Business requirement identification was not completed before entering the project development process. Recommendation: Identify and incorporate business requirements meeting customer needs prior to entering Development phase.	Management Response July 23, 2002 – No additional corrective action needed. The Business Systems Modernization Office (BSMO) approved the Enterprise Architecture which established the high-level requirements and standards for the business. Also, the Enterprise Life Cycle (ELC) already called for the identification of requirements early in the development of the projects.
Critical Processes and Dependencies Need to Be Addressed to Avoid Further Delays in Deployment of the Enterprise Systems Management Project (Reference Number 2002-20-084, dated May 2002).	Finding: The requirements management process needs to be strengthened and formalized. Recommendations: 1) Develop and implement formalized, proactive methods for assisting other modernization projects in refining their system requirements. 2) Address MITRE’s recommendations from the review of the ESM project’s System Requirements Report (SRR). The recommendations included making requirements more detailed so they can be tested.	1) Management Response April 4, 2002 – No corrective action taken. The Enterprise Systems Management (ESM) project team includes a requirements manager who proactively works with modernization projects to educate and gather ESM requirements from stakeholders. 2) Completed May 2, 2002 – MITRE’s role is to provide advice and guidance to the Internal Revenue Service (IRS) executives overseeing the Business Systems Modernization (BSM) projects. Since the audit, the IRS considered MITRE’s recommendations for ESM and incorporated them into guidance as appropriate.
Adhering to Established Development Guidelines Will Help to Ensure the Customer Account Data Engine Meets Expectations (Reference Number 2003-20-089, dated March 2003). Adhering to Established Development Guidelines Will Help to Ensure the Customer Account Data Engine Meets Expectations (Reference Number 2003-20-089, dated March 2003). (Continued)	Findings: 1) The balancing, control, and reconciliation process needs to be completed and tested prior to Release 1. 2) Release 1 entered the Development phase before all design requirements were identified. 3) File and job names need to be compatible with current tax processing systems. Recommendations: 1) Require the PRIME contractor to complete the balancing, control, and reconciliation process and fully test these processes to ensure they meet the design requirements. 2) Ensure the business system design is complete before entering the Development phase for Release 2 and future releases. 3) Ensure development of job and file naming standards is expeditiously completed by the IRS Enterprise Operations Services organization. Also, ensure these naming standards are used in the development of future Customer Account Data Engine (CADE) releases and all other IRS modernization projects.	1) Completed December 23, 2003 – Critical balancing, control, and reconciliation processes for Release 1 have been completed and tested. 2) Completed August 1, 2003 – The Release strategy has been revised to ensure the Development phase for Release 2 and future releases will not begin until the completion of system design. 3) Completed October 1, 2003 – The Modernization Information Technology Services organization’s naming standards have been approved, and the System Information Bulletins requiring the necessary programming are being implemented.
Improvements to the Modernized Infrastructure Are Needed to Support the Deployment of Business Systems Modernization Projects (Reference Number 2003-20-161, dated August 2003).	Finding: A decision to deploy was made even with significant cost increases and limitations on system performance. Recommendation: Require additional efforts to ensure performance and capacity planning are adequately addressed at an enterprise level and do not allow deployment of any BSM project without demonstration of the capability to meet performance requirements.	Completed May 12, 2004 – The IRS is developing an End-to-End Capacity and Performance process for the infrastructure projects to follow in support of application projects capacity and performance requirements.
Requirements Definition of the Integrated Financial System (Reference Number 2003-10-179, dated August 2003).	Finding: Not all Federal Government financial management system requirements addressed. Recommendations: The Chief Financial Officer (CFO) should – 1) Ensure the Joint Financial Management Improvement Program (JFMIP) requirements are fully addressed, the mandatory requirements are included in the SRR, and the value-added requirements are properly evaluated to ensure compliance with the JFMIP. 2) Take a more active role to ensure the contractor is meeting its responsibilities to deliver a JFMIP-compliant system.	1) Completed August 15, 2003 – Management will ensure the PRIME contractor includes or identifies the remaining 4 mandatory and 20 value-added requirements in the revalidated SRR during Milestone 2/3 of each of the appropriate future releases. 2) Completed August 29, 2003 – Management will respond immediately to identified gaps in requirements. Also, significant CFO resources were identified to participate in Application Quality Testing and Systems Integration Testing to ensure the system meets established requirements.
Oversight of the Business Systems Modernization Contractor Needs Improvement (Reference Number 2004-20-034, dated January 2004).	Finding: The IRS did not obtain written assurance that new computer systems will meet business requirements. Recommendations: 1) Ensure future task orders include the requirement that the PRIME contractor provide written assurance it and its major subcontractors performed adequate diligence in defining all significant business requirements and the proposed new systems will deliver all of the essential functional and operational capabilities needed by the systems’ users. 2) Conduct an analysis of future change requests to determine whether the change should have been part of the contractor’s normal requirements gathering. Task orders to define business requirements should be written to hold the PRIME contractor responsible for making such changes at no additional cost to the IRS.	1) Completed September 7, 2004 – Late last year, the CIO issued a directive requiring fixed-price contracting for all systems development and implementation projects. The new ELC Framework signed August 20, 2004, establishes a formal checkpoint, Milestone 4A, at which point mutually agreed detailed specifications are required and agreed to by the contractor by signing the firm fixed-price contract. 2) Corrective action in process – open as of November 18, 2004 – The BSMO will include contracting language that failure to identify any material requirement is the PRIME contractor’s responsibility and any work to include such a requirement is included under the agreed price.
Requirements Changes and Testing Delays Have Further Increased the Costs and Delayed the Benefits of the e-Services Project (Reference Number 2004-20-036, dated February 2004).	Finding: Previous business case estimates have been significantly inaccurate. Recommendation: Establish a more formal discipline to ensure system requirements are established prior to beginning development.	Completed September 30, 2003 – A Directive was issued to establish a Configuration Management process and procedures for monitoring change requests. Further improvement to the requirements management will be completed through actions within the Challenges Action Plan.
Modernized e-File Project Integration Difficulties Have Delayed Its Deployment (Reference Number 2004-20-072, dated March 2004).	Finding: Ineffective coordination about the project design has delayed deployment. Recommendation: Update ELC project development process to include provisions to certify the project’s physical design is in compliance with the Enterprise Architecture.	Completed August 20, 2004 – The CIO signed the revised ELC guidelines. This revised ELC requires a physical design to be completed (at Milestone 4A) prior to the commencement of development.
The Integrated Financial System Software Does Not Comply With Some Accounting Standards or Contain Certain Functionality as Originally Asserted by the Vendor (Reference Number 2004-10-187, dated September 2004). The Integrated Financial System Software Does Not Comply With Some Accounting Standards or Contain Certain Functionality as Originally Asserted by the Vendor (Reference Number 2004-10-187, dated September 2004). (Continued)	Findings: 1) Risk of noncompliance with the Federal Financial Management Improvement Act (FFMIA) of 1996. 2) Asserted out-of-the-box requirements were not met by vendor software. Recommendations: 1) The CFO should ensure all JFMIP and FFMIA requirements and accounting standards are operating before the Integrated Financial System (IFS) deployment. If this is not feasible, the IRS should move to the Federal Government version of the IFS as soon as possible, and the CFO should ensure these requirements are included in that version. 2) The CFO, in concert with the Director, Procurement, and the Associate CIO, BSM, should ensure a coordinated and united stance is taken when conducting any negotiations with the IFS contractor concerning the cost associated with the functionality of accounting requirements that were asserted to be ready for deployment by the software subcontractor. 3) The CFO should ensure all costs are considered when any negotiations are held concerning system requirement deferrals.	1) Corrective action in process – open as of November 18, 2004 – The CFO will ensure those requirements not satisfied in Release 1 are implemented in subsequent enhancements or upgrades. Upon successful implementation of the IFS, Release 1, the CFO will develop an action plan to address enhancements and future releases of the IFS–especially the upgrade to the Federalized version of the software which should close many of the existing JFMIP gaps. 2) Completed September 30, 2004 – The CFO has implemented the recommendation to coordinate the IFS contract negotiations with the Director, Procurement, and the Associate CIO, BSM. 3) Completed September 30, 2004 – The IRS has worked together to ensure all costs associated with requirements that were asserted to be ready for deployment by the software subcontractor, but not delivered in Release 1, were included in the negotiations.
To Ensure the Customer Account Data Engine’s Success, Prescribed Management Practices Need to Be Followed (Reference Number 2005-20-005, dated November 2004). To Ensure the Customer Account Data Engine’s Success, Prescribed Management Practices Need to Be Followed (Reference Number 2005-20-005, dated November 2004). (Continued)	Findings: 1) Manual processes within the CADE Release 1.1 need to be automated for future releases. 2) The CADE Program does not have a dedicated system architect. Recommendations: 1) Ensure inefficient manual processes, including the processes cited above, are automated in future CADE releases. 2) Ensure the BSMO makes the system architecture resources being acquired available to the CADE program on a full-time basis.	1) Corrective action in process – open as of November 22, 2004 – The CIO plans a number of specific changes for the January 2005 and July 2005 CADE release deliveries to address known operational inefficiencies. 2) Corrective action in process – open as of November 22, 2004 – The CIO has been recruiting for the last year to find qualified engineering resources, some of which will be dedicated to the CADE project. Until the additional staffing is on the rolls, the BSMO is providing the CADE engineering/architecture support from the existing Enterprise Architecture team and MITRE.

Report

Finding/Recommendation

Corrective Action

Modernization Project Teams Need to Follow Key Systems Development Processes
(Reference Number
2002-20-025, dated November 2001).

Finding: Business requirement identification was not completed before entering the project development process.

Recommendation: Identify and incorporate business requirements meeting customer needs prior to entering Development phase.

Management Response July 23, 2002 – No additional corrective action needed. The Business Systems Modernization Office (BSMO) approved the Enterprise Architecture which established the high-level requirements and standards for the business. Also, the Enterprise Life Cycle (ELC) already called for the identification of requirements early in the development of the projects.

Critical Processes and Dependencies Need to Be Addressed to Avoid Further Delays in Deployment of the Enterprise Systems Management Project (Reference Number
2002-20-084, dated
May 2002).

Finding: The requirements management process needs to be strengthened and formalized.

Recommendations:

1) Develop and implement formalized, proactive methods for assisting other modernization projects in refining their system requirements.

2) Address MITRE’s recommendations from the review of the ESM project’s System Requirements Report (SRR). The recommendations included making requirements more detailed so they can be tested.

1) Management Response April 4, 2002 – No corrective action taken. The Enterprise Systems Management (ESM) project team includes a requirements manager who proactively works with modernization projects to educate and gather ESM requirements from stakeholders.

2) Completed May 2, 2002 – MITRE’s role is to provide advice and guidance to the Internal Revenue Service (IRS) executives overseeing the Business Systems Modernization (BSM) projects. Since the audit, the IRS considered MITRE’s recommendations for ESM and incorporated them into guidance as appropriate.

Adhering to Established Development Guidelines Will Help to Ensure the Customer Account Data Engine Meets Expectations (Reference Number
2003-20-089, dated
March 2003).

Adhering to Established Development Guidelines Will Help to Ensure the Customer Account Data Engine Meets Expectations (Reference Number
2003-20-089, dated
March 2003). (Continued)

Findings:

1) The balancing, control, and reconciliation process needs to be completed and tested prior to Release 1.

2) Release 1 entered the Development phase before all design requirements were identified.

3) File and job names need to be compatible with current tax processing systems.

Recommendations:

1) Require the PRIME contractor to complete the balancing, control, and reconciliation process and fully test these processes to ensure they meet the design requirements.

2) Ensure the business system design is complete before entering the Development phase for Release 2 and future releases.

3) Ensure development of job and file naming standards is expeditiously completed by the IRS Enterprise Operations Services organization. Also, ensure these naming standards are used in the development of future Customer Account Data Engine (CADE) releases and all other IRS modernization projects.

1) Completed
December 23, 2003 – Critical balancing, control, and reconciliation processes for Release 1 have been completed and tested.

2) Completed
August 1, 2003 – The Release strategy has been revised to ensure the Development phase for Release 2 and future releases will not begin until the completion of system design.

3) Completed
October 1, 2003 – The Modernization Information Technology Services organization’s naming standards have been approved, and the System Information Bulletins requiring the necessary programming are being implemented.

Improvements to the Modernized Infrastructure Are Needed to Support the Deployment of Business Systems Modernization Projects (Reference Number 2003-20-161, dated August 2003).

Finding: A decision to deploy was made even with significant cost increases and limitations on system performance.

Recommendation: Require additional efforts to ensure performance and capacity planning are adequately addressed at an enterprise level and do not allow deployment of any BSM project without demonstration of the capability to meet performance requirements.

Completed May 12, 2004 – The IRS is developing an End-to-End Capacity and Performance process for the infrastructure projects to follow in support of application projects capacity and performance requirements.

Requirements Definition of the Integrated Financial System (Reference Number 2003-10-179, dated August 2003).

Finding: Not all Federal Government financial management system requirements addressed.

Recommendations: The Chief Financial Officer (CFO) should –

1) Ensure the Joint Financial Management Improvement Program (JFMIP) requirements are fully addressed, the mandatory requirements are included in the SRR, and the value-added requirements are properly evaluated to ensure compliance with the JFMIP.

2) Take a more active role to ensure the contractor is meeting its responsibilities to deliver a JFMIP-compliant system.

1) Completed
August 15, 2003 – Management will ensure the PRIME contractor includes or identifies the remaining 4 mandatory and 20 value-added requirements in the revalidated SRR during Milestone 2/3 of each of the appropriate future releases.

2) Completed
August 29, 2003 – Management will respond immediately to identified gaps in requirements. Also, significant CFO resources were identified to participate in Application Quality Testing and Systems Integration Testing to ensure the system meets established requirements.

Oversight of the Business Systems Modernization Contractor Needs Improvement (Reference Number 2004-20-034, dated January 2004).

Finding: The IRS did not obtain written assurance that new computer systems will meet business requirements.

Recommendations:
1) Ensure future task orders include the requirement that the PRIME contractor provide written assurance it and its major subcontractors performed adequate diligence in defining all significant business requirements and the proposed new systems will deliver all of the essential functional and operational capabilities needed by the systems’ users.

2) Conduct an analysis of future change requests to determine whether the change should have been part of the contractor’s normal requirements gathering. Task orders to define business requirements should be written to hold the PRIME contractor responsible for making such changes at no additional cost to the IRS.

1) Completed
September 7, 2004 – Late last year, the CIO issued a directive requiring fixed-price contracting for all systems development and implementation projects. The new ELC Framework signed August 20, 2004, establishes a formal checkpoint, Milestone 4A, at which point mutually agreed detailed specifications are required and agreed to by the contractor by signing the firm fixed-price contract.

2) Corrective action in process – open as of November 18, 2004 – The BSMO will include contracting language that failure to identify any material requirement is the PRIME contractor’s responsibility and any work to include such a requirement is included under the agreed price.

Requirements Changes and Testing Delays Have Further Increased the Costs and Delayed the Benefits of the e-Services Project (Reference Number 2004-20-036, dated February 2004).

Finding: Previous business case estimates have been significantly inaccurate.

Recommendation: Establish a more formal discipline to ensure system requirements are established prior to beginning development.

Completed
September 30, 2003 – A Directive was issued to establish a Configuration Management process and procedures for monitoring change requests. Further improvement to the requirements management will be completed through actions within the Challenges Action Plan.

Modernized e-File Project Integration Difficulties Have Delayed Its Deployment (Reference Number 2004-20-072, dated March 2004).

Finding: Ineffective coordination about the project design has delayed deployment.

Recommendation: Update ELC project development process to include provisions to certify the project’s physical design is in compliance with the Enterprise Architecture.

Completed
August 20, 2004 – The CIO signed the revised ELC guidelines. This revised ELC requires a physical design to be completed (at Milestone 4A) prior to the commencement of development.

The Integrated Financial System Software Does Not Comply With Some Accounting Standards or Contain Certain Functionality as Originally Asserted by the Vendor (Reference Number 2004-10-187, dated September 2004).

Findings:

1) Risk of noncompliance with the Federal Financial Management Improvement Act (FFMIA) of 1996.

2) Asserted out-of-the-box requirements were not met by vendor software.

Recommendations:

1) The CFO should ensure all JFMIP and FFMIA requirements and accounting standards are operating before the Integrated Financial System (IFS) deployment. If this is not feasible, the IRS should move to the Federal Government version of the IFS as soon as possible, and the CFO should ensure these requirements are included in that version.

2) The CFO, in concert with the Director, Procurement, and the Associate CIO, BSM, should ensure a coordinated and united stance is taken when conducting any negotiations with the IFS contractor concerning the cost associated with the functionality of accounting requirements that were asserted to be ready for deployment by the software subcontractor.

3) The CFO should ensure all costs are considered when any negotiations are held concerning system requirement deferrals.

1) Corrective action
in process – open as of November 18, 2004 – The CFO will ensure those requirements not satisfied in Release 1 are implemented in subsequent enhancements or upgrades. Upon successful implementation of the IFS, Release 1, the CFO will develop an action plan to address enhancements and future releases of the IFS–especially the upgrade to the Federalized version of the software which should close many of the existing JFMIP gaps.

2) Completed
September 30, 2004 – The CFO has implemented the recommendation to coordinate the IFS contract negotiations with the Director, Procurement, and the Associate CIO, BSM.

3) Completed
September 30, 2004 – The IRS has worked together to ensure all costs associated with requirements that were asserted to be ready for deployment by the software subcontractor, but not delivered in Release 1, were included in the negotiations.

To Ensure the Customer Account Data Engine’s Success, Prescribed Management Practices Need to Be Followed (Reference Number
2005-20-005, dated November 2004).

To Ensure the Customer Account Data Engine’s Success, Prescribed Management Practices Need to Be Followed (Reference Number
2005-20-005, dated November 2004). (Continued)

Findings:

1) Manual processes within the CADE Release 1.1 need to be automated for future releases.

2) The CADE Program does not have a dedicated system architect.

Recommendations:

1) Ensure inefficient manual processes, including the processes cited above, are automated in future CADE releases.

2) Ensure the BSMO makes the system architecture resources being acquired available to the CADE program on a full-time basis.

1) Corrective action in process – open as of
November 22, 2004 – The CIO plans a number of specific changes for the January 2005 and
July 2005 CADE release deliveries to address known operational inefficiencies.

2) Corrective action in process – open as of
November 22, 2004 – The CIO has been recruiting for the last year to find qualified engineering resources, some of which will be dedicated to the CADE project. Until the additional staffing is
on the rolls, the BSMO is providing the CADE engineering/architecture support from the existing Enterprise Architecture team and MITRE.

Source: TIGTA audit reports.

Table 2: TIGTA Reports With Requirements Traceability Issues

Report	Finding/Recommendation	Corrective Action
The Customer Communications Project 2001 Release Was Deployed, But Testing Processes Did Not Ensure All Applications Were Working As Intended (Reference Number 2002-20-056, dated March 2002).	Finding: Testing processes did not ensure all capabilities were working as intended. Recommendation: Perform reviews to ensure the BSMO receives documentation from the PRIME contractor showing project system requirements are traced to use cases, test cases, and test procedures.	Completed March 1, 2002 – On July 13, 2001, the PRIME contractor issued its Program Validation and Verification Plan that requires all test plans to include a requirements traceability matrix that maps all requirements to the test case and test phase. The IRS Product Assurance organization will conduct reviews of the requirements traceability matrices. Office of Audit Comment: Management’s response shows this action was completed in July 2001. We question whether that date is accurate, since we raised this issue after that date and were not provided information regarding corrective actions. If this corrective action was taken before we identified the issue, then we must conclude that the actions taken were either incomplete or ineffective.
Enhancements to the Internet Refund Project Need to Be Completed to Ensure Planned Benefits to Taxpayers Are Realized (Reference Number 2003-20-053, dated February 2003).	Finding: The Internet Refund Fact of Filing Project is not complete, has not provided all expected benefits, and has exceeded planned costs. Recommendation: Recommendations for corrective actions to address the issues were reported in previous TIGTA reports.	Not Applicable
Risks Are Mounting as the Integrated Financial System Project Team Strives to Meet an Aggressive Implementation Date (Reference Number 2004-20-001, dated October 2003).	Finding: Project testing practices can be improved. The Application Qualification Testing requirements traceability verification matrix (RTVM) was not complete. Recommendation: Ensure System Integration and Test practices are strengthened based on lessons learned during the initial Application Qualification Testing.	Completed April 30, 2004 – The Acquisition Project Manager performed an independent verification of the RTVM.
Requirements Changes and Testing Delays Have Further Increased the Costs and Delayed the Benefits of the e-Services Project (Reference Number 2004-20-036, dated February 2004)	Finding: Requirements changes were not easily traceable. Recommendation: Ensure the e-Services requirements database clearly reflects when a requirement is changed and which approval change request documents the change. Additionally, the change request document should indicate specific requirements affected.	Completed April 1, 2004 – Management prepared a change request to add additional fields to the requirements database to ensure clear traceability and level of approval indicator.
Improvements Are Needed for Subsequent Integrated Financial System Testing (Reference Number 2004-10-052, dated March 2004).	Finding: Some test cases and scripts were incomplete or inaccurate, and some system requirements could not be easily traced. Recommendation: The CFO should ensure subsequent System Integration and Test plans, cases, and scripts are complete and accurate and all applicable financial system requirements can be readily accounted for during the testing process.	Completed October 21, 2003 – The IRS created and is maintaining a current list of all requirements and has verified all requirements are being tested adequately. Completed October 1, 2003 – The RTVM has been reviewed and validated by the IRS to ensure all requirements have been properly mapped to test plans. Completed March 1, 2004 – The RTVM is reviewed weekly by the IRS to ensure all requirement change requests and any other changes have been properly reflected in the RTVM and documented. Completed March 1, 2004 – During testing, the IRS subject matter expert is responsible for reverifying that the requirements are mapped properly and have been met by the functionality tested in the test plan.
The Integrated Financial System Project Team Needs to Resolve Transition Planning and Testing Issues to Increase the Chances of a Successful Deployment (Reference Number 2004-20-147, dated August 2004). The Integrated Financial System Project Team Needs to Resolve Transition Planning and Testing Issues to Increase the Chances of a Successful Deployment (Reference Number 2004-20-147, dated August 2004). (Continued)	Finding: Testing practices continue to need improvement. 1) The RTVM and related documentation were incomplete. 2) Deviation forms did not include all signatures. 3) Some required security features are not being tested. Recommendations: 1) The RTVM should be updated to include a verification and validation method for all requirements. 2) Deviation forms should be updated to include a signature line for all required approving officials. 3) Trusted recovery and object reuse testing should occur or the risk of not conducting these tests should be documented.	1) Completed August 31, 2004 – The new ELC framework, signed by the CIO on August 20, 2004, requires updates to the RTVM beginning at Milestone 2 and updates at each successive milestone, as a condition of milestone exit. 2) Completed August 26, 2004 – The ELC requires the Program Office Director’s signature for test case waivers and deferrals. This signature line is included in all but the earliest waiver forms. 3) Rejected August 26, 2004 – The IRS is no longer required by the Department of the Treasury to test for object reuse capabilities. Office of Audit Comment: While the Department of the Treasury requirement may have changed, the Internal Revenue Manual still requires object reuse testing. Therefore, we believe the IRS should conduct the required testing, particularly since it plans to perform object reuse testing on one computer platform and not the other. In addition, the IRS’ response did not address trusted recovery testing.
System Requirements Were Not Adequately Managed During the Testing of the Custodial Accounting Project (Reference Number 2005-20-019, dated December 2004).	Finding: System requirements were not adequately managed during testing. Recommendation: Ensure the IRS and contractor implement appropriate requirements management practices, as required by the ELC, to adequately define, track, and report on system requirements for testing and delivery of future releases.	Completed June 30, 2004 – The CIO responded the corrective action has been completed with a strengthened requirements management process to ensure system requirements are defined, tracked, and reported.

Report

Finding/Recommendation

Corrective Action

The Customer Communications Project 2001 Release Was Deployed, But Testing Processes Did Not Ensure All Applications Were Working As Intended (Reference Number
2002-20-056, dated
March 2002).

Finding: Testing processes did not ensure all capabilities were working as intended.

Recommendation: Perform reviews to ensure the BSMO receives documentation from the PRIME contractor showing project system requirements are traced to use cases, test cases, and test procedures.

Completed March 1, 2002 – On July 13, 2001, the PRIME contractor issued its Program Validation and Verification Plan that requires all test plans to include a requirements traceability matrix that maps all requirements to the test case and test phase. The IRS Product Assurance organization will conduct reviews of the requirements traceability matrices.

Office of Audit Comment: Management’s response shows this action was completed in July 2001. We question whether that date is accurate, since we raised this issue after that date and were not provided information regarding corrective actions. If this corrective action was taken before we identified the issue, then we must conclude that the actions taken were either incomplete or ineffective.

Enhancements to the Internet Refund Project Need to Be Completed to Ensure Planned Benefits to Taxpayers Are Realized (Reference Number
2003-20-053, dated February 2003).

Finding: The Internet Refund Fact of Filing Project is not complete, has not provided all expected benefits, and has exceeded planned costs.

Recommendation: Recommendations for corrective actions to address the issues were reported in previous TIGTA reports.

Not Applicable

Risks Are Mounting as the Integrated Financial System Project Team Strives to Meet an Aggressive Implementation Date (Reference Number 2004-20-001, dated October 2003).

Finding: Project testing practices can be improved. The Application Qualification Testing requirements traceability verification matrix (RTVM) was not complete.

Recommendation: Ensure System Integration and Test practices are strengthened based on lessons learned during the initial Application Qualification Testing.

Completed April 30, 2004 – The Acquisition Project Manager performed an independent verification of the RTVM.

Requirements Changes and Testing Delays Have Further Increased the Costs and Delayed the Benefits of the e-Services Project (Reference Number 2004-20-036, dated February 2004)

Finding: Requirements changes were not easily traceable.

Recommendation: Ensure the e-Services requirements database clearly reflects when a requirement is changed and which approval change request documents the change. Additionally, the change request document should indicate specific requirements affected.

Completed April 1, 2004 – Management prepared a change request to add additional fields to the requirements database to ensure clear traceability and level of approval indicator.

Improvements Are Needed for Subsequent Integrated Financial System Testing (Reference Number
2004-10-052, dated
March 2004).

Finding: Some test cases and scripts were incomplete or inaccurate, and some system requirements could not be easily traced.

Recommendation: The CFO should ensure subsequent System Integration and Test plans, cases, and scripts are complete and accurate and all applicable financial system requirements can be readily accounted for during the testing process.

Completed
October 21, 2003 – The IRS created and is maintaining a current list of all requirements and has verified all requirements are being tested adequately.

Completed
October 1, 2003 – The RTVM has been reviewed and validated by the IRS to ensure all requirements have been properly mapped to test plans.

Completed March 1, 2004 – The RTVM is reviewed weekly by the IRS to ensure all requirement change requests and any other changes have been properly reflected in the RTVM and documented.

Completed March 1, 2004 – During testing, the IRS subject matter expert is responsible for reverifying that the requirements are mapped properly and have been met by the functionality tested in the test plan.

The Integrated Financial System Project Team Needs to Resolve Transition Planning and Testing Issues to Increase the Chances of a Successful Deployment (Reference Number
2004-20-147, dated
August 2004).

Finding: Testing practices continue to need improvement.

1) The RTVM and related documentation were incomplete.

2) Deviation forms did not include all signatures.

3) Some required security features are not being tested.

Recommendations:

1) The RTVM should be updated to include a verification and validation method for all requirements.

2) Deviation forms should be updated to include a signature line for all required approving officials.

3) Trusted recovery and object reuse testing should occur or the risk of not conducting these tests should be documented.

1) Completed
August 31, 2004 – The new ELC framework, signed by the CIO on August 20, 2004, requires updates to the RTVM beginning at Milestone 2 and updates at each successive milestone, as a condition of milestone exit.

2) Completed
August 26, 2004 – The ELC requires the Program Office Director’s signature for test case waivers and deferrals. This signature line is included in all but the earliest waiver forms.

3) Rejected
August 26, 2004 – The IRS is no longer required by the Department of the Treasury to test for object reuse capabilities.

Office of Audit Comment: While the Department of the Treasury requirement may have changed, the Internal Revenue Manual still requires object reuse testing. Therefore, we believe the IRS should conduct the required testing, particularly since it plans to perform object reuse testing on one computer platform and not the other. In addition, the IRS’ response did not address trusted recovery testing.

System Requirements Were Not Adequately Managed During the Testing of the Custodial Accounting Project (Reference Number 2005-20-019, dated December 2004).

Finding: System requirements were not adequately managed during testing.

Recommendation: Ensure the IRS and contractor implement appropriate requirements management practices, as required by the ELC, to adequately define, track, and report on system requirements for testing and delivery of future releases.

Completed June 30, 2004 – The CIO responded the corrective action has been completed with a strengthened requirements management process to ensure system requirements are defined, tracked, and reported.

Source: TIGTA audit reports.

Appendix VII

Comparison of Audit Findings, Requirements Development and Management Guidelines, and Proposed Requirements Management Office Responsibility Options

The following tables present a comparison of the audit findings reported by the Treasury Inspector General for Tax Administration (TIGTA) with associated requirements development and management guidelines and the proposed Business Systems Modernization (BSM) Office Requirements Management Office responsibility options related to the TIGTA report findings.

Table 1: Analysis of Findings and Guidelines to Complete Requirements Prior to Development

Finding/Guideline	Source	Addressed in Option			Requirements Management Office’s Responsibilities
Finding/Guideline	Source	1	2	3	Requirements Management Office’s Responsibilities
Requirements need to be completed before development begins.	TIGTA Reports: Key Processes 11/01; Enterprise Systems Management 5/02; Customer Account Data Engine 3/03; Oversight of BSM Contractor 1/04; e-Services 2/04	X			Cofacilitate business requirements with the requirements contractor to ensure use of standard requirements development techniques.
Requirements need to meet the Internal Revenue Service and Federal Government standards for information systems.	TIGTA Reports: Customer Account Data Engine, 3/03; Integrated Financial System 8/03; Modernized Infrastructure 8/03; Modernized e-File 3/04; Integrated Financial System 9/04	X			1) Cofacilitate business requirements with the requirements contractor to ensure use of standard requirements development techniques. 2) Assist in the identification and integration of enterprise requirements.
Obtain commitment to requirements.	CMMI-SE-SW-IPPD-SS pgs 86, 87 (Maturity Level 2) and BSMO-PD Req D&M Sect 5	X			Cofacilitate business requirements with the requirements contractor to ensure use of standard requirements development techniques.
Complete understanding of customers’ needs and requirements.	CMMI-SE-SW-IPPD-SS pgs 206, 210, 213-214, and 216-217 (Maturity Level 3)	X			1) Cofacilitate business requirements with the requirements contractor to ensure use of standard requirements development techniques. 2) Assist in the identification and integration of enterprise requirements.
Identify and involve the relevant stakeholders.	CMMI-SE-SW-IPPD-SS pg 91 (Maturity Level 2) and BSMO-PD Req D&M				None
Eliciting requirements needs an approach to proactively identify requirements not explicitly provided by customers.	CMMI-SE-SW-IPPD-SS pg 210 (Maturity Level 3)				None

Source: TIGTA audit reports.

Table 2: Analysis of Findings and Guidelines to Allow Project Requirements Traceability With Product Development Activities

Finding/Guideline	Source	Addressed in Option			Requirements Management Office's Responsibilities
Finding/Guideline	Source	1	2	3	Requirements Management Office's Responsibilities
Requirements traceability needs to be verified.	TIGTA Reports: Customer Communications, 3/02; Internet Refund Fact of Filing, 2/03; Integrated Financial System, 10/03; e-Services, 2/04; and Integrated Financial System, 3/04 and 8/04				None
Maintain bidirectional traceability of requirements and confirm the requirements are testable.	CMMI-SE-SW-IPPD-SS pg 88 (Maturity Level 2) and BSMO-PD Req D&M Sect 7.1				None
Validation demonstrates the product, as provided, will fulfill its intended use; whereas, verification addresses whether the work product properly reflects the specified requirements.	CMMI-SE-SW-IPPD-SS pg 282 (Maturity Level 3)				None

Source: TIGTA audit reports.

Appendix VIII

Enterprise Life Cycle Overview

The Enterprise Life Cycle (ELC) defines the processes, products, techniques, roles, responsibilities, policies, procedures, and standards associated with planning, executing, and managing business change. It includes redesign of business processes; transformation of the organization; and development, integration, deployment, and maintenance of the related information technology applications and infrastructure. Its immediate focus is the Internal Revenue Service (IRS) Business Systems Modernization (BSM) program. Both the IRS and the PRIME contractor must follow the ELC in developing/acquiring business solutions for modernization projects.

The ELC framework is a flexible and adaptable structure within which one plans, executes, and integrates business change. The ELC process layer was created principally from the Computer Sciences Corporation’s Catalyst^® methodology. It is intended to improve the acquisition, use, and management of information technology within the IRS; facilitate management of large-scale business change; and enhance the methods of decision making and information sharing. Other components and extensions were added as needed to meet the specific needs of the IRS BSM program.

ELC Processes

A process is an ordered, interdependent set of activities established to accomplish a specific purpose. Processes help to define what work needs to be performed. The ELC methodology includes two major groups of processes:

1. Life-Cycle Processes, which are organized into phases and subphases and address all domains of business change.

2. Management Processes, which are organized into management areas and operate across the entire life cycle.

Enterprise Life-Cycle Processes

The chart was removed due to its size. To see the chart, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

Life-Cycle Processes

The life-cycle processes of the ELC are divided into six phases, as described below:

· Vision and Strategy – This phase establishes the overall direction and priorities for business change for the enterprise. It also identifies and prioritizes the business or system areas for further analysis.

· Architecture – This phase establishes the concept/vision, requirements, and design for a particular business area or target system. It also defines the releases for the business area or system.

· Development – This phase includes the analysis, design, acquisition, modification, construction, and testing of the components of a business solution. This phase also includes routine planned maintenance of applications.

· Integration – This phase includes the integration, testing, piloting, and acceptance of a release. In this phase, the integration team brings together individual work packages of solution components developed or acquired separately during the Development phase. Application and technical infrastructure components are tested to determine whether they interact properly. If appropriate, the team conducts a pilot to ensure all elements of the business solution work together.

· Deployment – This phase includes preparation of a release for deployment and actual deployment of the release to the deployment sites. During this phase, the deployment team puts the solution release into operation at target sites.

· Operations and Support – This phase addresses the ongoing operations and support of the system. It begins after the business processes and system(s) have been installed and have begun performing business functions. It encompasses all of the operations and support processes necessary to deliver the services associated with managing all or part of a computing environment.

The Operations and Support phase includes the scheduled activities, such as planned maintenance, systems backup, and production output, as well as the nonscheduled activities, such as problem resolution and service request delivery, including emergency unplanned maintenance of applications. It also includes the support processes required to keep the system up and running at the contractually specified level.

Management Processes

Besides the life-cycle processes, the ELC also addresses the various management areas at the process level. The management areas include:

Background

Continuing Requirements Development and Management Problems Encouraged the Modernization Program to Propose a Requirements Management Office

Industry Guidance and Prior Audit Results Will Assist the Proposed Requirements Management Office in Effectively Developing and Managing Project Requirements

Recommendation

Detailed Cost and Resource Analyses Will Help the Proposed Requirements Management Office Determine the Extent of Services It Can Offer

Recommendation

Report Distribution List

ELC Processes

Life-Cycle Processes

Management Processes

Milestones