The Business Systems Development Organization’s Effective Process for Developing Information Systems Requirements Can Be Made More Efficient by Tracking and Analyzing Related Costs

 

May 2005

 

Reference Number:  2005-20-061

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

May 9, 2005

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:     Pamela J. Gardiner /s/ Pamela J. Gardiner

                 Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - The Business Systems Development Organization’s Effective Process for Developing Information Systems Requirements Can Be Made More Efficient by Tracking and Analyzing Related Costs (Audit # 200420040)

 

This report presents the results of our review of the Business Systems Development (BSD) organization’s efforts to effectively develop and manage the requirements for information services.  This review is part of our Fiscal Year (FY) 2005 Audit Plan for reviews to assess the adequacy of the Internal Revenue Service’s (IRS) information technology.

In summary, the BSD organization receives information services requests from the IRS’ business operating divisions and functional organizations.  These requests are usually received through the Request for Information Services (RIS) process.  The RIS process provides a common framework to document, control, monitor, and track requirement changes to IRS computer systems and requests for information technology support.

Requirements management is the process that controls and documents all project requirements.  It involves establishing the requirements, controlling all subsequent requirements changes, and maintaining agreement with the customers and providers of the requested products or services.  This process ensures requirements are unambiguous, traceable, verifiable, documented, and controlled.  An effective requirements development and management process can prevent potential problems before they become serious problems resulting in schedule delays and additional costs.

Our reviews of the RIS process and a sample of RISs submitted during FY 2004 identified the use of open and ongoing communications between the BSD organization and its IRS customers that enable them to sufficiently define RIS requirements.  Customers submitting requirements for RISs identify the business developments or changes they want to install.  The BSD organization assists by determining the information system requirements to meet the customers’ needs.

We analyzed the 1,183 original RISs submitted to the BSD organization during FY 2004.  Overall, 98 percent of the RISs included appropriate requirements when the customer submitted them to the BSD organization.  Subsequent to the RIS submission and approval, 13 percent needed further requirements development through a RIS amendment.  These amendments modified the existing requirements or added additional requirements as part of the RIS development process.  In our opinion, the amendment activity is relatively low and indicates the RIS process and the communications between the BSD organization and customers work to adequately develop the requirements.

Although the BSD organization and its customers work to adequately develop the requirements for information systems services, we found the BSD organization does not have the functionality in the cost accounting system or the RIS Tracking and Reporting System (RTRS) to compare the estimated and actual staff days for time spent developing, tracking, and testing requirements.  The Internal Revenue Manual requires the actual results and performance of software development projects be tracked against the software development plan.  The Carnegie Mellon Software Engineering Institute’s (SEI) Capability Maturity Model Integration (CMMI) states the project planning process should be monitored and controlled against the plan to include specific measures to address the estimated and actual effort and cost.  Project planning should include the ability to reconcile the project plan to reflect available and estimated resources.

Without measures to compare the estimated costs to the actual costs for requirements development, the BSD organization may not be able to effectively assign resources to support the future use and improvement of the organization’s processes and process assets.  To ensure the overall success of the requirements management process, we recommended the Chief Information Officer (CIO) have the BSD organization develop policies and procedures for requirements management and measurement. 

Management’s Response:  The CIO agreed that the BSD organization can further improve its efficiency by tracking costs for requirements development and management.  Since the audit fieldwork was completed, the CIO formed the Unified Work Request (UWR) initiative under the direction of the Associate CIO of Enterprise Services.  The UWR Working Group was formed on March 17, 2005.  It is tasked to develop a comprehensive process or set of processes, procedures, and system changes to be implemented enterprise-wide by combining or modifying the functionality of the existing work requests systems into a unified entity that will enable effective and efficient prioritization and coordination of all work requested of the Modernization and Information Technology Services (MITS) organization.  Further, the BSD organization will work with the MITS Management Information and Cost Analysis System project to correctly identify and enumerate all cost areas in the BSD organization.   Management’s complete response to the draft report is included as Appendix V.   

Copies of this report are also being sent to the IRS managers affected by the report recommendation.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

Table of Contents

Background

The Business Systems Development Organization Has an Effective Process to Develop Requirements for Information Services Requests

The Business Systems Development Organization Can Improve Its Efficiency by Tracking Costs for Requirements Development and Management

Recommendation 1:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Staff Day Estimates for the Request for Information Services Sample

Appendix V – Management’s Response to the Draft Report

 

Background

One of the Internal Revenue Service’s (IRS) major strategies contained in the IRS Strategic Plan for Fiscal Years (FY) 2000-2005 is to provide high-quality, efficient, and responsive information services.  This strategy plans continuing support for current operations with emphasis on increased quality and reduced costs of routine operations and continued support to the Business Systems Modernization program.  The Business Systems Development (BSD) organization defines, builds, tests, delivers, and maintains the IRS’ information systems.  The BSD organization’s work supports the Modernization and Information Technology Services (MITS) organization production environment in achieving the IRS’ business vision and objectives.

The BSD organization is one of the largest functions in the MITS organization in terms of both size and resources.  For FY 2004, the BSD organization had a budget of $336 million and 1,916 Full-Time Equivalent (FTE) positions.  The BSD organization’s budget for FY 2005 is $373 million and 2,175 FTE positions.  The BSD organization accomplishes its work with eight subordinate organizations:

·         Product Assurance Division.

·         Program Management and Release Readiness Office.

·         Compliance Systems Division.

·         Corporate Data and Systems Management Division.

·         Internal Management Systems Division.

·         Customer Applications Development Division.

·         Filing Systems Division.

·         Client Services Division.

The BSD organization receives information services requests from the IRS’ business operating divisions and functional organizations.  These requests are usually received through the Request for Information Services (RIS) process.  A RIS is a formal memorandum requesting BSD organization support for changes to current or planned programming, corporate hardware, commercial off-the-shelf software applications, system testing, and other MITS organization activities used in processing tax information.  The RIS process provides a common framework to document, control, monitor, and track requirement changes to IRS computer systems and requests for information technology support.

Requirements management is the process that controls and documents all project requirements.  It involves establishing the requirements, controlling all subsequent requirements changes, and maintaining agreement with the customers and providers of the requested products or services.  This process ensures requirements are unambiguous, traceable, verifiable, documented, and controlled.  An effective requirements development and management process can prevent potential problems before they become serious problems resulting in schedule delays and additional costs.

This review was performed at the BSD organization offices in New Carrollton, Maryland, during the period October 2004 through February 2005.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

The Business Systems Development Organization Has an Effective Process to Develop Requirements for Information Services Requests

The RIS process is initiated when a customer prepares and submits a Placeholder to the BSD organization to request services or support.  The primary BSD organization contact for the Placeholder initiates precoordination meetings with the customers and appropriate BSD organization personnel.  The Placeholder precoordination participants meet to get an early indication of the BSD organization’s ability to provide the requested support and/or service requirements.  The meeting results help the customer to determine whether to submit a RIS.

If the Placeholder precoordination meeting results indicate the BSD organization has the ability to provide the type of support and/or services requested, the BSD organization and the customer work together to define the initial RIS requirements.  Once the initial requirements are defined, the customer submits a RIS.  After the RIS is received, the primary BSD organization analyst must contact the customer to continue coordination in order to review, analyze, and agree upon the specific requirements outlined in the RIS.  The objective of RIS coordination is to determine whether the agreed upon requirements can be provided and to address any requirements/issues not resolved in Placeholder precoordination.

When all organizations involved in developing the RIS understand the requirements and agree to do the work, the BSD organization contact prepares a formal RIS response documenting the work to be performed.  An amended RIS is the vehicle to add or modify requirements to an existing RIS.  The work requested in an amended RIS must be integral to the work requested in the original RIS.  RIS amendments must be submitted through the regular  RIS process. 

In most cases, communications between the BSD organization and the customer do not necessarily end after the RIS response is issued.  Constant and ongoing communications usually take place through the implementation of the RIS and sometimes beyond, depending on the scope and complexity of the requested work.  Based on discussions and folder reviews, we found the RIS process and the communications between the BSD organization personnel and customers were an adequate means to develop the requirements.

The BSD organization works effectively with its customers in defining RIS requirements

Our reviews of the RIS process and a sample of RISs submitted during FY 2004 identified the use of open and ongoing communications between the BSD organization and its IRS customers that enable them to sufficiently define RIS requirements.  Customers submitting requirements for RISs identify the business developments or changes they want to install.  The BSD organization assists by determining the information system requirements to meet the customers’ needs.

We analyzed the RISs submitted during FY 2004.  During this period, the BSD organization received 1,183 original RIS submissions.

·         The IRS customers subsequently withdrew 66 RISs from further work because they decided the work was no longer necessary.

·         The BSD organization returned 53 RISs to the customers.  Of these, 19 (36 percent) were returned for requirements related reasons (e.g., insufficient requirements, requirements were already provided by another system, etc.).  The remaining 34 (64 percent) were returned due to an absence of resources.

Overall, 98 percent of the RISs included appropriate requirements when the customer submitted them to the BSD organization.

Subsequent to the RIS submission, 137 (13 percent) of the 1,064 approved RISs needed further requirements development through a RIS amendment.  These amendments modified the existing requirements or added additional requirements as part of the RIS development process.  In our opinion, the amendment activity is relatively low and indicates the RIS process and the communications between the BSD organization personnel and customers work to adequately develop the requirements.

The Business Systems Development Organization Can Improve Its Efficiency by Tracking Costs for Requirements Development and Management

The BSD organization does not have the functionality in the cost accounting system or the RIS Tracking and Reporting System (RTRS) to compare the estimated and actual staff days for time spent developing, tracking, and testing requirements.  We reviewed a sample of 20 RISs affecting  4 projects (16 RISs for Compliance Systems and 4 RISs for Corporate Data and Systems Management Divisions) and found the resources estimated to complete the requests ranged from 5 to 3,276 staff days.  Appendix IV provides the estimated staff days for each of the RISs sampled.

The Internal Revenue Manual (IRM) requires the actual results and performance of software development projects be tracked against the software development plan.  The Carnegie Mellon Software Engineering Institute’s (SEI) Capability Maturity Model Integration (CMMI) states the project planning process should be monitored and controlled against the plan to include specific measures to address the estimated and actual effort and cost.  Project planning should include the ability to reconcile the project plan to reflect available and estimated resources.  Table 1 presents the IRM and the CMMI measurement and analysis criteria for software development projects.

Table 1:  Project Requirement and Measurement Criteria

Source: The IRM and the CMMI.

In July 2004, the SEI reported the BSD organization did not refine, monitor, and adjust the estimated project measures (e.g., staff days) on a regular basis.  It recommended the BSD organization build a historical measurement database.  In response to this report, the BSD organization developed a process improvement plan which included the Measurement and Analysis Process Action Team (PAT).  As of December 2004, the BSD organization had not issued any policies or procedures as a result of the Measurement and Analysis PAT.

Without measures to compare the estimated costs to the actual costs for requirements development, the BSD organization may not be able to effectively assign resources to support the future use and improvement of the organization’s processes and process assets.

Recommendation

1.      To ensure the overall success of the requirements management process, the Chief Information Officer (CIO) should have the BSD organization develop policies and procedures for requirements management and measurement.  The Measurement and Analysis PAT should include measures to monitor the actual costs of the requirements development, tracking, and testing process against the estimates of cost, effort, and schedule.  The PAT can accomplish this by:

·         Identifying, from current processes, any existing data that could be used to compare actual costs against estimated costs.  For example, the RTRS currently has the estimated staff days for the development of the requirements.

·         Identifying the measures which need data accumulation not currently available.

·         Specifying how to collect and store the data for each required measure.

·         Creating automated data collection mechanisms, if not currently available, and providing training and guidance.

·         Determining how to measure, analyze, report, and use the data.

Management’s Response:  The CIO agreed that the BSD organization can further improve its efficiency by tracking costs for requirements development and management.  Since the audit fieldwork was completed, the CIO formed the Unified Work Request (UWR) initiative under the direction of the Associate CIO of Enterprise Services.  The UWR Working Group was formed March 17, 2005.  It is tasked to develop a comprehensive process or set of processes, procedures, and system changes to be implemented enterprise-wide by combining or modifying the functionality of the existing RIS RTRS, Change Request Tracking System, and other work requests systems or processes into a unified entity that will enable effective and efficient prioritization and coordination of all work requested of the MITS organization.  This is envisioned to be a multi-phased process.  Further, the BSD organization will work with the MITS Management Information and Cost Analysis System project to correctly identify and enumerate all cost areas in the BSD organization.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to evaluate the Business Systems Development (BSD) organization’s efforts to effectively develop and manage the requirements for information services.  To accomplish this objective, we:

I.             Evaluated the BSD organization’s overall policy for developing and managing requirements and the controls provided by the Internal Revenue Manual, the Enterprise Life Cycle-Lite (ELC-Lite), and the Carnegie Mellon Software Engineering Institute’s (SEI) Capability Maturity Model Integration (CMMI).

II.          Determined the assignment of responsibilities in the BSD organization for requirements development and management processes

A.    Interviewed BSD organization executive, division, and branch management to determine their roles and responsibilities for requirements development and management.

B.     Determined the potential impact on requirements development and management in the BSD organization with the imminent establishment of the Enterprise Services organization.

C.     Analyzed internal BSD organization reviews of project activities regarding requirements development and management.

III.       Judgmentally selected 4 projects and 20 Requests for Information Services (RIS) submitted during Fiscal Year (FY) 2004 from the Compliance Systems (2 projects and 16 RISs) and Corporate Data and Systems Management (2 projects and 4 RISs) Divisions.  This sample was selected from the 33 development and maintenance projects implemented by August 2004 and selected for testing by Product Assurance.  Our sample was used to evaluate the process to develop the requirements prior to the RIS approval and manage the requirements after a RIS is approved.  We used a judgmental sample because a precise projection of sample results over a population was not required.

A.    Interviewed division management and analysts to determine how the requirements were developed, tracked, and monitored prior and subsequent to the RIS approval.

B.     Reviewed project folders for the selected projects to determine how communications between the developer/management and the customer were documented and development of the requirements were tracked.

IV.       Analyzed amended and returned RISs from the 1,183 RISs the BSD organization managed during FY 2004 to determine the reason for the amendment or return.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary Hinkle, Director

Edward A. Neuwirth, Audit Manager

Tina Wong, Senior Auditor

Suzanne Noland, Auditor

Linda Screws, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Information Technology Services  OS:CIO:I

Director, Business Systems Development  OS:CIO:I:B

Director, Stakeholder Management  OS:CIO:SM

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaisons:

Director, Business Systems Development  OS:CIO:I:B

Manager, Program Oversight Office  OS:CIO:SM:PO

 

Appendix IV

 

Staff Day Estimates for the Request for Information Services Sample

Table 1 lists the staff day estimates for each Request for Information Services (RIS) included in our sample.  The RISs are grouped by their related Division and information systems program.

 

 

Table 1:  Staff Day Estimates for Sampled RISs

 

Source:  Treasury Inspector General for Tax Administration analyses of sampled RISs.

 

Appendix V

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.