The Business Systems Development Organizationís Effective Process for Developing Information Systems Requirements Can Be Made More Efficient by Tracking and Analyzing Related Costs

 

May 2005

 

Reference Number:2005-20-061

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

May 9, 2005

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:†††† Pamela J. Gardiner /s/ Pamela J. Gardiner

†††††††††††††††† Deputy Inspector General for Audit

 

SUBJECT:†††† Final Audit Report - The Business Systems Development Organizationís Effective Process for Developing Information Systems Requirements Can Be Made More Efficient by Tracking and Analyzing Related Costs (Audit # 200420040)

 

This report presents the results of our review of the Business Systems Development (BSD) organizationís efforts to effectively develop and manage the requirements for information services.This review is part of our Fiscal Year (FY) 2005 Audit Plan for reviews to assess the adequacy of the Internal Revenue Serviceís (IRS) information technology.

In summary, the BSD organization receives information services requests from the IRSí business operating divisions and functional organizations.These requests are usually received through the Request for Information Services (RIS) process.The RIS process provides a common framework to document, control, monitor, and track requirement changes to IRS computer systems and requests for information technology support.

Requirements management is the process that controls and documents all project requirements.It involves establishing the requirements, controlling all subsequent requirements changes, and maintaining agreement with the customers and providers of the requested products or services.This process ensures requirements are unambiguous, traceable, verifiable, documented, and controlled.An effective requirements development and management process can prevent potential problems before they become serious problems resulting in schedule delays and additional costs.

Our reviews of the RIS process and a sample of RISs submitted during FY 2004 identified the use of open and ongoing communications between the BSD organization and its IRS customers that enable them to sufficiently define RIS requirements.Customers submitting requirements for RISs identify the business developments or changes they want to install.The BSD organization assists by determining the information system requirements to meet the customersí needs.

We analyzed the 1,183 original RISs submitted to the BSD organization during FY 2004.Overall, 98 percent of the RISs included appropriate requirements when the customer submitted them to the BSD organization.Subsequent to the RIS submission and approval, 13 percent needed further requirements development through a RIS amendment.These amendments modified the existing requirements or added additional requirements as part of the RIS development process.In our opinion, the amendment activity is relatively low and indicates the RIS process and the communications between the BSD organization and customers work to adequately develop the requirements.

Although the BSD organization and its customers work to adequately develop the requirements for information systems services, we found the BSD organization does not have the functionality in the cost accounting system or the RIS Tracking and Reporting System (RTRS) to compare the estimated and actual staff days for time spent developing, tracking, and testing requirements.The Internal Revenue Manual requires the actual results and performance of software development projects be tracked against the software development plan.The Carnegie Mellon Software Engineering Instituteís (SEI) Capability Maturity Model Integration (CMMI) states the project planning process should be monitored and controlled against the plan to include specific measures to address the estimated and actual effort and cost.Project planning should include the ability to reconcile the project plan to reflect available and estimated resources.

Without measures to compare the estimated costs to the actual costs for requirements development, the BSD organization may not be able to effectively assign resources to support the future use and improvement of the organizationís processes and process assets.To ensure the overall success of the requirements management process, we recommended the Chief Information Officer (CIO) have the BSD organization develop policies and procedures for requirements management and measurement.

Managementís Response:The CIO agreed that the BSD organization can further improve its efficiency by tracking costs for requirements development and management.Since the audit fieldwork was completed, the CIO formed the Unified Work Request (UWR) initiative under the direction of the Associate CIO of Enterprise Services.The UWR Working Group was formed on March 17, 2005.It is tasked to develop a comprehensive process or set of processes, procedures, and system changes to be implemented enterprise-wide by combining or modifying the functionality of the existing work requests systems into a unified entity that will enable effective and efficient prioritization and coordination of all work requested of the Modernization and Information Technology Services (MITS) organization.Further, the BSD organization will work with the MITS Management Information and Cost Analysis System project to correctly identify and enumerate all cost areas in the BSD organization. ††Managementís complete response to the draft report is included as Appendix V.

Copies of this report are also being sent to the IRS managers affected by the report recommendation.Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

Table of Contents

Background

The Business Systems Development Organization Has an Effective Process to Develop Requirements for Information Services Requests

The Business Systems Development Organization Can Improve Its Efficiency by Tracking Costs for Requirements Development and Management

Recommendation 1:

Appendix I Ė Detailed Objective, Scope, and Methodology

Appendix II Ė Major Contributors to This Report

Appendix III Ė Report Distribution List

Appendix IV Ė Staff Day Estimates for the Request for Information Services Sample

Appendix V Ė Managementís Response to the Draft Report

 

Background

One of the Internal Revenue Serviceís (IRS) major strategies contained in the IRS Strategic Plan for Fiscal Years (FY) 2000-2005 is to provide high-quality, efficient, and responsive information services.This strategy plans continuing support for current operations with emphasis on increased quality and reduced costs of routine operations and continued support to the Business Systems Modernization program.The Business Systems Development (BSD) organization defines, builds, tests, delivers, and maintains the IRSí information systems.The BSD organizationís work supports the Modernization and Information Technology Services (MITS) organization production environment in achieving the IRSí business vision and objectives.

The BSD organization is one of the largest functions in the MITS organization in terms of both size and resources.For FY 2004, the BSD organization had a budget of $336 million and 1,916 Full-Time Equivalent (FTE) positions.The BSD organizationís budget for FY 2005 is $373 million and 2,175 FTE positions.The BSD organization accomplishes its work with eight subordinate organizations:

         Product Assurance Division.

         Program Management and Release Readiness Office.

         Compliance Systems Division.

         Corporate Data and Systems Management Division.

         Internal Management Systems Division.

         Customer Applications Development Division.

         Filing Systems Division.

         Client Services Division.

The BSD organization receives information services requests from the IRSí business operating divisions and functional organizations.These requests are usually received through the Request for Information Services (RIS) process.A RIS is a formal memorandum requesting BSD organization support for changes to current or planned programming, corporate hardware, commercial off-the-shelf software applications, system testing, and other MITS organization activities used in processing tax information.The RIS process provides a common framework to document, control, monitor, and track requirement changes to IRS computer systems and requests for information technology support.

Requirements management is the process that controls and documents all project requirements.It involves establishing the requirements, controlling all subsequent requirements changes, and maintaining agreement with the customers and providers of the requested products or services.This process ensures requirements are unambiguous, traceable, verifiable, documented, and controlled.An effective requirements development and management process can prevent potential problems before they become serious problems resulting in schedule delays and additional costs.

This review was performed at the BSD organization offices in New Carrollton, Maryland, during the period October 2004 through February 2005.The audit was conducted in accordance with Government Auditing Standards.Detailed information on our audit objective, scope, and methodology is presented in Appendix I.Major contributors to the report are listed in Appendix II.

The Business Systems Development Organization Has an Effective Process to Develop Requirements for Information Services Requests

The RIS process is initiated when a customer prepares and submits a Placeholder to the BSD organization to request services or support.The primary BSD organization contact for the Placeholder initiates precoordination meetings with the customers and appropriate BSD organization personnel.The Placeholder precoordination participants meet to get an early indication of the BSD organizationís ability to provide the requested support and/or service requirements.The meeting results help the customer to determine whether to submit a RIS.

If the Placeholder precoordination meeting results indicate the BSD organization has the ability to provide the type of support and/or services requested, the BSD organization and the customer work together to define the initial RIS requirements.Once the initial requirements are defined, the customer submits a RIS.After the RIS is received, the primary BSD organization analyst must contact the customer to continue coordination in order to review, analyze, and agree upon the specific requirements outlined in the RIS.The objective of RIS coordination is to determine whether the agreed upon requirements can be provided and to address any requirements/issues not resolved in Placeholder precoordination.

When all organizations involved in developing the RIS understand the requirements and agree to do the work, the BSD organization contact prepares a formal RIS response documenting the work to be performed.An amended RIS is the vehicle to add or modify requirements to an existing RIS. The work requested in an amended RIS must be integral to the work requested in the original RIS.RIS amendments must be submitted through the regular RIS process.

In most cases, communications between the BSD organization and the customer do not necessarily end after the RIS response is issued.Constant and ongoing communications usually take place through the implementation of the RIS and sometimes beyond, depending on the scope and complexity of the requested work.Based on discussions and folder reviews, we found the RIS process and the communications between the BSD organization personnel and customers were an adequate means to develop the requirements.

The BSD organization works effectively with its customers in defining RIS requirements

Our reviews of the RIS process and a sample of RISs submitted during FY 2004 identified the use of open and ongoing communications between the BSD organization and its IRS customers that enable them to sufficiently define RIS requirements.Customers submitting requirements for RISs identify the business developments or changes they want to install.The BSD organization assists by determining the information system requirements to meet the customersí needs.

We analyzed the RISs submitted during FY 2004.During this period, the BSD organization received 1,183 original RIS submissions.

         The IRS customers subsequently withdrew 66 RISs from further work because they decided the work was no longer necessary.

         The BSD organization returned 53 RISs to the customers.Of these, 19 (36 percent) were returned for requirements related reasons (e.g., insufficient requirements, requirements were already provided by another system, etc.).The remaining 34 (64 percent) were returned due to an absence of resources.

Overall, 98 percent of the RISs included appropriate requirements when the customer submitted them to the BSD organization.

Subsequent to the RIS submission, 137 (13 percent) of the 1,064 approved RISs needed further requirements development through a RIS amendment.These amendments modified the existing requirements or added additional requirements as part of the RIS development process.In our opinion, the amendment activity is relatively low and indicates the RIS process and the communications between the BSD organization personnel and customers work to adequately develop the requirements.

The Business Systems Development Organization Can Improve Its Efficiency by Tracking Costs for Requirements Development and Management

The BSD organization does not have the functionality in the cost accounting system or the RIS Tracking and Reporting System (RTRS) to compare the estimated and actual staff days for time spent developing, tracking, and testing requirements.We reviewed a sample of 20 RISs affecting 4 projects (16 RISs for Compliance Systems and 4 RISs for Corporate Data and Systems Management Divisions) and found the resources estimated to complete the requests ranged from 5 to 3,276 staff days. Appendix IV provides the estimated staff days for each of the RISs sampled.

The Internal Revenue Manual (IRM) requires the actual results and performance of software development projects be tracked against the software development plan.The Carnegie Mellon Software Engineering Instituteís (SEI) Capability Maturity Model Integration (CMMI) states the project planning process should be monitored and controlled against the plan to include specific measures to address the estimated and actual effort and cost.Project planning should include the ability to reconcile the project plan to reflect available and estimated resources.Table 1 presents the IRM and the CMMI measurement and analysis criteria for software development projects.

Table 1:Project Requirement and Measurement Criteria

IRM

CMMI

Software development projects should be tracked and have oversight to provide adequate visibility into actual progress so management can take effective actions when the projectís performance deviates from the plan.

The requirements management process includes monitoring and controlling the requirements against the plan for performing the process and taking appropriate corrective action (i.e., measures used in monitoring and controlling changes).

The actual results and performance of software development projects shall be tracked against the project plan.When actual results and performance significantly deviate from the plan, corrective actions shall be taken and managed to closure.

Project planning includes monitoring and controlling the actual values against the plan for performing the process (i.e., cost, effort, and schedule) and taking appropriate corrective action.

Changes to project commitments shall be agreed to by managers in affected organizational units.

An organization should collect work products, measures, measurement results, and improvement information derived from planning and performing the requirements management to support the future use and improvement of the organizationís processes and process assets.

Source: The IRM and the CMMI.

In July 2004, the SEI reported the BSD organization did not refine, monitor, and adjust the estimated project measures (e.g., staff days) on a regular basis.It recommended the BSD organization build a historical measurement database. In response to this report, the BSD organization developed a process improvement plan which included the Measurement and Analysis Process Action Team (PAT).As of December 2004, the BSD organization had not issued any policies or procedures as a result of the Measurement and Analysis PAT.

Without measures to compare the estimated costs to the actual costs for requirements development, the BSD organization may not be able to effectively assign resources to support the future use and improvement of the organizationís processes and process assets.

Recommendation

1.      To ensure the overall success of the requirements management process, the Chief Information Officer (CIO) should have the BSD organization develop policies and procedures for requirements management and measurement.The Measurement and Analysis PAT should include measures to monitor the actual costs of the requirements development, tracking, and testing process against the estimates of cost, effort, and schedule.The PAT can accomplish this by:

         Identifying, from current processes, any existing data that could be used to compare actual costs against estimated costs.For example, the RTRS currently has the estimated staff days for the development of the requirements.

         Identifying the measures which need data accumulation not currently available.

         Specifying how to collect and store the data for each required measure.

         Creating automated data collection mechanisms, if not currently available, and providing training and guidance.

         Determining how to measure, analyze, report, and use the data.

Managementís Response:The CIO agreed that the BSD organization can further improve its efficiency by tracking costs for requirements development and management.Since the audit fieldwork was completed, the CIO formed the Unified Work Request (UWR) initiative under the direction of the Associate CIO of Enterprise Services.The UWR Working Group was formed March 17, 2005.It is tasked to develop a comprehensive process or set of processes, procedures, and system changes to be implemented enterprise-wide by combining or modifying the functionality of the existing RIS RTRS, Change Request Tracking System, and other work requests systems or processes into a unified entity that will enable effective and efficient prioritization and coordination of all work requested of the MITS organization. This is envisioned to be a multi-phased process.Further, the BSD organization will work with the MITS Management Information and Cost Analysis System project to correctly identify and enumerate all cost areas in the BSD organization.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to evaluate the Business Systems Development (BSD) organizationís efforts to effectively develop and manage the requirements for information services.To accomplish this objective, we:

I.             Evaluated the BSD organizationís overall policy for developing and managing requirements and the controls provided by the Internal Revenue Manual, the Enterprise Life Cycle-Lite (ELC-Lite), and the Carnegie Mellon Software Engineering Instituteís (SEI) Capability Maturity Model Integration (CMMI).

II.          Determined the assignment of responsibilities in the BSD organization for requirements development and management processes

A.    Interviewed BSD organization executive, division, and branch management to determine their roles and responsibilities for requirements development and management.

B.     Determined the potential impact on requirements development and management in the BSD organization with the imminent establishment of the Enterprise Services organization.

C.     Analyzed internal BSD organization reviews of project activities regarding requirements development and management.

III.       Judgmentally selected 4 projects and 20 Requests for Information Services (RIS) submitted during Fiscal Year (FY) 2004 from the Compliance Systems (2 projects and 16 RISs) and Corporate Data and Systems Management (2 projects and 4 RISs) Divisions.This sample was selected from the 33 development and maintenance projects implemented by August 2004 and selected for testing by Product Assurance.Our sample was used to evaluate the process to develop the requirements prior to the RIS approval and manage the requirements after a RIS is approved.We used a judgmental sample because a precise projection of sample results over a population was not required.

A.    Interviewed division management and analysts to determine how the requirements were developed, tracked, and monitored prior and subsequent to the RIS approval.

B.     Reviewed project folders for the selected projects to determine how communications between the developer/management and the customer were documented and development of the requirements were tracked.

IV.       Analyzed amended and returned RISs from the 1,183 RISs the BSD organization managed during FY 2004 to determine the reason for the amendment or return.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary Hinkle, Director

Edward A. Neuwirth, Audit Manager

Tina Wong, Senior Auditor

Suzanne Noland, Auditor

Linda Screws, Auditor

 

Appendix III

 

Report Distribution List

 

CommissionerC

Office of the Commissioner Ė Attn:Chief of StaffC

Deputy Commissioner for Operations SupportOS

Associate Chief Information Officer, Information Technology ServicesOS:CIO:I

Director, Business Systems DevelopmentOS:CIO:I:B

Director, Stakeholder ManagementOS:CIO:SM

Chief CounselCC

National Taxpayer Advocate TA

Director, Office of Legislative AffairsCL:LA

Director, Office of Program Evaluation and Risk AnalysisRAS:O

Office of Management ControlsOS:CFO:AR:M

Audit Liaisons:

Director, Business Systems Development OS:CIO:I:B

Manager, Program Oversight OfficeOS:CIO:SM:PO

 

Appendix IV

 

Staff Day Estimates for the Request for Information Services Sample

Table 1 lists the staff day estimates for each Request for Information Services (RIS) included in our sample.The RISs are grouped by their related Division and information systems program.

 

 

Table 1:Staff Day Estimates for Sampled RISs

Corporate Data and Systems Management Division

Notice Review Processing System (NRPS) - The NRPS extracts notices for quality review against current data prior to mailing to the taxpayer.It provides notice data, which is to be reviewed by tax examiners on-line.

RIS Number

Estimated Staff Days

OPR30007A00

3,276

WIC40005A00

362

WIC40014A00

324

NRPS and On-Line Notice Review (OLNR) - The OLNR application gives tax examiners the ability to review and edit notices on-line.

WSP30172A00

123

 

Compliance Division

Integrated Collection System (ICS) - The ICS provides workload management, case assignment/tracking, inventory control, electronic mail, case analysis tools, and management information systems capabilities to support the collection fieldwork.

RIS Number

Estimated Staff Days

COL30129A00

2,731

COL30111A00

10

COL30018A00

76

COL30083A00

28

COL10071A00

107

COL20114A00

191

SCS10214A00

1,713

COL30099A00

5

WCA10100A00

235

COL30100A01

403

SCA30017A00

10

COL30039A00

46

COL20124A00

45

COL10047A00

169

COL20022A00

60

Refund Intercept Request - This process intercepts erroneous or questionable refunds that otherwise would be mailed or electronically deposited to taxpayers.

OPR20004A00

1,470

Source:Treasury Inspector General for Tax Administration analyses of sampled RISs.

 

Appendix V

 

Managementís Response to the Draft Report

 

The response was removed due to its size.To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.