Improvements to the Modernized Infrastructure Are Needed to
Support the Deployment of Business Systems Modernization Projects
August 2003
Reference Number: 2003-20-161
This report has cleared the Treasury
Inspector General for Tax Administration disclosure review process and
information determined to be restricted from public release has been redacted
from this document.
August
1, 2003
MEMORANDUM FOR
CHIEF INFORMATION OFFICER
FROM: Gordon C. Milbourn III /s/ Gordon C.
Milbourn III
Assistant Inspector General for Audit (Small Business and
Corporate Programs)
SUBJECT: Final Audit Report - Improvements to the Modernized Infrastructure
Are Needed to Support the Deployment of Business Systems Modernization Projects
(Audit # 200220047)
This
report presents the results of our review of the development of the Internal
Revenue Service’s (IRS) modernized infrastructure. The overall objective of this review was to determine whether the
modernized infrastructure being
developed by the Infrastructure Shared Services (ISS) program was adequate to
support the release of Business Systems Modernization (BSM) projects in
2003. To complete our objective, we
followed up on issues reported in an earlier audit of the Security and Technology
Infrastructure Release (STIR) and conducted new tests to evaluate the
progression of the ISS program.
The success of the BSM
program depends on establishing a strong foundation from which to build
business applications to support tax processing functions. This process begins with the development of
a modernized infrastructure. In
February 2002, the IRS established the ISS program to support other BSM
projects by building upon the basic infrastructure components installed during
2002.
In
summary, the ISS program has made progress in developing processes to improve
coordination between itself and the BSM project teams. In addition, improvements have been made in
documenting the impact of changes requested by the Business Systems
Modernization Office (BSMO), the PRIME contractor, and other stakeholders to
the infrastructure and the projects it supports. Progress was also made in identifying necessary staffing levels
for the ISS and working to fill those current vacancies.
However, we identified some critical
issues that need to be addressed quickly so BSM projects that are scheduled for
deployment during 2003 and beyond are not affected by significant schedule
delays or cost increases. For example,
critical coordination between the ISS program and the BSM project teams is
occurring too late in the life cycle process of these projects. Because development processes are not
clearly defined and consistently followed, certain project activities needed in
the projects’ development phase are not being completed until the deployment
phase, and dependencies between the projects and the infrastructure are not
being timely defined or worked. As a
result, significant unplanned cost increases and schedule delays have occurred
on the BSM projects. For example, the
Integrated Financial System project incurred additional unforeseen costs of
$22.6 million, and the e-Services project was delayed for nearly a year. Meeting earlier in the development process
to identify infrastructure dependencies would not necessarily have eliminated
all the additional costs or delays but would have minimized the impact on the
projects.
The first release of the STIR was
deployed even though the costs of the STIR increased substantially from the
amounts initially approved by IRS executives and before acceptable performance
of the system had been demonstrated.
Additional costs also were incurred to improve system performance after
the deployment was completed. In
addition, the capacity of the systems used to test the BSM projects was not
sufficient to handle the needs of the projects, and stronger disciplines were
needed to ensure that configurations in the development, test, and production
environments were better documented and managed. These issues have thus far resulted in deployment delays for
several BSM projects.
Finally,
change requests were not processed timely, and project risks were not
effectively identified and addressed.
These are key project management disciplines that are needed to help the
ISS program effectively address the needs of the BSM projects it supports.
Since
the completion of our audit work, the BSMO has taken actions to address some of
the above conditions. The BSMO published two documents that provide
detailed guidance on the steps needed for coordinating infrastructure issues
and the timing of when the project teams must take these key integration
steps. In addition, seminars have been
held with project teams to provide details on the infrastructure and how users
will interact with the systems. The
BSMO has also put in place a process to formally integrate Cross-Project
Dependencies from non-PRIME contract projects, and has developed a new process
to identify the roles and responsibilities of the IRS and the PRIME contractor
in analyzing, selecting, and acquiring third-party software and hardware
products in support of ongoing project initiatives.
To
address the remaining issues, we recommended the Chief Information Officer:
·
Initiate updates to the infrastructure processes guidance
and ensure that BSM project teams follow this guidance.
·
Require the ISS program to verify that key dependencies are
identified prior to moving dependent projects into subsequent phases of the
life cycle.
·
Hold the PRIME contractor accountable for cost and schedule
estimates produced at the end of the project design phase, and place additional
emphasis on enterprise-level performance and capacity planning.
·
Ensure that performance and capacity planning are adequately
addressed and that BSM projects sufficiently demonstrate acceptable performance
before being deployed.
·
Request sufficient funding to address test lab inadequacies.
·
Ensure that established test lab processes are followed and
that project teams identify test lab requirements earlier in the project life
cycle.
·
Require the PRIME contractor to improve its focus on
documenting and managing the configurations in the various testing
environments.
·
Require the BSMO and the PRIME contractor to focus on
critical and high-priority change requests with approaching or past due dates,
and ensure that the change dates are reasonable and included on all change
request forms.
·
Ensure that the BSMO and the PRIME contractor follow
established risk management processes.
Management’s Response: Management’s response was due on July 17, 2003. Management requested an extension to July 25, 2003, but as of July 30, 2003, had not responded to the draft report.
Copies of this report are also being sent to the IRS managers affected
by the report recommendations. Please
contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Acting
Assistant Inspector General for Audit (Information Systems Programs), at (202)
622-8510.
Critical Coordination Between the Infrastructure and Other Projects Is Occurring Too Late
The Capacity of the Test Lab Is Not Adequate to Support the Modernization Program
Change Requests Were Not Processed Effectively
Project Risks Were Not Effectively Identified and Addressed
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors to This Report
Appendix III – Report Distribution List
The Internal Revenue Service (IRS) is currently in the midst of a multiyear, multibillion dollar effort to update its core business systems, known as Business Systems Modernization (BSM). The IRS hired the Computer Sciences Corporation as its PRIME contractor to head an alliance of leading technology companies in assisting with the BSM program. The IRS also established the Business Systems Modernization Office (BSMO) to manage the BSM program and oversee the work of the PRIME contractor.
The success of the BSM program depends on establishing a strong foundation from which to build business applications to support tax return processing and other mission-related functions. One of the key components in this foundation is the development of a modernized infrastructure. The IRS’ modernized infrastructure is divided into three major functional areas, or sub-projects:
1)
Security and Technology Infrastructure Release
(STIR) provides the hardware and software necessary to deploy
and run a BSM project. It provides key
features such as web servers, security, messaging, and directory services.
2)
Enterprise Systems Management (ESM)
provides the overlying management capabilities to the STIR and provides network
and systems management to improve infrastructure availability and
performance. It also manages hardware
and software inventories and monitors the performance of various systems.
3)
Development, Integration, and Test Environment
(DITE) provides a complete project development and testing environment
that is meant to simulate actual operating conditions. It provides labs to: evaluate potential solutions and their
impact on business processes (Solutions Demonstration Lab), support all phases
of information systems development (Virtual Development Environment), and test
systems integration and final acceptance (Enterprise Integration and Test
Environment).
The development of a secure infrastructure to support the BSM program is a very complex undertaking for an organization as large as the IRS. The infrastructure under development is geographically dispersed over various sites and includes numerous pieces of hardware and software, which must effectively communicate and interact with each other as they support projects that provide benefits to taxpayers and IRS employees. Designing, acquiring, and managing this infrastructure is difficult and requires a significant amount of money, time, and effort.
In February 2002, the BSMO established the Infrastructure Shared Services (ISS) program to manage and build upon the basic infrastructure components installed during 2002. The BSMO planned to augment the existing infrastructure components in Fiscal Year (FY) 2003 by providing additional hardware and software to handle increased users and additional BSM projects. At the time of our audit, the ISS program included the STIR and ESM projects; however, the DITE functionality had not yet been included.
The following BSM projects have releases scheduled for deployment during FY 2003 and require support from the ISS program:
· e-Services will provide practitioners and business partners additional automated service over the Internet and enable the IRS to improve relationship management among this important set of customers.
· Internet Employer Identification Number will allow employers, tax practitioners, and financial institutions to apply for and receive a validated employer identification number directly from the IRS via the Internet.
· Customer Account Data Engine (CADE) will provide the modernized database that will hold all taxpayer account information and support other BSM projects.
· Human Resources Connect is the new human resources application being implemented by all agencies within the Department of the Treasury.
· Internet Refund Fact of Filing (IRFOF) provides tax refund information to taxpayers on the IRS’ Internet web site.
· Custodial Accounting Project/Enterprise Data Warehouse (CAP/EDW) will provide the IRS with integrated, timely, and accurate tax administration and internal management information to support decision analysis, performance measurement, and other management information needs.
These projects will begin the acceleration of the deployment of functionality that supports taxpayers and IRS employees. Although the ISS program provides a certain number of hours out of its own funds, primary funding for the projects’ infrastructure, including system design and development, must now come from the BSM projects seeking infrastructure support.
The audit was conducted at the BSMO facilities in New Carrollton, Maryland, from June 2002 to February 2003 in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
Generally, we saw progress in the development of processes used by the STIR and ESM infrastructure projects, specifically in the area of system development. In December 2002, the BSMO completed A Customer’s Guide to Infrastructure Shared Services describing the services offered by the ISS program and the associated processes to be used by the BSM projects for obtaining infrastructure support. In addition, the BSMO has used general information sessions, as well as more focused team approaches, to educate and work with the BSM project teams that plan to deploy releases in FY 2003.
We also saw improvements in the consistent preparation of impact assessments for infrastructure change requests. During a previous audit of the STIR, we reported that necessary impact assessments showing the cost of the change, as well as potential implications of making the change, were not always prepared.
Lastly, staffing needs for the ISS program have been identified, and the BSMO is working to fill current vacancies. However, staffing has been complicated by budget issues and other events across the IRS that have constrained staff increases, and it is taking longer than expected to fill several critical positions.
Although the IRS’ infrastructure program has seen improvements, we identified some critical areas that need to be addressed quickly so that any impact to BSM projects planned for development in FY 2003 and beyond is minimized.
Integration between the ISS program and the BSM projects continues to impede the development of system design. During our previous audit of the STIR, we determined that the STIR project team was encountering problems providing the complex integration required for the IRFOF project, which was the first BSM project implemented on the modernized STIR infrastructure. We found similar problems during this audit with coordination between the ISS program and BSM project teams scheduling releases this year.
Project teams are entering the development phase prior to obtaining key decisions about project design and configuration
Decisions about infrastructure designs and integration need to be made by project teams prior to the start of the development stage of a project. Activities, such as fully defining the configuration of the infrastructure components and determining the appropriate testing environments and procedures, are needed prior to the end of the design phase to ensure the BSM projects will integrate into the infrastructure. This information is also needed so the project teams can identify specific infrastructure costs to include in the business cases. Without the information on the infrastructure configurations and associated costs, it is difficult for the project teams to prepare accurate business cases. These business cases are used by IRS executives to approve funds to begin the project development phase, and without complete infrastructure data, project approval may be granted based on inaccurate cost and benefits information.
One reason these infrastructure issues have not been addressed sooner is because the Enterprise Life Cycle (ELC), which contains the processes and procedures the BSMO and the PRIME contractor follow on system development projects, does not provide clear, consistent, and cohesive guidance on coordinating infrastructure issues between the ISS program and the BSM projects it supports. Although there is technical information about the infrastructure in the ELC, the detailed steps to be taken to coordinate efforts between the individual BSM project teams and the ISS program are not clearly defined in this guidance. Also, the ELC does not require definition of the configuration until after the design phase ends and development begins.
Because the processes and guidelines for coordinating on infrastructure issues were not clear, the ISS program did not coordinate with some of the BSM project teams early enough to develop an adequate level of infrastructure definition prior to the projects entering the development phase. As a result, detailed infrastructure coordination was delayed until the projects were in development and, in several cases, unplanned additional costs were incurred to purchase and install new or different hardware and software components. For example, in early 2002, the CAP/EDW project spent approximately $6.5 million, and the e-Services project spent over $8.4 million, to revise application components to meet infrastructure requirements and to integrate with other projects being developed. Had the ISS program and project teams coordinated infrastructure issues sufficiently during the design phase of the projects, these application components could have been identified earlier and the costs to integrate them into the projects may have been less.
Delays also occurred in project deployment due in part to the late coordination. For example, the e-Services project moved into the development phase in August 2001; however, much of the technical infrastructure design that had been completed at that point was inaccurate due to delays in coordination. Updates of this documentation to create a more accurate design were produced in January and August 2002. The delay in completing infrastructure design contributed to the project’s extended development phase, which was still ongoing at the completion of our audit work in February 2003.
Management Actions: The BSMO’s Office of Infrastructure Modernization recently released A Customer’s Guide to Infrastructure Shared Services and A Customer’s Guide to Development, Integration, and Test Environment, which provide more details about the steps needed for coordinating infrastructure issues and the timing of when the project teams must take these key integration steps. In addition, seminars have been held with project teams to provide details on the infrastructure and how users will interact with the systems.
Dependencies between infrastructure and business projects are not consistently defined and timely addressed
We reviewed the Cross-Project Dependencies (CPD) listed on the IRS’ Integrated Master Schedule (IMS) between infrastructure and BSM projects and found that the ISS program is not consistently identifying dependencies.
· Of the 194 total CPDs listed on the IMS at the time we extracted the data, only 3 showed testing dependencies. Neither the CADE nor the IRFOF projects, which have both experienced testing delays, had any testing dependencies identified. Testing is a critical process in any project development, and most BSM projects have experienced delays in completing testing. Concerns regarding the lack of documentation of testing CPDs were reported to the PRIME contractor by the BSMO in the 3rd and 4th quarter 2002 Performance Evaluation Reports. In response to the 4th quarter report, the PRIME contractor DITE team indicated that it would begin keeping a master DITE schedule to better communicate scheduling risks and issues.
· There were no CPDs formally controlled in the IMS between the CAP/EDW project and the infrastructure in August and October 2002, when we obtained our extracts. We were later provided data that indicated several CPDs for the CAP/EDW project were documented in the IMS in mid-November, but we believe these should have been in place much earlier, considering the significance of that project, its upcoming deployment date, and its dependencies on the infrastructure.
· The ISS program was not consistently meeting the BSM projects’ need-by dates on the CPDs. For those dependencies we analyzed (those with need-by dates that were due), the ISS missed slightly more than one-half (28 of 55) of the projects’ revised need-by dates.
· For those dependencies identified as being supplied by the STIR, one-third (8 of 24) of the revised need-by dates were missed.
One reason testing CPDs were not consistently identified was that the PRIME contractor did not consider them accurate, as customers were basing their need-by dates on release windows, rather than specific dates. Additionally, the CAP/EDW project did not have documented dependencies on the infrastructure because a contractor outside of the PRIME alliance developed the project, and the CPD process was not always performed consistently between the two different contractors. There was no process for entering non-PRIME CPDs into the IMS. Lastly, we believe the ISS program was not consistently meeting the need-by dates because the complexities of completing the needed infrastructure items were not always considered when the need-by dates were developed.
Because documenting and tracking CPDs between the BSM projects and the infrastructure is so critical, we believe there may be a need for clearer guidance on what the project teams need on a consistent basis from the ISS program. Also, there is a need for better adherence to the guidance that presently exists on identifying and documenting dependencies.
The effect of the four issues reported above is that unforeseen costs can occur and cause budget problems and schedule delays. Significant changes are needed to a project’s Baseline Business Case (BBC) when the amount of infrastructure support required and infrastructure costs are not known prior to the completion of the design phase.
For example, over 55 percent ($49 million) of the unplanned increases to the BSM Expenditure Plan for FY 2002 were due to increases in infrastructure costs. ISS personnel indicated that the BSM project teams did not include all of the necessary infrastructure costs in the original business cases that are used to support the BSM Expenditure Plan. The following table shows the unplanned infrastructure costs. The table is divided to indicate the expenditures occurring prior to, during, and after a key executive meeting in which critical budget decisions were made.
|
|
Total |
Infrastructure Increases |
Percentage of Increase |
|---|---|---|---|
|
Before 6/25/02 |
$49,115,000 |
$40,779,000 |
83.03% |
|
At 6/25/02 CBS ESC |
$25,531,000 |
$7,431,000 |
29.11% |
|
After 6/25/02 |
$13,983,000 |
$845,000 |
6.04% |
|
Totals |
$88,629,000 |
$49,055,000 |
55.35% |
Source: FY 2002 BSM Budget Plan and Change Report, General Accounting Office Briefing, November 21, 2002.
In addition to the figures above, we found that the CAP/EDW project had to incur additional costs of over $230,000 for an interim solution to work around some of the infrastructure issues that came up during integration meetings that occurred after the project moved into its development phase. Even more recently, the BSMO had to add $22.6 million to the Integrated Financial System (IFS) project budget late in its design phase to cover unforeseen costs, a large portion of which related to unplanned infrastructure costs.
Unrealistic schedules are typically developed and delays occur when projects are not aware of the required technical effort and associated time that is required to integrate into the modernized infrastructure. Delays in the e-Services project of nearly a year, and delays in the CAP/EDW project of several months, are at least partly related to underestimating time required to work infrastructure issues.
Management Actions: Since the end of our audit work, the BSMO has put in place a process to formally integrate CPDs from non-PRIME contract projects (the newly established PRIME Account Management Function). In addition, a new process has been developed to identify the roles and responsibilities of the IRS and the PRIME contractor in analyzing, selecting, and acquiring third-party software and hardware products in support of ongoing project initiatives.
To improve the critical coordination between BSM projects and the ISS program, we recommend that the Chief Information Officer (CIO) require:
1.
The BSMO to update the ELC
with the details included in the guidance documents developed by the ISS
program, and the new processes related to CPDs and third-party software and
hardware products.
Management’s Response: Management’s
response was due on July 17, 2003.
Management requested an extension to July 25, 2003, but as of July 30,
2003, had not responded to the draft report.
2.
The BSMO to ensure BSM
project teams follow and adhere to ELC processes and supplemental guidance
provided by the ISS program. Special
emphasis should be placed on infrastructure performance and capacity
engineering processes.
3.
The ISS program to verify that key infrastructure dependencies
have been identified and documented before giving approval to exit the project
design phase and begin project development.
The initial release of one of the ISS program components, the STIR, was approved for deployment in February 2002. The Infrastructure Executive Steering Committee (IESC) approved the STIR deployment even though significant cost increases had occurred during the development phase and the STIR had not successfully demonstrated the ability to meet performance requirements.
Significant cost increases occurred after the BBC was
approved
Delays in developing accurate cost estimates resulted in the STIR life cycle cost estimates nearly doubling from those approved by IRS executives in the BBC. The total life cycle cost estimate in the BBC developed at the end of the STIR design phase was almost $230 million. At the end of the development phase, when a decision was made to deploy the system, the total cost estimate had grown to nearly $364.5 million, an increase of $134.5 million. In the final business case, which was produced at the end of the deployment phase, these costs had grown an additional $61.5 million to nearly $426 million.
We reviewed the updates to the original BBC. In those documents, the PRIME contractor identified the following explanations for the estimated cost increases:
· The cost of software to allow employees and taxpayers to access the STIR web servers (portal software) was not included in the original estimates.
· Initial costs included outdated vendor estimates.
· Necessary upgrades to telecommunications capacity were not included.
· The STIR project requirements were changed during development.
We believe that changes to project requirements would be sufficient justification to support a cost increase. However, the other reasons given for nearly doubling the life cycle costs of this project appear to result from inadequate estimating practices or insufficient requirements gathering.
We identified a similar issue in a report recently issued on the IRFOF project. The Customer Relationship Management Sub-Executive Steering Committee approved the IRFOF project for deployment even though project costs increased approximately 27 percent and the monetary benefits expected to be derived from the project were reduced by $42 million from the BBC.
Because decisions to invest significant funds to develop and deploy BSM projects are based on the BBC, it is crucial that cost estimates and business benefits be as accurate as possible. On the STIR and IRFOF projects, and probably other BSM projects, executives have been making these critical decisions based on inaccurate or incomplete information. Additionally, the Office of Management and Budget, which approves BSM funding, and the Senate Finance Committee have raised concerns over the quality of the IRS’ business cases. Continued problems with the accuracy and completeness of business cases could raise more questions and possibly affect future BSM budgets.
Performance testing was not completed prior to
deployment of the STIR
We reviewed the testing performed on the STIR project and its support for the IRFOF project to determine the extent and results of performance testing. In our earlier review of the IRFOF development, we reported that while it had been providing refund and filing information to taxpayers, at the time of deployment the project was not providing the level of performance required by the IRS. The PRIME contractor found during testing that the IRFOF project was unable to meet the required number of transactions per second and was, in fact, handling less than one-half of the capacity required. The PRIME contractor determined that the STIR web application servers were unable to provide the capacity needed for the IRFOF project requirements. To meet the capacity requirements, the PRIME contractor installed upgrades to the web servers at a cost to the IRS of over $400,000.
The ELC, and the detailed processes it is based on, document that requirements should be developed and managed
for system performance. Validation that these system requirements are met occurs
with the testing phase of a project.
While reviewing the documentation on the STIR project testing, we found that although performance tests were planned, all of these tests were either waived or deferred. The IRS agreed to waive or defer the performance tests because the IRFOF project was deployed prior to the peak tax refund period, and it believed the performance issues could be addressed by deploying the upgraded web application servers prior to the peak refund period. The IRS executives believed that the benefits of deploying the IRFOF project to get experience with it in a live environment outweighed the risks of not completing all planned tests.
We understand that the IRFOF project was initially deployed at a time when the capacity would not be an issue, but we are concerned that the IRS would agree to deploy the STIR without fully testing its performance and support for the IRFOF project. The infrastructure is a critical component for the BSM projects, and deploying the first release without fully testing it is a risky practice that can affect both the IRFOF and subsequent projects that will depend on the infrastructure, such as e-Services.
When we discussed this with ISS personnel, they agreed that this was risky. They pointed out that the waiver of this testing prior to deployment did not require ISS approval and, therefore, the Infrastructure Modernization Office had not approved the waiver. The IRS approving officials for this waiver included the IRFOF Project Manager, the BSM Program Director, and an individual in the BSM Release Management Office. This approval was granted to get the IRFOF project deployed and tested in an operational environment when the required capacity would be light. Implementation of upgrades was planned prior to the anticipated increase in capacity needs.
Capacity and performance issues continue within the BSM program. In January 2003, the IESC noted that the IRS was experiencing significant problems with development of the CADE project at the Martinsburg Computing Center (MCC) due to insufficient capacity and performance planning.
It appeared that the PRIME contractor had not developed an accurate projection of the required infrastructure capacity to run the BSM projects, such as the CADE and e-Services, which are being developed and deployed. The MCC was provided funding to purchase only a limited level of additional capacity for all BSM projects; however, according to MCC officials, the capacity requirements for just the projects that are being deployed in 2003 appear to already exceed this capacity level.
It is critical that accurate, reliable performance modeling across the board for all BSM projects be performed quickly, so that an accurate cost figure can be developed for the infrastructure required to run these projects. In addition, deploying BSM projects without having adequate capacity and performance capabilities at the MCC could jeopardize current processing of tax returns.
Discussions with a PRIME contractor representative in December 2002 indicated that enterprise-wide performance planning was just starting to be addressed. He indicated that although some performance work had been done on individual projects, the enterprise-wide planning is just beginning. He believed that delays in starting the enterprise work were due to inadequate funding. In addition, discussions with BSMO personnel indicated that they are working closely with individuals in capacity planning and other areas within the IRS to develop a matrix showing what performance testing should be done, who should do it, and at what stage of the project life cycle it should occur.
As new projects are added to the modernized infrastructure, performance testing and capacity analysis become even more critical to determine whether the infrastructure can handle the new requirements. Deployment of the e-Services project, which runs on the same infrastructure components as the IRFOF project, is planned for the 3rd quarter of FY 2003. Demonstration of the performance capabilities during the testing of these projects should be required, and waivers of these requirements should not be allowed during this testing.
Because the decision to deploy is so critical and the implications of significant changes to the business case could be damaging, we recommend that the CIO:
4.
Hold the PRIME contractor
accountable, within a reasonable percentage, to cost and schedule estimates
developed at the end of the design phase (BBC). This would help force the PRIME contractor to improve the
estimates provided to the IRS.
5.
Require that additional
efforts be undertaken to ensure that performance and capacity planning are
adequately addressed at an enterprise level and not allow deployment of any BSM
project without demonstration of the capability to meet performance
requirements.
Limitations in the DITE test environment have resulted in project deployment delays. Because the e-Services and IRFOF projects were developed with different versions of supporting software, and the test lab did not have sufficient equipment to provide separate environments for each software version, testing of the projects had to occur at different times of day. A painstaking process had to be developed to switch from one test lab configuration to another, and around-the-clock testing had to be conducted in the test lab to support both projects.
Delays occurred, particularly in the testing of the e-Services project, because the IRFOF project remained in the lab much longer than planned to conduct performance testing. As a result, the e-Services project team could test only part of each day, rather than having full control of the testing lab.
Another limitation of the test lab that has caused delays
and concern among the various projects is not having a separate infrastructure
environment in which to test changes prior to implementing the changes in the
full test lab. As a result, when an
upgrade or change to the infrastructure is needed, each project in the test lab
has had to discontinue testing while that change is tested and then conduct
some form of additional limited testing to ensure the change does not
negatively affect the project.
For example, if a change was needed to the STIR hardware or
software on which the e-Services project was operating, testing of the
e-Services project would have to stop until the STIR change was completed and
tested. Then the e-Services project
team would have to restart its testing to ensure the STIR changes did not
affect the e-Services project.
Additionally, certain errors
identified during testing have taken longer to resolve due to the limitations
of the infrastructure in the lab. For
example, when the e-Services project team was testing part-time and sharing the
environment with the IRFOF project, it encountered delays in being able to get
errors resolved because it had to test at night while the IRFOF project was
testing in the day. At times,
individuals who were key to the resolution of errors were unavailable at night,
and the errors that were encountered had to be addressed the following day. Because of the large number of errors that
were encountered in early e-Services testing, the delays in resolving
individual errors caused testing delays.
Lastly, the limited capacity in the test lab has impaired
the ability of the testers to conduct performance testing. This has resulted in waivers or deferrals of
this testing to subsequent test phases.
Because schedules are always tight during testing, deferring key tests
to later test phases increases the pressure to reduce the scope of these tests
or to waive them altogether (see the prior section, “Performance testing
was not completed prior to deployment of the STIR”).
One reason that the lab is limited is because when it was
originally developed, funds were tight and decisions were made by IRS
executives to limit lab funding and spend more money on developing and
implementing business projects.
Additionally, the DITE project personnel responsible for developing and
running the test lab have not been following the established testing processes
consistently or gathering project requirements for the test lab early in the
projects’ life cycles. As a result,
some delays have occurred in establishing the configurations in the lab to
support certain projects. The BSMO
reported concerns in this area to the PRIME contractor in its 3rd
and 4th Quarter 2002 Performance Evaluation Reports.
ISS personnel have indicated that efforts are underway to
add equipment and capabilities to the test lab. Even though budgets continue to be tight, it is very important
that these efforts continue because limitations in the testing environment can
cause delays with every project in the BSM program.
To ensure that test lab capacity can support future testing of modernized projects, we recommend that the CIO require:
6. Improvements to the test lab be made a priority for future funding requests.
7. The DITE project team to follow the testing processes consistently and gather test lab requirements from projects earlier.
An issue that delayed the deployment of the initial release of the STIR and IRFOF projects was the inability to track the differences between the configurations of the DITE development lab, the DITE test lab, and the production environment. As a result, a project would run in one environment but not in another. Tracking down the issues causing problems with the system was difficult because the configurations were not adequately documented.
Because of this earlier issue, we followed up to see if improvements had been made. We requested, but were unable to obtain, clear documentation of the processes the DITE team uses to determine the configurations in the test labs, and in production, and the differences between those configurations. Discussions with DITE personnel indicated that the test labs were not at the same level of robustness as the production environment and thus were not a complete mirror image of what was in production.
As we concluded our audit work, we were provided a preliminary “gap analysis” document that showed what types of software and hardware components were in the labs and in production. However, this document was at a very high level and did not give version numbers or details of configurations of the various components. Thus, it would be of limited use in tracking down issues that arise in moving a system from one environment to another.
One cause of this problem is that technical documentation of the STIR was accepted by the BSMO to allow the project to move from development into deployment, even though the documentation was inaccurate. As a result, these documents could not be relied upon to provide a clear description of what was being deployed, and additional documentation was needed to provide clear identification of what was running in the various environments and in production.
DITE personnel indicated that another reason documentation of the various configurations has not been completed was because the IRS has restrictions on access to the production environment that limit the view of the production environment to only certain individuals. As a result, some of the individuals who have needed access have been unable to access the production environment to review what is running in that environment.
In addition to delays in the IRFOF project deployment that occurred because of problems with configuration documentation, delays have occurred in converting the data from the current security logging system to the system that is being used in the modernized environment. These delays are due in part to not having good documentation of the various environments. If configurations are not clearly documented prior to moving a project into the production environment, problems that occur will continue to be difficult to trace and resolve. Problems that could occur in other systems already in production as a result of the new deployment will also be difficult to resolve if documentation is not available to clearly describe what was deployed.
To ensure documentation of the infrastructure configurations is available, we recommend that the CIO require:
8. The PRIME contractor to focus on documenting the infrastructure configurations in the various environments and begin managing those configurations. This should be done immediately to avoid future problems and delays in the testing and deployment of modernized systems. Additionally, appropriate views of the production environment should be provided to contractor personnel so that configurations can be documented.
A change request is the medium for requesting approval to
change a baselined product or other controlled item. The use of change requests is key to
ensuring proper control over a system.
This process is required by the ELC and is accepted by the BSMO and the
PRIME contractor as the proper method of controlling changes to modernized
systems. In our earlier audit of the
STIR project, we identified that most of the change requests we reviewed did
not have a key component—an impact assessment to show what cost, schedule, or
scope components are affected by the change.
We also found that change requests were not being processed timely. As a result, we decided to follow up to
determine if improvements had been made to address these concerns.
We found that significant
improvements had been made in the consistency of preparation of impact
assessments for the change requests.
However, change requests involving the infrastructure and the FY 2003
projects it supports were still not consistently approved timely, and many were
not processed and either approved or declined by the date the customer needed
them.
We analyzed the 62 change requests
initiated between October 1, 2001, and June 30, 2002, and found:
·
19 (31 percent) requests
without a documented date by which the customer needed the change.
·
25 (40 percent) requests
requiring BSMO approval that had not been signed as of August 2002.
·
9 (15 percent) requests
approved after the date by which the customer needed the change made.
·
9 (15 percent) requests
properly completed and timely approved or declined.
Of the 34 changes that were either
not approved by the BSMO or not approved in a timely manner, 18 were coded as
either critical or high-priority changes.
One reason the customers were not
including the dates the changes were needed is because this field is not listed
as a required field in the ELC. ISS
personnel indicated that one reason the changes were not always approved by the
due dates was because some of these changes were lower-priority. These changes were related to documentation,
and, to be cost-effective, they sometimes hold those until they have a
significant number and then make all the changes at once.
Improvements are needed in the
approval process to ensure change requests are processed efficiently and
high-priority requests receive the timely attention they need. Improvements are also needed in the quality
of the change request to ensure all necessary dates are included on the
form. Without identification of a clear
date on which the change is needed, the reviewers do not have any real impetus
to get the request processed.
Delays in change request
processing can cause corresponding delays in projects while they are awaiting
direction or approval for changes in direction from the change control
board. In addition, a lengthy change
approval process can result in project teams making efforts to avoid the
controls that this process provides.
To improve the processing of change requests, we recommend that the CIO require:
9. The BSMO and PRIME contractor to focus on critical and high-priority change requests that are approaching or are delayed beyond the dates by which the changes are needed. Additionally, organizations or individuals requesting changes should be required to develop reasonable dates by which changes are needed.
10. The need-by date field to be completed on all change request forms.
Risks and issues within the ISS program were not being effectively identified and managed. Of the 2 risks identified by the ISS, only 1 had been assigned, and 6 of the 7 issues identified had mitigation action items overdue by 10 to 232 days.
We believe risks exist in the areas of schedule, integration, and technology, and these risks need to be documented and addressed. One reason that risks are not being effectively identified and managed is because the project personnel are so busy working other project issues, they do not take time to document and manage the risks.
Risk management has been an area of concern for several years. In March 2001, we issued a report on another BSM project that identified problems with risk management. A recommendation was made that the PRIME contractor and BSMO managers complete the evaluation and implementation of the new risk tracking and reporting process. The BSMO stated that a bi-weekly risk forum had been established to review and approve proposed risks and mitigation plans.
In November 2001, we issued another report recommending that increased emphasis be placed on identifying, tracking, and mitigating risks at the BSMO program level. The BSMO stated that a Risk Process Action Team had been established to focus on improving implementation and execution of the risk management process across the BSM program.
Although these actions may have occurred, we believe that risks and issues with the ISS program are not consistently identified and tracked to closure. Tracking risks and issues, as well as their criticality ratings, is meaningful because risks and issues that are not dealt with can easily evolve into cost overruns, schedule slips, or serious technical problems. Preventing undesirable outcomes is one of the main goals of program management, and the monitoring and timely execution of the appropriate mitigation strategy are the primary means to this end.
To address issues with risk management in the infrastructure, we recommend that the CIO require:
11. The BSMO to ensure that risk management processes are being followed and that risks and issues are timely and effectively identified, tracked, and mitigated.
Appendix I
Detailed Objective, Scope,
and Methodology
The overall objective of this review was to determine whether the Internal Revenue Service’s modernized infrastructure being developed by the Infrastructure Shared Services (ISS) program would be adequate to support the release of Business Systems Modernization (BSM) projects in Fiscal Year (FY) 2003. To complete our objective, we followed up on issues reported in an earlier audit of the Security and Technology Infrastructure Release (STIR) and conducted new tests to evaluate the progression of the ISS program. To accomplish our objective, we:
I. Evaluated the plans developed by the infrastructure sub-projects to design, acquire, and test key components of the modernized infrastructure and the projects they support.
A. Obtained a high-level understanding of the infrastructure functionality to be provided to the various BSM projects in the FY 2003 release.
B. Evaluated the progress made in the design and acquisition of software and hardware, and in the preparation of the testing environment, for the FY 2003 release.
II. Determined whether key management disciplines have been developed and implemented.
A. Reviewed the infrastructure cost increases to the STIR and identified the causes of these increases as listed in the business case.
B. Reviewed the most current processes established for configuration management. We reviewed change requests initiated by the infrastructure sub-projects during October 1, 2001, through June 30, 2002, and determined whether proper configuration management controls were used.
C. Reviewed the most current processes established for risk management. We compiled a list of potential risks identified during our review of project documentation and evaluated mitigation plans for each documented risk identified by the project.
D. Obtained an organization chart for the ISS program and analyzed the chart to determine whether the modernized infrastructure team is appropriately staffed to support the FY 2003 release.
E. Reviewed the cross-project dependency analysis conducted in the design, acquisition, and test sections of Step I. above for each business project and determined whether the infrastructure projects were adequately supporting delivery of the FY 2003 release.
Appendix II
Major Contributors to This Report
Margaret E. Begg, Acting Assistant Inspector General for
Audit (Information Systems Programs)
Scott A. Macfarlane, Director
Tammy L. Whitcomb, Audit Manager
Michelle Griffin, Senior Auditor
Jimmie Johnson, Sr., Senior Auditor
Phung Nguyen, Senior Auditor
Charles Winn, Senior Auditor
Louis Zullo, Senior Auditor
Appendix III
Commissioner N:C
Deputy Commissioner for Operations Support N:OS
Associate Commissioner, Business Systems Modernization M:B
Deputy
Associate Commissioner, Systems Integration
M:B:SI
Chief, Information Technology Services M:I
Director, Infrastructure Modernization M:B:SI:IM
Acting Director, Portfolio Management M:R:PM
Chief Counsel
CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis N:ADC:R:O
Office of Management Controls N:CFO:AR:M
Audit Liaison: Associate Commissioner, Business Systems Modernization M:B