Executive Summary

Objective and Scope

Background

Results

conclusion

Detailed Objective and Scope

Management Response to Draft Report

Executive Summary

Based on prior Internal Audit recommendations, Service management has initiated actions to improve the Service’s Year 2000 certification efforts and improve the accuracy of the Service’s Applications Program Registry (APR) data. The Service is also working toward being classified as a capability Maturity Model Level 2 organization to institutionalize effective management processes for software projects, which will allow for repetition of successful practices developed on earlier projects. We recognize these efforts are not yet completed.

Our results emphasize the need for the Service to continue with its efforts to improve the accuracy of the APR. The Service also needs to ensure Service developers and testers document test activities as well as contractors, and the Service needs to give greater consideration to how Year 2000 programming changes will affect hardware capacity and system performance on Tier 2 and Tier 3 systems.

The overall objective of this audit was to perform an assessment of Phase 4 of the Service’s Year 2000 conversion and testing efforts.

In addition to the three specific findings of this audit, we noted that the century Date change (cDc) Project Office has not established any controls to verify the accuracy of Year 2000 compliance certifications. Since these are not validated, the cDc Project Office is unable to fully assess the risk that exists for programs that are not actually Year 2000 compliant.

At the end of our audit fieldwork, efforts to link components tracked on the APR with a standardized project name and phase had not yet been completed. In addition, as previously reported, conversion and testing dates on the APR did not accurately reflect the status of Phase 4 Year 2000 conversion activities.

The Product Assurance Division cannot effectively monitor the status of project conversion and testing efforts, follow up on projects falling behind schedule, accurately report the status of conversion and testing, or make sound project and configuration management decisions when inaccuracies exist on the APR.

Project files prepared by contractors were complete, cross-referenced, and easy to follow, unlike those prepared by Service developers and testers. The Service’s project files did not always reflect testing for invalid conditions or the reasons why invalid testing was not performed.

Insufficient documentation of programming and testing activities can delay Year 2000 project certification and hinder the Service’s efforts to be classified as a capability Maturity Model Level 2 organization. Given the significance of the Year 2000 programming efforts, testing for invalid conditions should be performed to identify and avert any potential system errors or failures.

The Service does not have a corporate plan in place to address the capacity management and performance evaluation issues that may arise due to Year 2000 conversions. Quarterly assessments of Year 2000 capacity and performance issues will be done for Tier 1 systems; however, adequate oversight consideration is not being given to how Year 2000 programming changes will affect hardware capacity and system performance of Tier 2 systems. In addition, no capacity or performance evaluations have been completed for systems that will be running Year 2000 compliant applications software on Tier 3 hardware, such as the Integrated collection System (IcS) which supports over 10,000 workstations.

If capacity and performance evaluations of major Tier 2 and Tier 3 systems are not performed, the potential exists that system timeliness could be impaired or that additional hardware capacity could be needed.

This report makes seven recommendations for the Service to ensure it can effectively meet its century date change needs. In summary, they are:

• The Product Assurance Division should monitor organizational efforts to re-certify the data on the APR.
• Service developers and testers should document and cross-reference programming and testing activities and prepare data to test for invalid conditions or document why such tests are not required.
• The Service needs to proactively consider and evaluate potential capacity and performance issues in preparing for its Year 2000 systems environment.
  

Management Response: Management’s response was not available for inclusion in the report at the time the final report was issued. We were informed that management is developing actions to address our concerns and will provide us with a written description of their proposed corrective actions at a later date.

Objective and Scope

This audit was initiated as part of Internal Audit’s continual emphasis on century date change issues. The overall objective was to perform an assessment of Phase 4 of the Service’s Year 2000 conversion and testing efforts; however, the scope of this audit did not include tests to assess code review activities for Information Systems components.

We conducted this audit at the National Office from March through June 1998. Audit work was performed in accordance with generally accepted government auditing standards. Attachment I contains the detailed objective and scope of our audit.

Management’s response was not available for inclusion in the report at the time the final report was issued. We were informed that management is developing actions to address our concerns and will provide us with a written description of their proposed corrective actions at a later date.

Background

The Year 2000 century change is one of the most critical problems facing many organizations’ data processing efforts. To maximize system processing capabilities and to preserve data storage space, many date fields in system programs and applications have been limited to a two-digit year representation (i.e., 97 for 1997). The Service uses this form of two-digit year representation for its programs and applications. As a result, the Service’s existing date routines will not be able to recognize the changes required by the Year 2000. This problem is extremely critical to the Service, as many of its tax processing and collection functions are date driven. Additionally, numerous other Service operations, including law enforcement, personnel, and procurement, are also highly date driven.

Results

Based on prior Internal Audit recommendations, Service management has initiated actions to improve the Service’s Year 2000 certification efforts, and improve the accuracy of the Service’s Applications Program Registry (APR) data. The Service is also pursuing capability Maturity Model (cMM) Level 2 classification. To achieve this, basic project management processes must be established for software managers to track system functionality. We recognize these efforts are underway and are not yet completed. Our continued appraisal of these conditions during Phase 4 emphasizes the need for completion of planned actions in these areas and has identified other issues to help the Service accomplish necessary actions for Year 2000 functionality.

To assess conversion activities for Phase 4 non-Information Systems (non-IS) components, we selected a judgmental sample of 100 non-IS components from the APR. We reviewed the components to evaluate their compliance with the Service’s Year 2000 standards. Our audit did not indicate there were programming problems with Phase 4 non-IS programs; however, we noted there is no control set up to verify the accuracy of Year 2000 compliance certifications.

The century Date change (cDc) Project Office has developed and issued procedures and standards for Year 2000 compliance. The procedures include a certification by the programmer that the component is Year 2000 compliant. After the component has undergone conversion, the code developer tests it. These certifications are used to update the Year 2000 status of the component on the APR; however, the cDc Project Office does not validate the certifications. By not validating these certifications, the cDc Project Office is unable to fully assess the risk that exists if programs are not actually Year 2000 compliant.

Our audit tests found the APR inventory data continues to be incomplete and inaccurate. Fourteen of the 100 components in our sample of Phase 4 non-IS components were not correctly recorded as retired or to be retired, or were rescheduled for conversion in Phase 5. Our audit work pertaining to Product Assurance Division activity also found incomplete and inaccurate APR inventory data.

Our review also found:

• Documentation of programming and testing activities needs to be consistent and complete (similar to issues previously reported).
• The Service needs to address the impact of Year 2000 programming changes on hardware capacity and system performance of Tier 2 and Tier 3 systems.

APR inaccuracies remain a recurring issue.

The APR does not reflect the accurate status of Year 2000 components. At the end of our audit fieldwork, efforts to link components tracked on the APR, which is part of the Integrated Network and Operations Management System with a standardized project name and phase, had not yet been completed. In addition, as previously reported during our Phase 3 audit, conversion and testing dates on the APR did not accurately reflect the status of Phase 4 Year 2000 conversion activities. For instance, we identified several components recorded as Phase 4 on the APR that were not actually Phase 4 projects. Nine of the 16 projects in the original sample of Phase 4 projects we chose to review were incorrectly documented as Phase 4 projects on the APR. In addition, we noted that many of the Systems Acceptability Testing (SAT) and Production Transmittal dates on the APR were inaccurate.

The Product Assurance Division serves as the central point of contact and control for Product Assurance testing activities relating to the Year 2000 conversion. It is primarily a coordinating entity since project-specific management remains the responsibility of the individual testing branches. The Product Assurance Division’s duties include ensuring unit and SAT tests are planned, scheduled, and conducted within specific time frames. To meet its objectives, the Product Assurance Division was required to develop and maintain a master test schedule, track testing progress, and provide testing status to the IRS cDc Project Office. The APR was developed to track component conversion progress and certification.

Prior to inception of the APR, components were not tracked on a centralized database. Once the APR was established, developers and testers did not update the APR on a regular basis to ensure data was accurate. In some instances, as previously noted, developers have not consistently linked components on the APR with a project name or claimed ownership of remaining components.

The Product Assurance Division cannot effectively monitor the status of project conversion and testing efforts, follow up on projects falling behind schedule, or accurately report the status of conversion and testing on the APR when inaccuracies exist. If developers and testers do not accurately update APR data, cDc Project Office and Service management will not be able to effectively oversee the Year 2000 conversion process and make sound project and configuration management decisions in the future.

1. The cDc Project Office should ensure that the Product Assurance Division monitors organizational efforts to re-certify the data on the APR, ensuring also that components tracked and reported on the APR are linked with the correct phase and status.

Service developers and testers do not always clearly and completely document test activities.

During our audit of Phase 4 Year 2000 components, some of which were subjected to systems acceptability testing by Product Assurance, we identified (from our review of available project files) that Service developers and testers do not always clearly and completely document test activities. A significant difference was noted while reviewing project files prepared by contractors secured to support systems acceptability testing. Files prepared by contractors were complete, cross-referenced, and easy to follow. In addition, contractors clearly tested invalid conditions or reasons why this testing was not necessary or performed. conversely, testing for invalid conditions was not always evident in Service project files, nor were testers documenting the reasons why invalid testing was not performed.

Established guidelines are available to provide developers and SAT testers with the requirements and guidance necessary to perform unit and systems acceptability testing. The Developer’s Testing Guidelines, SAT Guidelines, Procedures for Testing Year 2000 changes, IRM 2600: Quality Systems Testing Procedures and Guidelines, and cDc Project Office memorandums recommend that pre-determined results be prepared for every test condition specified in Program Requirements Packages, Functional Specifications Packages or other appropriate functional requirements documentation. As a "rule of thumb," pre-determined results should include data for invalid test case conditions.

According to the guidelines, project file documentation should contain clear and correct documentation related to the development and maintenance of a component. All testing material should be cross-referenced with appropriate identifying information. Documentation maintained in project files is necessary for review during the Year 2000 certification process and for an organization to be established and classified as Level 2 within the cMM.

contractors are required to follow established guidelines, under the terms of their contracts, in preparing project documentation. This documentation includes performing tests of invalid data. However, unit and SAT guidelines remain as recommended guidance to Service developers and testers and are not uniformly followed. Due to time constraints, Service developers and testers continue to rely on system tests that have been used for non-Year 2000 related SAT typically performed for tax year changes. In addition, Service programmers and testers continue to maintain project file documentation as they routinely have in the past.

Insufficient documentation of programming and testing activities can delay Year 2000 project certification and hinder the Service’s efforts to be classified as a cMM Level 2 organization. Given the significance of the Year 2000 programming efforts, testing for invalid conditions should be performed to identify and divert any potential system errors or failures.

Recommendations

The Deputy chief Information Officer, Systems Development, should ensure that Service developers and testers:

2. Be required to rigorously follow available guidelines to clearly and completely document and cross-reference programming and testing activities.

Greater consideration needs to be given to how Year 2000 programming changes will affect hardware capacity and system performance.

Although the Service has a capacity Management Program in place, adequate consideration is not being given to how Year 2000 programming changes will affect hardware capacity and system performance. For example, the mission of the Service’s capacity Management Program does not include a provision for Year 2000 capacity and performance evaluations. Also, the Service has not begun to consider how the Service center consolidation will impact the hardware capacity and system performance of Tier 2 systems that will be moved from non-Year 2000 compliant platforms to Year 2000 compliant platforms. In addition, no capacity or performance evaluations have been completed for systems that will be running Year 2000 compliant applications software on Tier 3 hardware, such as the Integrated collection System (IcS) which supports over 10,000 workstations.

capacity management in the Service is the process of maintaining adequate resources necessary to process a given workload and deliver a required level of performance. The capacity Management Program is responsible for providing centralized management for all computing centers and managing the capacity of multiple systems and architectures, including: mainframes (Tier 1 systems), minicomputers (Tier 2 systems), and workstations (Tier 3 systems). A capacity management program must incorporate the measurement and evaluation of performance, as well as the projection of future workload and performance requirements, for planning purposes.

The Service does not have a corporate plan in place to address the capacity management and performance evaluation issues that may arise due to Year 2000 conversions. Also, the capacity Management Program is not as visible as it could be. For example, the Program’s web page, published under the Tier 2 web page, is not linked to the Year 2000 home page. Additionally, the Program has not taken a proactive role in addressing Year 2000 capacity and performance issues when evaluating systems.

While the cDc Project Office tasked the Mainframe capacity Management Section with providing a yearly assessment of Year 2000 capacity and performance issues by site for Tier 1 systems, it has not tasked such studies of Tier 2 or Tier 3 systems. The Distributed capacity Management Section (Tier 2 systems) has only conducted one capacity/performance study, which was requested by the user. We did not identify any studies done for Tier 3 systems.

If capacity and performance evaluations of major Tier 2 and Tier 3 systems are not performed, the potential exists that system timeliness could be impaired or that additional hardware capacity could be needed. For example, with many systems being consolidated into centrally maintained sites, the system workload and number of users could significantly increase, warranting the need for additional capacity so that system performance is not hindered. The hardware required for the additional capacity needed may not be readily available because it must be purchased via a procurement vehicle.

If Tier 2 systems, such as the Integrated Submission and Remittance Processing Systems (ISRP), are not converted to the correct Year 2000 compliant platform based on the workload and number of users, a system’s functionality may also be impaired. For example, the Service center consolidation may have an effect upon a system’s functionality, regarding the speed in which workload is processed. In addition, confusion over the classification of platforms as Tier 2 or Tier 3 has made it difficult to associate hardware with software applications (and software applications with the hardware) necessary to run fully Year 2000 compliant versions of software.

5. For major Tier 2 and Tier 3 systems, such as the ISRP and IcS, capacity studies and performance evaluations may need to be conducted before a system is certified as Year 2000 compliant.
6. The capacity and Performance Management Technical Handbook should be updated to detail which sections are responsible for capacity studies and performance evaluations. The Handbook should also provide detailed instructions outlining the steps that need to be conducted so that field organizations would be able to conduct their own evaluations if there is a lack of National Office resources to conduct these studies.
7. The capacity Management Program should also create a link between their web site and the Service’s Year 2000 home page for higher visibility and access to information.

conclusion

Service management has accepted a number of Internal Audit recommendations and is in the process of implementing numerous other actions. Efforts to improve the accuracy of the APR should continue with additional emphasis on efforts to re-certify the data and ensure components are linked with the correct phase and status.

The Service also needs to act to ensure it can effectively meet its century date change needs by:

• Ensuring Service developers document test activities as well as what contractors are doing.
• Giving greater consideration to how Year 2000 programming changes will affect hardware capacity and system performance on Tier 2 and Tier 3 systems.

/s/ David H. Newman

David H. Newman

Audit Manager

Audit team members:

• Barbara Bartuska, Auditor
• Ed coleman, Audit Manager
• Eleonor Lindner, Auditor
• Joseph R. McGeehan, Auditor
• Kathryn K. MacMillan, Senior Auditor/Team Leader
• A. Michael Stevens, Senior Auditor/Team Leader
• Kimberly A. Woodard, Auditor

Detailed Objective and Scope

The overall objective of this audit was to perform an assessment of Phase 4 of the Service’s Year 2000 conversion and testing efforts. Specifically, we:

1. Assessed conversion activities for Phase 4 non-Information Systems (non-IS) controlled components by:
1. Selecting a judgmental sample of 100 modified Phase 4 components.
2. Reviewing source code (and related documentation) of 90 of those 100 components to determine if all Year 2000 changes have been identified and made. (We did not review the other 10 components because the owning organization reported that the component should have been marked as to be retired, had been retired, or was rescheduled for conversion in a later phase.)
3. Examining Source code compliance forms or comparable documentation for the 90 components reviewed to determine whether the completion status was accurately reported to the century Date change (cDc) Project Office.
4. Examining Unit Test checklists, Test Readiness Review Reports, or comparable documentation for each functional area owning the 90 components reviewed to assess efforts for performing code review before placing programs into production.
2. Assessed the quality review and testing of Phase 4 components and determined whether testing was sufficient before components were placed into production by:
1. Selecting judgmental samples of Phase 4 components for:
1. Nine Information Systems (IS) projects scheduled for Systems Acceptability Testing (SAT).
2. Five IS projects not scheduled for SAT by Product Assurance Division.
3. Three non-IS projects not scheduled for SAT by Product Assurance Division.
2. Analyzing the samples to determine whether Product Assurance had gathered adequate test data and prepared test scenarios for testing Phase 4 Year 2000 changes and by:
1. Determining whether the inventory accurately reflected Phase 4 components.
2. Determining whether testing was progressing as planned, based on a review of a cDc Tracking Status Report dated April 6, 1998.
3. Interviewing Service and contractor developers and testers.
3. Reviewing open Phase 4 problems identified from the 5534 System for any problems or trends being identified during Year 2000 testing.
4. Obtaining and reviewing the Problem Reporting Measurements Report, which identified 5534s by issue area.
3. Assessed the Service’s efforts to ensure the impact of Year 2000 changes have been considered and tested on its systems environment by:

1. Determining whether plans were prepared to test performance and capacity for Year 2000 changes that:
0. considered the effect field expansion will have on capacity.
1. Addressed the capacity implication on existing equipment for performance.
2. Included specifications or models to identify capacity requirements.
2. Determining whether plans were prepared to convert and test the hardware items necessary to support Year 2000 changes by ascertaining whether there were:
0. Plans in place to get additional equipment.
1. Plans to test the new equipment.

Management Response to Draft Report