Improvements
Are Needed in the Telecommunications Data Network Management Program
November 2001
Reference
Number: 2002-20-026
This report has cleared the Treasury Inspector General for Tax Administration (TIGTA) disclosure review process and information determined to be restricted from public release has been redacted from this document.
November
30, 2001
MEMORANDUM FOR
DEPUTY COMMISSIONER FOR MODERNIZATION & CHIEF INFORMATION OFFICER
FROM: Pamela J. Gardiner /s/ Pamela J. Gardiner
Deputy Inspector General for
Audit
SUBJECT: Final Audit Report - Improvements Are
Needed in the Telecommunications Data Network Management Program (#200120007)
This report
presents the results of our review of telecommunications data network
management and capacity planning. The
overall objective of the audit was to determine whether the Internal Revenue Service (IRS) Information
Technology Services (ITS) organization effectively implemented telecommunications data network capacity and
performance management policies and procedures.
The ITS Telecommunications Division is responsible for purchasing,
operating, and maintaining networks for more than 100,000 employees in over 750
locations. Several actions have been initiated to
accomplish the strategic goal of improving the IRS’ ability to manage telecommunications
services. These actions include
upgrading telecommunications technology, developing network performance
measures, and initiating service level agreements defining the network performance
to be provided to users. Additional
actions in the following areas would further improve the network management
program.
Network
management policies and procedures do not provide defined
network monitoring responsibilities. As
a result, duplicative monitoring activities may be costing the IRS about $1
million a year. Additional network
management program weaknesses include network monitoring tools that cannot
access all wide area network equipment, the lack of defined network problem
resolution timeliness standards, and the lack of periodic identification of
recurring network problems. Also,
network traffic is not proactively analyzed to optimize network
performance. For example, about 10
percent of the traffic routed through the Internet firewall, such as Internet
advertisements, could be eliminated.
Improving network management efforts would ensure the IRS effectively
uses resources and maintains a reliable network that meets users’ needs.
IRS policies
and procedures do not require network capacity planning and the use of
analytical modeling techniques when planning major network changes that could
affect current network capacity and performance. Due to ineffective network capacity planning
efforts, network traffic volumes on 12 of
306 upgraded circuits used less than 10 percent of the circuits’ capacity, and
37 upgraded circuits that cost approximately $1.9 million to install were
disconnected after being used for only 1 to 11 months.
In addition,
change management procedures designed to maintain network integrity and security were not followed. During September through November
2000, 14 of 28 Change Management Tracking System
(CMTS) records did not include documented management approval and/or
test results, and all 132 Treasury Communications System (TCS) service requests
were not documented on the CMTS as required.
Also, change management computer software was not implemented to enhance
oversight procedures.
Management’s
Response: IRS management
agreed with the recommendations presented in the report. Corrective actions taken and planned will
improve network monitoring and performance, capacity planning, and change
management procedures. Management’s
complete response to the draft report is included as Appendix V.
Copies of this
report are also being sent to the IRS managers who are affected by the report
recommendations. Please contact me at
(202) 622-6510 if you have questions or Scott Wilson, Assistant Inspector
General for Audit (Information Systems Programs), at (202) 622-8510.
Actions Were
Initiated to Accomplish the Telecommunications Strategic Goal
Network Management Efforts Could Be Improved
Network Capacity Planning Could Be Improved
Change Management Controls Should Be Enforced
Appendix I – Detailed Objective, Scope, and Methodology
Appendix II – Major Contributors To This Report
Appendix III – Report Distribution List
Appendix IV – Outcome Measures
Appendix V – Management’s Response to the Draft Report
The Fiscal Year (FY) 2001 – 2002 Internal Revenue Service
(IRS) Information Technology Services (ITS) Strategy and Program Plan includes
a strategy to “Improve IRS’ Ability to Manage Telecommunications
Services.” To accomplish the strategy,
the ITS function plans to leverage technology, implement organizational change,
adopt best practices, and invest in its
employees.
The ITS Telecommunications Division had a $139 million FY 2001 budget to provide data services to more than 100,000 employees in over 750 locations. The Telecommunications Division will pay Treasury Communications System (TCS) contractors $80 million (58 percent of the data services budget) for Wide Area Network (WAN) operations and maintenance services. Approximately $59 million will be spent on Department of the Treasury and TCS fees, services, and Program Management Office (PMO) support; the IRS National Network Management Center (NMC) contractor support; the purchase and maintenance of IRS-owned data network equipment; and other costs.
The Telecommunications
Division organizations with network responsibilities include:
·
The WAN Office –
Purchases and installs WAN circuits and equipment.
·
The National NMC –
Monitors, troubleshoots, and tests the WAN.
·
The
Telecommunications Engineering and Project Management Office – Designs network
changes and develops performance measures.
In addition, computing center, IRS center, and area office ITS employees monitor, troubleshoot, and test the WAN and Local Area Networks (LAN) to assure network availability. These employees currently are assigned to the Corporate Computing, Service Center Operations, and Desktop Management offices.
The International Organization for Standardization (ISO) established information technology standards that apply to data communication networks. Telecommunications management informed us that they are working to achieve key network management functions recommended by the ISO, including:
· Fault management – Monitoring and controlling the network to ensure that a problem in one location does not disable the remainder of the network.
· Configuration management – Controlling network changes to ensure that all changes are authorized.
· Performance management – Monitoring and controlling the network to meet the users’ quality of service needs (i.e., timeliness, availability, and reliability).
Audit work was conducted in the ITS National Headquarters; Martinsburg Computing Center (MCC); Tennessee Computing Center (TCC); Atlanta, Austin, and Memphis IRS Centers; and Midstates and Southeast Area Offices during November 2000 through July 2001. This audit was scheduled as part of the Treasury Inspector General for Tax Administration’s (TIGTA) Annual Audit Plan and performed in accordance with Government Auditing Standards. Detailed information on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed in Appendix II.
Telecommunications management recognized that accomplishment of the strategic goal to improve the IRS’ ability to manage telecommunications services required improvements in network performance and management. The actions initiated to improve the network include:
· Replacing older WAN circuits with current technology and redesigning the WAN configuration to increase capacity, and improve network management, problem resolution, and performance.
· Developing network performance measures that support the IRS’ Balanced Measures.
· Initiating development of service level agreements with IRS business units that will document the network service levels that users will be provided.
· Developing the Enterprise Help Desk, a new system that should improve network problem reporting.
Additional management actions in the following areas would further improve the network management program.
Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources,
requires management to ensure that networks are cost effective. The OMB Circular also requires management to
record and maintain sufficient
information to ensure management and accountability of programs and to
protect the IRS’ legal and financial rights.
The ISO standards state that the fault management function includes reacting to and correcting
problems as soon as possible to minimize the impact on the entire network. The performance management function includes
monitoring and controlling the network to meet the timeliness,
availability, and reliability requirements of users.
The IRS has
not implemented nationwide policies and procedures for network
monitoring responsibilities, established problem resolution timeliness
standards, and documented traffic and performance analyses.
Management advised us that these policies and procedures were not
implemented because their immediate priority was upgrading the network
infrastructure and completing the
ITS reorganization. Increased
management attention to the following areas would improve network management
effectiveness and allow staff resources to be assigned to work that is more
productive.
Network monitoring efforts are duplicated and network monitoring tools
cannot access all equipment
Current network monitoring procedures do not establish
nationwide network monitoring responsibilities, resulting in duplicate network
monitoring and the inability of network management tools to electronically
access all WAN equipment.
·
The
NMC, MCC, and TCC network monitoring operations are staffed around the clock
and monitor the network to identify problems.
Twenty-nine MCC and TCC Telecommunications Specialists (IRS employees)
and 20 NMC Network Engineers (IRS contractors) monitor the WAN and troubleshoot
and report problems. Review of the IRS
employees’ and contractors’ duties showed that 22 of the 29 MCC and TCC
employees’ duties duplicated the NMC Network Engineers’ duties. By eliminating
the overlapping duties, management could reassign the 22 employees to other
duties resulting in $4.8 million in resources being put to better use over 5
years (see Appendix IV for details on our calculation of the cost).
Management stated that the NMC is going to be moved to a
computing center. The
Telecommunications Division has recently assembled a team of managers that is
looking at various alternatives for this move including the possibility of
implementing a distributed network monitoring process. In addition, the IRS’ systems
modernization and consolidations at the MCC and TCC have increased their
network responsibilities.
·
MCC and TCC
employees and NMC and TCS contractors use similar
electronic tools to monitor the WAN.
These electronic tools identify each circuit and piece of equipment that
makes up the network and monitor the network to detect problems (e.g.,
interrupted service and slow response times).
The accuracy and completeness of this information is critical because
employees and contractors review the problem alerts generated by the tools and
network diagrams to troubleshoot problems and determine the appropriate
corrective actions.
However, the NMC and TCS tools could not electronically access all WAN circuits and equipment. NMC employees explained that their access was limited because
employees in local IRS offices had not shared with them the network equipment
access codes.
As of March 2001, the TCS contractor had completed 10
network diagrams for 2 of the 8 offices selected for the audit. The diagrams identified 137 WAN circuits but
did not provide all of the information (e.g., destination names) needed to
manage 46 of the circuits.
Telecommunications management subsequently advised us that NMC engineers
completed detailed network diagrams for approximately 90 percent of the WAN,
but their tools’ access to the remaining network equipment is still
limited. Problems that occur after
local office business hours in locations that cannot be monitored by the NMC
and TCS contractors may not be identified promptly, which could delay
resolution and prevent employees from conducting their work when they return
the following day.
Network problem resolution timeliness standards have not been defined and
incorporated into the status reporting process
Network
problems are resolved by TCS and NMC contractors and IRS employees. The TCS contract defines performance
standards, including various time periods for restoring interrupted service,
and the TCS contractor is required to design, implement, operate, and
maintain the network to meet the standards selected by the bureaus (e.g., IRS).
However, the IRS has not established problem resolution timeliness
standards for the TCS contractor and the IRS employees and NMC contractors who
resolve problems.
The TCS
contract minimum standard requires the contractor to restore interrupted
service within 8 business hours (or 1 business day). The minimum
TCS standard was applied in a review of 91 problem tickets randomly selected
from the 457 problem tickets recorded on the NMC’s Log Management System (LMS)
between September and November 2000.
The results of the review show that improvements are needed to more
timely resolve network problems (see Table 1).
|
Problem Resolved By: |
0 – 1 |
1 - 2 |
2 - 5 |
Over 5 |
CND |
Total |
|---|---|---|---|---|---|---|
|
TCS |
55 |
11 |
7 |
4 |
1 |
78 |
|
IRS and NMC |
7 |
2 |
0 |
2 |
2 |
13 |
Source: Sample of 91 of 457 problem tickets recorded
on the LMS between September and November 2000.
Note: CND (Could Not Determine) - Three
problem tickets and the associated documentation did not show the resolution
dates and times.
NMC procedures require contractors to prepare daily Network
Event Status Reports for Telecommunications management that include all
unresolved and recently resolved network circuit problems. Management advised us that they discuss
unresolved problems identified in the status reports with the Department of the
Treasury TCS PMO and contractors in monthly meetings. The 91 sampled problem tickets were also reviewed to determine
whether the problems were reported in the daily Network Event Status
Reports. The review found that
improvements are needed to ensure problems are reported in the Network Event
Status Reports (see Table 2).
|
Problem Resolved by: |
Problems Not in Status Reports |
Total Problems Reviewed |
|---|---|---|
|
TCS |
41 |
78 |
|
IRS and NMC |
4 |
13 |
Source: Sample of 91 of 457 problem tickets recorded
on the LMS and Network Event Status Reports between September and November
2000.
Timely resolution of network
problems is important to ensure the network remains operational and meets the
needs of users. Also, accurately
preparing status reports assures management is aware of network problems, has
the necessary information to assign the appropriate resources to address the
problems, and has complete information when meeting with the PMO and
contractors.
Analyses of recurring network problems are not periodically performed
Telecommunications management advised us that they analyze problem tickets to identify circuits with recurring problems (i.e., two or more problems occurring in a short period of time), which helps them ensure that recurring problems are properly resolved. However, management provided documentation of only one IRS analysis of recurring problems, dated September 2000, that was submitted to the PMO.
The analysis listed nine circuits with recurring problems, and the PMO subsequently reported back to Telecommunications management that a TCS contractor was working on one chronic circuit (a circuit that has four or more problems in the month) and would continue to monitor three others. In addition, the TCS contractor prepares monthly TCS Trouble Ticket System (TTS) Reports that identify chronic problems, but the TTS Reports were not timely received by the IRS. As of July 2001, the TTS report dated September 2000 (covering problems that occurred in August 2000) was the most recent report that the IRS received.
Periodic IRS
analyses would have identified additional circuits with recurring
problems. A review of the 457 problem tickets recorded on the
LMS between September and November 2000 identified 40 circuits with 2 to 5
problems that occurred after the September 2000 IRS analysis, including:
· One circuit with 5 problems reported in 28 days.
· One circuit with 4 problems reported in 12 days.
NMC procedures do not require periodic analysis of network problems. However, effective network performance management should incorporate periodic problem analyses to ensure the network is available and reliable in meeting users needs.
Network traffic is not analyzed to identify optimization opportunities
Telecommunications
management has not issued policies and procedures requiring periodic analysis
of network traffic to identify optimization opportunities. If network traffic is reviewed and results
shared with other organizations that could resolve the identified problems,
network performance could be improved.
For
example, our analysis of the IRS Internet firewall audit logs for September
through November 2000 identified unnecessary traffic that wasted network
resources, including:
·
Internet
advertisements that accounted for 123 billion of 1,449 billion bytes (8
percent) of traffic through the firewall.
In response to this analysis, the Telecommunications Division began
implementing controls that block Internet advertisements. Management’s action will conserve network
resources and result in improved network response times.
·
Misrouted
IRS Intranet traffic (e.g., traffic with sources and destinations on the IRS
internal network was routed through the external Internet firewall) that
accounted for 25 billion of 1,449 billion bytes (2 percent) of traffic through
the firewall. The vast majority of the
sources appear to be IRS employees’ workstations. However, the sources for 19 percent of the misrouted traffic were
5 network devices, including 3 proxy servers used to route users’ traffic to
the correct destination address.
Periodic
traffic analysis by Telecommunications Division personnel may also identify
other low-cost opportunities to further improve the network’s
effectiveness. Although
Telecommunications management advised us that they are not responsible for the
settings on systems and desktop computers connected to the network, they are
responsible for the efficiency of the network and communicating the results of
proactive analyses to the organizations that are responsible for the
equipment. By sharing pertinent
information and taking necessary corrective action, network resources could be
conserved and network response times improved.
The
Deputy Commissioner for Modernization & Chief Information Officer should:
1. Establish WAN monitoring responsibilities that eliminate duplicate monitoring efforts and reassign employees displaced by revised network monitoring responsibilities to more productive work.
Management’s Response: Management plans to review IRS network management capabilities and implement the necessary organizational, contractual, and procedural changes to complete realignment of responsibilities and functions.
2. Issue instructions to ensure network monitoring tools can electronically access all WAN equipment.
Management’s Response: Management plans to complete the deployment
and refinement of the network management tools.
3. Define problem resolution timeliness standards for telecommunications contractors and employees and monitor the resolution of reported problems.
Management’s Response: Management plans to review the problem identification process and align it with industry best practices. A procedural guide will be developed to assist network monitors to determine the severity of an outage. The guide will also assist in providing network restoration timelines. Furthermore, management will establish a process to periodically review trouble handling.
4. Incorporate problem resolution timeliness standards into the Network Event Status Report and ensure all problems are included in the daily status report.
Management’s Response: Management plans to continue to work
with the Department of the Treasury and the TCS vendor to assess problem
resolution responsiveness and timeliness.
Also, management plans to use the Enterprise Help Desk and the
Information Technology Asset Management System (ITAMS) network problem
reporting and tracking procedures to improve problem resolution. Furthermore, network outages will be
documented and compared to Service Level Agreement standards currently being
developed.
5. Conduct and document monthly network problem analyses to identify recurring problems.
Management’s Response: The Telecommunications Division established
a monthly forum with the Department of the Treasury and TCS vendors to discuss
network management issues, including chronic network problems. Management also plans to partner with the
Department of the Treasury and TCS vendors to analyze and resolve root causes
for repetitive problems.
6. Complete efforts to block Internet advertising.
Management’s Response:
The Telecommunications Division
prepared a plan to filter Internet traffic and block Internet
advertisements. Management is also
working to procure the needed hardware and software.
7. Require periodic network traffic analyses and initiate corrective actions or communicate the results with responsible organizations.
Management’s Response: The Telecommunications Division is establishing a capacity planning staff and working with the Business System Modernization Program Office (BSMO) to use an analytical modeling tool to baseline the current network and assist in periodically analyzing network traffic.
The Treasury Information Technology Manual, TD P 81-10, requires bureaus (e.g., the IRS) to implement planning processes that ensure investments reduce costs and improve the effectiveness of work processes. Analytical modeling is a capacity planning technique that uses historic network performance data and future requirements to simulate the proposed network changes and estimate their impact on network performance. IRS policies and procedures do not require network capacity planning and the use of analytical modeling techniques when planning major network changes that could affect current network capacity and performance.
The IRS purchased analytical modeling computer software and a Business Systems Modernization contractor is using it to help determine the future network capacity changes needed for the IRS’ future tax processing and compliance systems. However, the ITS function has systems development initiatives, such as the Tier II Consolidation Project and the National Print Strategy, that will significantly change current network traffic volumes and patterns and analytical modeling techniques were not being used. These modeling techniques were not being used by the ITS function because management had not allocated resources to train Telecommunications engineers or hired contractors with the required skills. Telecommunications management advised that, in May 2001, they started using modeling techniques to plan network changes for the employee travel reporting system that is part of the Tier II Consolidation Project.
Weekly TCS
circuit performance reports were analyzed to evaluate the effectiveness of
capacity planning. Analysis of the peak
traffic volume reports for February 2001 showed that some upgraded circuits and
associated ports might have been sized incorrectly. Specifically:
·
Twelve
of 306 WAN circuits never used more than 10 percent of the available capacity
during the entire month; a pattern indicating the circuits might be
oversized. Of these 12 circuits, 10
connected computing centers and IRS centers with each other or large offices.
·
Sixteen
of 68 ports discarded traffic on 2 or more occasions during the month when the
network traffic exceeded the ports’ capacity, a pattern indicating that the
ports might be undersized. As a result,
users could experience screen errors or slow response times.
In addition,
the WAN circuit inventory identified 37 circuits that were upgraded to current
technology between January and November 2000 at a cost of approximately $1.9
million. These circuits were
disconnected in December 2000, after being used for only 1 to 11 months, when
the IRS reconfigured the WAN (see Appendix IV for details on our calculation of
the total installation costs).
Telecommunications management informed us that this sequence of events
occurred because TCS contractors did not complete their work timely and the IRS
placed a moratorium on network changes during the tax filing season. However, improved network capacity planning
may have prevented the purchase of circuits that were abandoned within a year
after installation.
By not having policies and procedures requiring network
capacity planning and the use of analytical modeling techniques, the IRS is at
risk of wasting resources and building a network that does not meet users’
needs.
The Deputy Commissioner for Modernization & Chief Information Officer should:
8.
Issue
and implement capacity planning policies and procedures for major network
changes.
Management’s Response: The Telecommunications Division is
establishing a capacity planning staff and developing a capacity planning policy
and procedural guide for performing quarterly capacity planning studies. The Telecommunications Division is also
working with the BSMO to use an analytical modeling tool to baseline the
current network and assist in the required capacity planning work.
9. Ensure analytical modeling techniques are incorporated into network and system development planning methodologies.
Management’s Response: The Telecommunications Division is establishing a capacity planning staff to use the modeling techniques and is working with the BSMO to use an analytical modeling tool to baseline the current network and assist in the capacity planning and modeling effort.
The IRS’ WAN is used to access taxpayer accounts and to transmit tax return and payment information among the many IRS locations. The IRS annually assists about 128 million taxpayers and processes about 233 million tax returns and $2.1 trillion in payments. To maintain network integrity and security, the IRS requires all IRS employees and contractors implementing network changes to follow the Change Management Procedure for IRS Network Management (CMP). In addition, the Change Management Tracking System (CMTS) is to be used for documenting network changes including management approvals, test plans, and completed tests.
However, as of July 2001, Telecommunications management had not implemented oversight procedures to ensure that employees and contractors followed the CMP. A review of all 28 network change management requests recorded on the CMTS and all 132 requests to the TCS PMO for network changes that were approved by the PMO and/or completed by contractors between September and November 2000 determined that:
· Fourteen of the 28 CMTS records do not show that changes were properly approved and/or tested before the changes were implemented.
· None of the 132 TCS service requests were recorded on the CMTS and, therefore, the CMTS does not show that the changes were properly approved and/or tested before they were implemented.
In addition, computer software, designed to allow only authorized employees and contractors to implement network changes and to record on a computer file who accessed the equipment and what changes were made, has been purchased but not implemented because of problems identified during testing. However, when implemented, the software will work on only one vendor’s equipment. Management has been unsuccessful in identifying software that would provide the same control over other vendors’ equipment. Management also had not established who would be responsible for reviewing the computer files and taking action on unauthorized changes.
Without effective oversight of network changes and the implementation of electronic access tools, management cannot ensure network equipment changes are authorized and that network integrity and security are maintained.
The
Deputy Commissioner for Modernization & Chief
Information Officer should:
10.
Implement
management oversight procedures to ensure that employees and contractors follow
the CMP.
Management’s Response: Management plans to issue the Change
Management Tracking System document to establish procedures for making changes
to the operational network and implement access control software for one
vendor’s network devices.
11. Enhance change management oversight by:
a. Implementing the change management computer software already purchased.
b. Identifying, purchasing, and implementing software that is compatible with other vendors’ hardware.
c. Revising the CMP to establish responsibility for reviewing change management records and taking corrective actions on unauthorized changes.
Management’s Response: Management plans to issue the Change Management Tracking System document to establish procedures for making changes to the operational network, implement access control software for one vendor’s network devices, and research the availability of tools for other vendors’ network devices and recommend the appropriate purchases.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this audit was to determine whether
the Internal Revenue Service (IRS) Information Technology Services (ITS)
organization effectively
implemented telecommunications data network capacity and performance management
policies and procedures. To accomplish
the objective, we:
I.
Reviewed
IRS network management policies, procedures, and performance measures to
determine whether they are comprehensive and provide guidance to local and
National Headquarters network management personnel.
A. Reviewed National Network Management Center’s (NMC), Martinsburg and Tennessee Computing Centers’; Atlanta, Austin, and Memphis IRS Centers’; and Midstates and Southeast Area Offices’ network monitoring procedures to determine whether they:
1. Established consistent responsibilities for network monitoring that do not overlap or leave gaps in coverage.
2. Provided consistent procedures for monitoring and problem reporting.
3. Established criteria, thresholds, and procedures for elevating to high-level IRS management and/or the Treasury Communications System (TCS) Program Management Office (PMO) in the Department of the Treasury problems that are not resolved timely or that reoccur with some frequency.
B. Reviewed network capacity planning procedures to determine whether they:
1. Established responsibility for conducting capacity planning activities.
2. Established procedures for proactive analyses based on performance statistics, periodic reporting of the results, and elevating the results to appropriate management levels.
3. Established responsibility and procedures for identifying telecommunications network requirements during the planning phase for system development projects and determining the impact the requirements will have on network capacity and performance.
C. Reviewed the Change Management Procedure for IRS Network Management, revised September 2000, to determine whether it:
1. Required controls for all network changes.
2. Required management and technical reviews and approvals.
3. Required testing of all changes.
D. Reviewed the Telecommunications Division network performance measurements to determine whether:
1. They measured telecommunications program and improvement goals.
2. They are relevant to users’ requirements and expectations.
3. The information supporting the measurements is readily available and the measurements are properly recorded.
E. Obtained organization and staffing charts from the NMC, Martinsburg and Tennessee Computing Centers; Atlanta, Austin, and Memphis IRS Centers; and Midstates and Southeast Area Offices to determine the resources that these locations assigned to do the network management activities.
II. Reviewed a sample of problem tickets to determine whether network performance monitoring capabilities were implemented and problems that performance monitoring activities identified were resolved.
A. Sampled and tested problem tickets to determine whether problems were resolved timely.
1. Identified and downloaded to Treasury Inspector General for Tax Administration computers the 457 problem tickets recorded on the NMC Log Management System (LMS) between September 1 and November 30, 2000. Determined that the population contained problems with differing characteristics (e.g., circuit and non-circuit problems, and problems resolved by IRS employees, NMC or TCS contractors). The electronic data were validated by determining that the LMS numbered the problem tickets sequentially and verifying that the computer file contained all problem ticket numbers.
2. Due to the varying characteristics of the problem tickets, we decided not to project the results over the population. Therefore, we selected a judgmental sample of 91 problem tickets (interval sample of every fifth ticket) with a randomly selected starting point. A copy of the Integrated Network and Operations Management System (INOMS) record associated with each sample problem ticket was also obtained to complete the IRS information.
3. Using the LMS and INOMS information, measured the resolution time for each sampled problem ticket in business days from the date and time each problem was recorded until it was resolved and stratified the results (e.g., 0 – 1, 1 – 2, 2 – 5, and over 5 business days). For this analysis, we assumed that each location’s business hours were 8:00 AM to 5:00 PM, weekdays only, and that problems should be resolved within 8 business hours (1 business day), the minimum TCS contract standard.
4. Reviewed each sampled problem ticket that was not resolved within 1 business day to determine whether the problem ticket was included on daily Network Event Status Reports. Also, unresolved problem escalation procedures were discussed with Telecommunications Division managers.
B. Reviewed the 457 LMS problem tickets (selected in Test II.A.) to identify equipment and circuits with multiple events reported between September and November 2000. The problem tickets were reviewed to determine whether duplicate reporting occurred or repetitive problems were elevated to appropriate management levels for attention.
1. Reviewed the 457 LMS problem tickets and associated INOMS records to determine whether duplicate tickets were prepared and the cause was determined.
2. Reviewed the LMS problem tickets and associated INOMS records for circuits and equipment with multiple events to determine the time periods they occurred in and whether Telecommunications employees reported them as chronic problems.
3. Reviewed telecommunications documentation associated with repetitive problems to determine whether the problems identified by the auditors were elevated to IRS management.
C. Reviewed available TCS Trouble Ticket System (TTS) Reports (DRD-031) to determine whether results were reported for September through November 2000 and reviewed a Telecommunications Division and TCS contractor analysis of chronic problems.
III. Reviewed samples of performance statistics and network capacity planning documents to determine whether Telecommunications management implemented proactive capacity planning activities.
A. Obtained Internet firewall log data for September 1 through November 30, 2000. The data identified source and destination addresses, the size of the web objects, and dates and transmission times for each access through the firewall during the period. The data were analyzed for the following information:
1. Misrouted IRS Intranet Traffic – Determined the volume of IRS Intranet traffic (from Intranet address to Intranet address) that is misrouted through the Internet firewalls.
2. Advertising Banners – Determined the volume of data that known Internet advertisers transmitted to IRS addresses through the Internet firewall.
B. Discussed with Telecommunications management how capacity planning analyses are reported and reviewed the one capacity planning document that was produced for the Telecommunications Division to determine whether they:
1. Analyzed network statistics for misrouted IRS Intranet traffic or advertising banners or similar conditions and reported the results.
2. Used performance statistics to evaluate immediate and long-term capacity requirements and to support the business cases for acquisitions.
3. Evaluated the impact on the telecommunications network of system consolidation projects.
C. Reviewed WAN performance reports for February 2001, the TCS circuit inventory listing as of March 2001, and TCS Service Request listings for FY 2000 and FY 2001 to evaluate the effectiveness of capacity planning activities.
1. The performance reports identified 306 circuits and 68 ports that carried WAN traffic. Reviewed the performance reports for all circuits and ports to identify traffic volumes that exceeded the circuits’ and ports’ guaranteed capacity during peak periods or traffic that used less than 10 percent of the capacity.
2. Reviewed the performance reports, circuit inventory, and Service Request reports to identify the WAN circuits that were upgraded in FY 2000 and FY 2001 but were not shown in the February 2001 performance reports and to determine the purchase cost of circuits upgraded and disconnected during FY 2000 and FY 2001.
IV.
Reviewed all 28 Change Management Tracking System
(CMTS) records created between September 1 and November 30, 2000, and all 132
TCS Service Requests approved by the TCS PMO or completed during the same
period to determine whether the Change
Management Procedure for IRS Network Management was implemented and
enforced.
Appendix II
Scott E. Wilson, Assistant Inspector General for Audit
(Information Systems Programs)
Gary Hinkle, Director
Danny Verneuille, Audit Manager
Barbara Bartuska, Senior Auditor
Frank
Greene, Senior Auditor
Larry
Reimer, Senior Auditor
Olivia Jasper, Auditor
Linda Screws, Auditor
Appendix III
Commissioner N:C
Chief, Information Technology Services M:I
Deputy Chief, Information Technology Services M:I
Director,
Corporate Computing M:I:E
Director, Desktop Management M:I:F
Director, IT Midstates Area M:I:F:MS
Director, IT Service Center Operations M:I:SC
Director, IT Southeast Area M:I:F:SE
Director, Martinsburg Computing Center M:I:E:MC
Director, Telecommunications M:I:T
Director, Tennessee Computing Center M:I:E:TC
Director, Strategic Planning and Client
Services M:I:SP
Chief Counsel CC
National Taxpayer Advocate TA
Director, Legislative Affairs CL:LA
Director, Office Program Evaluation and Risk Analysis N:ADC:R:O
Office of
Management Controls N:CFO:F:M
Audit Liaisons:
Chief, Information Technology Services M:I
Director, Corporate Computing
M:I:E
Director, Desktop Management M:I:F
Director, IT Service Center Operations M:I:SC
Director, Telecommunications M:I:T
Appendix IV
This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration. These benefits will be incorporated into our Semiannual Report to the Congress.
Type and Value of Outcome Measure:
· Cost Savings, Recommendations That Funds Be Put to Better Use – Potential; $4.8 million in employee salaries (see page 3).
Methodology Used to Measure the Reported Benefit:
We reviewed information about the
Telecommunications Division National Network Management Center (NMC)
contractors’ and Martinsburg Computing Center (MCC) and Tennessee Computing
Center (TCC) employees’ network management duties; their network monitoring,
problem reporting, and troubleshooting methods; and their Position Description
job titles, pay grades, and tours of duty.
Based on the employees’ pay grades, we estimated the salaries that could
be saved if the overlapping
network monitoring and problem reporting duties were eliminated.
|
Total salaries of 29 MCC and TCC employees who monitor
Wide Area Network (WAN) circuits (actual grades, estimated step 5), rounded
to thousands |
$1,283,000 |
|---|---|
|
Percent of work hours spent on network monitoring
duties (estimated by MCC management) |
75 percent |
|
Estimated salary savings ($1,283,000 * 75 percent),
rounded to thousands |
$963,000 |
|
Estimated 5-year savings ($963,000 * 5 years) |
$4,815,000 |
|
Estimated number of employees (29 * 75 percent),
rounded to whole number |
22 |
Type and Value of Outcome Measure:
· Protection of Resources/ Reliability of Information – Actual; $1.9 million spent (see page 11).
Methodology Used to Measure the Reported Benefit:
The Internal Revenue
Service’s (IRS) WAN circuits were identified by reviewing the WAN circuit
performance reports for February 2001, the TCS circuit inventory report as of
March 2001, and the IRS Service Request reports for FY 2000 and FY 2001. The Service Request reports also identified
the purchase costs. Comparison of the
reports identified 37 WAN circuits that were upgraded to current technology
between January and November 2000 at a cost of approximately $1.9 million. These circuits were disconnected in December 2000, after being used for
only 1 to 11 months, when the IRS reconfigured the WAN.
Appendix
V
The response was removed due to its size. To see the complete response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.