Mid-range Computer Storage Resources Need Better Administration to Ensure Effective and Efficient Utilization and Accurate Reporting

 

July 2005

 

Reference Number:  2005-20-098

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

 

July 21, 2005

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

 

FROM:     Pamela J. Gardiner /s/ Pamela J. Gardiner

                 Deputy Inspector General for Audit

 

SUBJECT:     Final Audit Report - Mid-range Computer Storage Resources Need Better Administration to Ensure Effective and Efficient Utilization and Accurate Reporting (Audit # 200320016)

 

This report presents the results of our review of the Internal Revenue Service’s (IRS) mid-range computer storage management.  The overall objective of this review was to assess the effectiveness and efficiency of the IRS’ mid-range computer storage management practices.

In summary, the IRS requires a large and complex computer environment to process and store taxpayer, financial, and administrative data.  In October 2004, the IRS had approximately 340.2 terabytes of storage capacity on the mid-range computer storage area networks (SAN).  In Fiscal Years (FY) 2002 - 2005, the Enterprise Operations  function Distributed Systems Management Branch spent about $23.2 million to purchase 156.7 terabytes ($147,797 per terabyte) of mid-range computer storage hardware, software, maintenance, and other support.

Our review indicates mid-range computer SANs are not effectively managed to ensure efficient utilization of storage resources.  The Distributed Systems Management Branch monitors only the amount of storage that is allocated to applications and has not implemented the ControlCenter^® storage management tool or an alternative process to assess the utilization of its allocated SAN storage space.  To begin addressing its lack of storage utilization management information, the Distributed Systems Management Branch is currently establishing a SAN Development Laboratory to assess storage management tools.  Our analysis of allocated and utilized SAN storage determined the IRS has used approximately 35.5 percent of the allocated storage space.  Therefore, the IRS could avoid spending approximately $9.9 million for additional storage capacity by improving the efficiency of its use of the current storage capacity. 

In addition, duplicate and old files are not being routinely identified in the SANs because mid-range computer storage personnel do not monitor storage usage and consider it a responsibility of the application owners to identify and remove duplicate or old data.  To address prevention of duplicate data storage across existing and future application systems, the Modernization and Information Technology Services organization established the Enterprise Data Management Office in November 2002.  The Enterprise Data Management Office is currently working on the Enterprise Data Warehouse Strategy.

Storage capacity is not accurately, consistently, and completely reported.  Distributed Systems Management Branch personnel use computer software to detect and diagram mid-range computer SAN storage and manually prepare spreadsheets to track and report storage capacity and allocations.  However, comparison of the diagrams and spreadsheets identified significant differences.  For example, the SAN diagrams showed 38.6 terabytes of capacity that were not listed in the spreadsheets, and the spreadsheets identified 62.0 terabytes of capacity that were not shown on the SAN diagrams.  In addition, the amount of mid-range computer SAN storage reported by the Enterprise Operations function Capacity Management Branch and the Distributed Systems Management Branch are not consistent.  For example, the storage volumes reported on the Capacity Management Branch web site differed from those the Distributed Systems Management Branch reported by 10.1 terabytes.  The Distributed Systems Management Branch report also listed 148.6 terabytes of storage capacity for the Modernization mid-range computer systems that were not included in the Capacity Management Branch report.  As a result, the inaccurately and incompletely reported web site information may have affected storage investment and resource decisions.

Finally, storage continues to be purchased although it does not comply with the IRS Enterprise Architecture requirements.  In FY 2004, the Enterprise Operations function purchased approximately 5.0 terabytes of noncompliant direct attached storage capacity and maintenance costing $766,765 for 2 application systems the IRS expects to use indefinitely.  Enterprise Operations function management did not identify any plans to convert the systems to networked storage within 1 year, as required by IRS procedures. 

To improve mid-range computer storage management and ensure effective and efficient utilization and accurate reporting, the Chief Information Officer should ensure the implementation of the SAN Development Laboratory and the Enterprise Data Warehouse Strategy, the assessment and implementation of an appropriate storage management tool(s), and the identification and elimination of duplicate and old files.  In addition, the Chief Information Officer should ensure the accuracy and completeness of storage capacity reports and the compliance of application systems’ storage with the Enterprise Architecture.

Management’s Response:  IRS management agreed with our recommendations and has implemented several corrective actions including implementing the SAN Development Laboratory, submitting change requests to the Enterprise Architecture as required, and enforcing Enterprise Architecture waiver compliance.  IRS management plans to develop an Enterprise Data Warehouse Strategy and evaluate and test an automated tool that meets the IRS’ SAN management requirements.  The Enterprise Operations function will issue an interim policy to restrict further creation of duplicate data within the SAN, initiate a working group to establish policy related to routine removal of duplicate and outdated files, and create a one-time plan to remove existing duplicate and outdated files.  For the consolidation and transitioned Modernization systems, IRS management will ensure implementation of an automated tool that will meet the IRS’ SAN management requirements.  In addition, IRS management will ensure accuracy and completeness of web site storage capacity reporting and effective review of all major information technology acquisitions for Enterprise Architecture compliance.  Finally, noncompliant application systems owners will be required to submit an Enterprise Architecture waiver request and a mitigation plan.  The Enterprise Operations function will support and monitor the approved plan through to completion.  However, IRS management disagreed with the report’s cost savings estimates identified in Appendix IV based on concerns of including regular tape backup systems expenses in the cost, the database utilization figures, and the standard for storage utilization. Management’s complete response to the draft report is included as Appendix V.

Office of Audit Comment:  We do not agree with the IRS’ response to our outcome measures included in Appendix IV.  The costs savings and the methodology for calculating the savings were discussed with IRS management, and the audit report was revised based on management’s comments.  IRS management disagrees with including tape backup systems expenses in the basis of our calculations.  The information for calculating the cost per terabyte was provided to us by IRS personnel, and they advised the figures included all components required to make a technically redundant and functional SAN.

IRS management also expressed concern about the accuracy of the database utilization figures calculated and reported in Appendix IV.  The IRS database administrators and system administrators developed the computer programs to run against the databases and provided us with the utilization rates.  They also validated the methodology we used to calculate the utilization rates.  During our prior discussions, IRS management commented that other database processes such as application overhead and data staging also require storage.  However, they could not provide any documentation to show the impact on storage from the database processes.

Finally, IRS management took issue with the 85 percent capacity utilization goal we measured the IRS against based on our industry research.  The Modernization Information Technology and Security Services Strategy and Program Plan for FY 2004 – 2005 states while the development of new technology evolves, existing operations must continue and improvements must be made to meet the needs of tax administration and demonstrate IRS commitment to improved service to taxpayers.  The Plan further states the IRS will continue to respond to this challenge by continuing benchmarking efforts which allow comparison of IRS performance to that of similar efforts in private industry.  Our research shows this goal is one that companies in the financial services industry and government agencies have worked toward achieving and, therefore, seems to be a reasonable goal.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

 Table of Contents

Background

Mid-range Computer Storage Area Networks Are Not Effectively Managed to Ensure Efficient Utilization of Storage Resources

Recommendation 1:

Recommendation 2:

Recommendations 3 and 4:

Storage Capacity Is Not Accurately, Consistently, and Completely Reported

Recommendations 5 and 6:

Some Purchased Storage Did Not Comply With the Enterprise Architecture Requirements

Recommendation 7 and 8:

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measures

Appendix V – Management’s Response to the Draft Report

 

Background

The Internal Revenue Service (IRS) requires a large and complex computer environment to process and store taxpayer, financial, and administrative data.  A significant portion of this workload is performed in the IRS’ mid-range computer environment consisting of file servers and related hardware, software, maintenance, and services.  The Modernization and Information Technology Services organization is near the end of a 5-year consolidation of mid-range computer systems from multiple locations to 3 Computing Centers:  the Detroit Computing Center, the Enterprise Computing Center-Martinsburg, and the Enterprise Computing Center-Memphis.

The mid-range computer workloads produce large volumes of information (data) that require storage.  In October 2004, the IRS had approximately 340.2 terabytes of mid-range computer disk storage on the storage area networks (SAN) at the Computing Centers.  The SANs are fiber-based, dedicated, high-speed networks used to move large volumes of data between computer systems and the storage devices (see Figure 1).

Figure 1:  SAN Network Diagram

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

The Enterprise Operations function Distributed Systems Management Branch is responsible for technical and programmatic management of all IRS mid-range computer systems including the purchase, use, management, and disposition of mid-range computer storage.  The Enterprise Operations function Capacity Management Branch uses Distributed Systems Management Branch-provided information to prepare high-level storage briefing reports for IRS management.

In Fiscal Years (FY) 2002 - 2005, the Distributed Systems Management Branch spent about $23.2 million to purchase mid-range computer storage hardware, software, maintenance, and other support for consolidated mid-range computer systems.  The IRS has also projected 20 percent annual growth in its mid-range computer storage environment.  However, the IRS is faced with ever-shrinking budgets.  Figure 2 summarizes the actual expenditures for mid-range computer storage.

Figure 2:  Mid-range Computer Storage Expenditures for FYs 2002 - 2005

Source:  FYs 2002 – 2005 expenditure information obtained from IRS management.

This review was performed in the Enterprise Operations function offices at the IRS National Headquarters in New Carrollton, Maryland; the Detroit Computing Center in Detroit, Michigan; the Enterprise Computing Center-Martinsburg in Martinsburg, West Virginia; and the Enterprise Computing Center-Memphis in Memphis, Tennessee, during the period April 2004 through February 2005.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.

Mid-range Computer Storage Area Networks Are Not Effectively Managed to Ensure Efficient Utilization of Storage Resources

The Clinger-Cohen Act of 1996 states the Chief Information Officer is responsible for promoting the effective and efficient design and operation of all major information resources management processes for the executive agency.  The Internal Revenue Manual states the Chief Information Officer and Director, Enterprise Operations, within the Modernization and Information Technology Services organization are responsible for ensuring the effective and efficient use of the IRS automated information processing environment. 

The IRS Enterprise Architecture requires computer data storage to be provided through a mix of disk, tape, and optical drives configured to provide the fastest, most available solution consistent with application cost, performance, and availability constraints.  All storage will be centrally managed through a concept called storage virtualization.

Storage virtualization is the consolidation of multiple network storage devices into what appears to be a single storage unit (i.e., a virtual storage “pool”) and is often used in SAN environments.  It makes tasks such as archiving, backup, and recovery of data easier and faster.  When an application requires additional storage capacity, the storage manager can allocate the additional storage capacity from the pool.  Because the resources allocated to an application can be increased or decreased at will, there is no need to overallocate storage.  Storage virtualization and SAN architecture should result in the optimal placement of files to achieve the highest performance and availability.

Our review of the mid-range computer SAN storage and storage management practices identified opportunities for more effective storage utilization.

Allocated storage is not being used

Storage industry research shows that data centers historically only use 15 - 60 percent of their available storage capacity, while they should have as a goal to use closer to 85 percent.  In October 2004, the Distributed Systems Management Branch provided information showing the IRS had approximately 340.2 terabytes of mid-range computer SAN storage.  The Distributed Systems Management Branch also reported that 285.4 terabytes (84 percent) was allocated to applications for use.  However, actual utilization of allocated storage space is not currently reported and monitored. 

Therefore, we worked with Detroit Computing Center, Enterprise Computing Center-Martinsburg, and Enterprise Computing Center-Memphis database administrators and system administrators to obtain information about utilization of the storage allocated to the database applications.  The database administrators and system administrators wrote and manually executed scripts (i.e., small computer programs) to gather the information for each database application.

We received storage utilization information for 134.7 of the 285.4 terabytes of allocated mid-range computer SAN storage from October 25, 2004, through February 3, 2005.  Due to the number of storage allocations and database tables involved and the manual effort required, IRS management did not have the resources available to provide usage information for the remaining 150.7 terabytes of storage.

·         The information provided showed only 12.8 (9.5 percent) of the 134.7 terabytes of allocated SAN storage contained data. 

·         IRS computer storage personnel advised that the amount of data being stored is tripled to 38.4 terabytes during database application processing because the SAN creates mirror and business continuity copies of the data. 

·         In addition, IRS management explained the Enterprise Computing Center-Martinsburg storage allocated to the consolidated mid-range computer SAN is designated for disaster recovery.  Therefore, we included the 10.1 terabytes allocated to this SAN in our analysis.  However, 0.7 terabytes of the 10.1 terabytes of storage contained data and were included in the previous calculations regarding the 12.8 terabytes of storage.  Therefore, the remaining 9.4 terabytes not containing data were added to the 38.4 terabytes to determine storage utilization.

We concluded that 47.8 (35.5 percent) of the 134.7 terabytes are used.  We then applied the industry 85 percent storage utilization goal to the analysis and determined the IRS has approximately 66.7 terabytes (134.7 terabytes times 85 percent minus 47.8 terabytes) of available storage capacity. 

In addition, IRS management commented that storage space is used by application files and data files (e.g., Master File data) transferred into the SAN.  Storage space also is needed for overhead and workspace for the database application during processing.  However, IRS management could not provide any information showing the quantity of storage used by these files and processes. 

Based on the storage utilization information provided by the database administrators and system administrators and the average cost of $147,797 per terabyte (see Figure 2), the IRS could avoid spending approximately $9.9 million for additional storage capacity by improving the efficiency of the current storage capacity (see Appendix IV).

The IRS mid-range computer SAN storage utilization rate is low because the Distributed Systems Management Branch monitors only the amount of storage that is allocated to applications and does not have a process in place to analyze utilization of its allocated SAN storage space.  Further, Distributed Systems Management Branch management and employees have not completely familiarized themselves with the ControlCenter^® software management tool to determine its capabilities in monitoring utilization of the IRS’ mid-range computer SAN storage space. 

Storage industry research also shows the flexibility offered by SANs and comprehensive storage management tools can greatly benefit data centers.  The storage management tools can optimally improve utilization through increasing allocation efficiency, cost-effectively controlling the amount of storage purchased, and automating manual reporting processes to assess usage trends and storage growth.  The Distributed Systems Management Branch purchased storage for a SAN Development Laboratory it plans to use to determine how ControlCenter^® and/or other storage management tools can help it more effectively and efficiently manage SAN storage.

Duplicate and old files are not being routinely identified in the SANs

According to storage industry experts, about one-half of stored file server data has not been accessed in 90 days or more.  Therefore, the probability of files with these data being accessed or needed is very low.  In addition, duplicate and obsolete files contribute unnecessarily to overall storage growth.  At our request, Enterprise Computing Center-Martinsburg and Enterprise Computing Center-Memphis database administrators and system administrators created and manually ran scripts to list file characteristics (e.g., file names, file sizes, date of last file modification) for Oracle^® database files on the SANs.  The resulting file listings, dated October 14, 2004, through November 3, 2004, identified 4,405 files that totaled 17.9 terabytes on the SANs.  Our review of the script results identified:

·         Instances where file names were repeated two or more times with minor differences in the names (e.g., P1_DATA_LG_01.dbf, P1_DATA_LG_02.dbf, etc.).  To determine whether files contain duplicate data, the contents would have to be compared, which is outside the scope of Modernization and Information Technology Services organization personnel’s authority to access taxpayer information.  A Distributed Systems Management Branch frontline manager stated the IRS has never implemented file naming conventions that would facilitate identification of duplicate files. 

To address the prevention of duplicate data storage across existing and future application systems, the Modernization and Information Technology Services organization established the Enterprise Data Management Office in November 2002.  The Enterprise Data Management Office is currently working on the Enterprise Data Warehouse Strategy and has brought together representatives from the Modernization and current production areas of the Modernization and Information Technology Services organization.  However, Enterprise Operations function personnel stated the current effort is the third of this type to be established since 2000, and none of the previous efforts were completed.

·         A total of 844 “old” files, using 2.7 terabytes of storage, including 626 Electronic Fraud Detection System files totaling 2.5 terabytes.  See Figure 3 for the file information identified.

Figure 3:  “Old” Files Identified on the SANs

Source:  IRS personnel-created scripts.

Potentially duplicate and old data reside on the mid-range computer SANs because mid-range computer storage personnel do not monitor storage usage and consider it a responsibility of the application owners to identify and remove duplicate and old data. 

As a result of not having an implemented process or tool to ensure the IRS is effectively and efficiently using allocated storage and identifying duplicate and old files, the IRS continues to risk purchasing additional storage it does not need.

Recommendations

To improve SAN management, the Chief Information Officer should ensure:

1.      The implementation of the SAN Development Laboratory and the Enterprise Data Warehouse Strategy.

Management’s Response:  The Technical Systems Software Division implemented the SAN Development Laboratory for testing of SAN-related software to improve configuration, monitoring, and utilization reporting.  The Enterprise Architecture area will develop the appropriate Enterprise Data Warehouse Strategy.

2.      The assessment of the capabilities of ControlCenter^® or alternative tools to manage the IRS’ mid-range computer storage, including monitoring the usage rates of allocated storage.

Management’s Response:  The Enterprise Operations function will assess the capabilities and ensure documented evaluation and test of an automated tool that meets the IRS’ SAN management requirements for configuration, monitoring, and database utilization.  However, IRS management disagreed with the report’s cost savings estimates based on concerns of including regular tape backup systems expenses in the cost, the database utilization figures, and the standard for storage utilization.

Office of Audit Comment:  We do not agree with the IRS’ response to our outcome measures included in Appendix IV.  The costs savings and the methodology for calculating the savings were discussed with IRS management and the audit report was revised based on management’s comments.  While IRS management disagreed with including tape backup systems expenses in the basis of our calculations, the information for calculating the cost per terabyte was provided to us by IRS personnel, and they advised us the figures included all components required to make a technically redundant and functional SAN.

IRS management also expressed concern about the accuracy of the database utilization figures calculated and reported in Appendix IV.  The IRS database administrators and system administrators developed the computer programs to run against the databases and provided us with the utilization rates.  They also validated the methodology we used to calculate the utilization rates.  During our prior discussions, IRS management commented that other database processes such as application overhead and data staging also require storage.  However, they could not provide any documentation to show the impact on storage from the database processes.

Finally, IRS management took issue with the 85 percent capacity utilization goal we measured the IRS against based on our industry research.  The Modernization Information Technology and Security Services Strategy and Program Plan for FY 2004 – 2005 states while the development of new technology evolves, existing operations must continue and improvements must be made to meet the needs of tax administration and demonstrate IRS commitment to improved service to taxpayers.  The Plan further states the IRS will continue to respond to this challenge by continuing benchmarking efforts which allow comparison of IRS performance to that of similar efforts in private industry.  Our research shows that this goal is one that companies in the financial services industry and government agencies have worked toward achieving and, therefore, seems to be a reasonable goal.

3.      The implementation of ControlCenter^® or an alternative tool to collect and report accurate mid-range computer storage capacity and utilization information.

Management’s Response:  For the consolidation and transitioned Modernization systems, the Enterprise Operations function will ensure complete implementation of an automated tool that will meet the IRS’ SAN management requirements for configuration, monitoring, and database utilization.

4.      The identification of files (e.g., multiple replications of the same data and files with old data warranting removal) that can be eliminated and ensure their removal.

Management’s Response:  The Enterprise Operations function will issue an interim policy to restrict further creation of duplicate data within the SAN, initiate a working group to establish policy related to routine removal of duplicate and outdated files, and create a one-time plan to remove existing duplicate and outdated files.

Storage Capacity Is Not Accurately, Consistently, and Completely Reported

The Government Accountability Office document Assessing Risks and Returns:  A Guide for Evaluating Federal Agencies’ IT Investment Decision-making states that, despite making a huge investment in information technology, many Federal Government operations are still hampered by inaccurate data and inadequate systems.  Informed management decisions can occur only if accurate, reliable, and up-to-date information is included in the decision-making process. 

Distributed Systems Management Branch personnel use automated network management tools to detect and diagram storage arrays on the mid-range computer SANs.  They also manually prepare spreadsheets to track and report storage capacity and allocations.  As of October 2004, the spreadsheets indicated the IRS mid-range computer SAN capacity totaled approximately 340.2 terabytes.  Our comparison of the SAN diagrams to the spreadsheets found the following inaccuracies:

·         The SAN diagrams showed 1 storage array with 38.6 terabytes of capacity that was not listed in the spreadsheets.

·         The spreadsheets identified 2 storage arrays with 62.0 terabytes of capacity that were not shown on the SAN diagrams.

These inaccuracies indicate management does not have a reliable storage capacity and allocation reporting methodology.  Distributed Systems Management Branch management stated they do not use the ControlCenter^® storage management software reporting tool because their limited experience with the tool had shown it to be cumbersome to use.

In addition, the amount of mid-range computer SAN (i.e., Infrastructure SAN) storage reported by the Capacity Management Branch and Distributed Systems Management Branch are not consistent.  Results of our review of the storage capacity reported on the Capacity Management Branch web site differed from the amount the Distributed Systems Management Branch reported by 10.1 terabytes because the Capacity Management Branch was not timely updating its web site.  Capacity Management Branch personnel indicated they receive monthly information from the Distributed Systems Management Branch and are responsible for updating the Capacity Management Branch web site with the most current information.  Figure 4 shows the storage amounts reported by the Capacity Management Branch and Distributed Systems Management Branch, respectively.

Figure 4:  Mid-range Computer SAN Storage Volume Reports

Figure 4 was removed due to its size.  To see Figure 4, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

 

In addition, the Distributed Systems Management Branch report listed 148.6 terabytes of storage capacity for the Modernization mid-range computer systems at the Detroit Computing Center and Enterprise Computing Center-Martinsburg that were not included in the Capacity Management Branch report.  IRS management stated the storage information provided on the Capacity Management Branch web site is used for executive briefings.  As a result, the inaccurate and incomplete information may have affected storage investment and resource decisions.  Capacity Management Branch personnel stated they would be updating the mid-range computer storage capacity reported on their web site.

Without accurate and complete tracking and reporting of mid-range computer SAN storage, the IRS does not know how much storage it has and may be unable to make reliable investment and resource decisions.

Recommendations

To improve capacity and usage reporting, the Chief Information Officer should ensure:

5.      The implementation of an automated SAN management tool.

Management’s Response:  For the consolidation and transitioned Modernization systems, the Enterprise Operations function will ensure implementation of ControlCenter^® 5.2 (or alternative tool) to meet the IRS’ SAN management requirements for configuration, monitoring, and database utilization.

6.      The accuracy and completeness of Capacity Management Branch mid-range computer storage capacity reports.

Management’s Response:  The Enterprise Operations function will ensure accuracy and completeness of web site storage capacity reporting.

Some Purchased Storage Did Not Comply With the Enterprise Architecture Requirements

The Internal Revenue Manual states the Chief Information Officer and Director, Enterprise Operations, are responsible for ensuring the effective and efficient use of the IRS automated information processing environment.  The IRS Enterprise Architecture requires all data to be stored on networked storage.  The Enterprise Architecture prohibits application and data file storage on direct attached storage and allows, during a transition period, operating system software storage on direct attached storage.  To prevent compromise of the Enterprise Architecture standards, the Distributed Systems Management Branch implemented procedures limiting management approval of storage to requests that comply with the Enterprise Architecture.  Requests for noncompliant storage can receive a 1-year interim approval if, for example, the requestor is planning to become compliant with the Enterprise Architecture. 

The Modernization and Information Technology Services organization Enterprise Operations and End User Equipment and Services function managers indicated they understood the Enterprise Architecture storage network and virtualization requirements apply to current and forthcoming projects.  For example, Enterprise Operations function Distributed Systems Management Branch management stated all new mid-range computer consolidation projects are being configured for a SAN.  Also, storage for two systems currently operating with direct attached storage is being converted to a SAN.

However, the Enterprise Operations function purchased direct attached storage capacity and maintenance for two application systems that the IRS expects to use indefinitely.  Enterprise Operations function management did not identify any plans to convert the systems to networked storage within 1 year.  The two systems are:

·         The Offshore Credit Card Project/Offshore Voluntary Compliance Initiative, which is a new application placed into production in March 2004.  IRS management approved the direct attached storage requisition, justification, and Enterprise Architecture review without an explanation of why the storage does not comply with the Enterprise Architecture or any indication that the application system would ever be brought into compliance with the Enterprise Architecture.

In July 2004, the IRS paid $148,635 to purchase 4.1 terabytes of storage to implement production at the Enterprise Computing Center-Martinsburg.  In late 2004, the Offshore Credit Card Project/Offshore Voluntary Compliance Initiative production system was moved to the Detroit Computing Center and installed on a computer system and storage purchased in May 2003.  The computer system and storage were available for the Offshore Credit Card Project/Offshore Voluntary Compliance Initiative system because they were not being used for their initial purpose.

·         The Electronic Management System, which has operated at the Enterprise Computing Center-Memphis and Austin, Texas, Campus for many years, and currently supports storage for the Modernized e-File system.  In January 2004, the IRS paid $618,130 for 876 gigabytes (0.9 terabytes) of direct attached storage to replace the obsolete and at-capacity storage hardware and to relocate the Austin Campus Electronic Management System storage to the Enterprise Computing Center-Martinsburg.  However, the replacement storage is not Enterprise Architecture compliant, and the interim period is longer than 1 year.  The approved justification stated the Electronic Management System would migrate to the Modernization platform in 2009, and an interim migration to a mid-range computer consolidation platform was not planned.

As a result of not having an effective Enterprise Architecture review and effective management of mid-range computer SAN storage, the IRS unnecessarily purchased approximately 5.0 terabytes of storage costing $766,765 (see Appendix IV).  The IRS also continues to be at risk of unnecessarily purchasing direct attached storage, wasting limited resources, and compromising the Enterprise Architecture.

Recommendations

To improve compliance with the Enterprise Architecture, the Chief Information Officer should ensure:

7.      The effectiveness of all mid-range computer storage requisition reviews and, if Enterprise Architecture compliance is to be waived, the justification complies with IRS procedures.

Management’s Response:  The Enterprise Operations function will implement controls that ensure effective review of all major information technology acquisitions for Enterprise Architecture compliance.

8.      Application systems owners using storage methods that are not compliant with the Enterprise Architecture (e.g., direct attached storage) develop and implement plans to convert to Enterprise Architecture-compliant network storage.

Management’s Response:  The Enterprise Operations function has submitted change requests to the Enterprise Architecture as required and enforces Enterprise Architecture waiver compliance during the web Request Tracking System review approval process.  The Enterprise Operations function will require noncompliant application system owners to submit an Enterprise Architecture waiver request and the required mitigation plan to become compliant with the Enterprise Architecture.  The Enterprise Operations function will support and monitor the approved plan through to completion.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to assess the effectiveness and efficiency of the Internal Revenue Service’s (IRS) mid-range computer storage management practices.  To accomplish this objective, we:

I.                   Compared a listing of all mid-range computer disk storage and devices containing storage for each of the three Computing Centers to storage area network (SAN) design diagrams and to the amount of storage reported on the Enterprise Operations function Capacity Management Branch web site.  We reviewed storage volume allocations and usage reports to determine the amount of available SAN storage and the amount of SAN storage allocated to database applications.  We requested database and system administrators to run scripts (i.e., small computer programs) to identify the amount of the 340.2 terabytes of storage used by database applications.  Management reports showed 285.4 of the 340.2 terabytes were allocated to database applications.  We reviewed storage utilization information for 134.7 of the 285.4 terabytes of allocated mid-range computer SAN storage from October 25, 2004, through February 3, 2005.  Due to the number of storage allocations and database tables involved and the manual effort required, IRS management did not have the resources available to provide utilization information for the remaining 150.7 terabytes of storage.

We interviewed database and system administrators and Enterprise Operations function Distributed Systems Management Branch personnel to identify current needs, projected budget needs, and actual expenditures for mid-range computer storage for Fiscal Years 2002 - 2005.  We also researched storage industry information on storage utilization.

II.                Interviewed Distributed Systems Management Branch personnel responsible for overseeing the mid-range computer storage Enterprise Architecture, personnel from the Enterprise Operations function Distributed Systems Software Branch responsible for managing mid-range computer hardware and software, End User Equipment and Services function personnel, and other Enterprise Operations function personnel to determine whether policies and procedures have been developed for reviewing changes to the Enterprise Architecture.  We also identified and reviewed any information technology projects using direct attached storage to determine the cost of the storage and whether the storage was properly reviewed, approved, and justified.

III.             Reviewed the IRS’ as-built-architecture documentation and interviewed mid-range computer storage management at each Computing Center to identify potential duplication of files.  We also interviewed mid-range computer storage, system, and database administrators at each Computing Center to identify potential data duplication actions.  At our request, the database and system administrators generated and ran scripts providing the characteristics (e.g., file names, file sizes, date of last file modification) of files in the SAN and storage array capacity, table space allocation, and usage data.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary Hinkle, Director

Danny Verneuille, Audit Manager

Mark Carder, Lead Auditor

James Douglas, Senior Auditor

Frank Greene, Senior Auditor

Kim McManis, Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Associate Chief Information Officer, Business Systems Modernization  OS:CIO:B

Associate Chief Information Officer, Information Technology Services  OS:CIO:I

Director, Stakeholder Management  OS:CIO:SM

Director, Business Systems Development  OS:CIO:I:B

Director, End User Equipment and Services  OS:CIO:I:EU

Director, Enterprise Operations  OS:CIO:I:EO

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Management Controls  OS:CFO:AR:M

Audit Liaisons: 

            Deputy Commissioner for Operations Support  OS

Manager, Program Oversight Office  OS:CIO:SM:PO

 

Appendix IV

 

Outcome Measures

 

This appendix presents detailed information on the measurable impact that our recommended corrective actions will have on tax administration.  These benefits will be incorporated into our Semiannual Report to the Congress.

Type and Value of Outcome Measure:

• Cost Savings, Funds Put to Better Use – Potential $9,858,060 (see page 4).

Methodology Used to Measure the Reported Benefit:

We worked with Internal Revenue Service (IRS) Computing Center database administrators and system administrators to obtain storage array capacity, table space allocation, and usage data.  The IRS reported 285.4 of approximately 340.2 terabytes were allocated to databases.  We received storage utilization information from October 25, 2004, through February 3, 2005, for only 134.7 of the 285.4 terabytes of allocated mid-range computer Storage Area Network (SAN) storage.  The information provided showed only 12.8 (9.5 percent) of the 134.7 terabytes of allocated SAN storage contained data.  IRS computer storage personnel advised that the amount of storage in use would be tripled to 38.4 terabytes during database application processing because the SAN creates mirror and business continuity copies of the data.  In addition, 10.1 terabytes (0.7 terabytes included in 12.8 terabytes that contained data plus 9.4 terabytes that did not contain data) of the Enterprise Computing Center-Martinsburg storage allocated to the consolidated mid-range computer SAN is designated for disaster recovery.  Therefore, we included the 9.4 terabytes of allocated storage not containing data in our assessment of utilized storage and concluded that 47.8 (35.5 percent) of the 134.7 terabytes are used.  We then applied the industry goal of 85 percent utilization of storage to the analysis and determined the IRS has 66.7 terabytes of available storage capacity.  Table 1 shows how we estimated the available storage capacity.  Due to the number of storage allocations and database tables involved and the manual effort required, IRS management did not have the resources available to provide utilization information for the remaining 150.7 terabytes of storage. 

In addition, IRS management commented that storage space is used by application files and data files (e.g., Master File data) transferred into the SAN.  Storage space also is needed for overhead and workspace for the database application during processing.  However, IRS management could not provide any information showing the quantity of storage used by these files and processes.

Table 1:  Calculation of Available Mid-range Computer Storage Capacity (in terabytes)

Source:  IRS storage management reports and personnel-created scripts.

Type and Value of Outcome Measure:

• Inefficient Use of Resources – Actual; $766,765 (see page 14).

Methodology Used to Measure the Reported Benefit:

We identified applications and systems that use direct attached storage and reviewed requisition, justification, and delivery order documents for direct attached storage purchased for these applications and systems in Fiscal Years 2003 and 2004.  We determined the IRS purchased direct attached storage for the Offshore Credit Card Project/Offshore Voluntary Compliance Initiative and the Electronic Management System, but the approved justifications did not comply with IRS guidelines.  In addition, the purchases may not have been necessary given the storage utilization rate of only 35.5 percent of the 134.7 terabytes of SAN storage analyzed.  Table 2 shows the amount of direct attached storage purchased and its cost.

Table 2:  Direct Attached Storage Purchases and Costs

Source:  IRS purchase requisition information.

 

Appendix V

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.