TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION
The Modernization Vision and Strategy Program Is Achieving Desired Results, but Risks Remain
October 31, 2008
Reference Number: 2009-20-008
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone Number |
202-622-6500
Email Address | inquiries@tigta.treas.gov
Web Site |
http://www.tigta.gov
October 31, 2008
MEMORANDUM FOR CHIEF INFORMATION OFFICER
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – The Modernization Vision and Strategy Program Is Achieving Desired Results, but Risks Remain (Audit # 200820003)
This report presents the results of our review to determine whether the
Internal Revenue Service (IRS) Modernization Vision
and Strategy (MV&S) Program is achieving
the desired results. This review
was included in our Fiscal Year 2008 Annual Audit Plan coverage of IRS Business
Systems Modernization efforts.
Impact on the Taxpayer
Since the Business Systems Modernization program was initiated
in Fiscal Year 1999, substantial changes in the information technology and
budget environments have forced the IRS to shift its thinking and original
assumptions. The MV&S Program was
developed to implement a fundamentally different approach to the IRS
modernization effort and to provide guidance in the development of an
integrated portfolio of information technology investments. The MV&S Program has improved the
information technology investment decision processes, resulting in information
technology projects that 1) better support the IRS goals of improving customer
service and enforcement, and 2) make better use of technology funds. However, process improvements need to be
implemented to ensure the continued improvement of future investment proposal
cost estimates and standardized reporting of investment decisions’ performance
measures.
Synopsis
Overall, the decision-making processes used in the MV&S Program provide IRS executives with effective tools and information with which to make informed information technology decisions. Developed in October 2006, the MV&S Plan is an annually updated 5-year plan for the governance and planning processes for IRS information technology decisions. This approach leverages the IRS strategic plan and associated business plans to drive information technology decisions, address the priorities around modernizing front-line tax administration, and enable technical capabilities provided by the information technology infrastructure. The IRS also created several guiding principles for this new vision and strategy. The MV&S Program will guide the investment priorities of the Business Systems Modernization program for Fiscal Years 2007 through 2011.
Our review of the MV&S Program determined that, overall, processes and procedures have been effectively implemented. For example, the MV&S Plan is updated annually; strategic planning is effective; collaborative alliances have been established; and focused, more frequent delivery of information technology project releases is emphasized.
In addition, the IRS is using an approved estimation model for determining total cost of ownership. We determined that the cost estimates for a sample of 4 of 23 projects in the approved investment portfolio appear to be reasonable based on 1) the documentation provided, 2) the validation of the estimates by employees with expertise in the area, and 3) reviews by the team working on the proposed investments. However, IRS internal estimation guidance does not require comparison of actual investment costs to the original estimates. Therefore, the reliability of estimated total costs of ownership might be at risk because the estimates for prior projects were not validated by comparing the estimates to the actual costs.
MV&S Program performance
measures are being collected, but there is no standardized procedure for compiling
and reporting these measures. Through
review of documentation and discussions with personnel, we could not determine
whether performance measurement and reporting were being thoroughly
accomplished as required. The lack
of MV&S Program performance measures and reporting compliance could result
in inefficient accomplishment of strategic information technology acquisitions,
objectives, and expectations.
Recommendations
The Chief Information Officer should 1) ensure the reliability of the cost estimation process by implementing procedures to compare actual project operations and maintenance costs to initial estimates and revising the estimation process, if necessary, and 2) include guidelines for performance measurement and reporting in the updated MV&S Plan.
Response
IRS management agreed with all of our recommendations. Corrective actions planned include 1) implementing procedures to
compare actual project operations and maintenance costs to initial estimates
and revising the estimation process when necessary, and 2) updating the
MV&S Plan to reference performance measures and reporting. This update will be included in the Calendar Year
2009 annual publication. Management’s
complete response to the draft report is included as Appendix V.
Copies of this report are also being sent to the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Security and Information Technology Services), at (202) 622-8510.
The Process for Developing Total Cost Estimates
Can Be Improved
Modernization Vision and Strategy Program Performance
Measurement Standards Need Further Development
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix IV
– Glossary of Terms
Appendix V
– Management’s Response to the Draft Report
Abbreviations
|
IRS |
Internal Revenue Service |
|
MITS |
Modernization and Information Technology
Services |
|
MV&S |
Modernization Vision and Strategy |
Since the Business Systems Modernization program was initiated in Fiscal Year 1999, substantial changes in the information technology and budget environments have forced the Internal Revenue Service (IRS) to shift its thinking and original assumptions. In particular, the original goal to completely replace the existing IRS information technology environment within 10 years - 15 years has been deemed unrealistic due to resource limitations. The Modernization Vision and Strategy (MV&S) Program was developed to implement a fundamentally different approach to the IRS modernization effort and to provide guidance in the development of an integrated portfolio of information technology investments. Developed in October 2006, the MV&S Plan is an annually updated 5-year plan for the governance and planning process of IRS information technology decisions. This approach leverages the IRS strategic plan and associated business plans to drive information technology decisions, address the priorities around modernizing front-line tax administration, and enable technical capabilities provided by the information technology infrastructure. The MV&S Program will guide the investment priorities of the Business Systems Modernization program for Fiscal Years 2007 through 2011. It includes the following guiding principles:
The Portfolio
Planning, Estimation, and Delivery Services Office within the Modernization
and Information Technology Services (MITS)
organization leads and facilitates the MV&S Program. This Program consists of reviewing, prioritizing, and preparing proposed
information technology projects for funding approval through the MITS
organization Enterprise Governance Committee.
Three offices facilitate the MV&S Program:
This review was performed
at the MITS organization office in New Carrollton, Maryland, during the period
February through August 2008. We
conducted this performance audit in accordance with generally accepted government auditing standards. Those standards require that we plan and
perform the audit to obtain sufficient, appropriate evidence to provide a
reasonable basis for our findings and conclusions based on our audit objective.
We believe that the evidence obtained
provides a reasonable basis for our findings and conclusions based on our audit
objective. Detailed information
on our audit objective, scope, and methodology is presented in Appendix I. Major contributors to the report are listed
in Appendix II.
The Modernization Vision and Strategy Program Is an Effective Process for Making Information Technology Investment Decisions
Overall, the decision-making processes used in the MV&S Program provide IRS executives with effective tools and information with which to make informed information technology decisions. For example:
· The IRS annually updates the MV&S Plan.
· The MITS organization has established an effective approach for strategic planning.
·
The MITS organization
has established collaborative alliances internally and with other IRS business
organizations.
·
The IRS emphasizes
focused, more frequent delivery of information technology project releases.
The IRS annually updates the MV&S Plan
The MV&S
Program Office has a detailed project plan outlining efforts from MV&S Plan
preparation to communicating information affecting the Program. The
first MV&S Plan was prepared
in October 2006 and was updated in October 2007. Annual updates to the Plan will be ongoing
throughout the MV&S Program. The IRS advised us that the Plan is updated
to reflect changes that affect the business organizations. In addition, the annual update includes
changes in strategic objectives and incorporates new requirements. For example, our review of the 2006 and 2007 Plans
determined that the 2007 Plan was revised to reflect updated lists of potential
and future projects and to include information technology as a common service.
The
MITS organization has established an effective approach for strategic planning
The IRS begins the strategic planning process by having the business organizations define their goals and initiatives to meet their business objectives, which then aids in the accomplishment of IRS strategic goals. We confirmed that the Portfolio Planning, Estimation, and Delivery Services Office conducted portfolio investment planning efforts. In addition, the MITS organization completed several portfolio planning efforts for Fiscal Years 2007 through 2011. The Strategy processes involve a series of activities aimed at preparing, executing, and communicating the Strategy.
For each business area (e.g., the Submission Processing, Managing Taxpayer Accounts, Customer Service, Chief Counsel, and Internal Management organizations), the MV&S Plan defines the goals, benefits, opportunities, and expected business outcomes related to technology that will yield the greatest benefits to taxpayers and the IRS. The Plan identifies the specific projects, systems, and investments that will be necessary to achieve the business outcome. The business areas collaborate to develop a joint strategy for information technology investments. After strategic plans are reviewed, investment proposals are identified, evaluated, and prioritized for each business area. The investment proposals are then prioritized based on the IRS’ strategic needs. We reviewed the most recent enterprise portfolio prioritization document available and found that all internal business areas included in the MV&S Plan prioritized and presented their specific lists of projects.
Mapping of
investment proposals to IRS and other strategic goals is being
accomplished. Information technology vision sessions
are conducted within the business areas to identify scope, goals,
opportunities, recommendations, and strategic views for each office. Each investment proposal that goes through the
MV&S Program is given an investment review that shows 1) the business
proposal’s alignment with business area goals, 2) initiatives in the
President’s Management Agenda, and 3) Department of the Treasury and IRS
strategic goals.
The MITS organization has established collaborative alliances
internally and with other IRS business organizations
Collaborative alliances were formed between organizations for the MV&S Program. The Portfolio Planning, Estimation, and Delivery Services Office staff have developed relationships internally and with other business organizations to engage participants in the various activities in the MV&S Program. Additionally, the staff work closely with the Chief Financial Office and the Application Development Budget, Capital Planning and Investment Control, and Financial Management Services organizations to determine how to improve departmental processes. The Capital Planning and Investment Control and the Financial Management Services organizations are members of a working group structured to improve investment processes. The MV&S Program Office continues to search for ways to improve these processes. For example, the MITRE Corporation conducted lessons-learned sessions concerning the 2009 investment planning processes. The participants in these sessions reported that the planning process had improved from that in the prior year.
Another
example of the collaborative alliances that were developed is demonstrated by the
effort to ensure that all proposed solutions are in compliance with the
Enterprise Architecture and support business processes. We determined that a collaborative system
integration review was being accomplished and included representatives from
several organizations such as the Systems Architecture and Engineering,
Enterprise Architecture, Enterprise Data Management, and Information Technology
Security Engineering organizations.
The IRS emphasizes focused, more frequent delivery of information technology project releases
The IRS
continues to emphasize smaller, more frequent delivery of information
technology project releases. In our annual
assessment of the Business Systems Modernization program for Fiscal Year 2008,[2] we concluded that the IRS and its
contractors were performing better against cost and schedule estimates as a
result of breaking releases down into smaller, more manageable pieces. While our annual assessment included an
analysis of major investments in the Business Systems Modernization program, we
determined that the October 2007 MV&S Plan included language relevant to all investment projects. Further, our review of
meeting minutes and other supporting documents determined that the IRS is following
a policy of making more frequent releases.
This policy is the focus of continued emphasis within the IRS. As recently as July 2008, the
The
Government Accountability Office Standards
for Internal Control in the Federal Government[3] states that internal management
control is 1) comprised of plans, methods, and procedures used to meet
departmental mission, goals, and objectives, and 2) supports performance-based
management. The Clinger-Cohen Act
of 1996[4]
addresses controls over information technology investments and encourages
streamlined, phased approaches for information technology acquisitions.
The IRS Enterprise Life Cycle methodology stresses focus on
delivering quality results that produce smaller benefits early and
incrementally, throughout the life of the project.
The
MV&S Program has improved the information technology investment
decision processes, resulting in information technology projects that 1) better
support the IRS goals of improving customer service and enforcement, and 2)
make better use of technology funds. However,
process improvements need to be implemented to ensure the continued improvement
of future investment proposal cost estimates and standardized reporting of
investment decisions’ performance measures.
The Process
for Developing Total Cost Estimates Can Be Improved
The IRS is using internal guidance to estimate the total cost of ownership for information technology investment projects. However, the guidance does not require comparison of actual investment costs to the original estimates. Thus, these comparisons do not occur. We reviewed the cost estimates for 4 of the 23 projects in the approved investment portfolio and determined that the Estimation Program Office uses 1 of the approved estimation models for determining total cost of ownership. The cost assumptions and the resulting cost estimates are validated by employees with expertise in the area participating in the development of the Solution Concept document, and Estimation Program Office personnel conduct indepth reviews of the Basis of Estimate with the team working on the proposed investment. Based on the extent of the estimation process and the Basis of Estimate documents, the cost estimates appear to be reasonable. However, the reliability of the estimated total cost of ownership might be at risk because the estimates for prior projects were not validated by comparing the estimates to the actual costs.
Internal guidance, IRS Basis for Parametric Estimation Guidance, dated October 2007, provides a basic
requirement to update operations and maintenance estimates to actual costs for
future estimating purposes. However, it
does not include specific provisions for comparisons of actual costs to
original estimates. Additionally, in
July 2007, the Government Accountability
Office released an Exposure Draft, Cost
Assessment Guide: Best Practices for
Estimating and Managing Program Costs,[5]
that provides an outline for successful cost estimating in a 12-step estimation
process. The process provides specific
procedures to update cost estimates to actual costs.
IRS management informed us that there is no documented process to track and compare actual operations and maintenance costs to the original estimates. The systems modernization projects (i.e., major projects) track actual costs compared to the Earned Value Management budget for Capital Planning and Investment Control, but they do not consider original cost estimates. Management further indicated that processes to document other projects (i.e., non-major projects) focus on only development, modernization, and enhancement costs and schedules and do not consider operations and maintenance.
By not comparing initial cost estimates to the actual project costs to
validate the cost estimation model, the IRS is increasing the risk that its future
investment proposal cost estimates will not be reliable.
Recommendation
Recommendation
1: The Chief Information Officer should ensure the reliability of the
cost estimation process by implementing procedures to compare actual project
operations and maintenance costs to initial estimates and revising the
estimation process, if necessary.
Management’s Response: The IRS
agreed with this recommendation. It will implement procedures to compare actual project
operations and maintenance costs to initial estimates and revise the estimation
process when necessary.
Modernization Vision and Strategy Program Performance
Measurement Standards Need Further Development
MV&S Program performance
measures are being collected, but there is no standardized procedure for compiling
and reporting these measures. Through
review of documentation and discussions with personnel, we could not determine
whether performance measurement and reporting were being thoroughly
accomplished as required. Specifically,
the October 2007 MV&S Plan does not include guidelines for measuring
and reporting compliance. Also, when we reviewed the performance measurement briefing
slides and templates, it was unclear whether the data collection and reporting
cycle was accomplished. Additionally,
personnel stated that because the MV&S Program was in its early stages and was just gaining momentum, they
are now able to determine the types of data measurement and reporting that are
needed.
The Government
Performance and Results Act of 1993[6] requires agencies to set goals, measure performance, and report on
their accomplishments. A key tenet of
the Act is that agencies will develop 1) strategic plans–as well as annual
performance plans that are linked to the strategic plan–that establish the
organizational goals and objectives, and 2) strategies for achieving these
goals. Additionally, agencies are required
to establish performance measures and benchmarks to identify gaps between
actual and desired performance levels and mission outcomes.
The MV&S Program is a new
program that began with the first MV&S Plan in October 2006. Over the next 2 years, as the Program gained
momentum, management stated that formalized procedures were being consolidated
based on integrated involvement and lessons learned.
Determining
performance goals for information technology investments is a critical factor
for requesting funding in the budget formulation and submission process. Performance measures are necessary to support
the establishment of a sound strategic direction to ensure that decisive
information technology investment and acquisition decisions are accomplished through
business priority shifting and technological evolvement. The lack of MV&S Program performance
measures and reporting compliance could result in inefficient accomplishment of
strategic information technology acquisitions, objectives, and expectations.
Recommendation
Recommendation
2: The Chief Information Officer
should include
guidelines for performance measurement and reporting in the updated MV&S Plan.
Management’s Response: The IRS
agreed with this recommendation and will
update the MV&S Plan to reference performance measures and reporting. This update will be included in the Calendar Year
2009 annual publication.
Appendix I
Detailed Objective, Scope, and Methodology
The overall objective of this review was to determine whether the IRS MV&S Program is achieving the desired results. To accomplish our objective, we:
I. Determined whether the enterprise-wide investment portfolio approved by the MITS organization Enterprise Governance Committee[7] supports modernizing front-line tax administration and the technical infrastructure.
A. Conducted interviews with IRS personnel and requested and reviewed business organizations’ strategic plans.
B. Interviewed IRS personnel and determined whether technical, functional, and integration reviews were conducted by the IRS to ensure that the system projects identified in the investment portfolio support the technical infrastructure.
II. Determined whether the business organizations were involved in the investment strategy formulation process.
A. Interviewed Portfolio Planning, Estimation, and Delivery Services Office management to determine the process for soliciting input from the business organizations for the budget formulation process.
B. Reviewed documentation to determine whether data requests were sent to all business organizations.
C. Reviewed the participants list for the investment strategy formulation meeting.
III. Reviewed the coordination between the MV&S Program Office and other organizations responsible for portfolio management/governance, including the Capital Planning and Investment Control Office, Executive Steering Committees, and Financial Management Services organization.
A. Interviewed MV&S Program Office and Capital Planning and Investment Control Office personnel to determine the roles and responsibilities regarding monitoring, tracking, and reporting the progress of the MV&S Program.
B. Interviewed Financial Management Services organization personnel to determine the roles and responsibilities regarding budget formulation and adjustments within the MV&S Program.
C.
Reviewed the
Executive Steering Committees’ meeting minutes to assess coordination with the MV&S Program Office.
IV.
Determined
whether the IRS is frequently delivering smaller, more incremental releases.
A. Interviewed MV&S Program Office personnel and reviewed the investment portfolio and Executive Steering Committees’ meeting minutes to determine whether projects are developed and frequently delivered via incremental releases.
B.
Reviewed
project documentation for
supporting costs and schedules that are being estimated and approved at the
release level. We selected a
judgmental sample of 4 projects from the population of 23 projects in the
approved investment portfolio. This list of 23 projects provided by the IRS
included only projects for the MV&S Program for the Fiscal Year 2009 cycle.
We obtained estimated costs and business area
and respective owner information to determine whether projects were being estimated and approved at the release
level. The four projects were
judgmentally selected based on the projects’ titles, time period from development
to operations and maintenance, total cost, and project business area. We used a judgmental sample because we
did not plan to project the audit results.
V.
Determined
whether the IRS is creating costing models that include investment, operations, and
maintenance amounts.
A. Interviewed personnel in the MV&S Program Office; the Portfolio Planning, Estimation, and Delivery Services Office; the Solution Concept Office; the Estimation Program Office; and the Capital Planning and Investment Control Office regarding the estimation process and models used to prepare project costs estimates.
B.
Reviewed the
Government
Accountability Office Assessing Risks and Returns: A Guide for Evaluating Federal Agencies’
Information Technology Investment Decision-making[8] and the IRS operations and maintenance guidance.[9]
C.
Reviewed the
supporting project cost documentation
for the judgmental sample of four projects selected in Step IV.B.
VI.
Determined
whether guidelines, processes, and procedures were in place for measuring and
reporting results delivered by the MV&S Program.
A. Interviewed MV&S Program Office personnel and reviewed the MV&S Plan to identify MV&S Program goals, objectives, outcomes, and performance measures to ensure program accomplishment.
B.
Verified whether measures are linked to IRS long-term
strategic goals as defined in the 2005-2009 IRS Strategic Plan.
C. Interviewed MV&S Program Office personnel to determine whether performance measures are incorporated into the Business Performance Management System.
D. Interviewed MV&S Program Office personnel to determine whether specific guidelines were developed for periodically measuring and reporting MV&S Program accomplishments.
VII. Determined the validity of data. The IRS provided us with a report, identified as the project Basis of Estimate report, that contained support for each project’s total cost and timeline. We reviewed the information in each Basis of Estimate report for reasonableness. However, we did not conduct audit steps to verify the accuracy or completeness of the information because we did not use the information to make projections.
Appendix II
Major Contributors to This Report
Margaret E. Begg,
Assistant Inspector General for Audit (Security and Information Technology
Services)
Scott A. Macfarlane,
Director
Danny R. Verneuille, Audit Manager
Kimberly R. Parmley, Lead Auditor
Paul M. Mitchell, Senior Auditor
Wallace C. Sims, Senior Auditor
Charlene L. Elliston, Auditor
Appendix III
Commissioner C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Operations Support OS
Associate Chief Information Officer, Applications Development OS:CIO:AD
Associate Chief Information
Officer,
Director, Stakeholder Management OS:CIO:SM
Chief
Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director,
Office of Program Evaluation and Risk Analysis
RAS:O
Office of Internal Control OS:CFO:CPIC:IC
Audit
Liaison: Director,
Program Oversight Office OS:CIO:SM:
Appendix IV
|
Basis of Estimate |
A tool used to
estimate all task order proposals. It includes specific sections that must be
documented to justify the estimate proposal, such as work to be performed, scheduling
estimates, and staffing summaries. |
|
Business Performance Management System |
A System designed
to support the IRS’ Strategic Planning and Budgeting and Performance
Management processes. It supports executives
and the senior leadership team, division managers, division planners, and analysts. The System also brings value to the
organization by providing 1) a centralized source of agency-wide planning and
performance management data, 2) a “guided analysis tool” to facilitate
planning and performance management at the enterprise, division, and program
level, and 3) a “one-stop” source for information. |
|
Earned Value Management |
Involves measuring actual cost and work accomplished
against the budgeted cost and planned work scheduled. Variances are analyzed for decision making. |
|
|
A unifying overall design or structure for
an enterprise. |
|
|
The highest level recommending and decision-making
body to oversee and enhance enterprise management of information systems and
technology. It ensures that strategic
modernization and information technology program investments, goals, and
activities are aligned with and support 1) the business needs across the
enterprise, and 2) the modernized vision of the IRS. It oversees investments, including
validating major investment business requirements and ensuring that enabling
technologies are defined, developed, and implemented. |
|
|
A structured business systems development method that
requires the preparation of specific work products during different phases of
the development process. |
|
Major Project (Investment) |
Department of the Treasury specific criteria state that major information technology investments (or projects) have an annual cost equal to or greater than $5 million, or total lifecycle costs exceeding $50 million. |
|
MITRE Corporation |
Provides the IRS with independent, expert, and objective advice and guidance on strategic, technical, and program management issues. |
|
Non-Major Project (Investment) |
Any initiative or investment not meeting the definition of a major project. |
|
Release |
A specific edition
of software. |
|
Solution Concept Document |
Provides
the characteristics of the ability of the system. |
Appendix V
The response was removed due to its
size. To see the response, please go to
the Adobe PDF version of the report on the TIGTA Public Web Page.
[1] See Appendix IV for a glossary of terms.
[2] Annual Assessment of the Business Systems Modernization Program (Reference Number 2008-20-129, dated June 24, 2008).
[3] GAO/AIMD-00-21.3.1, dated November 1999.
[4] (Federal Acquisition Reform Act of 1996) (Information Technology Management Reform Act of 1996), Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).
[5] GAO 07-1134SP, dated July 2007.
[6]
Pub. L. No. 103-62, 107 Stat. 285 (codified as
amended in scattered sections of 5 U.S.C., 31 U.S.C., and 39 U.S.C.).
[7] See Appendix IV for a glossary of terms.
[8] GAO/AIMD-10.1.13, dated February 1997.
[9] IRS Basis for Parametic Estimation Guidance, dated October 23, 2007.