TREASURY INSPECTOR GENERAL FOR TAX ADMINISTRATION

 

 

The Modernization and Information Technology Services Organization’s Revised Post Implementation Review Procedure Can Be Improved

 

 

 

October 26, 2006

 

Reference Number:  2007-20-001

 

 

This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.

Phone Number   |  202-927-7037

Email Address   |  Bonnie.Heald@tigta.treas.gov

Web Site           |  http://www.tigta.gov

 

October 26, 2006

 

 

MEMORANDUM FOR CHIEF INFORMATION OFFICER

\

FROM:                            Michael R. Phillips /s/ Michael R. Phillips

                                         Deputy Inspector General for Audit

 

SUBJECT:                    Final Audit Report – The Modernization and Information Technology Services Organization’s Revised Post Implementation Review Procedure Can Be Improved (Audit # 200620039)

 

This report presents the results of our review of the Modernization and Information Technology Services organization’s revised Post Implementation Review (PIR) procedure.  The overall objective of this review was to determine the adequacy of the procedures for performing PIRs to assess benefits and capabilities of new information technology projects.  This review was part of our Fiscal Year 2006 audit plan for reviews of the Internal Revenue Service’s (IRS) Business Systems Modernization (BSM) efforts.

Impact on the Taxpayer

PIRs assess the impact of a new information technology project by comparing and evaluating actual project results to estimates of cost, schedule, performance, and mission improvement outcomes.  Once a PIR is completed, the results should be archived and distributed to affected parties, particularly those with decision-making authority who could most benefit from the recommendations and lessons learned.  The IRS is in the process of updating PIR guidance used in conducting project assessments.  However, additional enhancements to the PIR guidance would help ensure taxpayer funds spent on the program are being used effectively and efficiently.

Synopsis

In June 2000, the BSM Office (BSMO), which is now in the Applications Development office, issued PIR procedures as part of the Enterprise Life Cycle.[1]  In October 2004, the BSMO issued a BSMO Procedure updating the June 2000 procedures and changing the responsible office to the Program Performance Management office.[2]  This 2004 update provided additional details for launching and performing PIRs and added the requirement to perform Post Reviews of milestone completion activities.  The Program Performance Management office stated it added the Post Reviews of milestone completion activities to the 2004 guidance as a way to further enhance post implementation controls in response to preliminary Government Accountability Office (GAO) findings related to PIRs.  Following a formal GAO recommendation in November 2004,[3] the Program Performance Management office stated it would again update the PIR procedures by September 2006.

The Program Performance Management office needs to implement its PIR process to provide the IRS feedback about operating its new computer systems to provide better service to taxpayers.

The Program Performance Management office has initiated a PIR of the e-Services project.  This Review will serve as a pilot for implementing the updated PIR procedures.  The Program Performance Management office’s plans include having the IRS Office of Program Evaluation and Risk Analysis and the Wage and Investment Division Customer Account Services office assist in the review.  The Program Performance Management office planned to perform the e-Services project PIR between May and July 2006 and issue the PIR report by September 2006.  Completion of the PIR and any procedure updates in September 2006 were intended to help the IRS meet its commitment to the GAO to implement corrective actions making the PIR process more meaningful.

Our review of the June 2000 Enterprise Life Cycle procedures, the October 2004 BSMO Procedure, and the May 2006 draft procedures showed these guidance documents generally incorporate the requirements issued by the Office of Management and Budget and the GAO.[4]  The May 2006 draft procedures provide additional detail to clarify some PIR processes in the current Enterprise Life Cycle and BSMO procedures.  Although both the October 2004 and updated draft procedures provide adequate direction to perform a PIR, the Program Performance Management office has not scheduled reviews for all projects that have exited milestones or for deployed releases.  Further, it has not completed its process for capturing and managing documentation related to PIRs such as Post Review reports, PIR reports, and lessons learned documents.  Additionally, it has not identified the reviewer qualifications or training needed to prepare participants to perform Post Reviews and PIRs.

Without the information accumulated and developed from a PIR, senior management may not have necessary information about the issues with implemented systems to decide to continue, modify, or terminate their operations.  Management may also lack the information necessary to decide about further actions needed to achieve anticipated system benefits to meet agency mission requirements and projected return on investment.  The need for this information will become more critical as the IRS begins implementing its Information Technology Modernization Vision and Strategy, which itemizes numerous new projects.  Without the ability to readily access PIR documentation, reports, and lessons learned, project and program managers may not make the best decisions to promote project deployment and direct the modernization program.  Also, PIRs can be successful only if they are staffed with qualified and trained team members.

Recommendations

The Chief Information Officer should direct the Program Performance Management office to develop a schedule to perform PIRs for deployed releases; identify and obtain staffing resource commitments needed from the Program Performance Management office, the Office of Program Evaluation and Risk Analysis, and appropriate business unit representatives to effectively execute PIRs; eliminate the requirement to perform Post Reviews of milestone completion activities; implement a procedure to control PIR results that ensures appropriate executives and decision makers have access to these documents; identify skills and abilities desired for PIR team members and ensure team assignments consider these qualifications; and develop a training guide for PIR team members to provide them with an understanding of the purpose, objectives, and processes of the PIR.

Response

The IRS agreed with the three report recommendations and has successfully eliminated the requirement to perform Post Reviews of project milestone activities.  Due to other priorities, including enhancing the Modernization and Information Technology Services organization’s Governance and Program Control processes as mandated in the pending Fiscal Year 2007 Treasury/IRS Appropriations language, the IRS is uncertain when the remaining corrective actions will be completed.  The IRS will prioritize these recommendations with other urgent needs as part of the Modernization and Information Technology Services organization’s ongoing High Priority Initiative process.  Management’s complete response to the draft report is included as Appendix VIII.

Office of Audit Comment

Although the IRS is focusing its efforts on enhancing the Governance and Program Control processes as mandated in the pending Fiscal Year 2007 Treasury/IRS Appropriations language, it should be noted that established laws, regulations, policies, and procedures have required PIRs of new information technology projects since 1990.  These requirements are presented in the Background section and Appendix VI of this report.  For example, Office of Management and Budget Circular No. A-130, Management of Federal Information Resources, dated March 2006, requires agencies to develop a capital planning and investment control process that links mission needs, information, and information technology effectively and efficiently.  This process states that agencies must conduct PIRs of information systems and information resource management processes to (1) validate estimated benefits and costs and (2) document effective management practices for broader use.

The PIR process is an integral component of the Governance and Program Control process and should have been an ongoing activity to assess program accomplishments and performance.  Postponing corrective actions, which would have established a schedule and resource commitments to perform PIRs, will not assist the IRS in making informed decisions to continue, modify, or terminate information technology projects.

Copies of this report are also being sent to the IRS managers affected by the report recommendations.  Please contact me at (202) 622-6510 if you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs), at (202) 622-8510.

 

 

Table of Contents

 

Background

Results of Review

Post Implementation Review Guidance Is Being Updated to Provide Appropriate Direction to Develop Meaningful Project Assessments

Enhancements to Post Implementation Review Procedures Can Make Them More Effective and Efficient

Recommendation 1:

Recommendations 2 and 3:

Appendices

Appendix I – Detailed Objective, Scope, and Methodology

Appendix II – Major Contributors to This Report

Appendix III – Report Distribution List

Appendix IV – Outcome Measure

Appendix V – Enterprise Life Cycle Overview

Appendix VI – Authorities Requiring Post Reviews of New Information Technology Projects

Appendix VII – Glossary of Terms

Appendix VIII – Management’s Response to the Draft Report

 

 Abbreviations

 

BSM

Business Systems Modernization

BSMO

Business Systems Modernization Office

ELC

Enterprise Life Cycle

GAO

Government Accountability Office

IRS

Internal Revenue Service

PIR

Post Implementation Review

U.S.C.

United States Code

 

Background

 

Post Implementation Reviews (PIR) assess the impact of a new information technology project by comparing and evaluating actual project results to estimates of cost, schedule, performance, and mission improvement outcomes.  Federal Law,[5] Executive Branch guidance,[6] and the Internal Revenue Service (IRS) Enterprise Life Cycle (ELC)[7] establish the need to perform PIRs after project completion and after completion of major project releases.  PIRs include the following activities:

The key focuses of PIRs are to assess the impact of new projects on the IRS mission, customers, workforce, and technology.

·         Assessment of project effectiveness in meeting the original objectives.

·         Identification of benefits achieved to determine whether they match projected benefits, and reasons for any discrepancies.

·         Evaluation of original business assumptions used to justify a project.

·         Comparison of actual life cycle investment costs incurred to projections.

·         Assessment of the impact of project risks.

·         Determination of project timeline and implementation date achievements.

·         Identification of management and user perspectives on a project.

·         Evaluation of issues requiring continued attention.

PIRs are a vital part of the Investment Decision Management process.  Lessons learned developed as part of the PIR activity should be used to recommend changes that improve the investment process (e.g., selection, control, or evaluation) and the management of individual investments.  For example, PIR report recommendations may include suggested refinements of criteria to select future projects.  Once a PIR is completed, the results should be archived and distributed to affected parties, particularly those with decision-making authority who could most benefit from the recommendations and lessons learned.

In November 2004, the Government Accountability Office (GAO) recommended[8] the IRS Chief Information Officer ensure PIRs are performed after Business Systems Modernization (BSM) projects are deployed.  The recommendation suggested the PIRs include analyses of quantitative and qualitative investment data to determine, at a minimum, whether expected benefits were achieved.  The IRS responded with corrective action plans to implement new procedures that not only extend to the PIR process as recommended by the GAO but also look at lessons learned at the end of each project milestone and retain the results of these comprehensive reviews in a repository available to everyone in the BSM Program.  The IRS planned to complete its corrective actions in September 2006.

This review was performed at the Modernization and Information Technology Services organization’s facilities in New Carrollton, Maryland, during the period February through May 2006.  The audit was conducted in accordance with Government Auditing Standards.  Detailed information on our audit objective, scope, and methodology is presented in Appendix I.  Major contributors to the report are listed in Appendix II.  Appendix VI presents the authorities requiring post implementation review of new information technology projects.  Appendix VII presents a glossary of terms.

 

 

Results of Review

 

Post Implementation Review Guidance Is Being Updated to Provide Appropriate Direction to Develop Meaningful Project Assessments

The PIR process, first issued in June 2000, was updated in October 2004 and is again being updated by the Program Performance Management office.

In June 2000, the BSM Office (BSMO), which is now in the Applications Development office, issued PIR procedures as part of the ELC.  In October 2004, the BSMO issued a BSMO Procedure updating the June 2000 procedures and changing the responsible office to the Program Performance Management office.[9]  This 2004 update provided additional details for launching and performing PIRs and added the requirement to perform Post Reviews of milestone completion activities.  The Program Performance Management office stated it added the Post Reviews of milestone completion activities to the 2004 guidance as a way to further enhance post implementation controls in response to preliminary GAO findings related to PIRs.  Following a formal GAO recommendation in November 2004, the Program Performance Management office stated it would again update the PIR procedures by September 2006.

To assess modernization project capabilities and benefits, the ELC currently provides guidance that includes identifying review responsibilities, performing review activities, and communicating review results in the form of a post review report and/or lessons learned document.  This guidance directs the Enterprise Services Program Performance Management office to initiate PIR activities.

Current PIR procedures specify two types of reviews:

·         Post Reviews assess project processes and activities during the design, development, and deployment phases.  These Reviews involve analyses of (1) the project’s contract, including financial data and the contract schedule, and (2) performance metrics,  including cost and schedule variance analysis.  These Reviews also include analysis of survey and interview data from acquisition project management teams and assessments from the Architecture, Security, Risk Management, and other offices.  Post Reviews should be completed within 1 month to 4 months from project milestone exits and release deployments.  This type of Review is also performed for projects that have been cancelled.

·         Post Implementation Reviews assess the outcomes, capabilities, and benefits of deployed releases and projects.  The PIR not only includes analyses in a post review, it also involves analyses of the following project products:

o       Office of Management and Budget Exhibit 300, Capital Asset Plan and Business Case.

o       Case for Action and Baseline Business Case.

o       Project Description, Concept of Operations, Vision and Strategy documents.

o       Task Order or Contract project requirements (high level).

o       Responses from business owners using surveys and interviews.

o       Responses from end users using surveys and interviews.

o       Responses from system maintainers using surveys and interviews.

o       Business performance metrics (input, output, efficiency, and outcome).

o       Business process reengineering documentation.

The PIR should be completed within 6 months to 18 months after each project release deployment.

Performance of the PIR is followed by a post review session.  The Program Performance Management office hosts the post review session and uses the results to develop a lessons learned document.  The lessons learned document is incorporated into the final PIR report.  The Post Review and PIR reports, which include documentation of lessons learned, are finalized and distributed to the Acquisition Project Manager, business owner, external Federal Government “audit” agencies (i.e., the GAO and Treasury Inspector General for Tax Administration), and IRS senior leadership, as appropriate.  The Post Review report includes:

·         Highlights of post review project performance metrics and key assessments.

·         The top three “What Went Wrong.”

·         The top three “What Went Right.”

·         An overview of the top lesson developed and type of lesson.

·         Recommendations and insights regarding improvement of acquisition, investment review, or capital planning processes.

·         Areas requiring resolution.

During our audit, the Program Performance Management office initiated a PIR of the e-Services project.  This Review was intended to serve as a pilot to gather additional input for procedure updates.  The Program Performance Management office’s plan included assistance by the IRS Office of Program Evaluation and Risk Analysis in the Review.  During our audit, the Program Performance Management office was finalizing the PIR work plan and related survey and Review templates.  Review participants included representatives from the e-Services project team and the Wage and Investment Division Customer Account Services office.  The Program Performance Management office planned to perform the e-Services project PIR between May and July 2006 and issue the PIR report by September 2006.  Completions of the PIR and any procedure updates in September 2006 were intended to help the IRS meet its commitment to the GAO to implement corrective actions making the PIR process more meaningful.

Enhancements to Post Implementation Review Procedures Can Make Them More Effective and Efficient

The Program Performance Management office needs to implement its PIR process to provide the IRS feedback about operating its new computer systems to provide better service to taxpayers.

Our review of the June 2000 ELC procedures, the October 2004 BSMO Procedure, and the May 2006 draft procedures showed these guidance documents generally incorporate the requirements issued by the Office of Management and Budget and the GAO.[10]  The May 2006 draft procedures provide additional detail to clarify some PIR processes in the current ELC and BSMO procedures.  Although both the October 2004 and updated draft procedures provide adequate direction to perform a PIR, the Program Performance Management office has not scheduled reviews for all projects that have exited milestones or for deployed releases.  Further, it has not completed its process for capturing and managing documentation related to PIRs such as Post Review reports, PIR reports, and lessons learned documents.  Additionally, it has not identified the reviewer qualifications or training needed to prepare participants to perform the Post Reviews and PIRs.

All PIRs have not been identified and scheduled

In response to the GAO’s November 2004 recommendation to perform PIRs after project release deployment, the IRS planned corrective action to conduct the first PIRs under its new process by the end of the Calendar Year 2004 and within 45 calendar days of passing each subsequent project milestone.  The IRS has approved 15 milestone exits and 9 release deployments since the November 2004 GAO recommendation.  However, it has completed only three Post Reviews of milestone exits and no PIRs of release deployments.  Table 1 presents an analysis of post review activities for the milestones exited and releases deployed.

Table 1:  Modernization Project Milestone Exits and Release Deployments Since November 2004

Project

Release
Milestone

Milestone
Exit

Release

Release Deployment

Review Type

Report Date

Customer Account
Data Engine

 

 

1.2

January 18, 2005

None

 

 

 

 

1.3.1

September 19, 2005

None

 

 

 

 

1.3.2

January 17, 2006

None

 

 

Release 2.1/
Milestone 3

May 5, 2006

 

 

None

 

Custodial Accounting
Project

 

 

Cancelled

February 5, 2005*

Post Review

June 2005

e-Services

Release 2.0/ Milestone 4

May 30, 2005

2.0

August 31, 2005

PIR

In process

Filing and Payment Compliance

Release 1.1/
Milestone 3

July 5, 2005

 

 

None

 

 

Release 1.1/
Milestone 4A

August 30, 2005

 

 

None

 

 

Release 1.1/
Milestone 4B

January 23, 2006

1.1

January 24, 2006

None

 

 

Release 1.2/
Milestone 3

February 28, 2006

 

 

None

 

Integrated Financial
System

Release 1/
Milestone 4

 

1

November 29, 2005

Post Review

April 2005

Modernized
e-File

Release 3.1/
Milestone 4, Milestone 5

April 1, 2005

3.1

March 24, 2005

None

 

 

Release 3.2/
Milestone 3

February 4, 2005

 

 

None

 

 

Release 3.2/
Milestone 4A

April 28, 2005

3.2

March 22, 2006

Post Review

 

February 2006

 

Modernized
e-File (cont.)

Release 3.2/
Milestone 4, Milestone 5

March 22, 2006

 

 

None

 

 

Release 4/
Milestone 3

December 5, 2005

 

 

None

 

 

Release 4/
Milestone 4A

April 21, 2006

 

 

None

 

Totals

15

 

9

 

4

 

Source:  The BSMO and its online web site.
* Custodial Accounting Project cancellation Post Review report issued June 2005.

The three Post Reviews performed by the Program Performance Management office adequately assessed the project’s costs, schedule, and performance.  The Program Performance Management office has initiated a PIR for the e-Services project and is in the process of determining the resources needed to conduct the Post Review and PIR.  However, it has not yet received commitments from offices outside of the Program Performance Management office to participate as team members.

Limited resources may prevent the IRS from fully implementing planned corrective actions to the November 2004 GAO report, specifically performing a Post Review within 45 calendar days of passing project milestones.  The IRS already performs a detailed Milestone Exit Review before passing each project milestone, which provides executives sufficient information to assess a project’s progress and plans.  In preparation for the Milestone Exit Review, project teams develop a set of documents to provide executive management the ability to assess the project’s progress against ELC criteria prior to milestone exits.  This Review helps to ensure progressing to the next project phase is appropriate.  Milestone exits are approved by senior executives that form the Modernization and Information Technology Services Enterprise Governance board using the information developed in the Milestone Exit Review.

The Post Reviews of milestone activities performed by the Program Performance Management office in 2005 required approximately 63 staff days each to complete at a cost of approximately $28,000 per review.  To comply with current guidelines, the IRS would have had to perform an additional 12 Post Reviews of milestone activities by September 5, 2006 (see Table 1 above).  These Reviews would cost approximately $336,000.  In addition, the IRS would have to perform an estimated 26 additional Post Reviews of milestone activities over the next 5 years for projects currently in development.  We estimate these 26 Post Reviews of milestone completion activities would cost an estimated $728,000.  By no longer performing Post Reviews of milestone activities the IRS would eliminate the inefficient use of $1,064,000 (see Appendix IV for details).  Further, the Information Technology Modernization Vision and Strategy shows work for 54 projects in Fiscal Years 2007 and 2008.

The Program Performance Management office is also facing imminent deadlines for performing PIRs within 6 months to 18 months of release deployment.  Although the e-Services pilot PIR is being initiated, the Customer Account Data Engine project Release 1.2 (deployed January 18, 2005) and Modernized e-File project Release 3.1 (deployed March 24, 2005) will pass the 18-month time period during the planned performance of the e-Services project PIR.

A process to capture PIR documentation is in development

After a PIR, lessons learned may result in recommendations for changes to the current portfolio and/or the overall investment decision management process.

The IRS response to the GAO November 2004 report included agreement to capture the results of the PIRs in a repository that is available to everyone in the BSM Program.  The IRS planned to use the Document Management System as the repository in which to control all PIR reports.  The IRS planned to maintain PIR reports along with lessons learned documents developed during the various reviews of modernization activities in the Document Management System.  Since issuing this planned corrective action, the Program Performance Management office recognized a need for more control over PIR reports to ensure they are properly maintained and accessible to the appropriate level of management.  In September 2005, the IRS decided to stop using the Document Management System to control PIR reports and lessons learned documents due to an absence of adequate controls over access to these sensitive documents.  It postponed plans to control PIR reports in the Document Management System until it resolved the report access issues.

As of May 25, 2006, the Program Performance Management office had not yet developed a revised process to allow for the control and maintenance of current PIR reports or methods to capture and control future PIR reports.  These reports provide valuable information about the continuation, modification, or termination of a project.  The reports also present results about the project’s ability to meet agency mission requirements, assess whether guiding assumptions still hold in terms of meeting the projected return on investment, or provide a basis for recommendations about future investment decisions.

Qualifications and training for PIR participants have not been developed

The GAO presents as a prerequisite that individuals conducting PIRs are trained.  Thus, the PIR team members must be objective, well trained, and experienced when they conduct PIRs.  Also, the team leader should have past experience conducting similar investment reviews.

Neither the October 2004 nor the May 2006 draft PIR guidance identifies minimum qualifications or experience needed for PIR team members.  The Program Performance Management office also advised us that PIR training materials have not been developed.  It indicated it planned to have a briefing on the purpose and objectives of the PIR for team members in advance of the e-Services project PIR activities.

The value of the PIR will depend to a large degree on the credibility and competence of the team members conducting the study.

As the GAO presented in its prior report about the IRS’ execution of PIRs, a new procedure will not prove beneficial unless the IRS ensures it is followed, since the current procedure has not been fully implemented.  By completing the PIR procedure development, scope identification, repository controls, and training materials, the Program Performance Management office will be in a position to provide valuable feedback about the accomplishments and capabilities of new systems and applications.

Without the information accumulated and developed from a PIR, senior management may not have necessary information about the issues with implemented systems to decide to continue, modify, or terminate their operations.  Management may also lack the information necessary to decide about further actions needed to achieve anticipated system benefits to meet agency mission requirements and projected return on investment.  The need for this information will become more critical as the IRS begins implementing its Information Technology Modernization Vision and Strategy, which itemizes numerous new projects.  Without the ability to readily access and analyze PIR documentation, reports, and lessons learned, project and program managers may not make the best decisions to promote project development and deployment and direct the modernization program.  Also, PIRs can be successful only if they are staffed with qualified and trained team members.

Recommendations

Recommendation 1:  The Chief Information Officer should direct the Program Performance Management office to develop a schedule to perform PIRs for deployed releases and to identify and obtain staffing resource commitments needed from the Program Performance Management office, the Office of Program Evaluation and Risk Analysis, and appropriate business unit representatives to effectively execute the PIRs.  The Chief Information Officer should also direct the Program Performance Management office to eliminate the requirement to perform Post Reviews of milestone completion activities after milestone exits, with the option of identifying any lessons learned for further project progress through the Milestone Exit Reviews.

Management’s Response:  The IRS agreed with our recommendation.  The IRS has eliminated the requirement to perform Post Reviews of milestone activity and has developed and piloted a PIR process.  The IRS also agreed with the need to develop and implement a scheduling process and a process to identify resource requirements and request these resources.  While the IRS recognized the importance of this recommendation and subsequent corrective action, it stated that enhancing its Governance and Program Control processes, as mandated in the pending Fiscal Year 2007 Treasury/IRS Appropriations language, will require its immediate focus and available resources.  The IRS will complete the corrective action as soon as resources are available.  Until then, this corrective action is on hold with an implementation date of March 1, 2010, which reflects that this date is to be determined.  The implementation date will be modified to represent an accurate completion date once a date is established.  The IRS will prioritize this and the following recommendations with other urgent needs as part of its ongoing High Priority Initiative process.

Recommendation 2:  The Chief Information Officer should direct the Program Performance Management office to implement a procedure to control PIR results that ensures appropriate executives and decision makers have access to these documents.

Management’s Response:  The IRS agreed with our recommendation, responding that, using the Document Management System controls, it has already initiated a sensitivity process to control access to the lessons learned.  In addition, it has developed a draft revised process to allow for (1) the control and maintenance of current PIR reports and (2) methods to capture and control future PIR reports.  The IRS agreed with the need to finalize the draft revised process that ensures appropriate executives and decision makers have access to the proper documents.  As stated above, the IRS will prioritize the corrective actions for the recommendation with other urgent needs.

Recommendation 3:  The Chief Information Officer should direct the Program Performance Management office to identify skills and abilities desired for PIR team members and ensure team assignments consider these qualifications.  The Program Performance Management office should develop a training guide for PIR team members to provide them with an understanding of the purpose, objectives, and processes of the PIR.

Management’s Response:  The IRS agreed with our recommendation.  The IRS will update the Post Review procedure to identify skills and abilities desired for PIR team members, to ensure team assignments consider these qualifications.  Training materials for the e-Services PIR pilot were developed and used to provide the team members with an understanding of the purpose, objectives, and processes of the PIR.  Based on the results from the e-Services PIR pilot, these materials will be updated to complete a PIR Training Guide.  As stated above, the IRS will prioritize the corrective actions for the recommendation with other urgent needs.

Office of Audit Comment:  Although the IRS is focusing its efforts on enhancing the Governance and Program Control processes as mandated in the pending Fiscal Year 2007 Treasury/IRS Appropriations language, it should be noted that established laws, regulations, policies, and procedures have required PIRs of new information technology projects since 1990.  These requirements are presented in the Background section and Appendix VI of this report.  For example, Office of Management and Budget Circular No. A-130, Management of Federal Information Resources, dated March 2006, requires agencies to develop a capital planning and investment control process that links mission needs, information, and information technology effectively and efficiently.  This process states that agencies must conduct PIRs of information systems and information resource management processes to (1) validate estimated benefits and costs and (2) document effective management practices for broader use.

Additionally, we have concerns about the use of the High Priority Initiative process as a means for prioritizing the implementation of these recommendations because this process aims to address broad organizational policies and strategies rather than to develop and implement specific program control processes.  Therefore, we would expect the recommended improvements to the PIR process to not be selected as High Priority Initiatives.

The PIR process is an integral component of the Governance and Program Control process and should have been an ongoing activity to assess program accomplishments and performance.  Postponing corrective actions, which would have established a schedule and resource commitments to perform PIRs, will not assist the IRS in making informed decisions to continue, modify, or terminate information technology projects.

 

Appendix I

 

Detailed Objective, Scope, and Methodology

 

The overall objective of this review was to determine the adequacy of the procedures for performing PIRs to assess benefits and capabilities of new information technology projects.  This review was part of our Fiscal Year 2006 audit plan for reviews of the IRS’ BSM efforts.  To accomplish our objective, we reviewed available documentation and interviewed staff members from the Modernization and Information Technology Services organization Applications Development office and the Enterprise Services Program Performance Management office and business requirements representatives from the Wage and Investment Division Customer Account Services office.  Appendix VI presents the authorities requiring a PIR of new information technology projects, and Appendix VII presents a glossary of terms.  Specifically, we:

I.                   Determined whether the IRS had implemented key post implementation controls.

A.    Determined whether a PIR was performed for the e-Services project.

B.     Reviewed the e-Services project documentation, including the project management plan, and available e-Services project PIR plans and supporting information.

C.     Determined involvement in the e-Services project post implementation activities by the Program Performance Management office, Applications Development office, Office of Program Evaluation and Risk Analysis, and Wage and Investment Division Customer Account Services office.

II.                Reviewed PIR performance criteria issued by the IRS for compliance with Office of Management and Budget, GAO, and ELC procedures.

A.    Determined the status of the IRS corrective actions to address recommendations in performing PIRs as presented in the GAO report entitled Business Systems Modernization:  IRS’ Fiscal Year 2004 Expenditure Plan (GAO-05-46, dated November 2004).

B.     Determined the requirements for reporting PIR results to appropriate Modernization and Information Technology Services organization executives and governance boards.

C.     Determined how many modernization project release deployments and milestone exits have occurred since the November 2004 GAO report recommendation for performing PIRs.

 

Appendix II

 

Major Contributors to This Report

 

Margaret E. Begg, Assistant Inspector General for Audit (Information Systems Programs)

Gary V. Hinkle, Director

Edward A. Neuwirth, Audit Manager

Paul M. Mitchell, Senior Auditor

Glen J. Rhoades, Senior Auditor

Louis V. Zullo, Senior Auditor

 

Appendix III

 

Report Distribution List

 

Commissioner  C

Office of the Commissioner – Attn:  Chief of Staff  C

Deputy Commissioner for Operations Support  OS

Commissioner, Wage and Investment Division  SE:W

Associate Chief Information Officer, Applications Development  OS:CIO:B

Associate Chief Information Officer, Enterprise Services  OS:CIO:ES

Deputy Associate Chief Information Officer, Applications Development  OS:CIO:AD

Director, Stakeholder Management  OS:CIO:SM

Deputy Associate Chief Information Officer, Business Integration  OS:CIO:ES:BI

Deputy Associate Chief Information Officer, Systems Integration  OS:CIO:ES:SI

Chief Business Architect and Business Strategist  SE:W:CAS:BSBA

Chief Counsel  CC

National Taxpayer Advocate  TA

Director, Office of Legislative Affairs  CL:LA

Director, Office of Program Evaluation and Risk Analysis  RAS:O

Office of Internal Control  OS:CFO:CPIC:IC

Audit Liaisons:

Associate Chief Information Officer, Applications Development  OS:CIO:B

Director, Program Oversight  OS:CIO:SM:PO

 

Appendix IV

 

Outcome Measure

 

This appendix presents detailed information on the measurable impact our recommended corrective actions will have on tax administration.  This benefit will be incorporated into our Semiannual Report to Congress.

Type and Value of Outcome Measure:

·         Inefficient Use of Resources – Potential; $1,064,000 (see page 5).

Methodology Used to Measure the Reported Benefit:

In October 2004, the BSMO issued a BSMO Procedure updating PIR guidance.  This 2004 update added the requirement to perform Post Reviews of milestone completion activities.

In 2005, the Program Performance Management office performed Post Reviews of milestone completion activities on the Integrated Financial System, Modernized e-File project, and Custodial Accounting Project (post cancellation review).  The Post Reviews of milestone completion activities performed by the Program Performance Management office in 2005 required approximately 63 staff days each to complete, at a cost of approximately $28,000[11] per review, and totaled $84,000 for all 3 reviews.

To comply with current guidelines, the IRS will have to perform an additional 12 Post Reviews of milestone completion activities by September 5, 2006.  These Reviews would cost a total of approximately $336,000.  In addition, the IRS would have to perform an estimated 26 additional Post Reviews of milestone completion activities over the next 5 years for projects currently in development.  We estimate these 26 Post Reviews of milestone completion activities would cost an additional $728,000.  Further, the Information Technology Modernization Vision and Strategy shows work for 54 projects in Fiscal Years 2007 and 2008.

We recommended the IRS eliminate the requirement to perform Post Reviews of milestone completion activities after milestone exits.  Before passing each project milestone, the IRS already performs a detailed Milestone Exit Review that provides executives sufficient information to assess a project’s progress and plans.  By no longer performing Post Reviews of milestone completion activities, the IRS would potentially eliminate the inefficient use of $1,064,000.

 

Appendix V

 

Enterprise Life Cycle Overview

 

The ELC is the IRS’ standard approach to business change and information systems initiatives.  It is a collection of program and project management best practices designed to manage business change in a successful and repeatable manner.  The ELC addresses large and small projects developed internally and by contractors.

The ELC includes such requirements as:

·         Development of and conformance to an enterprise architecture.

·         Improving business processes prior to automation.

·         Use of prototyping and commercial software, where possible.

·         Obtaining early benefit by implementing solutions in multiple releases.

·         Financial justification, budgeting, and reporting of project status.

In addition, the ELC improves the IRS’ ability to manage changes to the enterprise; estimate the cost of changes; and engineer, develop, and maintain systems effectively.  Figure 1 provides an overview of the layers, paths, phases, and milestones (shown as “MS” in Figure 1) within the ELC Framework.

Figure 1:  ELC Framework

Figure 1 was removed due to its size.  To see Figure 1, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.

ELC Layers

The ELC is a framework for organizing and using IRS directives, processes, procedures, templates, and standards to accomplish business change.  It is organized as a set of six interacting layers.

·         The Management Layer specifies how to plan and control business change programs, projects, acquisitions, and solutions throughout the ELC.

·         The Governance Layer specifies additional controls imposed from outside the project or program.

·         The Solution Life Cycle Layer specifies what should be done but not how to do it.

·         The Solution Layer manages the solution as it is produced, including providing standards for consistent solution specification and formal review of solution content.  This Layer provides control over work products that may be produced by multiple internal and external developers using differing methodologies.

·         The Methodology Layer details how to do the work and specifies a unique set of work products to be produced.  Specific methodologies are not part of the ELC Framework.

·         The Specialty Areas Layer provides additional guidance for areas of particular importance within the IRS.  These areas include Enterprise Integration, Test, and Evaluation;[12] Business Rules Harvesting[13] and Management; Transition Management;[14] Enterprise Architecture; Capital Planning and Investment Control;[15] Security and Privacy; and Requirements Development and Management.

ELC Paths

A path specifies a unique “philosophy” or orientation for performing the work.  Although the ELC specifies a standard for the work required to produce and operate business change solutions, there are multiple ways to approach and accomplish the required work.  Paths are like alternate roads, each of which crosses different terrain, but all of which lead to the same destination.  The ELC provides five distinct paths or approaches to developing systems:

·         The Large Custom Path is for large projects.

·         The Small Custom Path is for small projects.

·         The Commercial-Off-the-Shelf Path is a commercial, software-based approach.

·         The Joint Application Development/Rapid Application Development Path is a highly accelerated, prototyping-based approach for very small, standalone solutions or solution components.

·         The Iterative Custom Path is a hybrid approach that combines elements of the other approaches.

ELC Phases and Milestones

A phase is a broad segment of work encompassing activities of similar scope, nature, and detail and providing a natural breakpoint in the life cycle.  Each phase begins with a kickoff meeting and ends with an executive management decision point (called a milestone) at which IRS executives make “go/no-go” decisions for continuation of a project.  Project funding decisions are often associated with milestones.

Figure 2:  ELC Phases and Milestones

Phase

General Nature

of Work

Concluding Milestone

Vision and Strategy/Enterprise Architecture Phase

High-level direction setting.  This is the only phase for enterprise planning projects.

0

Project Initiation Phase

Startup of development projects.

1

Domain Architecture Phase

Specification of the operating concept, requirements, and structure of the solution. 

2

Preliminary Design Phase

Preliminary design of all solution components.

3

Detailed Design Phase

Detailed design of solution components.

4A

System Development Phase

Coding, integration, testing, and certification of solutions.

4B

System Deployment Phase

Expanding availability of the solution to all target users.  This is usually the last phase for development projects.

5

Operations and Maintenance Phase

Ongoing management of operational systems.

System

Retirement

Source:  The ELC Guide.

 

Appendix VI

 

Authorities Requiring Post Reviews of New Information Technology Projects

 

·         Capital Asset Plan and Business Case (Office of Management and Budget Circular No. A-11 Exhibit 300, dated June 2002).

·         Chief Financial Officers Act of 1990, Pub. L. No. 101-576, 104 Stat. 2838 (codified as amended in scattered sections of 5 U.S.C., 31 U.S.C., and 42 U.S.C.).

·         Clinger-Cohen Act of 1996 (Federal Acquisition Reform Act of 1996) (Information Technology Management Reform Act of 1996), Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).

·         Conducting Post-implementation Reviews Using a Standard Methodology (Government Accountability Office Evaluation Process, dated February 2006).

·         Executive Order 13011, Sec. 2 (b) (3) Federal Information Technology.

·         Federal Acquisition Streamlining Act of 1994 Pub. L. No. 103-355 (1994).

·         Government Performance and Results Act of 1993, Pub. L. No. 103-62, 107 Stat. 285 (codified as amended in scattered sections of 5 U.S.C., 31 U.S.C., and 39 U.S.C.).

·         Investment Decision Management Post Implementation Review Procedure (Enterprise Life Cycle, dated June 2000).

·         Management of Federal Information Resources (Office of Management and Budget Circular No. A-130, dated March 2006).

·         Paperwork Reduction Act of 1995, Pub. L. No. 104-13.

·         Post Phase/Milestone and Post Release/Project Implementation Reviews (BSMO-BI-PPM-PIR Procedure, dated October 2004).

 

Appendix VII

 

Glossary of Terms

 

Term

Definition

Acquisition Project Manager

Acquisition Project Managers have overall responsibility for the technical and management aspects of projects in development, as well as changes to a contract or task order.

Baseline Business Case

A Baseline Business Case documents the business case for proceeding with a project beyond design and development and is intended to justify the need and benefits of a project to the investment decision makers.

Case for Action

A Case for Action is a documented communication providing compelling reasons to change business practices.

Concept of Operations

An organization develops a Concept of Operations to establish the desired
product-line approach it wishes to take.

Custodial Accounting Project

The Custodial Accounting Project planned to use a data warehousing approach for storing, analyzing, and reporting taxpayer account and collection information.

Customer Account Data Engine

The Customer Account Data Engine is the foundation for managing taxpayer accounts in the IRS modernization plan.  It will consist of databases and related applications that will replace the IRS’ existing Master File processing systems and will include applications for daily posting, settlement, maintenance, refund processing, and issue detection for taxpayer tax account and return data.

Document Management System

The Document Management System provides a central location and automated processes for storing, retrieving, and using Modernization and Information Technology Services organization document files.

e-Services

The e-Services project provides a set of web-based business products as incentives to third parties to increase electronic filing, in addition to providing electronic customer account management capabilities to all businesses, individuals, and other customers.

Exhibit 300

The Exhibit 300 is a Capital Asset Plan and Business Case required by the Office of Management and Budget.

Filing and Payment Compliance

The Filing and Payment Compliance project will provide support for detecting, scoring, and working nonfiler cases (filing compliance) and delinquency cases (payment compliance).

Information Technology Modernization Vision and Strategy

The Information Technology Modernization Vision and Strategy establishes a 5-year plan that drives investment decisions, addresses the priorities around modernizing front-line tax administration and supporting technical capabilities, and leverages existing systems (where possible) and new development (where necessary) to optimize capacity, manage program costs, and deliver business value on a more incremental and frequent basis.

Integrated Financial System

The Integrated Financial System is intended to address administrative financial management weaknesses.  The first release of the Integrated Financial System will include the Accounts Payable, Accounts Receivable, General Ledger, Budget Execution, Cost Management, and Financial Reporting activities.  A future Integrated Financial System release will be needed to fully resolve all administrative financial management weaknesses.

Metric

A metric is a standard of measurement.

Milestone

Milestones provide for “go/no-go” decision points in a project and are sometimes associated with funding approval to proceed.

Modernized e-File

The Modernized e-File project develops the modernized, web-based platform for filing approximately 330 IRS forms electronically, beginning with the U.S. Corporation Income Tax Return
(Form 1120), U.S. Income Tax Return for an S Corporation (Form 1120S), and Return of Organization Exempt From Income Tax (Form 990).  The project serves to streamline filing processes and reduce the costs associated with a
paper-based process.

Release

A release is a specific edition of software.

 

Appendix VIII

 

Management’s Response to the Draft Report

 

The response was removed due to its size.  To see the response, please go to the Adobe PDF version of the report on the TIGTA Public Web Page.


[1] Appendix V presents an overview of the Enterprise Life Cycle.

[2] The Program Performance Management office is currently part of the Modernization and Information Technology Service Enterprise Services organization.

[3] Business Systems Modernization:  IRS’ Fiscal Year 2004 Expenditure Plan (GAO-05-46, dated November 2004).

[4] Conducting Post-implementation Reviews Using a Standard Methodology (GAO Evaluation Process, dated February 2006).

[5] Clinger-Cohen Act of 1996 (Federal Acquisition Reform Act of 1996) (Information Technology Management Reform Act of 1996), Pub. L. No. 104-106, 110 Stat. 642 (codified in scattered sections of 5 U.S.C., 5 U.S.C. app., 10 U.S.C., 15 U.S.C., 16 U.S.C., 18 U.S.C., 22 U.S.C., 28 U.S.C., 29 U.S.C., 31 U.S.C., 38 U.S.C., 40 U.S.C., 41 U.S.C., 42 U.S.C., 44 U.S.C., 49 U.S.C., 50 U.S.C.).

[6] Management of Federal Information Resources (Office of Management and Budget Circular No. A-130, dated March 2006).

[7] See Appendix V for an overview of the ELC.

[8] Business Systems Modernization:  IRS’ Fiscal Year 2004 Expenditure Plan (GAO-05-46, dated November 2004).

[9] The Program Performance Management office is currently part of the Modernization and Information Technology Services Enterprise Services organization.

[10] Conducting Post-implementation Reviews Using a Standard Methodology (GAO Evaluation Process, dated February 2006).

[11] Review costs were calculated using the Andover, Massachusetts, Submission Processing Site’s Cost/Savings Analysis Calculator dated November 4, 2005.

[12] Enterprise Integration, Test, and Evaluation includes processes for integrating multiple components of a solution and conducting various types and levels of testing on the solution.

[13] A business rule is a statement that defines or constrains some aspect of the business.  Harvesting is a general term used to broadly describe the entire set of activities involved in gathering, formalizing, analyzing, and validating business rules for a particular scope.

[14] Transition Management helps ensure personnel and organizations are prepared to receive, use, operate, and maintain the business processes and technology provided by business change solutions.

[15] The Capital Planning Investment and Control process manages a central portfolio of information technology investments across the IRS.