Embedded Quality Is an Effective Measure of Field Collection Program Work, but Improvements Could Be Made
August 13, 2009
Reference Number: 2009-30-104
This report has cleared the Treasury Inspector General for Tax Administration disclosure review process and information determined to be restricted from public release has been redacted from this document.
Phone Number |
202-622-6500
Email Address | inquiries@tigta.treas.gov
Web Site |
http://www.tigta.gov
August 13, 2009
MEMORANDUM FOR COMMISSIONER, SMALL BUSINESS/SELF-EMPLOYED DIVISION
FROM: Michael R. Phillips /s/ Michael R. Phillips
Deputy Inspector General for Audit
SUBJECT: Final Audit Report – Embedded Quality Is an Effective Measure of Field Collection Program Work, but Improvements Could Be Made (Audit # 200830020)
This report presents the results of our review to determine whether the Embedded Quality[1] program in the Collection function is an
effective measure of the quality of the function’s work and how management uses
the results to assess program performance. This review was included in our Fiscal Year
2008 Annual Audit Plan under the major management challenge of Tax Compliance
Initiatives.
Impact on the Taxpayer
Embedded Quality measures employee performance on casework and identifies areas for organizational and individual improvement. Overall, Embedded Quality is an effective measure of the quality of Field Collection program work. Although group managers rated many of the attributes accurately in the employee performance review system of Embedded Quality, our review determined that 64 (10 percent) of 651 quality attributes tested were rated inaccurately. Without appropriate employee performance feedback, errors in collection procedures could continue to occur and affect taxpayer rights and payment of tax liabilities.
Synopsis
Recent Collection function operational priorities emphasize improving customer service to taxpayers by conducting continuous reviews of quality and ensuring consistency and accuracy of review systems. Embedded Quality consists of two distinct systems: the National Quality Review System (NQRS) for organizational case quality and the Embedded Quality Review System (EQRS) for employee case performance. Both systems rate how well employees are performing their work on cases by evaluating various attributes which are concise statements of expectations for quality on Collection function cases. Both systems rate most of the same attributes.
Our review of Embedded Quality determined that it is an overall effective measure of the quality of Field Collection program (Collection Field function and Offer-in-Compromise program) work. In Fiscal Year 2008, the attributes were adequately aligned with Collection function priorities, Collection function management identified deficiencies and implemented improvement strategies, and NQRS results were statistically valid. Our review of 93 NQRS cases showed that NQRS reviewers accurately rated 97 percent of attributes tested. In addition, Collection function management analyzed and used the quality results regularly to assess the program and help identify problems that required changes in such areas as collection procedures and training.
Our review of 93 EQRS cases for which we reviewed 651 attributes showed that group managers rated 587 (90 percent) of the attributes accurately. However, group managers incorrectly rated 64 (10 percent) of the 651 attributes. Incorrectly applying attributes could affect the accuracy of quality results and individual employee performance evaluations. In addition, when deficiencies are not identified, managers may not take proper corrective action to improve them. We determined that 1) frequent revisions to the guidance used by reviewers and 2) managers not performing all required consistency reviews may have impacted the accuracy of reviews.
Internal Revenue Manual guidelines prohibit NQRS data from being used to evaluate individual employees or used as a substitute for first-line managerial review responsibilities. However, individual NQRS case results are being provided to Territory managers, who share the results with group managers and staff. Sharing NQRS Data Collection Instrument information with group managers and individual employees makes it possible for performance feedback to be given based on the results. Although we did not identify this happening, it could be considered a type of performance feedback, even if only by appearance.
Management informed us that the information is used only as an objective, educational tool at group meetings; however, some managers had shared NQRS data with employees individually. Further, if EQRS attributes are rated accurately, they should provide similar results as the NQRS and be the basis for individual case examples, rather than being used in conjunction with specific case results from the NQRS. The Internal Revenue Service (IRS) Labor Relations function advised Collection function management that they should discontinue this practice so the information cannot be misused. In addition, we noted that this practice is not consistent within the Small Business/Self-Employed Division. Small Business/Self-Employed Division Examination function management requested advice on this same issue from the IRS Labor Relations function and followed their advice to not request NQRS Data Collection Instrument information.
Recommendations
We recommended that the Director, Collection, Small Business/Self-Employed Division, study alternative methods to ensure accuracy and consistency in EQRS reviews, such as requiring Territory managers, on a regular basis, to review a sample of EQRS case reviews performed by group managers for some revenue officers within each group. In addition, the Commissioner, Small Business/Self-Employed Division, should assess whether the risk is warranted for the process of sharing NQRS specific case information with group managers and set a consistent policy within the Division.
Response
IRS management agreed with our recommendations. The Director, Collection, will update the Territory Manager guidelines to include a review of a sample of cases previously reviewed by the group manager during operational reviews. The review data will be captured in the EQRS, allowing systemic comparison of case review attribute ratings. This change, coupled with existing expectations that are included in the Territory Manager Operational Review Template, will contribute to continued improvement in rating accuracy and consistency. In addition, policy has been established that individual NQRS case review information will not be shared with field personnel including managers and front-line employees. The Director, Collection has discontinued the practice of sharing NQRS individual case review data (Data Collection Instrument) with the field function for both the Collection Field function and the Offer-in-Compromise program. Management’s complete response to the draft report is included as Appendix VI.
Copies of this report are also being sent to
the IRS managers affected by the report recommendations. Please contact me at (202) 622-6510 if
you have questions or Margaret E. Begg, Assistant Inspector General for Audit (Compliance
and Enforcement Operations), at (202) 622-8510.
Embedded Quality Is an
Overall Effective Measure of the Quality of Field Collection Program Work
Managers Rated Most
Attributes Accurately, but We Identified Some Inaccuracies
National Quality
Review System Data Are Being Shared in a Different Way Than Intended
Appendices
Appendix
I – Detailed Objective, Scope, and Methodology
Appendix
II – Major Contributors to This Report
Appendix
III – Report Distribution List
Appendix IV
– Glossary of Terms
Appendix
V – Attributes Included in the Case Review
Appendix
VI – Management’s Response to the Draft Report
Abbreviations
|
CFf |
Collection Field function |
|
EQ |
Embedded Quality |
|
EQRS |
Embedded Quality Review System |
|
FY |
Fiscal Year |
|
IRM |
Internal Revenue Manual |
|
IRS |
Internal Revenue Service |
|
NQRS |
National Quality Review System |
|
OIC |
Offer-in-Compromise |
|
SB/SE |
Small Business/Self-Employed |
Embedded Quality (EQ)[2] is a quality review program that allows Internal Revenue Service (IRS) Small Business/Self-Employed (SB/SE) Division management to measure employee performance on casework and identify areas for organizational and individual improvement. The data from the EQ are used to evaluate case quality and determine the effectiveness of its programs. The SB/SE Division Collection function uses the EQ in its Field Collection program including the Collection Field function (CFf) and Offer-in-Compromise (OIC) program.
The SB/SE Division Strategic Plan
and Collection function’s strategies and operational priorities for Fiscal Year
(FY) 2008 reflected the EQ and its significance in affecting casework
quality. Strategies emphasized improving
customer service to taxpayers by conducting continuous reviews of quality and
ensuring consistency and accuracy of review systems. In FY 2009, Collection function
management continued to emphasize quality.
The EQ is comprised of two distinct systems: the National Quality Review System (NQRS) to measure organizational case quality and the Embedded Quality Review System (EQRS) to measure employee case performance. Reviewers in both programs evaluate collection cases using various attributes. Attributes are concise statements of expectations for quality relating to appropriate and timely case activity, and are based on Internal Revenue Manual (IRM) guidelines for CFf and OIC program casework. Both systems rate most of the same attributes. The attributes are grouped into five key categories that measure customer accuracy, regulatory accuracy, procedural accuracy, professionalism, and timeliness.
The CFf and OIC program have their own group of NQRS reviewers that independently rate case actions on closed cases based on the attributes. Collection function managers can analyze the results of NQRS reviews at various organizational levels such as national, Area Office, and Territory. NQRS results are statistically valid to the Area Office level on a quarterly basis.
The EQRS replaced various paper-based review systems in April 2007. Group managers rate individual employee performance on case actions in the EQRS and use the results to generate feedback and documentation for performance reviews. The attributes are linked to performance standards to allow managers to provide employees with specific examples of what they need to do to sustain or improve their performance. Managers choose from a list of reasons and/or explain in narrative form why attributes were met or not met.
This review was performed at the SB/SE Division Collection function
offices in
Embedded Quality Is an Overall Effective Measure of the Quality of Field Collection Program Work
To assess CFf and
OIC program work quality, NQRS reviewers evaluate actions on closed collection
cases. NQRS reviewers are required to
rate 30 CFf or 24 OIC program attributes for each case and choose from a list
of reasons and/or explain in a narrative why a specific attribute was not
met. It is important that reviewers rate
attributes accurately to produce reliable results.
The EQ is effective for measuring quality because
samples are statistically valid, NQRS reviewers rate attributes accurately, and
management uses the results to assess the program and make improvements.
To determine if NQRS
reviewers were rating cases accurately, we reviewed 93 NQRS cases (56 CFf and 37
OIC).[3] Our review
of 8 CFf and 5 OIC attributes[4] from each
case showed that, based on the criteria for rating the attributes, NQRS
reviewers in both the CFf and OIC program accurately rated 97 percent of the
attributes tested.
IRM guidelines state
that NQRS results are
statistically valid and reliable measurements of the overall quality of
casework completed by the CFf and OIC program at the Area Office level. The Collection Planning and Analysis Office
is required to update sample plans annually.
Our analysis of Area Office populations and recalculation of sample
sizes for FY 2008 showed that sampling plans were valid and an adequate number of
NQRS cases were reviewed.
All levels of Collection function management use EQ review findings to help identify problems that require changes in such areas as collection procedures and training. In the three field offices visited, our interviews of all levels of Collection function management showed that they analyzed EQ results regularly to assess the program and used the results to create and implement plans for improvement. For example:
Managers Rated Most Attributes Accurately, but We Identified Some Inaccuracies
Although managers accurately rated many attributes based on the criteria, our review of 93 cases involving 651 attributes showed that 10 percent of the attributes were rated incorrectly.
The EQRS is a tool used by managers to rate how well
employees are taking actions on their casework by evaluating certain case
attributes. The case attributes, which
are linked to employee performance standards, are based on IRM guidelines for
working CFf and OIC program cases. Group
managers use an Attribute Job Aid, along with the IRM, when performing EQRS
reviews. Although these scores are not statistically valid to
measure overall quality, the EQRS provides a structured basis for evaluating
employee performance in relation to attribute scores and providing appropriate
feedback. Managers provide results to
employees via the EQRS Employee Individual Feedback Report.
Our review of 93 EQRS cases (62 CFf and 31 OIC cases) in 3 Area Offices during FY 2008 showed that improvements are needed to ensure more accurate results. We reviewed 8 of the 32 CFf attributes and 5 of the 26 OIC program attributes in the EQRS.
Managers rated 587 (90 percent) of the 651 attributes in the 93 cases accurately according to the criteria; however, our review showed that managers incorrectly rated 64 (10 percent) of 651 attributes. In approximately half the cases we reviewed, there was at least one error. Figure 1 shows the number of cases, total attributes reviewed, and the number of errors identified for both CFf and OIC program cases.
Figure 1: Analysis of EQRS Cases
|
Collection Program |
Number of Cases Reviewed |
Number
of Cases With Errors |
Percent
|
Number of Attributes Reviewed |
Errors |
Percent of Attribute Errors |
|
CFf |
62 |
30 |
48% |
496 |
48 |
10% |
|
OIC |
31 |
13 |
42% |
155 |
16 |
10% |
|
Total |
93 |
43 |
46% |
651 |
64 |
10% |
Source: Treasury Inspector
General for Tax Administration case review results.[5]
Although Figure 1 shows that we identified rating errors in 10 percent of the total attributes reviewed, 55 (86 percent) of the 64 errors favored the employee. Specifically:
Incorrectly applying the attributes in EQRS case reviews could affect the accuracy of quality results and employee performance reviews. In addition, when deficiencies are not identified, managers may not take proper corrective action to improve them. Without appropriate employee performance feedback, errors in collection procedures could continue to occur and affect taxpayer rights and payment of tax liabilities.
One contributing factor to the errors may be that consistency reviews were not being conducted as required by the IRM, which states that Territory managers will schedule and conduct EQRS consistency reviews with group managers on a quarterly basis. Consistency reviews involve all group managers within a Territory reviewing the same case and comparing and discussing attribute results on a quarterly basis. This is useful to help achieve consistent understanding and accurate scoring of attributes. The Territory managers we interviewed did not perform the required consistency reviews of one per quarter with their group managers. During FY 2008, most of the Territory managers had performed only one consistency review and participated with their group managers in one national consistency review.
Another possible cause of rating errors is that in an effort to improve the EQ system, Collection function management has made continual changes in the attribute definitions since the EQRS was implemented in April 2007. Specifically, there have been four CFf and two OIC program major Attribute Job Aid revisions since the implementation. The managers we interviewed indicated that it is sometimes confusing and burdensome for employees, managers, and NQRS reviewers to keep up with learning new attribute rating guidance and that if one attribute is missed, the next few may also be missed if they are closely related. Until attribute definitions and Attribute Job Aids are firmly established, it will be more difficult for reviewers to understand the definitions and consistently rate the attributes.
Recommendation
Recommendation 1: The Director, Collection, SB/SE Division, should study alternative methods to ensure accuracy and consistency in EQRS reviews, such as requiring Territory managers, on a regular basis, to review a sample of EQRS case reviews performed by group managers for some revenue officers within each group.
Management’s Response: IRS management agreed with this
recommendation. The Director, Collection,
will update the Territory Manager guidelines to include a review of a sample of
cases previously reviewed by the group manager during operational reviews. The review data will be captured in the EQRS allowing
systemic comparison of case review attribute ratings. This change, coupled with existing
expectations that are included in the Territory Manager Operational Review
Template, will contribute to continued improvement in rating accuracy and
consistency.
National Quality Review System Data Are Being Shared In a Different Way Than Intended
Providing
individual case results from the NQRS to group managers and employees is not
using NQRS results as intended.
Prior to the EQ, the Collection Quality Management System was used to rate only organizational quality based on closed cases. The EQ was created to allow for two separate but related systems: the NQRS for organizational case quality and the EQRS for employee case performance. Both systems rate most of the same attributes, and the EQRS is also linked to employee performance standards. The idea is that as employee performance improves by following the same attributes, overall NQRS scores would also improve. IRM guidelines state that NQRS data are to be used as measures of organizational quality and not to evaluate individual employees, and the NQRS is not a substitute for first-line managerial review responsibilities.
In FY 2008, Collection function management implemented a process to provide individual NQRS reports (known as Data Collection Instruments) to Territory managers on a regular basis. The results can then be shared with group managers and their employees. The process began in response to requests from Field Collection program managers who wanted an independent, third-party review or backup for EQRS attribute ratings and NQRS Area Office results. Our discussions with Field Collection program managers showed that some agreed that reviewing employee casework in the EQRS provides more specific results for feedback and more objectivity than the prior paper-based system. However, some managers explained that the transition has not been easy for some employees, and the employees do not always agree with managers’ ratings on specific attributes. In addition, when NQRS Area Office scores show downward trends, group managers do not always think their groups are responsible for the downward trends. Collection function management informed us that the Data Collection Instrument is used only as an objective, educational tool at group meetings and is not usually shared with individual employees. However, some of the managers informed us that they have discussed results with individual employees.
Sharing the NQRS Data Collection Instrument information with group managers and individual employees makes it possible for performance feedback to be given based on the results. Although we did not identify this happening, it could be considered a type of performance feedback, even if only by appearance. The sharing of NQRS information with groups and individual employees suggests that the EQRS process may not be providing managers with sufficient information to effectively provide feedback to employees. If EQRS attributes are rated accurately and consistently, they should provide similar results to the NQRS at the individual level and be the basis for individual case examples, rather than being used in conjunction with specific case results from the NQRS. Further, NQRS data are not statistically valid at the group level and should not be considered representative of the group’s performance.
An agreement between the National Treasury Employees Union and the SB/SE Division Field Compliance function states that NQRS data will not be used as individual employee performance feedback. IRS management made assertions during negotiations with the National Treasury Employee Union that NQRS data would not be shared with first-line supervisors and there was supposed to be no direct link to individual employees or groups.
The IRS Labor Relations function has advised Collection function management that they should discontinue this practice so the information cannot be misused. Further, we noted that this practice is not consistent within the SB/SE Division. SB/SE Division Examination function management requested advice on this same issue from the Labor Relations function. Unlike Collection function management, Examination function management followed the Labor Relations function advice to not request NQRS Data Collection Instrument information.
Recommendation
Recommendation 2: The Commissioner, SB/SE Division, should assess whether the risk is warranted for the process of sharing NQRS specific case information (through the Data Collection Instrument) with group managers and set a consistent policy within the SB/SE Division.
Management’s Response: IRS management agreed with this recommendation. Policy has been established that individual NQRS case review information will not be shared with field personnel including managers and front-line employees. The Director, Collection, has discontinued the practice of sharing NQRS individual case review data (Data Collection Instrument) with the field function for both the CFf and OIC program.
Appendix I
Detailed Objective, Scope, and Methodology
The overall
objective of this review was to determine whether the EQ[6] program in the Collection function is an
effective measure of the quality of the function’s work and how management uses
the results to assess program performance. To accomplish our objective, we:
I.
Determined the effectiveness of EQ program controls,
procedures, and sampling process.
A.
Identified
NQRS and EQRS IRM procedures and key controls for the CFf and OIC program.
B.
Compared NQRS/EQRS quality attributes with CFf and OIC program
management priorities such as taxpayer rights, timeliness of work, and
professionalism to determine if they are adequate. We also compared the
EQRS quality attributes to employee performance standards to determine
adequacy.
C.
Determined whether
the number of cases reviewed by NQRS reviewers in the CFf and OIC program were
statistically valid at the Area Office level during FY 2008. We duplicated sample plan populations by
pulling the first 3 quarters of FY 2007 closed cases from Integrated Collection
System archived records and Automated OIC records using the Treasury Inspector
General for Tax Administration Data Center Warehouse. We relied on the Data Center Warehouse
validations of IRS data.
II.
Determined whether reviewers accurately assessed and scored
quality attributes.
A.
Obtained
a judgmental sample of 93 NQRS closed cases (56 CFf and 37 OIC program cases
from all Area Offices nationwide) at the
B.
Obtained
a download from IRS management of closed CFf cases (population of 721 CFf and
292 OIC cases) from 3 Area Offices reviewed by group managers in the EQRS between
April 1, 2008, and June 27, 2008, and closed OIC cases reviewed by group
managers in the EQRS between October 5, 2007, and June 30, 2008. We reviewed a judgmental sample of 8 CFf and 5
OIC program attributes for a judgmental sample of 93 EQRS (62 CFf and 31 OIC
program) cases. Due to the large number
of EQRS reviews, we used judgmental sampling for both attributes and cases because
of limited resources and time available to complete the audit in a reasonable
time period. Also, we did not anticipate
being able to project the results. We relied
on the data provided by the IRS and only validated the data by ensuring the
reviews were within our time period and the dates matched the dates on the
actual Employee Individual Feedback Report.
III.
Determined how EQ results are used to assess the Collection
function program.
A.
Analyzed
NQRS and EQRS reports to determine if any trends or common variables are
present for the 9 quarters beginning October 1, 2006, through December 31,
2008. We relied on the data provided by
the IRS.
B.
Identified
the top 10 NQRS and EQRS defects and successes for the same period as stated in
Step III.A. and compared past top 10 lists to determine the progress of
particular attribute ratings. We relied
on the data provided by the IRS.
C.
Interviewed
various levels of Collection function managers to ascertain:
1.
Management
expectations and responsibilities for the EQ.
2.
Reviewer
responsibilities, qualifications, and training requirements.
3.
Consistency
review requirements.
4.
How
EQRS cases are selected for review.
5.
How NQRS/EQRS
results are used in the field offices.
6.
How management
analyzed results, identified trends, disseminated information, and created and
implemented improvement plans.
Appendix II
Major Contributors to This Report
Margaret
E. Begg, Assistant Inspector General for Audit (Compliance and Enforcement Operations)
Carl
Aley, Director
Lynn
Wofchuck, Audit Manager
Phyllis
Heald London, Lead Auditor
Janis
Zuika, Senior Auditor
Frank
O’Connor, Program Analyst
Matthew
Schimmel, Program Analyst
Appendix III
Commissioner
C
Office of the Commissioner – Attn: Chief of Staff C
Deputy Commissioner for Services and
Enforcement SE
Deputy Commissioner, Small Business/Self-Employed Division SE:S
Director, Collection, Small
Business/Self-Employed Division SE:S:CCS
Chief Counsel CC
National Taxpayer Advocate TA
Director, Office of Legislative Affairs CL:LA
Director, Office of Program Evaluation and Risk Analysis RAS:O
Office of Internal Control OS:CFO:CPIC:IC
Audit Liaison: Commissioner, Small Business/Self-Employed Division SE:S
Appendix IV
Term
|
Definition
|
|
Area
Office |
A geographic organizational level used by IRS business units and
offices to help their specific types of taxpayers understand and comply with
tax laws and issues. Area Offices
consist of Territory offices which consist of groups. |
|
Attributes |
Concise statements of expectations for quality relating to appropriate and timely case action on Collection function cases. |
|
Attribute
Job Aid |
A reference tool providing support for NQRS and EQRS
reviewers in rating EQ attributes in a uniform and consistent manner. |
|
Campus |
The data
processing arm of the IRS. The
campuses process paper and electronic submissions, correct errors, and
forward data to the Computing Centers for analysis and posting to taxpayer
accounts. |
|
Collection
Field Function |
The unit in
the Area Offices consisting of revenue officers who handle personal contacts
with taxpayers to collect delinquent accounts or secure unfiled returns. |
|
Collection
Planning and Analysis Office |
The office that oversees the strategic and business
planning process for the Field Collection program in the SB/SE Division. |
|
Consistency
Reviews |
Reviews that involve all group managers with a Territory
reviewing the same case and comparing and discussing attribute results on a
quarterly basis. |
|
|
A centralized storage and administration of files that
provides data and access services to IRS data. |
|
Data
Collection Instrument |
The tool used by
reviewers, both managers and quality reviewers, to input item-by-item quality
review results. |
|
Embedded
Quality |
The program comprised
of two distinct systems: the NQRS for
organizational casework quality and the EQRS for employee casework
performance. Reviewers use this
program to evaluate quality of the actions taken on collection cases. |
|
Embedded
Quality Review System |
A web-based system used by managers to rate employee collection
case actions against EQRS attributes. |
|
Employee Individual
Feedback Report |
The report that provides a printed
record of manager ratings and narrative comments in which the employee’s
performance, in relation to specific EQRS attributes, is summarized. |
|
Field Collection Program |
A program
that includes the CFf and OIC program in the SB/SE Division (for the purposes
of this audit). |
|
The IRS function that
includes the Field Collection and Field Examination programs in the SB/SE
Division. |
|
|
An automated system used to control and monitor delinquent cases
assigned to revenue officers in the field offices. |
|
|
The
IRS function that provides guidance and representation to IRS management in
areas such as collective bargaining and interactions and contractual
obligations with the National Treasury Employees Union. |
|
|
National
Treasury Employees |
The Federal employee union representing IRS employees. |
|
National
Quality Review System |
A web-based system used by national reviewers to rate employee
collection case actions against NQRS attributes. |
|
Offer-in-Compromise
|
An agreement between a taxpayer and the Government that
settles a tax liability for payment of less than the full amount owed. |
Appendix V
Attributes Included in the Case Review
Collection Field
function
202 - Compliance on Initial Contact
203 - Request/Secure Financial Information
204 - Trust Fund Recovery Penalty Process
410 - Lien Determination/Filing
505 - Timely Employee Actions
506 - Timely Closing Actions
600 - Clear Action Dates
602 - Advised of Consequences
Offer-in-Compromise
Program
106 - Initial Offer Actions
504 - Timely Followup Actions
505 - Timely Employee Actions
602 - Advised of Consequences
703 - Closing Reports and Supporting Documents
Appendix VI
Management’s Response to the Draft Report
The response was
removed due to its size. To see the
response, please go to the Adobe PDF version of the report on the TIGTA Public
Web Page.
[1] See Appendix IV for a glossary of terms.
[2] See Appendix IV for a glossary of terms.
[3] Cases were reviewed by NQRS reviewers between July 26, 2008, and October 16, 2008, for the CFf and between August 18, 2008, and October 3, 2008, for the OIC program.
[4] See Appendix V for the attributes reviewed.
[5] Our analysis consisted of closed cases reviewed in the EQRS between April 1, 2008, and June 27, 2008, for the CFf and between October 5, 2007, and June 30, 2008, for the OIC program.
[6] See Appendix IV for a glossary of terms.
[7] See Appendix V for the attributes reviewed.