- 1 -

An Exploratory Investigation of the Determinants and

Ratings Implications of Performance Appraisal Plan

Characteristics

Christopher S. Armstrong

University of Pennsylvania

Christopher D. Ittner

University of Pennsylvania

David F. Larcker

Stanford University

April 17, 2014

Abstract: Performance appraisal is one of the cornerstones of management control

systems. Although this topic has been the subject of considerable prior

research, most of this work is based on a single observation per firm or

performance appraisal practices within a single organization. In contrast,

this study examines the design of performance appraisal systems using

detailed proprietary data of the actual performance goals and the extent to

which these goals are aligned among firm employees for 408,816

employees in 153 distinct firms. These novel data are analyzed using a

two-step hierarchical approach that allows the contextual firm-level

attributes to moderate the relations between the user characteristics and

appraisal plan design attributes.

We thank Erik Berggren and Michael Strezo for their assistance on this study. The research support of EY (Ittner),

and the Winnick Family Fund and Stanford Rock Center for Corporate Governance (Larcker) is gratefully

acknowledged.

- 2 -

I. Introduction

Performance appraisal is one of the cornerstones of management control systems. Surveys

indicate that more than 90 percent of U.S. firms have formal performance appraisal processes

covering some or all of their employees (WorldatWork and Sibson Consulting, 2010; Bruce,

2013). These appraisals can serve both administrative purposes (e.g., assigning tasks, distributing

rewards, and making promotions) and developmental purposes (e.g., identifying employee

strengths and weaknesses and assessing training needs). When designing performance appraisal

systems to achieve these ends, management must make a number of choices regarding the

number and types of goals to assign to employees, the extent of formal cascading and alignment

of goals throughout the organization, the degree of variation in appraisal plan characteristics

across individuals and employee groups, and the format of performance ratings (e.g., aggregated

vs. disaggregated), all of which may influence the distribution (i.e., leniency and

discriminability) of the resulting performance ratings.

Although performance appraisal has been the subject of considerable research, literature

reviews highlight the scarcity of studies on the determinants of performance appraisal practices.

Accounting and economics researchers have primarily focused on the factors influencing the

choice of performance measures (e.g., financial vs. nonfinancial, objective vs. subjective) for

performance evaluation or compensation purposes (Prendergast 1999, Bol 2008, Chenhall 2006),

generally ignoring the determinants of other performance appraisal plan characteristics. The

organizational psychology literature has emphasized the psychometric properties of appraisal

instruments and the effects of supervisor-subordinate interactions on performance appraisals,

with little attention paid to the determinants of appraisal practices (Murphy and Cleveland 1995,

Levy and Williams 2004). Even when studies examine the determinants of certain appraisal-

- 3 -

related practices, they do not take into account the multi-level nature of these choices.

Individuals’ appraisal plan characteristics are likely to be influenced both by the organizational

context of the firm and by the employees’ specific employment characteristics (Murphy and

Cleveland, 1995). Moreover, the higher-level organizational factors (e.g., organizational

structure) may moderate the relations between an individual employee’s attributes (e.g., position

in the hierarchy) and appraisal plan characteristics (DeNisi, 2000). Consequently, studies that

examine cross-sectional samples with a single observation per firm or performance appraisal

practices within a single organization provide incomplete pictures of the determinants of

performance appraisal practices.

We extend the performance appraisal literature by providing exploratory evidence on the

relationships between employee- and firm-level variables and a variety of performance appraisal

plan characteristics, as well as on the relation between these characteristics and performance

ratings distributions. We conduct these analyses using performance appraisal data from the users

of a leading performance management software package. The data cover 408,816 employees

from 153 firms (median = 1,193 employees per firm). The availability of actual usage data

overcomes the difficulties of accurately assessing appraisal practices across multiple individuals,

organizational levels, and companies using other research methods such as surveys or public

disclosures (Bretz et al., 1992). In addition, our sample’s use of a common software package

alleviates concerns about differences in technological capabilities that may influence

performance appraisal practices (Farr, Fairchild, and Cassidy, 2014).

We examine four primary attributes of annual appraisal plans: the number of goals per

employee, the number of appraisal plans per employee, the extent to which these goals are

explicitly cascaded and aligned with the goals of others in the firm, and the goal rating format

- 4 -

(e.g., providing individual ratings for each performance goal versus providing only a single,

overall rating).1 We investigate the extent to which these attributes vary with differences in

firms’ operating environments and organizational structures, as well as with the individual

employee’s position (executive, manager, non-manager), relative organizational level, number of

subordinates, tenure, and gender.

Our initial firm-level analyses find positive associations between the average number of

annual goals per employee and the firm’s size, number of departments, percentage of employees

evaluated by the appraisal system, and percentage of system users who are managers. Explicit

goal alignment declines with the number of divisions, while taller organizational hierarchies are

associated with more plans per employee and more frequent use of ratings for specific goals.

Greater market volatility is associated with fewer plans per employee, less goal alignment, and

fewer rated goals. We find little or no relation between firm-level appraisal measures and a

firm’s competitive environment, growth, or other characteristics of the system users.

One limitation of our firm-level analyses is that they ignore variation in appraisal plan

characteristics within the firms. We therefore estimate a series of firm-specific models in which

the individual employee is the unit of analysis. We find that many of the employee

characteristics that were insignificant in the aggregated firm-level test, including the employee’s

gender, relative level in the firm, tenure, executive position, and number of reports, are

significantly related to within-firm variation in many of the appraisal choices. In particular,

executives and other employees who are higher in the hierarchy generally receive less complex

appraisal plans, while employees with longer tenure generally receive more complex plans.

1 Employees can participate in more than one appraisal plan during the year. For example, employees in one firm are

evaluated and rewarded on a quarterly basis for performance relative to objective goals, and also participate in an

annual plan that evaluates them relative to more subjective goals. Lower-level employees in another firm have both

individual-level and team-level appraisal plans. See Gibbs et al. (2009) for an analysis of automobile dealership

employees’ participation in multiple annual incentive plans.

- 5 -

However, despite these central tendencies, the significant firm-specific coefficients linking a

given appraisal practice to an individual employee attribute are positive in some firms and

negative in others, indicating that firms adopt very different approaches when tailoring

performance appraisal plans to employee characteristics.

We take the observed within- and across-firm variations in performance appraisal plan

choices into account using two-step hierarchical estimation. We first estimate firm-specific

regressions of individuals’ appraisal plan features on their user characteristics. The coefficients

from the first stage serve as the dependent variables in the second stage, with the aggregated

firm-level variables as predictors. This nested approach allows us to examine whether the

contextual firm-level attributes moderate the relations between the user’s characteristics and his

or her appraisal plan attributes. We find that many of the firm-level variables are associated with

the extent to which firms tailor plan characteristics to individual employee attributes. The firm-

level predictors’ explanatory power is greatest in models examining variations in appraisal

characteristics with the respect to non-executive employee’s relative level in the hierarchy.

Among the most consistent firm-level moderators are firm size, which tends to increase appraisal

plan complexity (e.g., number of goals and plans, goal alignment, and the provision of rated

goals) for executives and other employees higher in the hierarchy, managers with more reports,

more tenured employees, and men; divisionalization, which is related to more complex appraisal

practices for these groups; operating uncertainty, which is associated with fewer goals and plans

for executives, males, more tenured workers, and employees with more subordinates receive; and

past accounting returns, particularly with respect to the appraisal plans for more tenured

employees and those with more reports.

- 6 -

Finally, we examine the association between appraisal plan characteristics and

performance ratings distributions. We find no evidence that our appraisal plan measures

influence either mean or median firm-level ratings. However, firms that give more goals exhibit

greater positive skewness in ratings (a proxy for leniency bias) and larger standard deviations in

ratings differences between supervisors and their subordinates (i.e., less contagion bias).

Employee participation in more plans is associated with lower leniency bias. In contrast,

providing ratings for individual goals, which is more common for objective criteria, increases

discriminability biases, but is negatively related to ratings skew. Greater explicit goal alignment

is also related to lower ratings discriminability, suggesting that explicitly cascading and aligning

goals can induce discriminability biases by making it more difficult to differentiate performance

using inter-related goals.

Our study makes several contributions to the performance appraisal, performance

evaluation, and management control literatures. First, the detailed, individual-level data from a

diverse set of firms allows us to provide a rich description of performance appraisal choices, and

to provide evidence on the extent to which firms have adopted specific practices, such as the use

and explicit cascading of multiple performance goals, that are increasingly recommended by

academics and practitioners. Second, our analyses provide some of the first comprehensive

evidence on the determinants of a broad set of appraisal plan characteristics. In doing so, we

respond to calls for greater focus on the factors that drive performance appraisal choices (Levy

and Williams 2004, Brown and Heywood 2005). Finally, our tests highlight the importance of

taking the nesting of individual-level factors within firm-level factors into account when

examining management control system choices. Whereas prior studies have typically employed

cross-sectional samples with a single observation per firm, or performance appraisal practices

- 7 -

within a single organization, our results indicate that future studies will need to take both across-

and within-firm variation into account if we are to get a better understanding of the choice and

implications of management control practices.

The remainder of the paper is organized as follows. The next section reviews related

literature on the tradeoffs in appraisal plan choices and their potential determinants. Sections 3

and 4 discuss our sample and variables, respectively. Section 5 presents our results, followed by

our conclusions in Section 6.

2. Literature Review

2.1 TRADEOFFS IN APPRAISAL PLAN CHOICES

Firms face a number of tradeoffs when selecting appraisal plan characteristics. For

example, economic theories indicate that performance evaluations should include any costless

performance measure that provides incremental information on an employee’s actions (e.g.,

Holmstrom 1979). Incorporating multiple measures and goals in performance appraisal plans can

promote goal congruence by directing effort toward the multiple actions needed to achieve the

organization’s objectives (Kernan and Lord 1990, Feltham and Xie 1994, Kaplan and Norton

1996). However, experimental research indicates that having too many goals can cause

information overload on the part of both the employee and the evaluator, increase goal conflicts

and job-related tension, and lead employees and evaluators to focus on a small subset of the

measures (Shah et al. 2002, Emsley 2003, Locke 2004, Cheng et al. 2007, Luft et al. 2010).

Similarly, employee participation in more than one appraisal plan in a year (e.g., quarterly plans

or both individual and group appraisal plans) allows firms to change goals more rapidly, to focus

efforts on objectives with different criteria, units of analysis, and timing, or to use one plan to

- 8 -

compensate for shortcomings in another. But multiple plans can also reduce performance by

providing conflicting goals and promoting overly diffuse efforts (Rubenfeld and David 2006).

The extent of formal “cascading” and alignment of goals throughout the organization is

another important appraisal plan choice. As illustrated in Figure 1, formal goal cascading and

alignment involves decomposing the organization’s overall goals into a series of smaller goals

that each unit or department must achieve for the overall goals to be reached. These smaller

goals are then broken down further until each employee has his or her own performance goals.

According to proponents of explicit goal alignment, formally cascading goals and

communicating goal linkages in this manner builds consensus, aligns employees throughout the

organization around the same strategic objectives, and makes individual employees’ goals more

meaningful by linking them to the organization’s overall mission and strategy (Austin and Bobko

1985, Kaplan and Norton, 2006). In contrast, others argue that a formal, top-down goal

cascading and alignment process can actually be detrimental due to the substantial time and

difficulty in directly connecting each individual’s goals to the firm’s overall goals, the lack of

flexibility and employee participation in this top-down goal-setting process, the possibility that

the cascading process results in goals being fabricated or created specifically for the purpose of

cascading, and the availability of more informal and simpler methods for communicating and

aligning actions (Pulakos and O’Leary 2011, Skapinker, 2012).

Firms must also decide on the performance rating format. Organizations can provide

ratings for some or all of the employees’ individual performance objectives, or can provide only

overall performance ratings. Providing employees with clear, objective goals and feedback on

performance against these goals has been shown to improve motivation (Locke and Latham

2002). However, in some circumstances, such as when an employee’s tasks are extremely

- 9 -

complex or the desired performance objectives are more subjective, companies may not provide

quantitative ratings for each of the employee’s performance objectives. Instead, it may be

desirable to simply urge employees to “do their best” rather than evaluate their performance

against specific goals, to focus on behaviors rather than targets, or to provide narrative rather

than quantitative evaluations (Locke and Latham 2002, Brutus 2010).

Further, companies must decide whether to provide ratings for any individual goals when

multiple goals are included in the plan. The simplest approach for the cognitively difficult task of

combining multiple goals is for the evaluator to provide only an overall rating (Heneman 1986),

which has the advantages of not requiring an explicit weighting scheme for combining the

multiple measures and allowing the rater to make an overall judgment based upon all information

known about the individual (Lyness and Cornelius 1982). However, only providing an overall

rating of job performance gives employees little specific feedback on the performance

dimensions driving the overall evaluation or requiring improvement, and allows the evaluator to

omit relevant performance information or include extraneous sources of performance variation in

the rating (Landy and Farr 1980).

2.2 DETERMINANTS OF PERFORMANCE APPRAISAL PLAN CHARACTERISTICS

Much of the literature on performance appraisal and employee performance evaluation

contends that the choice of performance appraisal plan characteristics should be a function of the

firm’s monitoring ability and the informativeness of available performance metrics. Although no

comprehensive theory addresses the specific determinants of each of the appraisal plan choices

outlined above, the performance appraisal, performance measurement, and management control

system literatures suggest a number of firm- and employee-level factors that are expected to

influence performance appraisal plan design.

- 10 -

Firm-Level Factors

Organizational Structure. Studies have long indicated that organizational structure

influences monitoring ability and information transfer costs, and thereby performance appraisal

practices. As organizations get larger, it becomes more difficult to directly monitor employees’

actions, increasing the need for more formalized performance appraisal as a substitute for more

informal performance evaluation methods (e.g., Astley 1985, Brown and Heywood 2005). In

particular, researchers suggest that larger firms require a broader set of performance goals and

greater explicit goal alignment in order to stimulate more effective communication and

monitoring (e.g., Hoque and James 2000).2

At some point, the increased information processing costs associated with firm size make

it optimal to decentralize authority, which then requires changes in control systems to achieve

integration and minimize agency problems (Bohn 1987, Christie et al. 2003, Chenhall 2006).

Key attributes of decentralization are the adoption of divisionalized and/or departmentalized

structures, the number of levels in the organizational hierarchy, and managerial spans of control.

Researchers suggest that these organizational choices have implications for the design of

performance appraisal systems. For example, allocating more decision rights to divisions is said

to call for the use of fewer, more aggregated performance goals since top management no longer

dictates the specific actions to be taken by division-level employees (e.g., Chenhall and Morris

1986, Christie et al. 2003). In departmentalized structures that group jobs by function,

2 Survey-based studies find that larger firms are more likely to conduct formal performance appraisals (Suutri and

Tahvanainen 2002, Brown and Heywood 2005, Grund and Sliwka 2009), to evaluate expatriate managers’

performance more frequently (Suutri and Tahvanainen 2002), and to use a broader set of performance measures

related to the Balanced Scorecard categories (though not necessarily for performance evaluation purposes) (Hoque

and James 2000), but find no firm-level association between size and the use of objective or subjective criteria in

appraisal (Suutri and Tahvanainen 2002, Bayo-Moriones et al. 2012).

- 11 -

performance appraisal can be tailored to the work in each functional unit (Murphy and Cleveland

1995), leading to greater variation in the firm’s appraisal plans.

In relatively flat or less hierarchical structures with fewer organizational levels and wider

spans of control, managers have less ability to closely supervise subordinates. As a result, some

argue that flatter organizations require more structured performance appraisal and more

performance goals due to the difficulty in personally managing and monitoring operations

(Murphy and Cleveland 1995, Mia and Goyal 1991, Judge and Ferris 1993). Lee and Yang

(2011) contend that firms with fewer levels and narrower spans of control should also make

greater use of causal models that link performance measures across levels. Because these

organizations assign decision rights lower in the organization and have greater integration needs,

the enhanced goal alignment from causal models is needed to promote greater awareness of

performance drivers and to evaluate relations between inputs and outputs. In contrast, others

argue that taller organization structures are more bureaucratic, relying on top-down goal setting

and more formal, routine performance monitoring to improve effectiveness (e.g., Astley 1985),

suggesting that more complex performance appraisal practices will be found in taller rather than

flatter organizations.

These discussions suggest that variations in appraisal plan characteristics are likely to be

associated with differences in firm size, decentralization into departmental and divisional

structures, number of levels in the organizational hierarchy, and managerial spans of control.

Operating Uncertainty. Considerable empirical evidence indicates that operating

uncertainty influences control system design and benefits. Chenhall and Morris (1986), for

example, find that greater perceived environmental uncertainty is associated with a preference

for broad scope performance measures. Chong (1996) finds that in high task uncertainty

- 12 -

environments, a broader set of performance measures is associated with higher managerial

performance, but in low uncertainty environments more performance measures lead to

information overload that impedes managerial performance. Bourgeois (1985) adds that a large

number of goals can reduce risk in volatile environments, with more environmental uncertainty

requiring a greater number of goals to guide strategic decisions. More specifically, the need to

continually reappraise goals in uncertain environments leads to new goals being articulated and

added to existing goals. By subjecting decisions to a large set of performance targets, the

likelihood of hasty commitment of resources to a particular course of action is reduced.

Greater operating uncertainty may also reduce the benefits of formal goal cascading and

alignment. Voelpel et al. (2006) contend that in dynamic environments, the formal, top-down

goal setting, cascading, and alignment in the balanced scorecard process can lead to entropy,

limit initiative and actions to specified goals that may be difficult to prespecify, and lead to a

mechanistic mindset in a setting where adaptability is required. This argument is supported by

Kellermanns et al.’s (2011) meta-analysis of studies on the performance benefits of strategic

consensus. They conclude that strategic consensus (one of the claimed benefits from greater goal

alignment) is more beneficial in organizations operating in stable environments because high

levels of consensus can undermine performance in highly dynamic environments, where too

much agreement on a course of action might impede decision makers’ ability to consider new

alternatives and respond quickly to unforeseen events. These discussions suggest that

performance appraisal plan characteristics will be associated with operating uncertainty and

environmental dynamism.

Competitive Environment. Related to organizational uncertainty is the firm’s

competitive environment. Contingency research suggests that firms in competitive markets are

- 13 -

more likely to use multiple performance measures that allow the organization to monitor both

their static competencies (e.g., current operating efficiency) and dynamic capabilities (e.g.,

ability to anticipate and respond to changes in the competitive landscape), and to make greater

use of causal models to link and align performance goals (e.g., Hoque et al. 2001, Lee and Yang

2011).

Competitive success, as reflected in higher profits or market share growth, may also have

a positive effect on a firm’s willingness to invest in more sophisticated performance appraisal

practices, leading to a positive association between economic performance and appraisal plan

sophistication (Wright et al. 2005). In addition, stronger economic performance has been found

to be associated with variations in the types of performance measures used in incentive plans

(e.g., Ittner and Larcker 2002), and may lead to greater diversity in goals because the resulting

organizational slack provides enough space to avoid conflicts between multiple goals that may

conflict in the short-term (Bourgeois 1985). This evidence leads us to examine the relations

between the various plan characteristics and the firm’s competitive environment and past

economic performance.

Individual Employee Characteristics

Performance appraisal frameworks and economic theories contend that appraisal plan

characteristics are influenced by individual employee attributes as well as by firm characteristics.

Lazear’s (1979, 1981) analytical work indicates that the need for monitoring decreases with

tenure as deferred compensation raises the cost of dismissal for low effort. Learning about

abilities and assigning workers to the right jobs is also more important for employees earlier in

their career (Jovanovic 1979), while strategic alignment and agreement on strategic priorities

increase with tenure (Joshi et al. 2003). These factors are expected to reduce the benefits from

- 14 -

complex performance appraisal as tenure increases. In contrast, Brown and Heywood (2005)

argue that if the purpose of performance appraisal is to promote worker identification with

organizational objectives and develop human capital, the performance appraisal process for long-

tenured workers should be more detailed and complex.

Other theories maintain that gender influences performance appraisal practices. These

theories argue that men and women sort into different types of jobs with different monitoring

requirements. Goldin’s (1986) model contends that women have shorter expected tenure and are

less motivated by the deferred rewards from longer careers. As a result, firms with more women

rely on more routine and extensive monitoring and short-term rewards. Bayo-Moriones et al.

(2012) claim that women are frequently short-term workers and are often assigned to simple jobs

for which it is easier to implement routine monitoring processes based on more frequent,

objective criteria. Jirjahn and Stephan (2004) further argue that women self-select into jobs with

more objective evaluation methods because this leaves less room for gender discrimination.3

The empirical literature has also identified links between an employee’s organizational

level and performance appraisal practices. Top management positions are found to be associated

with greater use of quantitative, formally rated goals (Bretz et al. 1992, Suutari and Tahvanainen

2002). At higher organizational levels, fewer, more aggregate goals that do not pre-specify the

actions needed to achieve the organization’s objectives may be more appropriate, while at lower

levels it is easier to specify a broader, more specific set of goals. The benefits from formal goal

cascading and alignment are also claimed to be higher for lower-level workers whose

understanding of the organization’s objectives and their contribution to those objectives are

3 Surveys find that firms with a larger percentage of women are more likely to conduct formal performance

appraisals (Brown and Heywood 2005, Addison and Belfield 2008, Grund and Sliwka 2009). However, Bayo-

Moriones et al. (2012) finds no differences in appraisal frequency or the use of objective measures (but greater use

of subjective measures) when women represent a larger percentage of the workforce.

- 15 -

lower (Boswell and Bodreau 2001). Given this broad set of empirical and theoretical evidence,

we examine the influence of employee characteristics on appraisal practices.

2.3 APPRAISAL PLAN CHARACTERISTICS AND RATINGS BIASES

The ultimate outcome of the appraisal process is the employee’s performance rating.

Ideally, performance ratings are an accurate reflection of the employee’s relative performance in

the firm. However, substantial research indicates that performance ratings are frequently plagued

by leniency and discriminability biases. Leniency bias refers to the tendency for ratings to be

inflated. Discriminability refers to the ability of ratings to separate good performance from bad.

Discriminability bias can take two forms: (1) centrality biases, in which ratings cluster around a

point on the ratings scale and little use is made of the extreme ends of the scale, and (2)

contagion biases, in which subordinates’ ratings are heavily influenced by their superiors’

ratings, making it difficult to distinguish the individual contributions of the two parties (e.g.,

Latham et al. 2008).

Prior research suggests that plan characteristics can influence the extent of ratings biases.

Providing multiple unrated goals or only providing overall ratings of job performance allows the

evaluator to omit relevant performance information or to include extraneous sources of

performance variation in the rating (Landy and Farr 1980, Ittner et al. 2002). Economic models

also indicate that uncertainty in performance and the variance in the performance signal lead to

compressed ratings, suggesting that ratings based on well-defined, unambiguous criteria and

goals will be less biased (MacLeod 2001, Golman and Bhatia 2012). Moers’ (2005) empirical

study of a single firm further predicts and finds that subjectivity in performance measurement, as

well as greater diversity in objective performance measures (as proxied by the number of

measures), leads to more lenient and compressed ratings, which Moers argues is due to greater

- 16 -

diversity and subjectivity in performance measures affording managers more discretion in

performance evaluations.

Several experimental studies have examined whether the provision of causal models that

explicitly cascade and link performance measures can minimize biases found in performance

evaluations based on multiple financial and nonfinancial measures (e.g., Banker et al. 2004,

Wong-On-Wing et al. 2007). These studies indicate that the provision of causal models mitigates

the over-emphasis on certain types of measures, suggesting that ratings biases will be lower

when performance goals are explicitly aligned. However, explicitly cascading goals from one

organizational level to the next may introduce contagion biases because a superior’s performance

rating on a cascaded goal is formally linked to the subordinate’s performance goal and rating.

Given the potential ratings biases introduced by appraisal plan choices, we examine the

association between these choices and ratings leniency and discriminability.

3. Sample

Our sample consists of data from 153 clients of a leading performance management

software company. The software company maintains its clients’ performance management data

on a cloud basis and provided us with selected information from the clients’ data files. The initial

sample consisted of publicly-traded clients located in the United States and Canada that had at

least one year of experience using the software. Each firm was then contacted and given the

option to opt out of the study (four firms declined to participate). We determined the firms’ most

recent annual goal and review periods ending on or before December 31, 2009, and use that

year’s performance appraisal data in our tests. In addition, we interviewed eight system users to

- 17 -

gather additional insight into system usage practices. These users were selected by the software

company to maximize diversity in firm and appraisal plan characteristics.

A number of steps were taken to preserve the clients’ confidentiality. Prior to providing

us with the data files, the software company replaced the firms’ names with anonymous

identifiers and removed individuals’ names, job titles, and specific goal descriptions. We were

allowed a one-time request for specific financial variables, which were independently pulled

from Compustat and CRSP and provided to us in a file containing the anonymous company

identifiers for merger purposes.

Table 1 compares our sample with the larger population of Compustat firms in fiscal

2008 (the year prior to the 2009 performance appraisal data used in our tests to account for the

plan characteristics being chosen prior to the appraisal plan year). Panel A presents the industry

composition and indicates that our sample is fairly representative of the larger Compustat

population, with the exception of greater representation of durable manufacturers and textile,

printing, and publishing firms and lower representation of financial institutions.

Panel B of Table 1 presents means and medians for selected firm characteristics. Relative

to the Compustat population, our firms tend to be larger (measured by the total number of

employees, total assets, market capitalization, and total annual sales), more levered, more

profitable, and operate in less concentrated industries. Because the sample relates to 2009

performance appraisal plans, prior year (2008) returns of both groups are large and negative.

However, our sample had somewhat better market performance but greater market volatility.

4. Variables and Descriptive Statistics

4.1 EMPLOYEE-LEVEL APPRAISAL PLAN VARIABLES

- 18 -

We compute a number of variables to capture different characteristics of individual users’

2009 appraisal plans and performance ratings. Number of Goals equals the natural logarithm of

the total number of annual performance goals assigned to each employee. Number of Plans is the

number of different appraisal plans covering the employee during the year (e.g., a single annual

plan versus multiple quarterly plans). Proportion of Aligned Goals equals the fraction of each

employee’s goals that are explicitly aligned with other employees’ goals. Similar to the example

in Figure 1, a standard feature of the software is the ability to explicitly link each goal to the

goals of other employees. Clients may choose to use or not use the alignment feature in the

software, or may choose to use it for some types of goals or employees but not others.

As illustrated in Figure 2, the firms in our sample follow a number of ratings approaches.

Some companies provide individual performance ratings for some or all of the specific goals

assigned to the employee, as well as an overall rating. Others provide an overall goal objective

rating for the achievement of performance goals (with or without also providing ratings for

specific performance goals) together with a subjective overall “competency” rating.4 Still others

only provide an overall rating regardless of the number of specific performance goals or the

evaluation of competencies. We use two variables to capture these distinctions. Has Rated Goals

equals one if the employee has any rated performance goals and zero otherwise. Our discussions

with system users and the software company indicate that rated goals tend to be “harder” than

non-rated goals, in the sense of being more objective, quantitative, and easily measurable.

4 A competency rating assesses the extent to which an employee has achieved the specified knowledge, skills,

attitudes, traits, values, or other personal characteristics essential to perform the job. 48.7% of our sample firms give

competency ratings to at least some users. To examine the determinants of competency rating provision, we estimate

a Probit model with the dependent variable equal to one if any user received a competency rating and zero

otherwise. We also estimate a Tobit model with the proportion of users who receive competency ratings as the

dependent variable. Both dependent variables are negatively associated with the number of organizational levels

covered by the system and the number of months the firm has used the system, and positively related to prior stock

returns. The proportion of users who are managers is also positive and significant in the Tobit model, but not the

Probit model.

- 19 -

Multiple Ratings equals one if the employee receives multiple performance ratings (i.e., ratings

for overall goal objectives and/or overall competencies together with an overall rating) and zero

if only an overall rating is given.5

4.2 FIRM-LEVEL APPRAISAL PLAN VARIABLES

For our firm-level tests, we aggregate the employee-level data to capture the average

level of and variation in appraisal plan characteristics.6 Avg. Number Goals is the firm’s average

annual number of goals per user, and Std. Dev. Number Goals is the firm’s standard deviation in

the number of goals per user. Avg. Number Plans and Std. Dev. Number Plans are the firm’s

average and standard deviation in the number of different plans per user during the year,

respectively. Avg. Mean Goals Per Plan and Std. Dev. Mean Goals Per Plan are the average and

standard deviation in the mean number of goals an employee has per goal plan, respectively.7 In

addition to these employee-based measures, we include the firm-level variable Number Plans to

capture the total number of different plans operated by the firm during the year.

We construct the following aggregate variables to measure firm-level goal alignment

characteristics. Avg. Aligned is the firm’s average number of goals per user that are aligned, and

Std. Dev. Aligned is the standard deviation of number of goals that are aligned. Pct. Has Aligned

is the percentage of the firm’s users that have at least one aligned goal. Finally, we measure the

firm’s performance rating practices using Pct. Has Rated Goal (the percentage of the firm’s

5 An employee can receive multiple performance ratings (Multiple Ratings = 1), but have no rated performance

goals (Has Rated Goals = 0). This occurs when the employee receives an “overall goal objective rating” and/or an

“overall competency rating” in addition to an overall rating, even though no individual goals are rated. Alternatively,

an employee can have individually rated goals (Has Rated Goals = 1), but only receive an overall performance

rating and no separate overall goal objective or overall competencies rating (Multiple Ratings = 0). 6 Our analyses include firm-level standard deviations in plan characteristics because the extent to which plans are

varied across employees is a major appraisal plan choice. We examine this issue in greater detail in our employee-

level tests. 7 The mean number of goals an employee has per plan is used because employees can participate in multiple plans,

each with a different number of goals.

- 20 -

system users that have at least one rated performance goal) and Pct. Multiple Ratings (the

percentage of the firm’s users who receive multiple performance ratings).

Since many of the firm-level appraisal plan measures capture similar characteristics, we

use exploratory Principal Components Analysis (PCA) with oblique rotation to isolate the unique

underlying constructs captured by the data. Table 2 presents the resulting factor structure. Four

factors with eigenvalues greater than one emerge that collectively explain roughly 85% of the

variation in the data. We interpret the factors based on the variables with loadings greater than

0.40. The first principal component, labeled Goal Number Factor, captures the quantity of and

variation in the number of performance goals, as reflected in the large positive loadings on Avg.

Number of Goals, Std. Dev. Number of Goals, Avg. Mean Goals Per Plan, and Std. Dev. Mean

Goals Per Plan. The second principal component, labeled Goal Plan Factor, has large positive

loadings on Avg. Number Plans and Std. Dev. Number Plans and captures variation in the

number of plans for each employee. The third principal component, labeled Alignment Factor,

represents the degree to which goals are aligned, as indicated by the large negative loadings on

Avg. Aligned , Std. Dev. Aligned, and Pct. Has Aligned. 8

The fourth principal component,

labeled Rated Goal Factor, captures how employee performance is rated, as reflected in a large

negative loading on Pct. Multiple Ratings and Pct. Has Rated Goal.

The coefficient alphas for the variables loading greater than 0.40 are 0.894 for Goal

Number Factor, 0.717 for Goal Plan Factor, 0.949 for Alignment Factor, and 0.510 for Rated

Goal Factor. The four firm-level appraisal plan constructs represent the sums of the standardized

values of each variable loading heavily on that factor. The constructs are coded so that higher

8 The fact that means and standard deviations in the number of goals, the number of plans, the percentage of aligned

goals load on the same factors is primarily due to the tendency for the firms that average only one plan, few goals,

and little or no alignment to do this for all of their employees. We examine the lack of variation in some firms’

appraisal practices later in subsequent analyses.

- 21 -

values represent more complex appraisal plans (i.e., higher values when the number and

variation in goals (Goal Number Factor), number and variation in plans (Goal Plan Factor), and

goal alignment (Alignment Factor) are greater, and when employees have some rated

performance goals and do not receive an overall rating alone (Rated Goal Factor)).

4.3 PERFORMANCE RATING DISTRIBUTIONS

We compute a variety of variables to examine firm-level distributions in 2009

performance ratings. Since the firms in our sample exhibit substantial variation in the number of

scale points used in their performance ratings, the software company standardized each firm’s

overall ratings to range from zero (the lowest possible rating) to one (the highest rating). We use

the standardized scores to investigate potential biases in the firms’ ratings. Leniency biases are

examined using mean and median overall ratings (denoted Avg. Rating and Med. Rating,

respectively) and skewness in ratings (denoted Ratings Skew). Following prior literature, we

assume that mean and median ratings that are significantly higher than 0.50 (the midpoint of the

standardized ratings scales) or ratings that are skewed upwards exhibit leniency bias. Centrality

bias is assessed using the standard deviation in ratings (denoted Std. Dev. Rating) and the

percentages of ratings that are in the lowest and highest rating categories (denoted Pct. Min.

Rating and Pct. Max. Rating, respectively). Firms with larger standard deviations in ratings and

more ratings in the extreme ratings categories are assumed to have lower centrality bias.

Differences between supervisors’ and their subordinates’ ratings are used to examine contagion

biases. Avg. Rating Difference is the average difference in ratings between supervisors and

subordinates, and Std. Dev. Ratings Difference is the standard deviation in these differences.

4.4 PREDICTOR VARIABLES

Organizational Structure

- 22 -

Five variables capture different aspects of organizational structure. Size is measured by

log(Market Capitalization), which equals the natural logarithm of the firm’s market value at the

end of fiscal 2008. The other variables are pulled from the firms’ data files. Log(Number of

Departments), log(Number of Divisions), and log(Number of Levels) are the natural logarithms of

the number of departments, divisions, and hierarchical levels, respectively, covered by the

software. Avg. Span of Control is the natural logarithm of the average number of subordinates

per manager.

Operating Uncertainty

Growth and environmental volatility are two key factors that impact firm operating

uncertainty (Dess and Beard, 1984). We use six variables from Compustat and CRSP to measure

these constructs. Exploratory Principal Components Analysis of these indicators reveals two

underlying factors that serve as our proxies for operating uncertainty. Volatility Factor is

calculated as the sum of the standardized values of (1) Earnings Volatility (the standard deviation

of Return on Assets during the previous five years), (2) R&D Expense (annual research and

development expense scaled by total annual sales, with missing values of research and

development expense set to zero), and (3) Return Volatility (the standard deviation of monthly

stock returns over the previous 36 months). Growth Factor is the sum of the standardized values

of (1) New Investment (capital expenditures plus acquisitions less sales of property and

depreciation, scaled by average total assets), (2) Sales Growth (the percentage change in

revenues from the previous fiscal year), and (3) Employee Growth (the percentage change in the

total number of number of employees from the previous fiscal year). Coefficient alphas are 0.622

for Growth Factor and 0.496 for Volatility Factor. In addition, we include Book-to-Market (the

- 23 -

book value of total equity divided by market capitalization, both measured at fiscal year-end), a

common inverse proxy for more uncertain growth opportunities and innovation strategies.

Competitive Environment

Three variables from Compustat and CRSP measure the firms’ competitive

environments. Sales Concentration, a proxy for market competition, is the uncorrected sum of

squares of total annual sales divided by the squared sum of total annual sales of all firms in the

same one-digit SIC during fiscal 2008. Prior Return on Assets is net income divided by average

total assets. Prior Return is the cumulative stock return over the previous twelve months.

System User Characteristics

We examine a variety of user characteristics that earlier studies suggest can influence

performance appraisal practices. These include: (i) Male, an indicator that equals one if the

employee is male and zero otherwise, (ii) Executive, an indicator that equals one if the employee

is in level one or two of the organizational hierarchy and zero otherwise, (iii) Manager, an

indicator that equals one if the employee holds any managerial position and zero otherwise, (iv)

Relative Level in Firm, measured as a non-executive employee’s level in the organizational

hierarchy divided by the total number of hierarchical levels, rescaled so that larger values

represent non-executives higher in the hierarchy, and (v) Tenure, the natural logarithm of the

number of months the employee has worked at the firm. In firm-level tests, we aggregate these

variables into measures representing the percentages of employees who are male (Pct. Male),

executives (Pct. Executive), and managers (Pct. Managers), and average user tenure (Avg.

Tenure). Pct. Users controls for the proportion of the firm’s employees that use the system.

Other Controls

- 24 -

Since learning effects may occur through system usage, we include Months Used, which

equals the number of months the firm had used the software as of the end of 2009. The

performance appraisal literature also indicates that performance appraisal purpose has a

significant impact on appraisal practices, particularly whether or not the ratings are used for

compensation and promotion decisions. Although we do not have information on how our

sample firms use their performance appraisal information, we proxy for its explicit use in

compensation decisions using the variable Compensation Module, an indicator that equals one if

the firm uses the software’s compensation module in conjunction with the performance appraisal

module, and zero otherwise. Our discussions with system users and the software company

indicate that clients who link the software’s performance appraisal and compensation modules

are also likely to explicitly link performance ratings to compensation decisions (especially for

incentive pay), while those without the software link are more likely to either treat performance

appraisals and compensation decisions separately or to make the link less explicit.

4.5 DESCRIPTIVE STATISTICS

Panel A of Table 3 presents firm-level descriptive statistics for our appraisal plan

variables, and Figure 3 displays the empirical densities for a number of the plan characteristics

examined in our analyses. Plan characteristics vary widely across firms, and the variables’

distributions are far from normal. The average number of goals per employee ranges from one to

more than 20 (median = 5.93 goals). The median firm operates two plans with goal periods of

one year each (364 days), but the majority of employees participate in only one plan. The median

(mean) firm rates at least some individual goals for 80% (55%) of their users, but the distribution

is bi-modal, with large concentrations of (i) firms that do not rate any individual goals for any of

their employees and (ii) firms that rate at least some individual goals for 85% or more of their

- 25 -

employees. A similar distribution is seen for the percentages of employees who receive only an

overall performance rating (i.e., do not receive multiple performance ratings). Large variations

are seen in the extent to which goals are explicitly cascaded and aligned with higher-level goals.

In 22% of the firms, none of the users has an aligned goal in the system, and in only 19.5% of the

firms do more than 50% of the users have any of their goals explicitly aligned. In addition to

differences in central tendencies across firms, the sample also exhibits significant variation in

goal plan characteristics within many of the firms, as seen in the standard deviations in number

of goals and plans and the percentages of employees with rated goals and aligned goals.

System usage characteristics are provided in Panel B of Table 3. The mean firm has

36.82 months experience using the system, and 45% explicitly link their performance appraisals

to compensation through the system’s compensation module.9 On average, the systems cover

49% of the firms’ employees, 62% of whom are male, 24% managers, and 1% executives. The

average system has 2672 users in 297 departments, 28 divisions, and 8 organizational levels.

As shown in Panel C, mean and median performance ratings are 0.58 and 0.56,

respectively, both of these are significantly higher than the rescaled midpoint of 0.50 (p < 0.01,

two-tailed). The median (mean) percentage of users receiving the minimum rating is 0.76%

(5.2%) and the median (mean) receiving the maximum rating is 7.6% (9.2%). Supervisors’

ratings tend to be slightly higher than their subordinates’ ratings.

4.6 CORRELATIONS

Table 4 provides a correlation matrix containing the four firm-level appraisal plan

constructs and our firm-level predictor variables. Alignment Factor and Rated Goal Factor are

9 The compensation module percentage is similar to survey results in WorldatWork and Sibson (2010), which

indicate that 51% of firms link performance appraisals to short-term incentives and 31% to long-term incentives.

- 26 -

positively correlated, indicating that firms that explicitly cascade and align performance goals

also tend to provide ratings for at least some individual performance goals. Goal Number Factor

has a positive association with the number of departments and the percentage of employees

covered by the system. Volatility Factor is negatively associated with goal alignment and the

provision of rated goals. Rated Goal Factor is also negatively associated with the percentage of

users who are managers and male, and positively associated with prior stock returns. System

experience is positively correlated with the number of performance goals and goal alignment.

5. Results

5.1 FIRM-LEVEL TESTS

We examine the determinants of the four firm-level appraisal plan constructs found in

Table 5. Because the distributions of these four variables are censored, we estimate Tobit

models. The Goal Number Factor and Goal Plan Factor models are left-censored, the Alignment

Factor model is right-censored, and the Rated Goal Factor model is both left- and right-

censored. The resulting models are statistically significant (p < 0.01) but exhibit only modest

explanatory power (pseudo adjusted R2s ranging from approximately 3.0% to 7.0%).

At least one of the organizational design variables is significantly associated with each of

the four appraisal plan constructs. Consistent with claims that larger firms require more complex

performance appraisals to compensate for direct monitoring difficulties, Goal Number Factor is

positively related to firm size, as well as with the number of departments covered by the system.

However, the number of departments is negatively related to the Rated Goal Factor, implying

that the larger number of goals in these firms tend to be more subjective and less amenable to

formal performance rating. The number of divisions is negatively associated with goal

- 27 -

alignment, suggesting that alignment becomes more difficult or less beneficial as firms

decentralize decision-rights to stand-alone divisions (particularly when they operate

independently and do not need to coordinate actions). Firms with more hierarchical levels have

more plans and are more likely to provide rated goals, supporting claims that taller, more

mechanistic structures employ more structured and bureaucratic appraisal systems (e.g., Astley

1985).

The coefficient on Volatility Factor is negative in all four models and significant in all

but the Goal Number Factor model. In addition, Book-to-Market, an inverse proxy for

uncertainty related to innovation and growth opportunities, has a significant positive association

with Goal Number Factor. These results run counter to claims that greater volatility and

uncertainty require more performance goals (Bourgeois 1985, Chong 1996), but the significant

negative relation between Volatility Factor and Alignment Factor is consistent with top-down

goal cascading and alignment being problematic in dynamic environments (Voelpel et al. 2006).

Neither Growth Factor nor any of the competitive environment variables are significantly

associated with the appraisal plan constructs.

Systems that cover a larger percentage of the firm’s employees and a larger percentage of

users who are managers tend to give employees more performance goals. Avg. Tenure is

negatively associated with the number of plans. The percentages of users who are executives or

males appear to have little relation to overall, firm-level goal plan characteristics. Firms with

more experience using the system users have more performance goals but are less likely to rate

individual goals. Lastly, fewer goals are used when the appraisal system is explicitly tied to the

system’s compensation module, but the goals are more likely to be rated.

5.2 VARIATIONS IN APPRAISAL PLAN CHARACTERISTICS WITHIN FIRMS

- 28 -

Although our firm-level tests provide some evidence that performance appraisal plans

vary across companies in response to differences in organizational, environmental, and user

characteristics, they provide little insight into whether these characteristics influence individual

appraisal plans within firms. We begin investigating this issue by estimating separate models for

each firm, with individual users as the unit of analysis and employee-specific characteristics as

predictor variables. Since we are examining individual plans, we use employee-specific plan

characteristics as dependent variables rather than the aggregated firm-level constructs. In

particular, we examine (i) the natural logarithm of the employee’s number of annual goals, (ii)

the number of plans in which the employee participates, (iii) the proportion of the employee’s

goals that are aligned, (iv) an indicator for whether the employee has any rated performance

goals, and (v) an indicator for whether the employee only receives an overall performance rating.

Employee-specific predictor variables are the employee’s gender (where one equals male),

whether he or she is an executive, and his or her relative level in the firm, tenure at the firm, and

total number of reports (which equals zero if the user is not a manager). We do not include a

separate indicator for managers because this distinction is subsumed by the number of reports

variable.

Summary statistics from these firm-specific tests are provided in Table 6. We report

means, medians, and standard deviations for the coefficients on the employee-specific predictors,

as well as the percentages of coefficients that are positive, negative, and zero (which occurs

when a firm has no employee-level variation in a goal plan characteristic). Two aggregate test

statistics are provided: z-statistics for whether the mean coefficient is statistically different than

zero and non-parametric sign tests for differences in the direction of the coefficients.

- 29 -

The firm-specific tests support claims that different job responsibilities and monitoring

needs require different appraisal practices. Employees higher in the hierarchy generally have less

complex appraisal plans. Executives tend to receive fewer goals that are less likely to be

explicitly aligned or rated, and are more likely to receive only an overall rating. Similarly, non-

executive employees higher in the hierarchy generally have a lower proportion of goals that are

aligned, a lower probability of receiving at least some rated goals, and a lower probability of

receiving multiple ratings. When managers have more reports, the number of performance goals

tends to increase, reflecting the broader set of activities they manage. Men also tend to have

more goals, and the proportion of their goals that are aligned is generally higher, which is

inconsistent with the theory that women sort into jobs with more formal and explicit monitoring.

In contrast to theories that longer tenured employees require less monitoring, Tenure is positively

associated with the number of goals, number of plans, having rated goals, and receiving multiple

ratings.

Despite these central tendencies, there are substantial variations in how firms respond to

these employee characteristics, with some firms having significant coefficients that differ in sign

from the significant central tendencies, or having significant associations even though the overall

association is insignificant. For example, Relative Level in Firm has an insignificant overall

association with the number of performance goals (Table 6, Panel A), but this variable is

negative and significant in 49 firms and positive and significant in 45.

5.3 EXAMINING STANDARDIZATION OF PLAN CHARACTERISTICS

One question raised by the preceding analyses is why some firms exhibit no variation in

various appraisal plan measures. For example, 45% of firms display no differences in whether or

not employees receive multiple ratings. If monitoring ability or performance objectives vary

- 30 -

significantly across employees or employee groups within a firm, we would expect to see at least

some within-firm differences in this and other plan measures. We examine this issue by

estimating Probit models with dependent variables equal to one if the firm exhibits no variation

across users in a given appraisal plan characteristic and zero otherwise. Independent variables are

the firm-level predictor variables previously considered. Since all of the firms vary the number

of goals to some extent, this characteristic is not included in these tests.

The results (not reported in the tables) indicate that larger firms and firms with systems

that are more focused on managers (Pct. Managers) are less likely to vary the number of plans,

the proportion of goals that are rated, and the proportion of employees that receive multiple

performance ratings. Firms that have greater market volatility are also less likely to vary the

number of plans and the provision of rated goals. The Pct. Managers and Volatility Factor

results are consistent with Arya et al.’s (2005) model which suggests that standardized

performance evaluations are more advantageous when the employees being evaluated are less

diverse and when reducing measurement noise through relative performance evaluation is more

beneficial. Firms with more divisions are less likely to vary the percentage of aligned goals

(generally aligning no goals), consistent with the earlier finding that explicit alignment may be

less beneficial in decentralized organizations with stand-alone divisions. Firms with more

organizational levels and wider managerial spans of control are more likely to vary the number

of plans and average goal alignment, respectively.10

We find little evidence that the other

operating uncertainty, competitive environment, or system user variables explain the lack of

variation in our appraisal plan variables.

10

When there is no variation in Rated Goal, it is always because no goals are rated, and lack of variation in the

number of plans is always because all employees have only one plan. Of the firms that exhibit no variation in

whether they provide multiple performance ratings, some only give an overall rating to all of their users while others

provide ratings in addition to overall rating to all users.

- 31 -

5.4 DETERMINANTS OF FIRM-SPECIFIC VARIATIONS IN PLAN CHARACTERISTICS

We next examine whether firm-level characteristics are associated with within-firm

differences in plan characteristics in the subsample of firms that exhibit employee-level

variations in these variables. The sample size in these tests ranges from 77 to 153 firms

depending upon the plan characteristic. Because theories and prior empirical studies provide

little guidance regarding the moderating effects of firm-level characteristics on the relationship

between individual employee attributes and their appraisal plan, these tests are inherently

exploratory.

Since we are interested in understanding how appraisal plan characteristics vary with two

distinct levels of analysis (i.e., firm and individual employee), our tests are based on a two-step

hierarchical approach that is common in multilevel modeling. In the first stage, we estimate the

following firm-specific model where individual employees are the unit of analysis:

Plan Characteristici = α + β1 Malei + β2 Executivei + β3 Relative Leveli

+ β4 Tenurei + β5 Reportsi + εi (1)

where Plan Characteristic is either (i) the natural logarithm of the number of goals, (ii) the

number of plans, (iii) the proportion of the employee’s goals that are aligned, (iv) an indicator for

whether the employee has a rated performance goal, or (v) an indicator for whether the employee

receives multiple performance ratings, and i is the individual employee.

In the second stage, we estimate the following cross-sectional model in which the first-

stage, firm-specific coefficients from equation (1) serve as dependent variables:

βnj = γ + δ1…5 Organizational Structurej + δ6…8 Operating Uncertaintyj

+ δ9…11 Competitive Environmentj + δ12…16 System User Characteristicsj

+ δ17…18 System Usagej + εj (2)

where βnj is a coefficient estimated in the first stage, Organizational Structure is the five

organizational structure proxies, Operating Uncertainty is the three operating uncertainty

- 32 -

proxies, Competitive Environment is the three competitive environment proxies, System User

Characteristics is the five aggregated user characteristic proxies, System Usage is the two system

usage control variables, and j is the firm. Significant δ coefficients indicate that firm-level

characteristics help to explain cross-sectional differences in the extent to which firms vary plans

to account for differences in individual user characteristics.

Since the dependent variables in the second-stage regression are estimated rather than

observed (i.e., so called “estimated dependent variables”), the residual in the second-stage model

inherits sampling uncertainty from the first-stage regressions. To ensure that our second-stage

estimates are consistent and efficient, we follow the approach outlined by Lewis and Linzer

(2005) for the case where the sampling variances of the observations on the dependent variable

are known. This approach generally relies on feasible GLS (or FGLS) estimation. However, in

our research setting, our second-stage sample is small relative to the average first-stage sample

(i.e., our sample consists of relatively few firms that have relatively many system users) and, in

turn, the estimated variance of the regression residual that is not due to estimating the dependent

variable is small. Thus, the estimation in our case becomes the special case of weighted least

squares (WLS) described by Saxonhouse (1976) in which the observations in the second-stage

are weighted according to the inverse of their standard error from the first stage.

The results from our two-stage estimations are summarized in Table 7 and full results are

presented in the appendix. The models’ adjusted R2s range from 12.3% to 73.6% (mean = 30.4%,

median = 26.0%), indicating that these firm-level factors have a substantial moderating influence

on the relation between an individual employee’s attributes and his or her appraisal plan

characteristics. The firm-level predictors’ explanatory power is greatest in models examining the

influence of these factors on variations in appraisal characteristics with the respect to non-

- 33 -

executive employees’ hierarchical level (mean adj. R2 = 44.7%, median = 42.7%, maximum =

73.6%).

A positive sign in Table 7 indicates that a given employee characteristics (i.e., Male,

Executive, Relative Level, Tenure, Reports) is associated with greater appraisal plan complexity

(i.e., more goals and plans, greater alignment, and greater likelihood of receiving rated goals and

multiple ratings). The organizational design variable that most consistently moderates the

relations between individual employee and plan characteristics is firm size (Panel A). As market

capitalization increases, firms exhibit greater gender differences in appraisal plans, giving more

goals to men than to women, and providing men with rated goals and multiple ratings more

frequently. Firm size is also associated with variations in plan design based on organizational

level, with executives in larger firms being more likely to receive rated goals and to have greater

explicit alignment than non-executives. Non-executive employees higher in larger firms’

hierarchies are also more likely to have aligned goals, rated goals, and multiple performance

ratings. In addition, larger firms are more likely to provide rated goals to longer tenured

employees.

Both greater divisionalization and greater departmentalization tend to reduce the use of

more complex appraisal practices (i.e., plans, alignment, and rated goals) for men, executives,

and longer tenured employees. However, having more divisions and departments is positively

related to the use of rated goals for non-executives higher in the hierarchy and employees with

more reports. Firms with larger spans of control tend to vary their appraisal practices based on

the shape of the hierarchy and the employee’s position in that hierarchy. Firms with larger spans

increase the number of plans and their use of rated goals for employees with more reports, and

align goals to a greater extent for non-executives higher in the hierarchy. At the same time, these

- 34 -

firms reduce the provision of rated goals for executives and other employees higher in the

hierarchy, as well as the number of goals for male employees. In firms with more organizational

levels, executives and other employees higher in the hierarchy are more likely to only receive an

overall rating, while longer tenured employees and employees with more reports receive more

plans and goals, respectively.

Panel B of Table 7 examines measures of operating uncertainty. Market volatility

consistently moderates the relations between the number of goals and employee characteristics.

In firms with greater market volatility, males, executives, longer tenured employees, and

employees with more reports all receive fewer goals. Greater volatility is also associated with

men having fewer plans and a lower percentage of aligned goals, with longer tenured employees

and those with more reports being less likely to receive rated goals. The only situations in which

greater volatility is related to increased appraisal plan complexity is an increase in plans for

employees with more reports, and a greater likelihood of non-executives higher in the hierarchy

receiving multiple performance ratings. Consistent with these market volatility results, higher

book-to-market ratios (an inverse measure of operating uncertainty) are positively related to the

number of goals given to men, executives, more tenured employee, and employees with more

reports. Book-to-market ratios also have a significant relation with the percentage of aligned

goals (positive for Relative Level and negative for Executive and Reports) and the provision of

multiple ratings (negative for Reports and positive for Male and Relative Level). Growth is

primarily associated with the appraisal plans given to men, where it has a significant positive

relation with the number of goals, plans, and rated goals. Stronger growth is also associated with

executives only receiving an overall rating and with employees with more reports having fewer

plans.

- 35 -

Measures of firms’ competitive environment are primarily related to the extent to which

appraisal plans are varied by tenure and number of subordinates (Panel C of Table 7). When past

accounting and/or stock market performance is higher, more tenured employees tend to have

more plans and more aligned goals, but are less likely to receive rated goals. Stronger financial

performance is also associated with employees with more reports receiving more goals and

plans, lower explicit goal alignment, and less likelihood of receiving rated goals. In firms that

operate in industries with more concentrated sales, tenure is positively associated with the

number of plans and rated goals and the likelihood of receiving multiple performance ratings.

The number of reports is also positively associated with the number of goals in more

concentrated industries.

The performance appraisal literature suggests that a more diverse user base should be

associated with greater individual variation in plan characteristics. Consistent with this claim,

one or more system user variables is significant in 76% of the models, including all of the

Relative Level and Tenure models (Table 7, Panel D). The most frequently significant

moderators are Male and Pct. Users. When a larger percentage of users are men, non-executives

higher in the hierarchy have more goals but fewer plans, lower alignment, and a greater

likelihood of only receiving an overall rating; men are more likely to receive rated goals; more

tenured employees have greater goal alignment; executives receive fewer goals; and employees

with more subordinates are more likely to receive rated goals. As the appraisal system covers a

larger percentage of employees, men, executives, and more tenured employees receive more

goals than other users; men and employees in higher levels have more aligned goals (as well as a

greater likelihood of rated goals in the latter group); and executives and more tenured employees

become more likely to receive multiple ratings.

- 36 -

Panel E examines the system usage control measures. Firms using the software’s

compensation module tend to give employees with more reports a larger number of goals and

plans, give executives and non-executives higher in hierarchy only an overall rating, and give

males fewer goals plans. Longer software usage is associated with fewer goals for executives,

more aligned goals for men, a greater likelihood that employees with more reports receive rated

goals, and less likelihood that managers receive multiple ratings.

In sum, our results using hierarchical linear models that accommodate multiple sources of

variation (i.e., within-firm and across-firm) provide strong evidence that firm-level factors

moderate the relations between individual employee characteristics and their performance

appraisal plans. This is particularly true for firm size, which tends to increase appraisal plan

complexity (e.g., number of goals and plans, goal alignment, and the provision of rated goals) for

the various employee groups captured in our measures of employee attributes; divisionalization,

which is related to more complex appraisal practices for these groups; operating uncertainty,

which has a significant, negative relation with the number of goals and plans that executives,

males, more tenured workers, and employees with more subordinates receive; and past

accounting returns, particularly with respect to the appraisal plans for more tenured employees

and those with more reports.

Appraisal Plan Characteristics and Performance Rating Biases

We examine the ratings implications of the appraisal plan characteristics in Table 8.

Since the distribution of performance ratings is a firm-level attribute, we relate our four appraisal

plan factors to a variety of firm-level proxies for leniency and discriminability biases that are

based on employees’ overall performance ratings. Following prior literature, we assume that

mean and median ratings that are significantly higher than 0.50 (the midpoint of the standardized

- 37 -

ratings scales) or ratings that are skewed upwards exhibit leniency bias. Firms with larger

standard deviations in ratings and more ratings in the extreme ratings categories are assumed to

have lower centrality bias. Smaller differences between supervisors’ and their subordinates’

ratings and smaller standard deviations in these differences are assumed to reflect contagion

biases.

We include three control variables in these tests. Number Rating Points controls for the

possibility that our ratings bias proxies are influenced by the number of ratings scale points used

by the firm. Compensation controls for the more compressed and lenient ratings found in firms

that use performance appraisals for compensation purposes (e.g., Jawahar and Williams 1997).

360 Module is an indicator that equals one if the firm uses the software package’s 360 degree

feedback module and zero otherwise. 360 degree (or multi-source) feedback is an evaluation

method that incorporates feedback from the worker and his or her peers, superiors, subordinates,

and customers, and is claimed to reduce biases in performance appraisals (e,g., Grint 1993).

The intercepts in both the mean and median ratings models are significantly greater than

the midpoint of 0.50, consistent with leniency biases in performance ratings. However, neither

model is statistically significant, nor are the coefficients on any of the four appraisal factors.

However, the appraisal practices do appear to influence our other ratings bias proxies. Goal

Number Factor is positively associated with ratings skewness (a proxy for leniency bias) and

standard deviations in ratings differences between supervisors and their subordinates (consistent

with less contagion bias). Goal Plan Factor is positively related to standard deviations in ratings

and greater use of minimum and maximum ratings, reflecting less centrality biases. In contrast,

Rated Goal Factor is associated with greater discriminability bias, as seen in the significant

negative relations with standard deviations in ratings, the percentage of employees who receive

- 38 -

the minimum or maximum rating, and the smaller ratings differences between supervisors and

their subordinates. Thus, greater provision of rated goals (which tend to be based on more

objective criteria) appears to increase discriminability biases, contrary to claims that more

objective performance criteria and evaluations that decompose overall evaluations into smaller

subcomponents reduce ratings biases. However, Rated Goal Factor is negatively related to

ratings skew, suggesting that providing ratings for individual goals can reduce leniency biases.

Greater explicit goal alignment is also negatively associated with standard deviations in both

ratings and ratings differences between supervisors and subordinates, consistent with lower

ratings discriminability. This result suggests that explicitly cascading and aligning goals, as

espoused in the balanced scorecard and other literatures, can induce discriminability biases by

making it more difficult to differentiate performance based on inter-related (and therefore

correlated) goals.

Firms that formally link performance appraisals to compensation decisions through the

software’s compensation module exhibit lower mean and median ratings, but smaller standard

deviations in both overall ratings and supervisor-subordinate ratings differences. Surprisingly, a

larger number of available ratings scale points is negatively associated with ratings

discriminability. The use of 360 degree feedback, on the other hand, is associated with greater

variations in ratings, supervisor-subordinate ratings differences, and the percentage of employees

receiving the lowest rating, suggesting that multi-source feedback can increase ratings

discriminability.11

11

To examine whether employee-level factors influence these findings, we repeated the tests using two additional

sets of variables. First, we included the five firm-level system user variables as additional controls. The only

significant variable was Pct. Executives, which was negatively associated with ratings skew, average supervisor-

subordinate ratings difference, and the standard deviation in ratings difference. However, the inclusion of these

variables had no effect on our plan characteristic results. Second, we included the number of significant relations

from the firm-specific tests in Table 6 to examine whether firms that tailor their appraisal practices for individual

employees exhibit differences in ratings biases. Separate variables were computed for number of goals, number of

- 39 -

6. Conclusions

This large-sample study uses actual performance appraisal data to examine the relations

between firm- and employee-level factors and a variety of performance appraisal plan

characteristics, as well as the relations between these plan characteristics and performance

ratings distributions. Our aggregated firm-level tests indicate that appraisal practices are

influenced by organizational structure and environmental volatility, but provide little evidence

that these practices are associated with industry competition, past performance, or overall

employee characteristics. However, when we estimate separate firm-specific models, we find

that employee attributes such as organizational level, tenure, gender, and position play

significant roles in shaping appraisal plan characteristics within firms. Further analysis indicates

that firm-level attributes moderate the relations between individual employees’ characteristics

and many attributes of their appraisal plans. Finally, the various appraisal plan characteristics are

associated with differences in some types of performance rating biases, suggesting that firms can

minimize these biases through their choice of plan attributes.

There are three important limitations to our study. First, in order to maintain client

confidentiality, the software company did not provide us with the actual measures and goals in

the appraisal plans. As a result, we do not know whether the goals are objective or subjective, a

major attribute of performance appraisal. Second, we do not know the employees’ specific

positions or functions, which are likely to influence appraisal plan characteristics. Third, we do

not know the process used to determine overall performance ratings (e.g., formula-based or

subjective). Notwithstanding these limitations, the detailed employee-level data from a diverse

plans, Percentage of goals aligned, has rated goals, and receives multiple performance measures. The values ranged

from 0 to 5 depending upon the number of coefficients on the employee-level variables that were statsticially

significant. None of these variables was significant in the bias tests and their inclusion did not alter our reported

inferences.

- 40 -

set of firms allows us to examine the extent to which firms have adopted various appraisal

practices that have recently been advocated by academics and practitioners, and to provide some

of the first comprehensive evidence on the determinants of a broad set of appraisal plan

characteristics. More importantly, the multi-level nature of the data allows us to incorporate the

nesting of individual-level factors within firm-level factors in our analysis. The multi-level

results demonstrate that hierarchical modeling is an important methodological consideration that

needs to be taken into account if we are to increase our understanding of the factors driving the

design and implications of management control practices both within and across firms.

- 41 -

REFERENCES

ADDISON, J. AND C. BELFIELD. “The Determinants of Performance Appraisal Systems: A Note (Do

Brown and Heywood’s Results for Australia Hold Up in Britain?”. British Journal of Industrial

Relations 46 (2008): 521-531.

ARYA, A., J. GLOVER, B. MITTENDORF, AND L. YE. “On the Use of Customized Versus Standardized

Performance Measures.” Journal of Management Accounting Research 17 (2005): 7-21.

ASTLEY, W. G. “Organizational Size and Bureaucratic Structure.” Organization Studies 6 (1985):

201-228.

AUSTIN, J. AND P. BOBKO. “Goal-setting Theory: Unexplored Areas and Future Research Needs”.

Journal of Occupational Psychology 58 (1985): 289-308.

R. BANKER, H. CHANG, AND M. PIZZINI. “The Balanced Scorecard: Judgmental Effects of

Performance Measures Linked to Strategy”. The Accounting Review 79 (2004): 1-23.

BAYO-MORIONES, A., J.E. GALDON-SANCHEZ, AND S. MARTINEZ-DE-MORENTIN. “Performance

Appraisal: Dimensions and Determinants”. IZA Discussion Paper No. 5623 (2012).

BOHN, H. “Monitoring Multiple Agents: The Role of Hierarchies”. Journal of Economic Behavior

and Organization 8 (1987): 279-305.

BOL, J. “Subjectivity in Compensation Contracts”. Journal of Accounting Literature 27 (2008): 1-

32.

BOSWELL, W. AND J. BOUDREAU. “How Leading Companies Create, Measure and Achieve Strategic

Results through “Line of Sight,”. Management Decision 39 (2001): 851-859.

BROWN, M. AND J. HEYWOOD. “Performance Appraisal Systems: Determinants and Change”. British

Journal of Industrial Relations 43 (2005): 659-679.

BRETZ, R. G. MILKOVICH, AND W. READ. “The Current State of Performance Appraisal Research and

Practice: Concerns, Directions, and Implications”. Journal of Management 18 (1992): 321-352.

BRUCE, S. 2013. “Performance Management Survey Results In; How Do You Compare?”. HR Daily

Advisor, June 13.

BRUTUS, S. “Words Versus Numbers: a Theoretical Exploration of Giving and Receiving Narrative

Comments in Performance Appraisal”. Human Resource Management Review 20 (2010): 144-

157.

BOURGEOIS, L.J. “Strategic Goals, Perceived Uncertainty, and Economic Performance in Volatile

Environments”. Academy of Management Journal 28 (1985): 548-573.

- 42 -

CHENG, M., P. LUCKETT, AND H. MAHAMA. “Effect of Perceived Conflict among Multiple

Performance Goals and Goal Difficulty on Task Performance”. Accounting and Finance 47

(2007): 221-242.

CHENHALL, R.H. “The Contingent Design of Performance Measures”. In A. Bhimani (Ed.)

Contemporary Issues in Management Accounting. Oxford: Oxford University Press (2006): 92-

116.

CHENHALL, R. AND D. MORRIS. “The Impact of Structure, Environment, and Interdependence on the

Perceived Usefulness of Management Accounting Systems”. The Accounting Review 61 (1996):

16-35.

CHONG, V. “Management Accounting Systems, Task Uncertainty and Managerial Performance: A

Research Note”. Accounting, Organizations and Society 21 (1986): 415-421.

CHRISTIE, A., M. JOYCE, AND R. WATTS. “Decentralization of the Firm: Theory and Evidence”.

Journal of Corporate Finance 9 (2003): 3-36.

DENISI, A.S. “Performance Appraisal and Performance Management: A Multilevel Analysis”. In

K.J. Klein and S. Kozlowski (Eds), Multilevel Theory, Research and Methods in Organizations

(2000). San Francisco: Jossey-Bass: 121-156.

DESS, G. AND D. BEARD. “Dimensions of Organizational Task Environments”. Administrative

Science Quarterly 29 (1984): 52-73.

EMSLEY, D. “Multiple Goals and Managers’ Job-Related Tension and Performance”. Journal of

Managerial Psychology 18 (2003): 345-356.

FARR, J., J. FAIRCHILD, AND S. CASSIDY. 2014. “Technology and Performance Appraisal. The

Psychology of Workplace Technology, M. Coovert and L. Foster Thompson (Eds.), New York:

Routledge: 77-93.

FELTHAM, G. AND J. XIE. “Performance Measure Congruity and Diversity in Multi-Task

Principal/Agent Relations”. The Accounting Review 69 (1994): 429–453.

GIBBS, M., K. MERCHANT, W. VAN DER STEDE, AND M. VARGUS. “Performance Measure Properties

and Incentive System Design”. Industrial Relations 48 (2009): 237-264.

GOLDIN, CLAUDIA. “Monitoring Costs and Occupational Segregation by Sex: A Historical Analysis”.

Journal of Labor Economics 4 (1986): 1-27.

GOLMAN, R. AND S. BHATIA. “Performance Evaluation Inflation and Compression”. Accounting,

Organizations and Society 37 (2012): 534-543.

- 43 -

GRINT, K. “What’s Wrong with Performance Appraisals? A Critique and a Suggestion”. Human

Resource Management Journal 3 (1993): 61-77.

GRUND, C. AND D. SLIWKA. “The Anatomy of Performance Appraisals in Germany”. International

Journal of Human Resource Management 20 (2009): 2049-2065.

HENEMAN, R. “The Relationship between Supervisory Ratings and Results-Oriented Measures of

Performance: A Meta-Analysis”. Personnel Psychology 39 (1986): 811-826.

HOQUE Z. AND W. JAMES. “Linking Balanced Scorecard Measures to Size and Market Factors:

Impact on Organizational Performance”. Journal of Management Accounting Research 12

(2000): 1-17.

HOQUE, Z., L. MIA, AND M. ALAM. “Market Competition, Computer- Aided Manufacturing and Use

of Multiple Performance Measures: An Empirical Study”. British Accounting Review 33 (2001):

23-45.

HOLMSTROM, B. “Moral Hazard and Observability”. Bell Journal of Economics 10 (1979): 74-91.

ITTNER, C. AND D. LARCKER. “Determinants of Performance Measure Choices in Worker Incentive

Plans”. Journal of Labor Economics 20 (2002): S58-S90.

ITTNER, C., D. LARCKER, AND M. MEYER. “Subjectivity and the Weighting of Performance

Measures: Evidence from a Balanced Scorecard”. The Accounting Review 78 (2002): 725-758.

JAWAHAR, I. AND C. WILLIAMS. “Where All the Children are Above Average: the Performance

Appraisal Purpose Effect”. Personnel Psychology 50 (1997): 905-925.

JOSHI, M., R. KATHURIA, AND S. PORTH. “Alignment of Strategic Objectives and Performance: an

Integration of Operations and Strategic Management Perspectives”. Journal of Operations

Management 21 (2003): 353-369.

JIRJAHN, U. AND G. STEPHAN. “Gender, Piece Rates and Wages: Evidence from Matched Employer-

Employee Data”. Cambridge Journal of Economics 28 (2004): 683-704.

JOVANOVIC, B. “Job Matching and the Theory of Turnover”. Journal of Political Economy 87

(1979): 972-990.

JUDGE, T. AND G. FERRIS. “Social Context of Performance Evaluation Decisions”. Academy of

Management Journal 36 (1993): 80-105.

KAPLAN AND NORTON. The Balanced Scorecard: Translating Strategy into Action. Boston, MA:

Harvard Business School Press, 1996.

KAPLAN AND NORTON. Alignment: Using the Balanced Scorecard to Create Corporate Synergies.

Boston, MA: Harvard Business School Press, 2006.

- 44 -

KELLERMANNS, F., J. WALTER, S. FLOYD, C. LECHNER, AND J. SHAW. “To Agree or Not to Agree? A

Meta-Analytical Review of Strategic Consensus and Organizational Performance”. Journal of

Business Research 64 (2001): 126-133.

KERNAN, M.C. AND R.G. LORD. “Effects of Valence, Expectancies, and Goal-Performance

Discrepancies in Single and Multiple Goal Environments”. Journal of Applied Psychology 75

(1990): 194-203.

LANDY, F. AND J. FARR. “Performance Rating”. Psychological Bulletin 87 (1980): 72-107.

LATHAM, G., M. BUDWORTH, B. YANAR, AND G. WHYTE. “The Influence of a Manager’s Own

Performance Appraisal on the Evaluation of Others”. International Journal of Selection and

Assessment 16 (2008): 220-228.

LAZEAR, E. “WHY IS THERE MANDATORY RETIREMENT?” Journal of Political Economy 87 (1979):

1261-1284.

LAZEAR, E. “AGENCY, EARNINGS PROFILES, PRODUCTIVITY, AND HOURS RESTRICTIONS”. American

economic Review 71 (1989): 606-620.

LEE, C.-L. AND H.-J. YANG. “Organization Structure, Competition and Performance Measurement

Systems and their Joint Effects on Performance”. Management Accounting Research 22 (2011):

84-104.

LEVY, P. E., & WILLIAMS, J. R. “The Social Context of Performance Appraisal”. Journal of

Management 30 (2004): 881–905.

LEWIS, J.B., LINZER, D.A. “Estimating Regression Models in Which the Dependent Variable is

Based on Estimates”. Political Analysis 13 (2005): 345-364.

LOCKE, E. “Linking Goals to Monetary Incentives”. Academy of Management Executive 18 (2004):

130-133.

LOCKE, E. AND G. LATHAM. “Building a Practically Useful Theory of Goal Setting and Task

Motivation”. American Psychologist 57 (2002): 705-717.

LUFT, J., M. SHIELDS, AND T. THOMAS. “Multiple Performance Measures, Motivated Reasoning, and

Subjective Performance Measures”. Working paper, Michigan State University, 2010.

LYNESS, K. AND E. CORNELIUS III. “A Comparison of Holistic and Decomposed Judgment Strategies

in a Performance Rating Simulation”. Organizational Behavior and Human Performance 29

(1982): 21-38.

MACLEOD, W.B. “On Optimal Contracting with Subjective Evaluation.” Working paper, University

of Southern California, 2001.

- 45 -

MIA, L. AND M. GOYAL. “Span of Control, Task Interdependence and Usefulness of MAS

Information in Non-Profit Government Organizations”. Financial Accountability & Management

7 (1991): 249-266.

MOERS, F. “Discretion and Bias in Performance Evaluation: The Impact of Diversity and

Subjectivity”. Accounting, Organization and Society 30 (2005): 67-80.

MURPHY, K. R. AND J. CLEVELAND. 1995. Understanding Performance Appraisal. Thousand Oaks,

CA: Sage.

PRENDERGAST, C. “The Provision of Incentives in Firms”. Journal of Economic Literature 37

(1999): 7-63.

PULAKOS, E. AND R. O’LEARY. “Why is Performance Management Broken?” Industrial and

Organizational Psychology 4 (2011): 146-164.

RUBENFELD, S. AND J. DAVID. “Multiple Employee Incentive Plans: Too Much of a Good Thing?”

Compensation & Benefits Review 38 (2006): 35-40.

SAXONHOUSE, G.R. “Estimated Parameters as Dependent Variables”. American Economic Review 66

(1976): 178-183.

SHAH, J., R. FRIEDMAN, AND A. KRUGLANSKI. “Forgetting All Else: On the Antecedents and

Consequences of Goal Shielding”. Journal of Personality and Social Psychology 83 (2002):

1261-1280.

SKAPINKER, L. 2012. Why cascading goals doesn’t work. http://work.com/blog/2012/07/why-

cascading-goals-doesnt-work/.

SUUTARI, V. AND M. TAHVANAINEN. “The Antecedents of Performance Management among Finnish

Expatriates”. International Journal of Human Resource Management 13 (2002): 55-75.

VOELPEL, S., M. LEIBOLD, AND R. ECKHOFF. “The Tyranny of the Balanced Scorecard in the

Innovation Economy”. Journal of Intellectual Capital 7 (2006): 43-60.

WONG-ON-WING, B., L. GUO, AND W. LI, AND D. YANG. “Reducing Conflict in Balanced Scorecard

Evaluations”. Accounting, Organizations and Society 32 (2007): 363-377.

WORLDATWORK AND SIBSON CONSULTING. 2010. “2010 Study on the State of Performance

Management”. Washington, D.C.: WordatWork and The Siegal Group.

WRIGHT, P., T. GARDNER, L. MOYNIHAN, AND M. ALLEN. “The relationship between HR practices

and firm performance: examining causal order”. Personnel Psychology 58 (2005): 409-446.

- 46 -

Figure 1

An illustration of Performance Goal Cascading and Alignment

Source: B. Kolar. 2011. Goal Setting: Cascading versus Avalanching, February 24,

http://leaderquest.blogspot.com/2011/02/goal-setting-cascading-versus.html

- 47 -

Figure 2

Performance Appraisal Ratings Practices

Firms in our sample follow a number of approaches to providing performance ratings to employees, and

these approaches can vary across employees or employee groups. Some firms do not rate individual goals,

instead providing only one or more overall rating type. All three overall rating types need not be

provided. Common patterns are providing only an overall rating; providing a separate overall goal

objective rating (or overall competency rating in the subsample of firms where competencies are a

component of performance appraisals) together with an overall rating; or providing individual goal ratings

together with an overall rating.

- 48 -

Figure 3

Empirical Densities of Appraisal Plan Characteristics

This Figure presents histograms (and superimposed kernel density) of the Average Number of Goals, Average

Number of Plans, Percent Has Aligned Goal, Percent Has Rated Goal, and Percent Multiple Ratings, which are

defined in the caption of Table 3.

Average Number of Goals Average Number of Plans

Percent Has Aligned Goal

Average Percent Aligned Goals

Percent Has Rated Goal Percent Multiple Ratings

- 49 -

Table 1

Comparison of Proprietary Sample versus Compustat

This table presents a comparison of the industry composition (Panel A) and firm characteristics (Panel B) of the 153

firms in the proprietary sample and the Compustat population during fiscal year 2008. The variables in Panel B are

defined as follows. Employees is the total number of employees at the fiscal year end (EMPt). Employee Growth is

the percentage change in the total number of number of employees from the previous fiscal year ((EMP t/EMPt-1)–1).

Total Assets is total assets at the fiscal year end (ATt). Market Capitalization is the number of common shares

outstanding multiplied by the price per share at the fiscal-year end (CSHOt*PRCC_Ft). Sales is total annual sales

during the fiscal year (SALEt). Sales Concentration is calculated as the uncorrected sum of squares of total annual

sales divided by the squared sum of total annual sales of all firms in the same one-digit SIC during fiscal year 2008.

Return on Assets is net income scaled by average total assets (NIt/((ATt+ATt-1)/2)). Earnings Volatility is the

standard deviation of Prior ROA during the previous five years. Book-to-market is Total Shareholders’ Equity

divided by Market Capitalization, both measured at the fiscal year end ((ATt–LTt)/(CSHOt*PRCC_Ft)). Leverage is

long-term debt divided by Total Assets, both measured at the fiscal year end (LTt/ATt). R&D Expense is annual

research and development expense scaled by total annual sales (XRDt/SALEt) and missing values of research and

development expense are set to zero. Prior Return is the cumulative stock price return over the previous twelve

months. Return Volatility is the standard deviation of monthly stock returns over the previous 36 months.

Panel A: Industry Composition

Proprietary Sample Compustat

Industry Number Frequency Number Frequency

1. Mining and Construction 1 0.7% 200 2.8%

2. Food 1 0.7% 145 2.0%

3. Textiles, Printing, and Publishing 13 8.5% 221 3.1%

4. Chemicals 5 3.3% 169 2.4%

5. Pharmaceuticals 13 8.5% 476 6.7%

6. Extractive Industries 2 1.3% 304 4.3%

7. Durable Manufacturers 35 22.9% 1,237 17.5%

8. Computers 13 8.5% 369 5.2%

9. Transportation 6 3.9% 444 6.3%

10. Utilities 10 6.5% 328 4.6%

11. Retail 10 6.5% 519 7.3%

12. Financial Institutions 12 7.8% 1,040 14.7%

13. Insurance and Real Estate 4 2.6% 349 4.9%

14. Services 28 18.3% 1,122 15.9%

15. Other 0 0.0% 152 2.1%

Total 153 100% 7,075 100.00%

- 50 -

Table 1 (cont’d)

Panel B: Descriptive Statistics

Proprietary Sample Compustat

Industry Mean Median Mean Median

Employees 23,451 4,275 9,559 613

Employee Growth 0.11 0.03 1.28 0.01

Total Assets 15,101 2,625 12,033 389

Market Capitalization 4,789 1,391 3,011 150

Sales 6,163 1,571 3,388 186

Sales Concentration 0.06 0.04 0.23 0.16

Prior ROA 0.01 0.04 -3.04 0.01

Earnings Volatility 0.05 0.03 1.87 0.04

Book-to-Market 0.73 0.62 -23.88 0.72

Leverage 0.20 0.17 0.35 0.10

R&D Expense 0.11 0.01 3.54 0.00

Prior Return -0.37 -0.39 -0.70 -0.54

Return Volatility 0.71 0.66 0.14 0.12

- 51 -

Table 2

Factor Structure for Principal Components Analysis of Appraisal Plan Characteristics

This table presents the factor structure for the Principal Components Analysis (Oblimin Rotation with Kaiser

Normalization) of the goals and alignment measures. All variables are defined in the caption of Table 3. The

analysis produced four factors with eigenvalues greater than one, and the resulting factors are labeled Goal Number

Factor, Goal Plan Factor, Alignment Factor, and Rated Goal Factor, respectively.

Goal Number

Factor

Goal Plan

Factor

Alignment

Factor

Rated Goal

Factor

Avg. Number of Goals 0.865 0.380 -0.086 -0.077

Std. Dev. Number of Goals 0.928 0.235 -0.133 0.159

Avg. Number Plans 0.103 0.919 0.179 -0.200

Std. Dev. Number Plans -0.060 0.885 0.065 -0.005

Avg. Mean Goals per Plan 0.896 -0.331 -0.245 0.061

Std. Dev. Mean Goals Per Plan 0.872 -0.319 -0.244 0.256

Avg. Aligned 0.134 -0.102 -0.979 -0.110

Std. Dev. Aligned 0.179 -0.176 -0.934 -0.126

Pct. Has Aligned 0.204 -0.018 -0.973 -0.129

Pct. Multiple Ratings -0.128 0.249 -0.190 -0.660

Pct. Has Rated Goal -0.054 -0.072 -0.058 -0.857

Eigenvalue 3.679 2.039 2.569 1.098

% Variance Explained 33.4% 18.5% 23.4% 10.0%

Cronbach coefficient alpha 0.894 0.717 0.949 0.510

- 52 -

Table 3

Descriptive Statistics

This table presents descriptive statistics (mean, standard deviation, and 10th

, 25th

, 50th

, 75th

, and 90th

percentiles) for

goals and alignment variables (Panel A) and user characteristics and system usage (Panel B) for the 153 sample

firms. Avg. Number Goals is the average annual number of goals per system user. Std. Dev. Number Goals is the

standard deviation of the annual number of goals per system user. Avg. Number Plans is the average number of

different plans per system user at the firm during the year. Std. Dev. Number Plans is the standard deviation of the

number of different plans per system user at the firm during the year. Avg. Mean Goals Per Plan is the firm-level

average of its system users’ mean number of goals per goal plan. Std. Dev. Mean Goals Per Plan is the firm-level

standard deviation of its system users’ mean number of goals per goal plan. Avg. Aligned is the average number of

system users’ goals that are aligned. Std. Dev. Aligned is the standard deviation of the system users’ goals that are

aligned. Number Plans is the number of different plans at the firm during the year. Pct. Has Rated Goal is the

percentage of the firm’s system users that have a rated performance goal. Pct. Multiple Ratings is the percentage of

the firm’s users who receive multiple performance ratings. Number of Departments is the number of departments at

the firm that use the goal-setting software. Number of Divisions is the number of divisions at the firm that use the

goal-setting software. Number of Levels is the number of levels in the hierarchy of goal-setting software users.

Number of Users is the number of goal-setting software users. Pct. Male is the percentage of system users who are

male. Pct. Managers is the percentage of system users who are managers. Avg. Reports is the firm-level average

number of subordinates per manager. Pct. Users is the percentage of the firms’ total employees who use are system

users. Pct. Executives is the percentage of system users who are in either the first or second level of the hierarchy.

Months Used is the number of months experience with the software. Compensation Module is an indicator that takes

a value of one if the firm uses the compensation module and zero otherwise. Mean Rating is the average overall

rating in the firm, Median Rating is the median overall rating, Ratings Skew is the skew in overall ratings, Std. Dev.

in Ratings is the standard deviation in overall ratings, Pct. Min. Rating and Pct. Max. Rating are the percentages of

ratings that are in the lowest and highest rating categories respectively, Avg. Rating Difference is the average

difference in ratings between supervisors and subordinates, and Std. Dev. Ratings Difference is the standard

deviation in performance supervisor-subordinate ratings differences.

Panel A: Goals and Alignment

Mean Std. Dev.

10th

Percentile

25th

Percentile

50th

Percentile

75th

Percentile

90th

Percentile

Avg. Number Goals 6.52 3.08 3.77 4.87 5.95 7.43 9.42

Std. Dev. Number Goals 3.53 2.07 1.57 2.25 2.94 4.31 5.72

Avg. Number Plans 1.25 0.44 1.00 1.00 1.00 1.31 1.90

Std. Dev. Number Plans 0.20 0.24 0.00 0.00 0.06 0.40 0.49

Avg. Mean Goals per Plan 5.42 2.21 3.05 4.05 5.30 6.29 7.89

Std. Dev. Mean Goals Per Plan 2.89 1.76 1.19 1.65 2.46 3.73 4.97

Avg. Aligned 0.17 0.17 0.00 0.00 0.13 0.29 0.42

Std. Dev. Aligned 0.23 0.16 0.00 0.04 0.28 0.38 0.42

Pct. Has Aligned 0.25 0.24 0.00 0.01 0.20 0.45 0.57

Number Plans 1.94 1.25 1.00 1.00 2.00 2.00 3.00

Pct. Has Rated Goal 0.55 0.41 0.00 0.00 0.80 0.93 0.97

Pct. Multiple Ratings 0.42 0.48 0.00 0.00 0.01 1.00 1.00

- 53 -

Table 3 (cont’d)

Panel B: User Characteristics and System Usage

Mean Std. Dev.

10th

Percentile

25th

Percentile

50th

Percentile

75th

Percentile

90th

Percentile

Number of Departments 297 500 19 43 129 305 707

Number of Divisions 28.61 90.31 2.00 5.00 11.00 22.50 40.50

Number of Levels 7.99 1.67 6.00 7.00 8.00 9.00 10.00

Number of Users 2672 3443 205 473 1193 3482 7143

Pct. Male 0.62 0.16 0.39 0.51 0.65 0.75 0.78

Pct. Managers 0.24 0.08 0.16 0.18 0.23 0.28 0.33

Avg. Span of Control 21.18 16.32 8.50 13.42 18.12 25.57 33.43

Pct. Users 0.49 0.35 0.03 0.18 0.44 0.84 0.97

Pct. Executives 0.01 0.02 0.00 0.00 0.01 0.02 0.03

Months Used 36.82 16.14 17.00 24.00 35.00 47.00 57.00

Compensation Module 0.45 0.50 0.00 0.00 0.00 1.00 1.00

Panel C: Performance Rating Distributions

Mean Std. Dev.

10th

Percentile

25th

Percentile

50th

Percentile

75th

Percentile

90th

Percentile

Mean Rating 0.58 0.12 0.41 0.52 0.60 0.65 0.70

Median Rating 0.56 0.14 0.39 0.50 0.54 0.66 0.73

Ratings Skew 0.00 0.61 -0.85 -0.29 0.07 0.44 0.76

Std. Dev. in Ratings 0.17 0.05 0.11 0.13 0.16 0.19 0.23

Pct. Max. Rating 0.09 0.14 0.00 0.01 0.04 0.11 0.27

Pct. Min. Rating 0.05 0.13 0.00 0.00 0.01 0.03 0.15

Avg. Rating Difference 0.03 0.05 -0.02 0.01 0.03 0.05 0.08

Std. Dev. Ratings Difference 0.18 0.06 0.10 0.14 0.18 0.22 0.26

- 54 -

Table 4

Correlation Matrix of Firm-Level Appraisal Plan Factors and Firm Characteristics

This table presents the correlations between the four factors (i.e., Goal Number Factor, Goal Plan Factor, Alignment Factor, and Rated Goal Factor) and firm

characteristics. Pearson product-moment (Spearman rank-order) correlations are presented above (below) the diagonal and correlations that are significant at less

than the 10% level are bold. The four factors are as defined in Table 3. Volatility Factor is the sum of the standardized values of (1) Earnings Volatility, R&D

Expense, and (3) Return Volatility. Growth Factor is the sum of the standardized values of (1) New Investment, defined as capital expenditures plus acquisitions

less sales of property and depreciation, all scaled by average total assets (CAPXt + AQCt – SPPEt – DPCt)/((ATt+ATt-1)/2), (2) Sales Growth ((SALEt/SALEt-1)-

1), and (3) Employee Growth. Avg. Tenure is the average number of months the system users have worked at the firm. Comp Seat Usage is an indicator that takes

a value of one if the firm uses the system’s Compensation Module and zero otherwise. The remaining variables are defined in the caption of Table 2.

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20) (21) (22)

(1) Goal Number Factor 0.01 0.21 -0.13 0.08 0.16 0.05 0.09 -0.07 -0.05 -0.05 0.12 -0.09 -0.07 -0.05 0.21 0.00 -0.06 0.06 -0.05 -0.11 0.17

(2) Goal Plan Factor -0.06 -0.12 0.11 -0.01 0.07 -0.05 0.11 -0.03 -0.09 0.07 0.04 0.06 0.07 0.00 0.01 -0.07 -0.05 0.02 -0.07 0.12 0.03

(3) Alignment Factor 0.36 -0.06 0.16 0.05 0.04 -0.08 0.05 0.06 -0.20 0.01 0.07 0.07 -0.08 0.07 0.02 -0.09 -0.09 0.00 0.19 -0.03 0.18

(4) Rated Goal Factor -0.06 0.06 0.17 -0.09 0.11 0.02 -0.03 0.10 -0.12 0.01 0.08 -0.09 0.13 -0.06 0.11 -0.19 0.02 -0.10 0.08 0.06 -0.09

(5) log(Market Cap) 0.04 -0.08 0.10 -0.07 0.26 0.22 0.34 0.11 -0.39 -0.02 -0.40 -0.09 0.36 0.38 -0.21 -0.03 -0.20 0.07 0.17 0.04 0.24

(6) log(Number Departments) 0.17 0.00 0.05 0.13 0.29 0.36 0.42 0.19 -0.24 -0.02 -0.01 -0.22 -0.06 0.17 0.18 -0.30 -0.40 -0.04 -0.02 -0.04 0.26

(7) log(Number Divisions) 0.08 0.02 -0.04 0.01 0.22 0.37 0.36 0.22 -0.12 -0.07 0.10 -0.09 -0.03 -0.06 0.19 0.04 -0.20 0.06 0.03 -0.01 0.16

(8) log(Number Levels) 0.09 0.06 0.05 -0.03 0.42 0.39 0.34 0.23 -0.19 0.03 -0.02 -0.23 -0.07 0.11 0.23 -0.04 -0.40 -0.07 0.03 0.01 0.08

(9) Avg. Span of Control 0.09 0.03 0.23 0.14 0.29 0.42 0.36 0.54 -0.13 0.22 -0.02 -0.17 -0.13 0.10 -0.03 0.09 -0.06 0.10 -0.03 0.03 0.01

(10) Volatility Factor 0.05 -0.05 -0.18 -0.21 -0.49 -0.18 -0.09 -0.23 -0.15 0.07 -0.03 -0.06 -0.07 -0.52 0.31 0.05 0.19 0.04 -0.45 0.11 -0.14

(11) Growth Factor -0.02 -0.02 -0.04 -0.05 0.11 -0.06 -0.08 0.01 0.16 0.08 -0.10 -0.06 -0.01 0.16 0.01 0.04 0.02 0.11 -0.20 0.11 -0.07

(12) Book-to-Market 0.03 0.19 0.03 0.10 -0.27 -0.05 0.05 -0.09 0.00 -0.02 -0.18 -0.04 -0.43 -0.24 0.04 -0.10 -0.12 -0.07 0.15 -0.16 0.07

(13) Sales Concentration -0.06 0.13 0.07 -0.04 -0.18 0.01 -0.11 -0.06 -0.11 0.01 -0.08 -0.02 -0.05 0.00 -0.21 0.11 0.07 0.02 0.09 -0.10 -0.04

(14) Prior Return -0.10 0.01 -0.04 0.14 0.34 -0.02 -0.05 -0.02 -0.05 -0.31 0.11 -0.41 -0.14 0.11 -0.03 0.13 0.15 -0.01 0.04 0.17 -0.07

(15) Prior ROA -0.09 -0.04 0.04 -0.06 0.35 0.17 -0.07 0.11 0.12 -0.29 0.31 -0.42 0.07 0.21 -0.27 -0.02 -0.13 -0.06 0.12 -0.07 0.04

(16) Pct. Users 0.24 -0.01 0.03 0.15 -0.21 0.21 0.21 0.22 0.23 0.36 0.01 -0.09 -0.22 -0.04 -0.23 -0.25 -0.08 -0.09 -0.27 0.11 -0.01

(17) Pct. Managers -0.02 -0.01 -0.09 -0.23 -0.10 -0.36 -0.10 -0.10 -0.40 -0.03 -0.05 -0.07 0.03 0.18 -0.05 -0.30 0.34 -0.02 0.03 0.08 -0.22

(18) Pct. Executive -0.06 0.09 0.00 -0.05 -0.39 -0.54 -0.36 -0.55 -0.48 0.13 0.01 0.04 0.00 0.07 -0.19 -0.16 0.54 -0.02 -0.11 0.01 -0.32

(19) Pct. Male 0.10 0.01 -0.01 -0.14 0.07 -0.05 0.05 -0.06 0.07 0.10 0.16 0.09 -0.03 -0.01 -0.01 -0.05 -0.08 -0.08 0.11 0.04 0.18

(20) Avg. Tenure -0.08 -0.04 0.11 0.12 0.22 0.07 0.09 0.11 0.09 -0.51 -0.24 0.17 -0.04 0.04 0.05 -0.31 -0.01 -0.12 0.12 -0.06 0.11

(21) Compensation Module -0.08 0.03 -0.01 0.06 0.02 -0.03 0.00 0.00 0.01 0.06 0.12 -0.03 -0.18 0.17 -0.05 0.11 0.04 0.01 0.06 -0.04 -0.08

(22) log(Months Used) 0.20 0.00 0.15 -0.13 0.28 0.29 0.18 0.16 0.19 -0.05 -0.05 -0.06 0.00 -0.06 0.03 0.05 -0.26 -0.33 0.17 0.07 -0.08

- 55 -

Table 5

Firm-Level Tests of the Determinants of Performance Appraisal Plan Characteristics

This table presents estimates from Tobit regressions (the first two Factors are left-censored, the third Factor is right-

censored, and the fourth Factor is both left- and right-censored) of the 153 sample firms. Alignment and Goal

Factors constructed based on the Principal Components Analysis described in Table 2 and the independent variables

are defined in the captions of Tables 2, 3, and 4. Statistical significance at the 0.01, 0.05, and 0.10 levels is denoted

***, **, and *, respectively.

Goal Number

Factor

Goal Plan

Factor

Alignment

Factor

Rated Goal

Factor

Firm Complexity

Log(Market Capitalization) 0.50** -0.37 0.13 0.09

(2.076) (-1.219) (0.507) (1.041)

Log(Number of Departments) 0.44* 0.03 0.01 -0.14*

(1.832) (0.092) (0.054) (-1.666)

Log(Number of Divisions) -0.48 -0.13 -0.82** -0.08

(-1.647) (-0.352) (-2.577) (-0.751)

Log(Number of Levels) -1.20 5.81** -0.29 1.26**

(-0.698) (2.432) (-0.159) (2.049)

Avg. Span of Control -0.02 -0.03 0.03 -0.00

(-0.924) (-1.011) (1.640) (-0.121)

Operating Uncertainty

Volatility Factor -0.13 -0.48* -0.35** -0.12**

(-0.835) (-1.946) (-2.093) (-2.156)

Growth Factor 0.02 0.12 0.06 0.01

(0.142) (0.763) (0.477) (0.247)

Book-to-Market 0.61 0.66 -0.14 -0.27*

(1.526) (1.272) (-0.322) (-1.889)

Competitive Environment

Competition

Sales Concentration -3.51 9.19 7.20 -0.27

(-0.658) (1.363) (1.268) (-0.139)

Prior Return -0.89 2.03 -0.61 -0.43

(-0.810) (1.400) (-0.524) (-1.104)

Prior ROA -2.52 -2.68 -2.86 0.12

(-0.916) (-0.712) (-0.956) (0.123)

System User Characteristics

Pct. Users 2.94*** -0.73 1.42 0.43

(2.951) (-0.562) (1.332) (1.204)

Pct. Managers 8.67** -10.00 -1.40 0.64

(2.105) (-1.631) (-0.310) (0.438)

Pct. Executive -0.06 20.73 -2.53 -1.28

(-0.005) (1.075) (-0.164) (-0.254)

Pct. Male 2.35 1.04 -0.37 0.25

(1.318) (0.444) (-0.195) (0.392)

Avg. Tenure -0.00 -0.02* 0.00 0.00

(-0.110) (-1.697) (1.518) (0.966)

System Usage

Compensation Module -0.94* 0.81 0.14 0.34*

(-1.709) (1.138) (0.245) (1.745)

Log(Months Used) 0.95 0.62 1.12 0.57**

(1.428) (0.717) (1.570) (2.392)

Pseudo Adj. R-squared 0.035 0.036 0.031 0.074

- 56 -

Table 6

Aggregated Firm-Specific Regressions of Employee-Level Plan Features on Employee

Characteristics

This table presents summary statistics of firm-specific, employee-level regressions of various plan characteristics

variables on employee characteristics. The dependent variable in Panel A is the natural logarithm of the employee’s

number of annual goals. The dependent variable in Panel B is the number of plans in which the employee

participates. The dependent variable in Panel C is the proportion of the employee’s goals that are aligned. The

dependent variable in Panel D is the proportion of the employee’s goals that are rated. The dependent variable in

Panel E is an indicator variable that takes a value of one if the employee only receives an overall performance rating

and zero otherwise. Male is an indicator that takes a value of one if the employee is male and zero otherwise.

Executive is an indicator that takes a value of one if the employee is in the first or second level in the organizational

hierarchy. Relative Level in Firm is the level of the employee in the organizational hierarchy scaled by the number

of levels in the organizational hierarchy, excluding executive; higher scores represent employees higher in the

hierarchy. Tenure is the natural logarithm of the number of months the employee has worked at the firm. All Reports

is the natural logarithm of one plus the number of total subordinates who report to the employee. Mean Coefficient

and Median Coefficient are the mean and median values of the firm-specific coefficients, respectively. Std. Error is

the standard error of the coefficient estimates. Z-stat is the standardized z-statistic of the coefficient estimates.

Statistical significance at the 0.01, 0.05, and 0.10 levels is denoted ***, **, and *, respectively. Number Positive

(Significant) and Number Negative (Significant) are the number of coefficient estimates that are positive and

negative, respectively, and the number that are significantly different from zero at the 10% level in parentheses.

Number Zero is the number of coefficients that are zero because there is no within-firm variation in its employees’

Number of Goals. Sign-test (p-value) is the p-value of a sign test that the median number of coefficients is different

from zero.

Panel A: Number of Goals

Male Executive

Relative

Level in Firm Tenure Reports

Mean Coefficient 0.02 -0.20 0.01 0.03 0.06

Median Coefficient 0.01 -0.16 0.02 0.02 0.05

Std. Error 0.01 0.03 0.04 0.01 0.01

Z-stat 2.06* -6.80*** 0.19 5.80*** 11.20***

Number Positive (Significant) 88 (42) 38 (5) 45(72) 110 (77) 128 (98)

Number Negative (Significant) 65 (26) 105 (53) 49(80) 42 (12) 24 (5)

Number Zero 0 0 0 0 0

Sign-test (p-value) 0.075 0.000 0.570 0.000 0.000

Panel B: Number of Plans

Male Executive

Relative

Level in Firm Tenure Reports

Mean Coefficient -0.01 -0.05 0.07 0.03 0.01

Median Coefficient -0.01 -0.02 -0.00 0.02 0.00

Std. Error 0.01 0.04 -0.06 0.01 0.01

Z-stat -0.76 -1.10 1.17 3.20*** 1.43

Number Positive (Significant) 32 (7) 30 (11) 22(41) 58 (35) 42 (23)

Number Negative (Significant) 51 (16) 50 (18) 27(42) 25 (14) 41 (19)

Number Zero 70 70 70 70 70

Sign-test (p-value) 0.048 0.033 1.000 0.000 1.000

- 57 -

Table 6 (cont’d)

Panel C: Percentage of Goals Aligned

Male Executive

Relative

Level in Firm Tenure Reports

Mean Coefficient 0.02 -0.04 -0.14 0.01 0.00

Median Coefficient 0.02 -0.02 -0.06 0.00 0.00

Std. Error 0.01 0.02 -0.03 0.00 0.00

Z-stat 2.19** -2.22** -4.31*** 1.50 0.75

Number Positive (Significant) 80 (43) 40 (11) 54(76) 70 (37) 63 (35)

Number Negative (Significant) 37 (11) 70 (13) 22(40) 47 (15) 54 (18)

Number Zero 36 36 36 36 36

Sign-test (p-value) 0.000 0.005 0.001 0.042 0.460

Panel D: Has Rated Goals

Male Executive

Relative

Level in Firm Tenure Reports

Mean Coefficient 0.01 -0.35 -0.17 0.02 0.00

Median Coefficient 0.00 -0.34 -0.13 0.01 0.00

Std. Error 0.01 0.03 -0.05 0.00 0.00

Z-stat 0.93 -10.61*** -3.84*** 5.33*** -0.67

Number Positive (Significant) 61 (17) 18 (3) 63(83) 82 (52) 48 (26)

Number Negative (Significant) 52 (14) 91 (64) 15(30) 30 (9) 64 (27)

Number Zero 40 40 40 40 40

Sign-test (p-value) 0.452 0.000 0.000 0.000 0.156

Panel E: Receives Multiple Performance Ratings

Male Executive

Relative

Level in Firm Tenure Reports

Mean Coefficient 0.01 -0.19 -0.08 0.01 0.01

Median Coefficient -0.00 -0.09 -0.02 0.01 -0.00

Std. Error -0.01 -0.03 0.04 -0.01 -0.00

Z-stat 1.47 -5.88*** -2.20** 2.80*** 1.50

Number Positive (Significant) 8(40) 31(61) 27(12) 6(20) 10(33)

Number Negative (Significant) 10(37) 2(12) 50(31) 33(56) 17(43)

Number Zero 76 76 76 76 76

Sign-test (p-value) 0.820 0.000 0.012 0.000 0.302

- 58 -

Table 7

Summary of Significant Coefficients from Two-Step Hierarchical Estimation Examining

the Determinants of Variations in Individual Employees’ Goal Plan Characteristics

This table summarizes the statistically significant results from the cross-sectional regressions presented in the Appendix. The

dependent variables are the coefficient estimates obtained from the following firm-specific, employee-level regressions of five

Plan Characteristics (i.e., Number of Goals, Number of Plans, Percent Aligned, Rated Goals, and Multiple Ratings) on five

employee-level characteristics (i.e., Male, Executive, Relative Level, Tenure, and Reports):

Plan Characteristici = β0 + β1Malei + β2Executivei + β3RelativeLeveli + β4Tenurei + β5Reportsi + εi

where i denotes an individual user of the firm’s performance appraisal system, the independent variables are the aggregated firm-

level variables used in the firm-level tests in Table 5 (variable definitions are provided in Tables 2, 3, and 4), and standard errors

are adjusted as described in the text to account for the use of an estimated dependent variable. Panels A – E of this table report

the statistically significant coefficient estimates from those regressions corresponding to one of five categories of independent

variables: Firm Complexity (Panel A), Operating Uncertainty (Panel B), Competitive Environment (Panel C), System User (Panel

D), and System Usage (Panel E). Plus (+) and minus (-) signs denote that the firm-level independent variable has a significant

positive or negative association (p < 0.10, two-tailed) with the extent to which the firm varies that plan characteristic with

differences in the specified employee attribute. For example, (+) Number Goals in the column adjacent to Male in the

Moderated by Market Cap column of Panel A indicates that firms with larger market capitalizations tend to give more goals to

their male employees than to their female employees.

Panel A: Organization Structure Variables Moderated by

Market Cap

Moderated by

Divisions

Moderated by

Departments

Moderated by

Levels

Moderated by

Span of Control

Male (+) Number Goals

(+) Number Plans

(+) Rated Goals

(+) Multiple Ratings

(-) Number Plans

(-) %Aligned

(-) Rated Goals (-) Number Goals

Executive (+) Number Goals

(+) %Aligned

(+) Rated Goals

(-) Number Plans

(-) %Aligned

(-) Rated Goals

(-) Multiple Ratings

(-) Rated Goals

Relative Level (+) %Aligned

(+) Rated Goals

(+) Multiple Ratings

(-) Number Plans

(+) Rated Goals

(-) Multiple Ratings

(+) %Aligned

(-) Rated Goals

Tenure (+) Rated Goals

(+) Number Plans

(-) %Aligned

(-) Number Plans

(+) Number Plans

Reports (+) Rated Goals (+) Number Goals (+) Number Plans

(+) Rated Goals

Panel B: Operating Uncertainty Variables

Moderated by

Volatility

Moderated by

Book-to-Market

Moderated by

Growth

Male (-) Number Goals

(-) Number Plans

(-) %Aligned

(+) Number Goals

(+) Multiple Ratings

(+) Number Goals

(+) Number Plans

(+) Rated Goals

Executive (-) Number Goals (+) Number Goals

(-) %Aligned

(+) Rated Goals

(-) Multiple Ratings

Relative Level (+) Multiple Ratings (+) %Aligned

Tenure (-) Number Goals

(-) Rated Goals

(+) Number Goals

Reports (-) Number Goals

(+) Number Plans

(-) Rated Goals

(+) Number Goals

(+) Number Plans

(-) %Aligned

(-) Multiple Ratings

(-) Number Plans

- 59 -

Table 7 (cont’d)

Panel C: Competitive Environment Variables

Moderated by

Prior ROA

Moderated by

Prior Return

Moderated by

Sales Concentration

Male (-) Number Plans

(-) Rated Goals

Executive (-) Number Goals

Relative Level (-) Multiple Ratings (+) Multiple Ratings (+) Number Plans

Tenure (+) %Aligned

(-) Rated Goals

(+) Number Plans

(-) Rated Goals

(+) Number Goals

(+) Multiple Ratings

Reports (+) Number Plans

(-) %Aligned

(-) Rated Goals

(+) Number Goals

(+) Number Goals

Panel D: System User Variables

Moderated by

%Users

Moderated by

%Male

Moderated by

%Executives

Moderated by

%Managers

Moderated by

Average Tenure

Male (+) Number Goals (+) %Aligned

(+) Rated Goals

(+) Rated Goals

Executive (+) Number Goals (+) Multiple Ratings

(-) Number Goals

(-) %Aligned

(-) Rated Goals

Relative Level (+) %Aligned

(+) Rated Goals

(+) Number Goals

(-) Number Plans

(-) %Aligned (-) Multiple Ratings

(-) Multiple Ratings

(+)Multiple Ratings

Tenure (+) Number Goals

(+) Multiple Ratings

(+) %Aligned (-) Number Goals

(-) Rated Goals

Reports (+) Number Goals (+) Number Plans (+) Rated Goals

(+) Number Plans (+) Number Plans

Panel E: System Usage Control Variables

Moderated by

Compensation

Moderated by

Months Used

Male (-) Number Plans

(+) %Aligned

Executive (-) Multiple Ratings (-) Number Goals

Relative Level (-) Multiple Ratings (-) Multiple Ratings

Tenure (+) Number Plans

Reports (+) Number Goals

(+) Number Plans

(-) Rated Goals

- 60 -

Table 8

The Associations Between Firm-Level Appraisal Plan Characteristics and Performance Rating Distributions

This table examines the associations between the four firm-level appraisal plan factors and the firm-level distributions of overall performance ratings. Mean

Rating is the average overall rating in the firm, Median Rating is the median overall rating, Ratings Skew is the skew in overall ratings, Std. Dev. in Ratings is the

standard deviation in overall ratings, Pct. Min. Rating and Pct. Max. Rating are the percentages of ratings that are in the lowest and highest rating categories

respectively, Avg. Rating Difference is the average difference in ratings between supervisors and subordinates, and Std. Dev. Ratings Difference is the standard

deviation in performance supervisor-subordinate ratings differences. Statistical significance at the 0.01, 0.05, and 0.10 levels is denoted ***, **, and *,

respectively.

Mean

Rating

Median

Rating

Ratings

Skew

Std. Dev.

in Ratings

Pct. Max.

Rating

Pct. Min.

Rating

Avg. Rating

Difference

Std. Dev. Ratings

Difference

Intercept 0.579*** 0.550*** 0.143 0.201*** 0.176*** 0.101*** 0.056*** 0.252***

Goal Number Factor -0.005 -0.005 0.034** 0.000 -0.001 0.005 -0.001 0.002*

Goal Plan Factor 0.001 0.002 -0.012 0.002* 0.007* 0.009** -0.002 0.000

Alignment Factor 0.003 0.003 -0.002 -0.003** -0.001 -0.002 0.000 -0.003*

Rated Goal Factor 0.004 0.006 -0.070* -0.006** -0.026*** -0.023*** -0.006** 0.000

Number Rating Points 0.006 0.011 -0.058 -0.011*** -0.028*** -0.020** -0.007*** -0.023***

Compensation Module -0.039* -0.047* 0.150 -0.016** -0.034 0.008 -0.005 -0.016*

360 Module -0.011 -0.013 -0.136 0.022*** 0.055** 0.016 -0.011 0.018*

Adj. R-squared 0.0% 0.0% 2.6% 16.8% 12.0% 9.6% 2.7% 28.1%

- 61 -

Appendix

Two-Step Hierarchical Estimation Results

This appendix provides results from two-step hierarchical modelling of the moderating influence of firm-level factors on the association between employee attributes and their appraisal plan

characteristics. The dependent variables are the coefficient estimates obtained from the following firm-specific, employee-level regressions of five Plan Characteristics (i.e., Number of Goals, Number of Plans, Percent Aligned, Rated Goals, and Multiple Ratings) on five employee-level characteristics (i.e., Male, Executive, Relative Level, Tenure, and Reports):

Plan Characteristici = β0 + β1Malei + β2Executivei + β3RelativeLeveli + β4Tenurei + β5Reportsi + εi

where i denotes an individual user of the firm’s performance appraisal system, the independent variables are the aggregated firm-level variables used in the firm-level tests in Table 5 (variable definitions are provided in Tables 2, 3, and 4), and standard errors are adjusted as described in the text to account for the use of an estimated dependent variable. Panels A – E of this table report

regressions examining the extent to which the relation between a given appraisal characteristics and employee attribute are moderated by the firm-level variables in the left-hand column. For example,

the significant coefficient of 0.011 in the first cell indicates that larger firms give their male employees more goals.

Panel A: Number of Goals

Panel B: Number of Plans

Male Executive

Relative

Level Tenure Reports Male Executive

Relative

Level Tenure Reports

Firm Complexity

Log(Market Capitalization) 0.011** 0.117*** 0.035 0.000 -0.005 0.008* 0.051 0.001 0.006 -0.002

Log(Number of Departments) -0.001 -0.019 0.028 -0.002 0.003 0.000 0.011 0.017 -0.015*** 0.001

Log(Number of Divisions) -0.002 -0.022 -0.016 -0.002 -0.002 -0.009*** -0.055** -0.151*** -0.007* 0.001

Log(Number of Levels) -0.046 0.014 0.197 0.009 0.076*** 0.007 0.231 0.008 0.105* -0.016

Avg. Span of Control -0.001** 0.002 -0.003 -0.000 0.000 -0.000 -0.005 -0.004 -0.000 0.001***

Operating Uncertainty

Volatility Factor -0.008** -0.053* 0.012 -0.007** -0.006** -0.008** 0.006 -0.010 -0.005 0.003**

Growth Factor 0.005** 0.006 -0.007 0.000 -0.003 0.003** -0.015 0.025 0.001 -0.001*

Book-to-market 0.010* 0.126** 0.064 0.009** 0.009* -0.012 0.108 -0.070 0.022 0.013**

Competitive Environment

Sales Concentration -0.092 0.236 0.185 0.067 0.263** 0.089 -1.145 1.754* 0.324** -0.068

Prior Return -0.023 0.087 -0.153 0.005 0.030* -0.031 0.126 0.007 0.088*** 0.008

Prior ROA -0.060 -0.942** 0.012 -0.029 0.021 -0.127*** -0.183 -0.785 -0.096 0.071***

System User Characteristics

Pct. Users 0.035* 0.241* 0.135 0.035** 0.024 0.014 -0.153 -0.192 0.016 0.006

Pct. Managers 0.109 -0.834 0.149 0.044 0.050 0.050 -0.945 -1.840 -0.229 0.120***

Pct. Executives 0.313 2.533 0.650 -0.135 0.439 0.040 5.338 -2.154 0.846 0.643*

Pct. Male 0.034 -0.391* 0.397** 0.028 0.018 -0.013 -0.218 -0.563* -0.063 0.007

Avg. Tenure

0.000 -0.001 0.001 -0.000*** 0.000 -0.000 -0.001 0.001 -0.000** 0.000**

System Usage

Compensation Modules 0.011 0.005 -0.068 -0.001 0.018** -0.027*** -0.067 -0.059 0.006 0.012***

Log(Months Used) -0.003 -0.141* 0.078 0.001 0.001 -0.008 -0.000 0.153 0.029** 0.009**

Adj. R-squared 0.218 0.286 0.233 0.406 0.639 0.310 0.210 0.602 0.245 0.308

- 62 -

Appendix (cont’d)

Panel C: Percentage Aligned

Panel D: Rated Goals

Male Executive Relative

Level Tenure All

Reports Male Executive Relative

Level Tenure All

Reports

Firm Complexity

Log(Market Capitalization) 0.002 0.043** 0.026** -0.001 0.000 0.006*** 0.042* 0.118*** 0.004** -0.002

Log(Number of Departments) -0.002 -0.040** 0.004 -0.000 -0.000 0.000 -0.038* -0.005 -0.001 0.003*

Log(Number of Divisions) -0.007** 0.012 -0.021 -0.004* 0.001 -0.004 0.016 0.100*** 0.001 -0.002

Log(Number of Levels) -0.025 0.011 -0.066 0.013 -0.006 -0.039** 0.318 0.130 0.006 -0.003

Avg. Span of Control 0.000 -0.002 0.001* 0.000 0.000 0.000 -0.007** -0.007*** -0.000 0.000*

Operating Uncertainty

Volatility Factor -0.006** -0.025 0.003 0.001 -0.001 -0.001 -0.032 0.012 -0.006*** -0.004**

Growth Factor 0.001 -0.007 0.004 -0.000 -0.000 0.004*** 0.005 0.015 -0.001 -0.001

Book-to-market -0.000 -0.095** 0.091*** 0.002 -0.009*** 0.004 0.060* 0.080 0.003 0.003

Competitive Environment

Sales Concentration 0.012 0.031 0.224 -0.029 0.026 0.026 0.999 -0.426 0.159* 0.035

Prior Return 0.001 -0.015 0.005 -0.003 0.003 -0.021** 0.087 -0.034 -0.020* -0.007

Prior ROA -0.028 -0.254 -0.030 0.039* -0.039* -0.016 -0.267 -0.292 -0.048* -0.054**

System User Characteristics

Pct. Users 0.022** 0.016 0.086* 0.010* 0.000 0.002 0.051 0.252** 0.013 0.003

Pct. Managers 0.013 -0.832** -0.041 0.019 -0.013 0.005 -0.674 0.799 0.044 0.009

Pct. Executives -0.111 0.549 0.335 0.015 -0.049 0.307 4.792 -1.127 0.156 0.689**

Pct. Male -0.031 -0.192 -0.201** 0.005 0.002 0.031** -0.082 -0.317 -0.001 0.034***

Avg. Tenure

0.000 0.000 0.000 0.000 -0.000 -0.000 -0.002** 0.000 -0.000** -0.000

System Usage

Compensation Module 0.007 -0.048 -0.027 0.003 -0.002 -0.007 -0.062 0.005 0.001 -0.005

Log(Months Used) 0.020*** 0.003 0.032 0.005 0.000 -0.007 -0.072 -0.082 -0.008 -0.009*

Adj. R-squared 0.260 0.245 0.234 0.185 0.169 0.298 0.193 0.428 0.306 0.275

- 63 -

Appendix (cont’d)

Panel E: Multiple Performance Ratings

Male Executive

Relative

Level Tenure

All

Reports

Firm Complexity

Log(Market Capitalization) 0.006* 0.013 0.053** 0.002 -0.002

Log(Number of Departments) 0.001 -0.018 0.033 -0.002 0.001

Log(Number of Divisions) -0.002 0.023 0.037 0.002 0.001

Log(Number of Levels) -0.002 -0.193* -0.289* -0.028 -0.005

Avg. Span of Control 0.000 -0.002 0.002 0.000 0.000

Operating Uncertainty

Volatility Factor 0.001 -0.011 0.107*** -0.002 0.000

Growth Factor 0.001 -0.012* 0.003 0.000 0.000

Book-to-market 0.007* -0.007 0.232*** -0.002 -0.004*

Competitive Environment

Sales Concentration 0.066 -0.576 0.268 0.226** -0.005

Prior Return 0.008 -0.069 0.420*** -0.007 -0.010

Prior ROA -0.002 0.024 0.958*** -0.009 0.006

System User Characteristics

Pct. Users -0.007 0.178** -0.034 0.022* 0.003

Pct. Managers 0.000 -0.135 -1.904*** 0.002 0.040

Pct. Executives 0.120 3.471 2.588 0.348 -0.193

Pct. Male 0.020 -0.069 -0.444** 0.003 0.000

Avg. Tenure

-0.000 -0.001 0.003*** 0.000 0.000

System Usage

Compensation Module 0.009 -0.126*** -0.174*** -0.008 -0.001

Log(Months Used) -0.005 -0.024 -0.212*** -0.003 0.002

Adj. R-squared 0.170 0.265 0.736 0.257 0.123