RESEARCH AND PRACTICE
IN HUMAN RESOURCE MANAGEMENT

Highlight, copy & paste to cite:

Manoharan, T. R., Muralidharan, C. & Deshmukh, S. G. (2009). Employee Performance Appraisal Using Data Envelopment Analysis: A Case Study, Research and Practice in Human Resource Management, 17(1), 92-111.

Employee Performance Appraisal Using Data Envelopment Analysis: A Case Study

T. R. Manoharan, C. Muralidharan & S. G. Deshmukh

Abstract

Evaluating and ranking the employees working in organisations are challenging tasks involving several factors. Each employee achieves certain performance levels in various factors and the resulting information can be overwhelming. This paper demonstrates how data envelopment analysis (DEA) can be applied as a fair evaluating and sorting tool to support the performance appraisal (PA) as well in the decision making process. DEA focuses on the best practices of efficient employees for the purpose of improving overall performance. Unlike traditional performance appraisals DEA searches for the efficient employees who will serve as peers. The DEA process identifies inefficient employees, magnitude of inefficiency and aids to eliminate inefficiencies with a relatively easy to employ framework. This study supports the ideas that rating formats need reexamination with a focus on computer based models as an alternative to traditional rating methods. Earlier adopted methods have seldom identified and quantified the individual factors for inefficiency whereas DEA could overcome these shortfalls. Based on the results of DEA the improvement of employees’ performance are possible by way of providing training, talent enhancement and further qualification wherever required.

Introduction

In the wake of the industrial developments and to increase the productivity at the same time to maintain quality strategic tools are vital to evaluate the practices of the employees. Performance appraisal (PA) is an important management tool to assess employees’ efficiency in the workplace, and may be defined (Pearce & Porter 1986), as a structured formal interaction between a subordinate and supervisor that usually takes the form of a periodic interview (annual or semi annual) to evaluate the work performance. PA is intended to engage, align, and coalesce individual and group effort to continually improve overall organisational mission accomplishment (Grubb 2007). It provides a basis for identifying and correcting disparities in performance. Thus, it is activities oriented and is a rational, formalised, legitimate test using observation and judgment. Systematically, PA reviews each employee’s work performance during a specific period, evaluates and records it for future reference. Essentially, weaknesses and strengths of individuals are examined and discussed to identify opportunities in view of establishing improvement and skills development. Many authors (Oberg 1972, Colby & Wallace 1975) have pointed out the shortcomings of existing appraisal systems. They have noted that many appraisal systems are: (a) not relevant to organisational objectives, (b) subject to personal bias, and (c) are often influenced more heavily by personality than by performance. With a view to eliminate these shortcomings, an attempt with a computer based tool called Data Envelopment Analysis (DEA) is used in the present work to evaluate the performance. Nevertheless, several have agreed that well designed and properly used appraisal systems are essential for effective functioning of organisations (Slusher 1975).

For a long period, PA has been one of the most researched topic areas in the fields of Industrial- Organisational Psychology and Human Resource Management. The face of traditional Human Resource (HR) services in progressive organisations is currently undergoing a dramatic change. According to Renton (2000), the focus in future will be much more on guiding and implementing business strategy and much less on managing established HR functions. As a result, professionals in the HR field are increasingly being challenged to take a more strategic perspective on their role in the organisation. According to Becker, Huselid and Ulrich (2001), as HR professionals respond to this challenge, measuring HR performance consistently emerges as a key theme. Grote (2002) indicated PA enables managers to mobilise the energy of the people to achieve strategic goals. A PA system can tightly link strategy (mission, vision and values) with daily performance. Many companies pay close attention to the hard science of performance measurement, particularly the financial and operational sides, while successful companies play equal attention to the art, which is the softer aspect of selecting and applying performance measures (Singh & Finn 2003).

An effective PA programme should do more than set salary and promotion decisions on past performance. It should aid in the development of a performance improvement plan that utilises coaching from the department supervisor or manager to increase skills development. This puts it in the same category as training, which is all about looking ahead and developing practical programmes that result in improved performance. The interest is perceptible in as much as organisations appraise employee performance for a variety of purposes. PA forms the foundation for many HR functions, effectively setting the standards to drive recruiting efforts, and it is customary to use these criterion in hiring, promoting, evaluating and equitably compensating employees, and forming the basis for many employee training programmes (Gibson, Harvey & Harris 2007).

There is an obvious need for tools to improve the PA process. A great deal of effort has been made in this direction to develop suitable software tools, which can act as ‘consultants’ for managers. The advances in computer technology and the computer based techniques for handling information allow the development of decision support systems that can play a crucial role in the progress of a firm (Alexouda 2005).

Organisations are preparing reports or profiles periodically for each employee. The profiles include detailed information regarding wages/salaries costs, utilisation of resources, and outcome quality (e.g., cost per product, and reworking rates). These indicators are compared against performance in other organisations. Multiple factors involved profiling can effectively identify underutilisation of inputs, uncover problems with the efficiency and quality of work, and assess an individual’s performance (Sherman 1984). These profiles are designed to generate a specification, if the performance indicators for a particular employee differ from the average by a certain amount. And appraisal results are used, either directly or indirectly, to determine reward outcomes (Colby & Wallace 1975) as well as to identify the poor performers who may require some form of counselling, or training, or in extreme cases, demotion, dismissal or a decrease in pay. Since employees consume sizable portions of investment, the better management of employees can have a significant impact on the overall efficiency of the organisation.

This paper describes the use of DEA to improve methods of measuring employees’ efficiency for a small manufacturing industry. The objectives of current study are fivefold.

The structure of this paper is as follows. The first section presents the literature review, the traditional appraisal techniques with their advantages and limitations. The next section outlines the DEA and variant models. In the third second a methodology delineates the study site, the respondents, and the measures that were used to capture data. This section closes with an outline how these data were evaluated. In the fourth section the results explain the procedure for ranking employees. A discussion component, the fifth section, expresses how the study findings relate to the current relevant literature, and the last section offers conclusions in terms of the relevance of the study findings for HRM policies and practices in contemporary organisations.

Literature Review

This PA system is schematically illustrated in Figure 1. In general, the PA (Scheneier, Richard & Lloyd 1986) is concerned with three possible measures namely assessing results, behaviours, and personal characteristics. Each dictates a specific type of appraisal format based on competency or job related behaviour. These forms of appraisals are made by single or multi rater (two or more of supervisor/ peer/self/subordinate/outsider).

Figure 1
Schematic representation of performance appraisal system
Schematic representation of performance appraisal system

Table 1 explains the classification of the traditional methods of performance appraisal. It is based on qualitative features, quantitative dimensions and is objective in nature. The former two elements take the category of either an absolute or a relative standard. These forms of appraisals are normally made by a supervisor, team members, peers, self, a subordinate or even an outsider. Organisation managements establish performance standards and devise instruments and methods that can be used to measure and appraise an employee’s performance. The traditional measurement methods are explained along with their advantages and limitations in Table 2.

Table 1
Classification of performance appraisal methods
Appraisal by Qualitative Quantitative Objective
AS RS AS RS
Supervisor Essay appraisal
Critical incident appraisal
Behaviourally anchored rating scale
360 degree feedback
Group order ranking
Individual ranking
Graphic rating scale Paired comparison
Team Check list
Forced choice
360 degree feedback
Management by objective (MBO)
Self Essay appraisal
360 degree feedback
Peer Essay appraisal
360 degree feedback
Group order ranking Graphic rating scale Paired comparison
Subordinate Essay appraisal

Note: AS = Absolute Standard and RS = Relative Standard.

Table 2
Traditional measurement techniques, advantages and limitations.
Methods Description of the methods Advantages Limitations
1. Essay appraisal (Ref. 1) The rater writes a narrative description on employee’s strengths, weaknesses, past performance, potential, and suggestions for improvement.
  • Simple.
  • No complex forms or extensive training is needed.
  • Unstructured and likely to vary widely from person to person.
  • Difficult to compare individuals.
  • Provide qualitative data
2. Critical incident appraisal (Ref. 1) It focuses the rater’s attention on those critical or key behaviours that make the difference between doing a job effectively and doing ineffectively. The rater writes anecdotes that describe what the employee did that was especially effective or not.
  • Looks at behaviours.
  • Describes which behaviours are desirable and which are to be improved.
  • Time consuming and burdensome method.
  • Ranking of subordinates is difficult.
3. Checklist (Ref. 1) The evaluator uses a list of behavioural descriptions and checks off those behaviours that apply to the employee. The evaluator merely goes down the list and gives yes or no responses. Since the rater and the scorer are different, it reduces bias. May be inefficient if there are a number of job categories
4. Graphic rating scale (Ref. 2) Each employee characteristic is rated on a scale that has several points ranging from poor to excellent. The nature and scope of the character is limited by the organisation’s need to know.
  • Less time consuming
  • Presents quantitative analysis and comparison.
  • Comparability in diverse job categories is possible.
Does not provide the depth of information.
5. Forced choice (Ref. 1) It is a special type of checklist and the rater has to choose between two or more statements which are more descriptive of the individual being evaluated. The personnel department scores the answers based on a key. Reduces bias and distortion.
  • Tends to be disliked by appraisers. Raters may become frustrated, as they do not know what represents a good or bad answer.
6. Behaviourally anchored rating scales.(Ref. 3) The appraiser rates the employees based on items along a continuum, but the points are examples of actual behaviour on the given job. These behavioural examples are then retranslated into appropriate performance dimensions. Tend to reduce rating errors. Its major advantage stems from the dimensions generated.
  • Time consuming method.
7. Group order ranking (Ref. 1) The evaluator places employees into a particular classification, such as top one fifth. Prevents raters from inflating their evaluations. The disadvantage surfaces when the number of employees being compared is small.
8. Individual ranking (Ref. 1) Listing the employees in an order from higher to lowest. Only one can be best. Prevents raters from inflating their evaluations. The disadvantage surfaces when the number of employees being compared is small.
9. Paired comparison (Ref. 1) A score is assigned to each employee by simply counting the number of pairs in which the individual is the preferred member. It ranks each individual in relationship to all others on a one-on-one basis. Each employee is compared against every other. Can become unwieldy when large numbers of employees are being compared.
10. Management by Objectives (Ref. 1) It divides organisational objectives into individual objectives. It seeks to measure employee performance by examining the extent to which predetermined work objectives have been met.
  • Emphasis results oriented.
  • It assists the planning and control functions and provide motivation.
  • Takes an inordinate amount of the manager’s time.
  • Increase the volume of paperwork in an organisation.
11. 360 degree feedback (Ref. 4&5) The feedback on employees’ activities would come from subordinates, peers, and managers in the organisational hierarchy, as well as self assessment, and in some cases external sources such as customers and suppliers or other interested stakeholders. ‘360’refers to the 360 degrees in a circle. The person receiving the feedback will plan the training and development process.
  • Gaining acceptance of the principle of multiple stakeholders as a measure of performance.
  • Supporting a climate of continuous improvement.
  • Confidentiality of reviews is less.
  • The reports may not be alike

Notes: a. Ref 1 = Decenzo, D. A., & Robbins S. P. (2003). Personnel/human resource management, EEE Publication, 363-389.
b. Ref 2 = Paterson, D. G. (1972). The Scott Company graphic rating scale, The Journal of Personnel Research, 1, 361-376.
c. Ref 3 = Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales, Journal of Applied Psychology, 47(2), 149-155.
d. Ref 4 = Yammarino, F. J., & Atwater, L. E. (1997). Do managers see themselves as others see them? Implications of self-other rating agreement for human resources management, Organisational Dynamics, 25(4), 35-44.
e. Ref 5 = Brutus, S., & Derayeh, M. (2002). Multisource assessment programs in organisations: An insider’s perspective. Human Resource Development Quarterly, 13(2), 187-202.

A review of the literature shows that performance evaluation systems are criticised for failing to achieve employees’ expectations. Despite the large body of published work on the subject of PA there are still gaps in empirical investigations of performance evaluation. The main components (such as employee participation, continuous improvement) of a quality driven HR performance evaluation are less widely researched (Soltani, Van Der Meer, Gennard & Williams 2003). According to Dulewicz (1989) there are shortcomings of the traditional performance evaluation in quality organisational environments.

The qualitative type PA methods (as shown in Table 1) involve various types of errors (Locher & Teel 1977). These blemishes are evidenced as leniency error, halo error, similarity error, central tendency, and low appraiser motivation. Other drawbacks include such features as the results are qualitative, employment of the method is time consuming and the methods incorporate burdensome and inefficient procedures, or have insufficient information, are appraiser dependent, and suitable for a small volume of data. The PA system with these flaws will lead to so many unwanted consequences, and can leave people bitter, crushed, despondent, dejected and feeling inferior, with some even depressed, and unfit for work for weeks after receipt of a poor rating. Further, while using these methods, some form of rating is derived, which may be a numerical score or a per cent, a category (e.g., outstanding, valued contributor, needs improvement), or a narrative descriptive summary evaluation. In all cases, the rating is at best a subjective and partial judgment of individual performance against specific prescribed criteria (Grubb 2007). This unease is taken care by DEA, which provides the result on employees’ performance as a single quantity, which is derived relatively, and not against any specific prescribed criteria.

The inclusion of qualitative factors is one of the novelties of DEA. DEA is technique for assessing the relative efficiency of comparable units with a view to improving their performance. Differences among like units exist, but they are measurable (Golany & Roll 1989). The research articles by Golany and Roll (1989), and Cook and Zho (2006) use qualitative and quantitative data together to evaluate various performance measures in different environments. Allocation of values to phenomena using five point Likert scale and their mathematical manipulation, in DEA, as established by Cook and Zho (2006), have been used. Surrogates are used to convert the qualitative data into quantitative information and are finalised with the consensus of managers, supervisors and representatives of employees. As the qualitative factors are reduced the subjectivity is also minimised.

Some of the limitations of traditional qualitative methods are overcome by traditional quantitative methods. However, traditional quantitative methods of PA have its own limitations. In these approaches input/output factors and ratios are individually considered. For example, a large number of rejections per given volume of raw material would stand as an indication of inefficiency. Nevertheless, indicators are limited to one measure of input and/or output only. The proposed DEA model of PA, which is based on linear programming, converts multiple input and output measures into a single, comprehensive measure of efficiency without requiring the relative weights of the measures be known a priori.

A literature review shows little treatment of HR performance as a quantitative approach. The proposed DEA model removes this concern and also elicits five other features.

Data Envelopment Analysis

Traditionally, PA or efficiency measurement has been a major managerial concern in both the manufacturing sector and the service industry. Consequently, a wide variety of methods are used to measure efficiency. One of the methods is Frontier approach, which evaluates efficiency against production functions. A production function defines the maximum levels of outputs attainable with a certain combination of inputs or the minimum possible level of inputs for certain level of outputs. The engineering based approach defines productivity by comparing the current performance to a suitable set of engineering standards (Sueyoshi 1992). In both these methods controversy arises when the analyst attempts to assign relative weights to factors. Thus, prior assumptions on weights have reservations, and this problem is eliminated in the use of DEA, as the weights are assigned voluntarily by the method.

DEA measures efficiency by estimating an empirical production function, which represents the highest values of outputs that could be generated by relevant inputs, as obtained from observed and input output vectors for the analysed Decision Making Units (DMU). The efficiency of a DMU is then measured by the distance from the point representing its input and output values to the corresponding reference point on the production function (Mohamed & Luc 2008). DEA defines the relative efficiency for each DMU (bank branches, employees in engineering teams, hospitals, schools) by comparing its input and output data to all other DMUs in the same cultural environment. In addition to relative efficiency measures, a DEA study provides the following four properties (Paradi, Smith & Schaffnit-Chatterjee 2002).

DEA is today one of the most successful methods of operational research with a wide range of applications and an extensive bibliography is available (Giokas & Pentzaropoulos 2000). For instance, Emrouznejad, Parker, and Tavares (2008), in their extensive searches, have identified more than 4000 research articles published in journals or book chapters. They also enlightened that the evolution of DEA as a worldwide accepted operations research / management science tool and has been tracked in terms of the increases of publications and applications. The results of DEA are relative performance measures. With respect to the efficiency frontier, which is built by the efficient DMUs, the amount of improvement required for the inefficient DMUs are determined. The flexibility of DEA has been demonstrated successfully in numerous performance appraisals in real environments.

DEA is an appropriate method of evaluation of employees. Apart from giving individuals an efficiency score DEA is also able to identify the following five features.

DEA is a powerful technique for performance measurement (Cook & Seiford 2009). There is considerable evidence of the strengths of DEA (Ramanathan 2003).

Thanassoulis (1995) has given an account of a DEA application to the assessment of policing performance. The application was in the context of a major study into crime management in England and Wales, undertaken by the Audit Commission. Assessing police forces using DEA has not only more confidence in the results obtained, but also performance on specific area such as manning levels as distinct from crime clear ups gained. It identified potentially weak and strong Forces on performance, their efficient peers and the levels of clear ups that would render inefficient Forces efficient.

By utilising good data from Bell Canada on engineering design teams, Paradi, Smith, and Chatterjee (2002), were able to offer valuable advice to management, based on both the mathematical power of DEA and the managerial input. Their paper presents the performance analysis of 39 access network engineering design personnel at Bell Canada using 1994 data. The limitations of traditional performance measurement approaches used by Bell were reviewed and contrasted with the benefits offered by DEA. The primary contribution of this study, from a managerial point of view, is a demonstration of the opportunity available to the company that by repeatedly applying these DEA models it could redraw the boundaries of its service areas and increase their efficiency.

There are three basic DEA models. These frameworks are: (a) CCR (Charnes, Cooper & Rhodes 1978), (b) BCC (Banker, Charnes & Cooper 1984), and (c) CCGSS (Charnes, Cooper, Golany, Seiford & Stutz 1985). These models have different mathematical formulations, but all share the principle of envelopment (Golany & Roll 1989). In DEA both output maximisation or input minimisation are possible. The present study focuses on the output maximisation BCC model to improve the efficiency of existing inefficient employees. The BCC model has two main elements.

The BCC Model

Indices: j – DMUs, j = 1, …, n
r – outputs, r = 1, …, t
i – inputs, 1, …, m
Data: yrj – the value of the rth output of the jth DMU
xij – the value of the ith input for the jth DMU
ε – a small positive number (non-Archimedean constant, order 10-5 or 10-6)
Variables: si, σr – slacks corresponding to input i, output r respectively (>= 0)
λj – weight of DMU in the facet for the evaluated DMU (>= 0)
μr, vj – virtual multipliers for output r, input i respectively (>= ε)
hk – relative efficiency of DMUk
uk – returns to scale, is an indicator interpreted by BCC

The linear programming problem (LPP) formulation for each individual DMU to solve is
Equation 1

The objective here is to find the largest sum of weighted outputs of individual while keeping the sum of its ratio of the sum of weighted outputs to the sum of weighted inputs for any individual to be less than one. This ratio corresponds to the classical engineering ratio definition of efficiency. Consequently, the dual formulation for DMU is written as and solved.

Equation 2

The objective function of this model attempts to find a minimal value for an intensity factor (θk) which indicates the potential of an individual. In addition, the objective function seeks the largest slack values in all input output dimensions. In other words, it finds the reference point on the empirical production function which portrays individual in the worst efficiency characterisation.

Methodology

Site

The motor vehicle sector in India is rapidly growing. For example, the annual sales of motorcycles is expected to cross the 10 million mark by 2010, and the car statistics indicate that India will soon become one of the top 10 car manufacturing countries. Expectedly, the car production capacity will exceed the mark of two million units at the end of this fiscal year. Sustaining this sector of the Indian economy provides the imperative to focus on the strategies to preserve company growth and profitability. Consequently, a major managerial challenge is to address employee performance.

The study site is a small company located in southern India, which is involved in the manufacturing of automobile parts. This company was established six years ago and is involved in manufacturing and supplying components of carburettors (for two and four wheelers) to a manufacturing firm. Its annual turnover is INR 1.2 millions. Sixteen different components for various types of carburettors are manufactured.

Respondents

The company employs 23 people. There are two managers under the managing director. One is in charge of manufacturing and the other has responsibility for sales/purchases. Under the control of the manager (manufacturing), there are two supervisors, one each per shift of eight hours of duty. For each shift nine employees are working who are engaged in metal machining using lathes, and drilling machines. All these 18 employees underwent a PA within a framework of DEA that provided data for this study. The managers and supervisors are not included for PA.

Procedure

The main focus of the study is to improve the working efficiencies of the employees and to determine their training needs. Employee rankings will be used to decide the types of incentives and promotions during future expansion of the company. The factors (dataset) considered for the evaluation process are classified into input and output factors. One of the major advantages of the DEA is the inputs and outputs can be measured and used in their own units (Sami-Mahgary & Lahdelma 1995). No universally applicable rational template is available for the selection of factors. However, in general, the inputs must reflect the resources used and the outputs must reflect the service levels of the utility and the degree to which the utility is meeting its objective (Richards 2003, Thakur 2005).

The dataset is decided upon, by having discussions and brainstorming sessions with the managers, supervisors and representatives of employees. While considering input and output factors the isotonicity relations are assumed for DEA (i.e., an increase in any input should not result in a decrease in any output). Consequently, the values of some factors may have to be inverted before they are entered into the analysis. Another group of factors is the qualitative ones. These have to be assigned numerical values in order to participate in the mathematical evaluation of efficiency. Any number of input or output factors which are relevant and have an impact on the efficiency of employees could be considered for DEA. But the number of employees in the analysis should be at least twice the number of inputs and outputs considered (Golany & Roll 1989).

Measures

To evaluate efficiency scores of employees the following factors are used: job knowledge, customer relations, interpersonal relations, and work habits as input factors; and quality, and quantity of products produced as output factors. Among the input factors customer relations and interpersonal relations are qualitative. In a wide range of problem settings to which DEA can be applied qualitative factors are often present. Marketing’s interest in consumer perception and expectation, and human resources’ desire to explore and describe employees’ skills are two areas that routinely involve the quantification of qualitative concepts (Dyson, Allen, Camanho, Podinovski, Sarrico & Shale 2001). Only quantitative measures are used in DEA hence, qualitative factors need to be converted into quantitative scores. Such factors may be legitimately quantifiable, but very often such quantification is superficially forced, as a modelling convenience. Typically, a qualitative factor is captured either on a Likert scale, or is represented by some quantitative surrogate such as plant downtime or percentage sick days by employees (Cook, Kress & Seiford 1996). Many authors, Roman, Wigand and Wolfgang (2003), Wong, Yang and Greatbanks (2004), Biehl, Cook and Jonston, (2006), Cook and Zho (2006) utilised a five point Likert scale to convert qualitative data into quantitative used for the evaluations of performance using DEA.

Years of experience of employees is considered to represent the job knowledge (Ross & Droge 2002) and work habits is measured with a surrogate, percentage of employees’ attendance. The qualitative input factors Customer Relations and Interpersonal Relations are assessed by using a five point Likert scale with high scores reflecting better relations. In the case of Customer Relations; 1 = school final, 2 = industrial training, 3 = diploma, 4 = degree, and 5 = post graduate; and Interpersonal Relations is measured using: 1 = fair, 2 = satisfactory, 3 = good, 4 = very good, and 5 = excellent.

The characterisation of the dataset in explained in Table 3.

Table 3
Characterisation of input and output factors
Dataset Factors Description Type Unit Source
Output Quality The ability to set high standards for own personal performance; strive for quality work; put forth extra effort to ensure quality work. Avoiding rejections and reworking (Ref 1). Quantitative Percentage of acceptable units produced Production records
Quantity Volume of work produced with speed, accuracy and consistency of output (Ref 1). Quantitative Production volume in numbers
Input Job knowledge Considers the degree of job knowledge relative to length of time in the current position (Ross & Droge, 2002, Ref 1). The level of job relevant knowledge and skill an employee has. It includes awareness, practices, manual skills and techniques (Ref 2). Quantitative Years of experience Experience
Customer relations The degree to which the employee takes the initiative to meet internal and external customer needs in a timely and courteous manner (Ref 1). If an employee fails to have good customer relations with fellow employee (Internal customer) problems arise which may include failure of co-ordination efforts, internal inefficiencies, and increased costs (Ref 2). Qualitative Five point Likert scale Education
Work habits How an employee maintains regular attendance, effective time management skills and meets productivity standards (Ref 2). Quantitative Percentage of attendance Work log book and production records
Interpersonal relations Interpersonal relations refers to the effect that the employee has on others, including their ability to establish and maintain positive and productive working relationships. Employees with high levels of interpersonal skill tend to inspire goodwill, cooperation, and mutual respects, accept and acknowledge suggestions, works cooperatively and effectively with others to achieve unit goals. Often involved in conflict and misunderstandings with peers, clients, supervisors and others are with low interpersonal skill (Ref 2). Qualitative Five point Likert scale History of employee

Notes: a. Ref 1 = www.hr.arizona.edu/, accessed 5 August 2007.
b. Ref 2 = www.performance-appraisal.com/home.htm, accessed 20 September 2007.

Analysis

Frontier Analyst software version 3 is used to analyse the data. A series of regression analyses (Min, Min & Joo 2008) for classifying a factor as input or output was conducted. The statistical details of the dataset used in this study are shown in Table 4. Any factor with a weak relation to inputs and strong relation to outputs indicates a preference towards classifying the factor as an input. A weak relation to all the factors may indicate a need to reexamine the factor. Alternatively, a strong relation may indicate that the information contained in that factor is already represented by other factors and again its participation should be reexamined. The regression analyses are helpful in eliminating redundancies and reducing the number of factors for investigation. The correlation between the factors of the DEA dataset is also presented in Table 4.

Table 4
Summary of statistical details of dataset
Parameter Mean Std Max Min Correlation Between Factors
X2 X3 X4 Y1 Y2
Job knowledge (X1) 4.030 1.658 6.00 1.5 0.94 0.28 0.29 0.64 0.74
Customer relations (X2) 0.830 0.127 1.00 0.6 0.44 0.35 0.67 0.80
Work habits (X3) 0.805 0.085 0.90 0.6 0.18 0.42 0.24
Interpersonal relations (X4) 0.863 0.070 0.95 0.7 0.28 0.09
Quality (Y1) 0.940 0.016 0.97 0.9 0.69
Quantity (Y2) 737.560 11.758 755.00 722.0

Once the consensus decision on dataset is taken by managers, supervisors and representatives of employees, the required data only need to be fed into the DEA software. Most of the data are quantitative and available from databases. There is an obvious need for such software tools to improve the performance appraisal process. Lots of efforts have been made in this direction to develop suitable software tools, which can act as consultants for managers (Alexouda 2005). The advances in computer technology and the computer based techniques for handling information allow the development of decision support systems that can play a crucial role in the progress of a firm. Even though the computational part seems to be cumbersome, the software used will take care of the same. The purpose of explanation of the method is to show how DEA works.

Results

Table 5 presents the results in the case study by BCC model. Employees with an efficiency score of 100 per cent are efficient and less than that value are categorised as inefficient. An RTS with 1 and 0 refers to increasing (IRS) and constant (CRS) return to scale, respectively. In IRS an increase of all inputs leads to more than proportionate increase of all outputs and CRS, increase of all inputs leads to a small increase of all outputs. The next column indicates the peers for underperformers and the next six columns indicate the level of slacks in input and output factors. How many times each employee is referred as peer is given in the last column. Overall, the content of Table 5 expresses the less efficient employees to be 1, 6, 9, 11, 13, 14, 16, and 18.

Table 5
Summary of outcomes across DMUs (BCC Model)
Emp Efficiency RTS Peers Slacks in Refs
X1 X2 X3 X4 Y1 Y2
1 99.44 1 5, 17 0.000 0.071 0.030 0.071 0.030 0.000 0
2 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 1
3 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 1
4 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 2
5 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 3
6 99.18 1 2, 3 0.000 0.000 0.000 0.055 0.023 0.094 0
7 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 1
8 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 1
9 99.91 1 7, 8 0.027 0.033 0.000 0.074 0.019 0.000 0
10 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 1
11 99.76 1 4, 15, 17 0.006 0.007 0.000 0.120 0.000 0.000 0
12 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 1
13 99.69 1 10, 15, 17 0.000 0.014 0.090 0.000 0.014 0.000 0
14 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 0
15 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 2
16 97.30 1 5, 12 0.166 0.000 0.000 0.000 0.046 0.000 0
17 100.00 0 0.000 0.000 0.000 0.000 0.000 0.000 4
18 98.96 1 4, 5, 17 0.027 0.000 0.052 0.000 0.000 0.392 0

The content of Table 5 has four main elements. First, the efficiency scores of each DMU (Emp n) are given in the second column. Employees (Emp 1, 6, 9, 11, 13, 16 and 18) with less than 100 percent efficiency are designated as inefficient. This efficiency is a relative term, and it is related with a subgroup of individuals (peers, in fourth column) with whom the examined DMU is best compared. For example, Emp 5 and 17 are peers for Emp 1. The identification of inefficient DMU and peers can be found by using efficiency frontier or by solving the LPP Equation (2). The efficiency frontier can only be shown graphically as either, two inputs and one output for output maximisation, or one input and two outputs for input minimisation. Adding more inputs or outputs the problem becomes multi dimensional and can no longer be shown graphically.

Second, the third column in Table 5 indicates the Return to Scale (RTS). Three possible values of RTS are 1, 0 and -1. The RTS values 1, 0 and -1 correspond to an increasing, constant and decreasing RTS respectively. If an increase of all inputs leads to more than proportionate increase of all outputs, or an increase of all inputs leads to less than proportionate increase of all outputs, then the DMU exhibits VRS. As the DMU changed its scale of operations could either result in an increase or decrease of efficiency. An increase of all inputs leads to a proportionate increase of all outputs then the DMU exhibits constant returns to scale.

Third, the slacks (in case of output shortfalls or input surpluses) for inefficient DMUs are given as non zero values in column 5 to column 10, in Table 5. The input surpluses show that the corresponding DMU’s input factors are in excess for the amount of their output results. The unutilised input factors should be suitably used to improve their outputs. For example, the Emp 6 has a slack of 0.055 in X4 (i.e., the outputs produced are comparatively less for the DMU job knowledge). The focus is to enhance the output, rather than decreasing the inputs, and hence, the DMU has to improve outputs.

Last, the column Refs indicates the number of times the DMU is peer referred. Each peer group possesses similar combinations of factors. The representation of DMU in more than one peer groups shows that the DMU has many desirable characters (Doyle & Green 1991). For example, Emp 5 is present in the peers groups of Emp 1, 16, and 18, while Emp 14 is not at all a peer to any other DMU. This is because no other employee is present in the efficiency frontier plane of Emp 14, which indicates that the input values of this DMU are comparatively higher than other peers for the outputs produced. In DEA terms it is identified as weakly efficient DMU (Ramanathan 2003). Employees in each peer groups are having similar and comparable inputs.

Ranking of Employees

DEA allows for complete weight flexibility (i.e., the weightages are assigned by DEA without any prior assumptions) in the evaluation of the efficiency score (i.e., column 2 in Table 5). This may result in identifying a DMU with an unrealistic (extreme) weighing scheme to be efficient (Dyson & Thannassoulis 1988). The DMUs with extreme weights have the potential of being false positive candidates. A false positive DMUs score reaches a relative efficiency of 100 per cent by weighing heavily on few favourable input/output factors, and ignoring the other factors. Such DMUs are performing well with respect to few input/output measures and they may not have good overall practices. Very often having an efficiency score of 100 per cent by DEA may not be completely true. Consequently, a measure more than the simple efficiency score, is required in the ranking process to overcome such problems.

Many ranking methods are available in DEA. Table 6 explains them with their applications. Each technique may be useful in a specific area, but no one methodology can be prescribed as the complete solution to the question of ranking. The application of cross efficiencies found at many places and it can effectively used to overcome the problems associated with simple efficiency scores for ranking (Serrano-Cinca, Fuertes-Callen & Mar-Molinero 2005).

Table 6
Types and applications of DEA ranking methods
Method Criteria Applied in
1. Cross efficiency The units are self and peer evaluated Many areas of manufacturing, including engineering design, flexible manufacturing systems, industrial robotics and business process reengineering also been used heavily in project and R&D portfolio selection.
2. Super efficiency Ranks through the exclusion of the unit being scored from the dual linear program A wide range of papers from financial institutions and industry to public regulation, education and healthcare.
3. Benchmarking DMU is highly ranked if it is chosen as a useful target for many other DMUs Extensively in the field of utilities, industry and agricultural productivity.
4. Multivariate and DEA Developed connection between multivariate statistical techniques and DEA Universities and industry
5. Inefficient units ranking Ranking of inefficient units through proportional measures of inefficiency In many areas of both the public and private sectors
6. MCDM and DEA Crosses multi criteria decision making models with DEA. Agriculture and oil.

Cross efficiencies of a DMU provide information on how well DMU is performing with the optimal DEA weights of remaining n - 1 DMUs. The cross efficiencies of all DMUs are specifically arranged in a cross efficiency matrix (CEM) as shown in Table 7. Doyle and Green (1994) had adopted an effective way of measuring the false positiveness of DMUs by evaluating false positive index (FPI) using the following equation. The FPI relates to the percentage increment in efficiency while a DMU achieves on moving from peer appraisal to self appraisal. Peer appraisal for a DMU is how the DMU is rated by other DMUs. A good measure for this is use of average score.

Table 7
Cross efficiency matrix
Emp. Score 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Average 96.65 98.89 97.75 98.12 99.07 97.78 98.73 98.81 97.43 98.25 97.89 98.20 97.99 98.67 98.57 95.15 98.71 95.91
1 99.44 99.44 99.55 98.73 99.60 100.00 98.73 99.90 99.43 99.60 99.46 99.60 99.43 99.54 98.83 99.22 96.32 100.00 96.32
2 100.00 93.88 100.00 97.87 95.10 100.00 98.94 97.92 95.05 92.16 96.84 94.12 94.06 96.91 98.92 93.94 92.00 94.12 95.00
3 100.00 96.10 99.59 100.00 93.18 99.46 98.77 100.00 97.91 95.31 98.92 94.24 93.50 95.60 99.45 100.00 92.09 94.61 91.05
4 100.00 95.98 99.25 97.13 100.00 100.00 98.19 98.65 99.26 96.91 96.84 98.97 98.23 98.36 97.43 96.73 95.41 98.97 98.52
5 100.00 95.98 99.25 97.13 100.00 100.00 98.19 98.65 99.26 96.91 96.84 98.97 98.23 98.36 97.43 96.73 95.41 98.97 98.52
6 99.18 98.14 100.00 99.18 96.66 100.00 99.18 99.46 96.89 96.66 99.46 96.66 96.89 98.66 99.72 97.50 94.26 97.04 94.26
7 100.00 96.10 99.59 100.00 93.18 99.46 98.77 100.00 97.91 95.31 98.92 94.24 93.50 95.60 99.45 100.00 92.09 94.61 91.05
8 100.00 95.09 97.41 95.58 100.00 98.45 96.37 97.66 100.00 97.41 95.34 99.22 97.94 96.91 95.34 97.13 95.09 99.22 97.94
9 99.91 98.01 96.99 96.49 99.30 97.79 96.19 98.36 100.00 99.91 97.25 99.60 98.77 97.45 95.93 99.37 95.63 100.00 95.33
10 100.00 99.04 99.77 99.08 98.96 99.82 98.77 100.00 99.49 99.38 100.00 99.08 99.51 99.40 99.80 100.00 96.21 100.00 95.92
11 99.76 97.13 97.36 96.22 100.00 98.26 96.47 98.18 100.00 99.10 96.68 99.76 98.84 97.64 95.89 98.36 95.71 100.00 96.80
12 100.00 92.50 94.32 93.54 94.00 92.00 92.00 93.77 95.04 95.55 97.86 94.00 100.00 95.35 100.00 99.06 93.28 99.21 91.74
13 99.69 99.27 99.88 99.06 99.21 100.00 98.93 99.98 99.22 99.34 100.00 99.21 99.62 99.69 99.76 99.52 96.30 100.00 96.18
14 100.00 95.77 100.00 97.87 99.72 99.84 98.49 98.92 99.44 97.06 98.44 98.69 99.71 99.04 100.00 98.24 96.02 100.00 98.71
15 100.00 94.18 98.79 97.15 98.26 98.14 96.97 98.18 100.00 96.86 98.09 97.70 99.42 97.42 100.00 100.00 95.37 100.00 97.26
16 97.30 98.75 99.27 98.46 99.47 100.00 98.41 99.42 99.87 99.51 99.64 99.47 100.00 99.46 99.27 99.96 97.30 100.00 96.40
17 100.00 98.75 99.27 98.46 99.47 100.00 98.41 99.42 99.87 99.51 99.64 99.47 100.00 99.46 99.27 99.96 97.30 100.00 96.40
18 98.96 95.50 99.65 97.53 100.00 100.00 98.24 98.60 100.00 97.24 98.22 98.97 100.00 98.95 99.64 98.59 96.84 100.00 98.96

Self appraisal is the simple efficiency score of an employee.

FPIp = (θpp - (Σi θip/n))/(Σi θip/n)

where θpp is the simple efficiency of Emp p and Σi θip / n is the average score of Emp p obtained from the CEM.

Table 8 depicts the mean scores and the FPIs for the 18 DMUs used in the analysis. It is shown in Table 8 that some of the efficient DMUs are in fact false indicators. Efficient Emp 5, 2, 8, 7, 17, 14, 15, and 10 exhibited FPIs of 0.94, 1.13, 1.20, 1.29, 1.31, 1.35, 1.45, and 1.79 per cent, respectively. It is also shown in Table 8 that Emp 5 had the least FPI of 94.2 per cent as well as the highest mean score. A low FPI for a DMU indicates the least the DMU benefited when moving from peer appraisal to self appraisal. Based on these results the optimal choice of Emp 5 is a good overall candidate performing well in many dimensions. This methodology allows the decision maker to rank the DMUs based on their overall performance. The optimal choice may not necessarily be one with the highest column mean in CEM, but it can be one that the best meets other intangible requirements of the decision maker. The ranks of DMUs are given in the last column of Table 8.

Table 8
Ranking of DMUs based on the FPI
Emp Simple efficiency Mean cross efficiency FPI Ranking
5 100.00 99.07 0.94 I
2 100.00 98.89 1.13 II
8 100.00 98.81 1.20 III
7 100.00 98.73 1.29 IV
17 100.00 98.71 1.31 V
14 100.00 98.67 1.35 VI
6 99.18 97.78 1.43 VII
15 100.00 98.57 1.45 VIII
13 99.69 97.99 1.74 IX
10 100.00 98.25 1.79 X
12 100.00 98.20 1.83 XI
11 99.76 97.89 1.92 XII
4 100.00 98.12 1.92 XIII
16 97.31 95.15 2.27 XIV
3 100.00 97.75 2.30 XV
9 99.91 97.43 2.55 XVI
1 99.44 96.65 2.89 XVII
18 98.96 95.91 3.18 XVIII

Discussion

Previous research on HR performance evaluation in quality organisational environments is mainly confined to the literature and theory based studies. The main components such as employee participation, continuous improvement of a quality driven HR performance evaluation are less widely researched (Soltani, et al. 2003). This paper provides a partial answer to the question: ‘what is a typical quality oriented HR performance evaluation system’? It is shown in this study, the key generic criteria of a quality driven HR performance evaluation system is possible through DEA, by incorporating quality factor in the analysis. Thus, the most important issues in HR performance evaluation in a quality management context is given due importance. In addition, the study analyses the degree of effectiveness of the currently conducted HR performance evaluation in identifying training needs, improvement in future performance and overall performance of the organisation.

Some of the limitations of traditional qualitative methods such as time intense, insufficient information for improvement, and difficulty in ranking when large numbers of employees are being compared are overcome by traditional quantitative methods. However, these methods of PA have its own limitations such as input/output factors and ratios are individually considered, which do not provide overall efficiency. The ratio based indicators used in traditional quantitative methods of PA attempt to highlight employees’ performances that are exceptionally high or low. Indicators are limited to one measure of input and/or one measure of output, and they cannot easily accommodate situations where multiple outputs are produced using multiple inputs (Wagner, et al. 2003). To compensate for the one dimensional nature of the indicators, a large set of ratios and normative values needs to be calculated in the profile reports. Unfortunately, with multiple indicators, there is no objective way of identifying inefficient employees. For example, an employee whose performance is greater than the average value might be considered potentially efficient. Nevertheless, it is not possible to determine how much larger than the average an employee must be to be considered efficient or even if the average itself is efficient. Additionally, with multiple indicators, an employee may appear efficient for one group of measures, but inefficient for another group. Without an objective means of prioritising these indicators, identification of a truly efficient employee becomes difficult. Also, existing methods based on multiple ratios provide very little guidance on how employees can change their practices to improve their overall performance. In order to overcome the limitations associated with multiple indicators, employees’ performance is explored using the tool DEA. Each input and output variable can be measured independently in any useful unit, without being transformed into a single metric, provided the same variables are utilised for every DMU. In this way, non comparable measures can be incorporated into the analysis. Unlike statistical methods of performance analysis, DEA is non parametric in the sense that it does not require an assumption of a functional form relating inputs to outputs. DEA produces a single score for each unit, which makes the comparison easy (Ramanathan 2003).

Conclusion

Unlike traditional performance appraisals, DEA searches for the efficient employees who will serve as role models. The efficiency of a machine can be determined by comparing its actual output to its engineering specifications. However, when considering human service generally, the optimum efficiency is unknown, and, therefore, cannot be determine whether an employee is absolutely efficient (Sowlati & Paradi 2004). DEA can be used to identify employees, who are relatively inefficient, measure the magnitude of the inefficiency, and aids to select the alternative paths to eliminate inefficiencies. More efficient employees, who can act as trainers to the less efficient employees, can have a stake in the employee performance improvement process. A DEA aided appraisal process has four potential benefits.

Alexouda (2005) had pointed out that advances in computer technology and the computer based techniques for handling information allow the development of decision support systems that can play a crucial role in the progress of a firm. DEA assists decision making on employee training, recruitments, and dispersion of bonuses and incentives. There is an obvious need for such tools, which can improve human resource decision making. Many efforts have been made to apply suitable software tools that can act as consultants for HR managers.

The employees’ data are available in the database of the organisation hence, at anytime current appraisal results are made available. In this analysis consensus decisions are applied (like factors to be considered for analysis, criteria for data collection) thus, acceptance of the results among the employees is increased. Periodic appraisals support continuous improvement in performance and status of the employees. Well defined procedures and guidelines incorporated in DEA aided performance appraisal, reduces the bias and subjectivity. Identifying peers and utilising their services for cross training rather than outsiders, to improve the efficiencies of others is a novel way of training. Subordinates from the same working environment could easily identify causes of inefficiency and facilitate to rectify them. Thus, DEA present an alternative perspective on assessing PA systems.

Authors

T. R. Manoharan is a faculty member in the Department of Manufacturing Engineering at Annamalai University. His research interests are in Human Resource Management, CIMS.

Email : manoharantr@yahoo.com

C. Muralidharan is a faculty member in the Department of Manufacturing Engineering at Annamalai University. His research interests are in Manufacturing Management and Quality Management.

Email : muralre@yahoo.co.in

S. G. Deshmukh is a faculty member in the Department of Mechanical Engineering at IIT Delhi (now on Deputation as the Director of ABV-IIITM, Gwalior). His research interests are in Supply Chain Management, Quality Management, Information Systems, and System Optimisation.

Email : deshmukhsg@hotmail.com

References

Alexouda, G. (2005). A user-friendly marketing decision support system for the product line design using evolutionary algorithms. Decision Support System, 38(4), 495-509.

Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical and scale inefficiencies in data envelopment analysis. Management Science, 30(9), 1078-1092.

Becker, B. E., Huselid, M. A., & Ulrich, D. (2001). The HR scorecard, linking people, strategy, and performance. Harvard Business School Press: Boston.

Biehl, M., Cook, W., & Jonston, D. A. (2006). The efficiency of joint decision making in buyer-supplier relationships. Annals of Operations Research, 145(2), 15-34.

Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429-444.

Charnes, A., Cooper, W. W., Golany, B., Seiford, L., & Stutz, J. (1985). Foundations of DEA for Pareto-Koopmans efficiency empirical production functions. Journal of Econometrics, 30(1), 91-107.

Colby, J. D., & Wallace, R. L. (1975). Performance appraisal - help or hindrance to employee productivity. Personnel Administrator, 26(2), 37-49.

Cook, W. D., Kress, M., & Seiford, L. M. (1996). Data envelopment analysis in the presence of both quantitative and qualitative factors. Journal of the Operational Research Society, 47(7), 945-953.

Cook, W. D., & Zho, J. (2006). Rank order data in DEA: a general framework. European Journal of Operational Research, 147(2), 1021-1038.

Cook, W. D., & Seiford, L. M. (2009). Data envelopment analysis - thirty years on. European Journal of Operational Research, 192(1), 1-17.

Doyle, J. R., & Green, R. H. (1991). Comparing products using DEA. Omega, 19(6), 631-638.

Doyle, J. R., & Green, R. H. (1994). Efficiency and cross-efficiency in DEA: derivations, meanings and uses. Journal of the Operational Research Society, 45(5), 567-578.

Dulewicz, V. (1989). Assessment and selection in organisations: methods and practices for recruitment and appraisal. New York: John Wiley & Sons.

Dyson, R. G., Allen, R., Camanho, A. S., Podinovski, V. V., Sarrico, C. S., & Shale, E. A. (2001). Pitfalls and protocols in DEA. European Journal of Operational Research, 132(2), 245-259.

Dyson, R. G., & Thannassoulis, E. (1988). Reducing weight flexibility in data envelopment analysis. Journal of the Operational Research Society, 39(6), 563-576.

Emrouznejad, A., Parker, B. R., & Tavares, G. (2008). Evaluation of research in efficiency and productivity: a survey and analysis of the first 30 years of scholarly literature in DEA. Socio-Economic Planning Sciences, 42(3), 151-157.

Gibson, S. G., Harvey R. J., & Harris, M. L. (2007). Holistic versus decomposed ratings of general dimensions of work activity. Management Research News Journal, 30(10), 724-734.

Giokas, D. I., & Pentzaropoulos, G. C. (2000). Evaluating productive efficiency in telecommunications: evidence from Greece. Telecommunications Policy, 24(9), 781-794.

Golany, B., & Roll, Y. (1989). An application procedure for DEA, Omega, 17(3), 237- 250.

Grote, D. (2002). Performance appraisal. Executive Excellence, 19(12), 12-13.

Grubb, T. (2007). Performance appraisal reappraised: it’s not all positive. Journal of Human Resources Education, 1(1), 1-22.

Locher, A. H., & Teel, K. S. (1977). Performance appraisal: a survey of current practices. Personnel Journal, 56(5), 245-257.

Min, H., Min, H., & Joo, S. J. (2008). A DEA based balanced scorecard for measuring the comparative efficiency of Korean luxury hotels. International Journal of Quality and Reliability Management, 25(4), 349-365.

Mohamed, A., & Luc, C. (2008), Cost and profit efficiency of Chinese banks: a non-parametric analysis. China Economic Review, 19(2), 260-273.

Oberg, W. (1972). Make performance appraisal relevant. Harvard Business Review, 50(1), 18-32.

Paradi, J. C., Smith, S., & Schaffnit-Chatterjee, C. (2002). Knowledge worker performance analysis using DEA: an application to engineering design teams at Bell Canada. IEEE Transactions on Engineering and Management, 49(1), 161-172.

Pearce, J. L., & Porter, L. W. (1986). Employee responses to formal performance appraisal feedback. Journal of Applied Psychology, 71, 211-218.

Ramanathan, R. (2003). An introduction to data envelopment analysis: a tool for performance measurement. Sage Publications India Private Ltd: New Delhi.

Renton, M. (2000). New roles for HR part 1. People Dynamics, (January), 26-29.

Richards, R. C. (2003). Setting benchmarks and evaluating balanced scorecards with data envelopment analysis. Benchmarking: An International Journal, 10(3), 226-245.

Roman, B., Wigand, R. T., Wolfgang, K. (2003). Beyond electronic commerce: efficiency prevails. [Online]. Centre for Research on IT and organisations - University of California. Available http://www.crito.uci.edu/publications/pdf/gec/ECIS_0303251.pdf [2008, September 5th].

Ross, A., & Droge, C. (2002). An integrated benchmarking approach to distribution centre performance using DEA modelling. Journal of Operations Management, 20(1), 19-32.

Sami-Mahgary, E. I., & Lahdelma, R. (1995). Perspectives for practice DEA: visualising the results. European Journal of Operational Research, 85(3), 700-710.

Scheneier, C. E., Richard, W. B., & Lloyd, S. B. (1986). Creating a performance management system. Training and Development Journal, 40(5), 74-79.

Serrano-Cinca, C., Fuertes-Callen, Y., & Mar-Molinero, C. (2005). Measuring DEA efficiency in internet companies. Decision Support Systems, 38(4), 557-573.

Sherman, H. D. (1984). Hospital efficiency measurement and evaluation: empirical test of a new technique. Medicine Care, 22(10), 922-938.

Singh, P., & Finn, D. (2003). The effects of information technology on recruitment. Journal of Labor Research, 24(3), 395- 408.

Slusher, E. A. (1975). A systems look at performance appraisal. Personnel Journal, 54(2), 114-117.

Soltani, E., Van Der Meer, R., Gennard, J., & Williams, T. (2003). A TQM approach to HR performance evaluation criteria. European Management Journal, 21(3), 323-337.

Sowlati, T., & Paradi, J. C. (2004). Establishing the practical frontier in data envelopment analysis. Omega, 32(4), 261-272.

Sueyoshi, T. (1992). Measuring technical, allocative and overall efficiencies using a DEA algorithm. Journal of the Operational Research Society, 43(2), 141-155.

Thakur, T. (2005). Benchmarking study for the Indian electric utilities using data envelopment analysis. Paper appears in the proceedings of the 37th Annual North American Power Symposium, 545-549.

Thanassoulis, E. (1995). Assessing police forces in England and Wales using data envelopment analysis. European Journal of Operational Research, 87(3), 641-657.

Wagner, J. M., Shimshak, D. G., & Novak, M. A. (2003). Advances in physician profiling: the use of DEA. Socio-Economic Planning Sciences, 37(2), 141-163.

Wong, B. Y., Yang, J., & Greatbanks, R. (2004). Using DEA and the ER approach for performance measurement of UK retail banks. MCDM, (August), 6-11.