ICT for Performance Assessment of Emergent

Technologies Operators

Oleksandr Burov

Institute of Information Technologies and Learning Tools

ayb@iitlt.gov.ua

Abstract. The paper discusses main ideas of design principles and results of prac-

tical implementation of ICT for assessment and prediction of power plant opera-

tor fitness-for-duty. They are discussed reasons of discrepancy of operator psy-

chophysiological opportunities to occupational requirements that can reduce

his/her reliability and general fitness-for-duty. Principles of functioning of the

system for a pre-shift assessment of operator fitness-for-work are settled in defi-

nite terms. The structure of the computer system is described. Results of indus-

trial check of the system work in real-settings (fossil and hydro power plants) are

discussed. The main results are: (1) very high relationship between indices of

operators task performance when working with the system and indices of their

professional work that were registered by the technological automatic system of

the plant, (2) substantially higher accuracy of the prognosis built by the system

(85-90%) in comparison with other means of an operator check known from pub-

lished data (70% as the best).

Keywords: operator, fitness-for-duty, ICT, prediction.

Key Terms: capability, technology, ICTEnvironment, PSI Core Time ontology.

1 Introduction

Modern life and activity of a human depends on electric power - at work, at home, in

general purpose places and recreation zones. Process of manufacture and transfer of the

electric power to consumers is accompanied by unproductive losses as energy carriers,

and electric power produced. It is caused, on the one hand, by restrictions of used tech-

nologies and technical devices, and, on the other hand, by errors of the personnel serv-

ing these processes and the units (equipment). According to the world experts, failures

and incidents in power industry are caused by erroneous actions of human-operator [4],

who uses more and more actively information and communication technologies (ICT)

at present. Those errors result generating engines shutdown, underproduction of elec-

tricity and its deficiency by users, death of equipment and even ecological problems,

because power industry is industry with a relatively high ecological risk. Besides, it is

necessary to mention inefficient controlling technological units by operators that is ac-

companied by fuel losses [10]. To some extent, this could be explained by the activity

theory and its role in study of human performance [1].

According to data known, both simulation of operator work (in laboratory condi-

tions) and objective requirements of the job (in real settings) have shown importance

of individual approach to data analysis to assess and predict an operator’s fitness-for-

duty, as well as the use of his/her individual “norm” [6]. Those theoretical and applied

findings were confirmed in further researches and implementations and proved that a

simple cognitive test can secure a high accuracy of prediction of an operator fitness-

for-duty for the forthcoming shift [9].

However, it is obvious that such an approach needs to fulfill a lot of calculations

and to use an optimized set of indices that are very informative in relation to the psy-

chophysiological pattern of the operator functional state and his/her holistic assessment

as a system [8]. Besides, to be useful for applied needs it should be made ex tempore,

but not a posteriori. The method and tool for an operator fitness-for-duty assessment

and prediction must be prompt and accurate [7]. Achieved level of accuracy of fitness-

for-duty prediction is evaluated as 70% as maximal, though this means a high risk of

incidents and accidents, i.e., 30% of prediction error.

To reduce the prediction risk of a human performance degradation it is necessary to

develop principles of functioning of the computer system (ICT) for a pre-shift assess-

ment of operator fitness-for-work, the structure of the computer system realized that

approach and the algorithm of the system work.

The aim of the paper is to design and to implement the ICT for assessment and

prediction of operator fitness-for-duty with accuracy not less than 85% as well as to

check its work in real-settings.

2 Method

It is known that about 10% of all accidents are produced by operators having high

knowledge and skills, high motivation and good health [5]. This fact is neglected by

managers as a rule, but cannot be explained by insufficient training of operators or psy-

chological reasons only. The reason of this fact is quite complex.

The research carried out in the Institute for Occupational Medicine (Kiev) has

shown that duties of operator of thermal power plant consist in the control and the reg-

ulation of modes of operation of the equipment demanding the high responsibility, a

significant effort of attention and operative memory [7]. This effort arises owing to

necessity of the great capacity perception of the information coming from devices, shift

supervisor, chief operator, other operators and the subordinated personnel.

They were earlier developed the basic methodological features of the computer-

based system for assessment and to prediction operator fitness-for-duty [3]:

An individual-based approach to evaluation of operational efficiency and psy-

chophysiological state, which allows one to establish individual norms.

Use of a temporal biorhythmic structure of task performance as a correlate of a

human psychophysiological condition, on the one hand, and his or her professional

serviceability on the other.

The system is self-adjusting and uses the adaptive model of an operator's psycho-

physiological state.

The basic principles of the system for operator fitness-for-duty individual monitoring

were developed and validated, and allowed to create a new class of systems for pre-

shift (pre-flight, pre-mission, etc.) control on the basis of a human psychophysiological

state assessment and prediction [3].

3 Development and Validation of the System Serviceability

The way of an error may arise as a result of operator psychophysiological parame-

ters’ fluctuation that become quite clear if we take into account that the functional state

and fitness-for-duty are formed under simultaneous exposure of three levels of psycho-

physiological maintenance of professional work [3]:

General – relatively constant, genetic-defined, ensuring the principal profes-

sional aptitude to the occupation.

Age-related – slow changeable, defines changes in the dynamics of profession-

ally important qualities.

Current (operational) – day-to-day and over working day changes, ensuring an

operator fitness-for-duty for a particular mission.

Each occupation makes its demands to psychological and physiological abilities of

a human [11]. In general case, they have two limits for psychophysiological parameters

– upper and bottom. If parameters’ values are out-of-range, a human does not meet the

occupation requirements and his/her work can result a high risk of errors. Such psycho-

physiological professionally important qualities as intelligent quality, audio-motor rate

etc. [6] can have one-side limitation, but some parameters can have two-side limita-

tions. F.e., high adaptation rate and ability to creative work are not favorable, if they

are higher or lower than occupation requirements. It was set that fitness-for-duty and

functional states are two sides of manifestation of psychophysiological maintenance

under operator work. These parameters can be in permissible intervals, but their “inte-

gral” value of psychophysiological fitness-for-duty (IVofPF) can lie near limits.

Slow changeable parameters can very around the general level that moves summary

value closer to limits. An acute case arises when even marginal changes in fluctuating

psychophysiological parameters of the particular state turn IVofPF out of limits. This

means that the risk of an operator erroneous performance increases significantly and

this operator needs to correct his/her functional state. Such a situation is typical when

changes at all three levels of psychophysiological maintenance of professional work

are not critical by themselves if they are measured in traditional manner (by physiolog-

ical indices or by subjective tests), but their manifestation at the system level (profes-

sional performance) are recorded as a human erroneous or inefficient actions. That is

why the approach based on assessment of the operator test performance indices that

contains an information concerned physiological “cost” of the performance is more ef-

fective and fruitful [2]. It has been demonstrated that biorhythmic structure of opera-

tor’s task performance time reflects both physiological maintenance of cognitive work

and his/her professional efficiency [2].

In contrast to the operator work concerned to continuous tracking, work of power

unit operators has two most marked forms: the concentrated supervision over a course

of technological process (making 60-70 % of working hours) and active actions (30-40

% of working hours). If necessary to maintain a stable mode of the unit operation op-

erator work has a monotonous character. Thus in dynamics of the work shift some

changes become evident in the higher nervous activity which is characterized by fluc-

tuation of the period of simple visual-motor reaction, stability and switching attention,

decrease of ability to short-term storing. This needs some functional strain and the op-

erator psychophysiological features have to meet occupational requirements. It is

impossible to provide in real settings without the use of ICT accounting the above-

mentioned fluctuations and changing of psychophysiological parameters. Even more,

human-system integration needs to use automated system of data measurement and

storing every individual operator.

The developed automated system for a pre-shift assessment of operator fitness-for-

work (SPAOF) should meet the following requirements:

The prognosis of operator fitness-for-work should be constructed on the basis

of his/her individual characteristics with automatic correction of individual "norm" ac-

cording to the operator fitness-for-duty changes in dynamics of year and age, i.e. to

adapt to slow changes of the human psychophysiological state.

Because any kind of professional work needs training and a human adaptation

to specificity of problems to be solved, parameters of test performance as a model of

professional work vary in due course as well. Therefore the model of the operator fit-

ness-for-work should take proper account of an operator training level for test perfor-

mance and should be corrected ("be trained") in accordance to this level.

As external criteria to correct the models of an operator state and fitness-for-

work, they should be used parameters of his/her real work - technical and economic

parameters (TEP) of success of professional duties performance.

The system should be utilized by both the operator to correct his/her functional

state and the supervisor (chief operator) to make organizational decisions.

3.1 Principles of Functioning of the System SPAOF

The active current structure of the system SPAOF is determined at any moment

by functionality dependently on results of test performance. In other words, the infor-

mation which deals with the current operator state and is received by the system (task

performance time TPT and number of errors O) determines change of internal structure

of the system. The external information (about efficiency of professional work) deter-

mines dynamics of realignment of models of operator fitness-for-work. Thus, the sys-

tem provides permanent interaction of the external and internal information environ-

ment that imposes the certain restrictions on the use of information and makes demands

to it. This requires observance of the following principles:

Strict individuality of information models and formation of individual "norm"

of the psychophysiological state and fitness-for-work of an operator, regularly cor-

rected by software tool.

The multivariate analysis of test performance parameters as it is necessary to

describe a human state, as interaction of hierarchy of regulator systems, each of which

also is a complex multiparametric system.

Regular verification of the prognosis according to external criterion (parameters

of real industrial performance of operator).

Model adaptability to slow changes of functional state of the operator in dynam-

ics of the professional work having both seasonal, and age tendencies.

Special value for reliability and accuracy of the monitoring system has a principle

of use of external criterion for verification of the forecast as target parameter of system

SPAOF. Any method of forecasting may not be solvent and reliable if not the external

criterion is used. Construction of models should be based on application of external

criteria. In the system SPAOF, the role of an external criterion is played by technolog-

ical information which is not only predicted parameter of fitness-for-work, but also a

model training element (criterion of reliability).

According to experts’ opinion for power industry, quality of work of thermal power

plant operators is characterized by various parameters among which the most informa-

tive are parameters of over-burning. They were selected 10 parameters most sensitive

to influence of the operator from several dozen parameters registered by the power

plant. Further it was selected one parameter included in system for industrial operation

– over-burning for auxiliary. That parameter was an "integrated" estimation of effi-

ciency of technological process control by the operator. The external criterion in the

system for pre-shift prediction of operator’s fitness-for-work has been improved as

well. Thus, the system "was retrained" during some time on the estimation by that cri-

terion.

3.2 Structure of the System for a Pre-Shift Assessment of Operator Fitness-for-

Work

Implementation of the specified requirements has caused structure of the developed

system including subsystems:

training and re-training,

test performance,

express-analysis of test performance results,

constructions of the model of the current functional state and prediction of fit-

ness-for-work,

database control system,

storing.

Subsystem for training and re-training. It provides training the operator to work

with the system at acquaintance to the system and training after a long break in work.

The operator is considered as a “pupil", that is his/her data do not collect in database

(DB) up to double in series (in succession) performance by the criterion of “trained”.

The criterion is defined by a group "norm" of success work with the SPAOF: maximum

allowed TPT (Tmax) and maximum allowed number of errors (Omax). The operator is not

considered as a "pupil" at simultaneous fulfillment of conditions

Ti ≤ Tmax, Oi ≤ Omax, Ti-1 ≤ Tmax, Oi-1 ≤ Omax, (1)

where T i, T i-1 - TPT and O i, O i-1 - number of the allowable errors at “i” and “i-1”

test performance, accordingly.

After fulfillment of this criterion the operator got access to DB by input of the pass-

word and then the system begins to accumulate the data on his/her work in real condi-

tions with the subsequent construction of prognosis.

Subsystem for test performance. It includes software tools as follows: preparations

and presentations of a sequence of test tasks that allows simulating acceptance of an

information stream from the unit at real professional work; registration of answers of

the operator, their input in DB; the check of test performance results. As the test task,

it was used the cognitive test developed in the laboratory research [2]. The choice of

the test was co-coordinated with experts of the power plant and operators themselves.

Subsystem for processing of test performance results. It is a complex of the software

providing calculations of variation statistics, carrying out time series analysis, pro-

cessing test information flow by the operator. The received data are stored in the DB:

number of incorrectly solved tasks, parameters of information processing time (mean

TPT, standard deviation, factors of asymmetry and excess), oscillatory parameters of

TPT and the service information (date, time of day etc.) as well.

Subsystem for the construction of a model and forecast. It is intended for construc-

tion of the models connecting the characteristics of the test performance {x} with pa-

rameters describing professional fitness-for-work of the operator - by technical and

economic parameters. In a simple case, it is used one, the most dependent on an operator

state, parameter Y. Linear multi-regression models were used as models which order

was determined by experimental way [3].

As the operator functional state is subject to change in time (many days and season),

a model has an informational "noise" from the out-of-date information. To eliminate

decreasing accuracy of models owing to information “aging” they are imposed limita-

tions for a length of learning sample (it is determined by practical consideration for

each professional group of operators). The model is corrected by construction of a new

model after each session of test performance on learning sample {x}. Such adaptive

forecasting results in dependence as parameters of test performance and factors of re-

gress from time t, and the model in a general view becomes:

Y(t) = b0(t) + ∑ bi(t)xi(t), (2)

where bi - coefficients of regression of the test performance characteristics.

Prognosis is carried out by substitution of a subset of test performance characteristics

in the last model stored in DB of the system. This subsystem allows realizing the prin-

ciple of adaptive prediction of an external parameter of operator work.

Database control system. It is intended to collect and use the objective information

related to operator work both in modeling, and in real industrial conditions. As a data-

base for maintenance of information system SPAOF the database of relational type was

developed. Data collected are not simple data concerned the operator fitness-for-work,

but they are used for the automatic control of the subsystems and SPAOF work in gen-

eral included into the corporate ICT of the power plant.

Subsystem of storing. It is intended for a conclusion of results of test performance.

The report of test performance of operators group (for example, of the working shift) is

printed and becomes accessible to operation personnel.

3.3 Algorithm of the System Work

The described structure of the system SPAOF allows to create it not only as adaptive

to change of a functional state in dynamics (changes) of the professional biography of

the operator, but also as an self-adjusting system in which the structure and the param-

eters connected to parameters of regulatory realignment variations of physiological

maintenance of activity are changed automatically. In other words, the structure and

parameters of the monitoring system for operator state depend on regulation of physi-

ological maintenance of the organism. SPAOF may be referred to combined self-ad-

justed systems with automatic adjustment of parameters as it uses for self-adjustment

signals from both external influences (the technical and economic information related

professional fitness-for-work) and the information concerned dynamic characteristics

of the object. Thus, results of operator test performance have essential meaning for the

system functioning. The principle of use of test performance results as criterion of the

active functional subsystems selection is realized in SPAOF. This is provided by the

following algorithm of the system work.

Step 1. Psychophysiological test performance. Its purpose is modeling of dynamics

of operator work by presentation on the screen a task sequence with registration of

performance time Tl and accuracy O.

Step 2. Supervision of the operator readiness to work with the system. If he/she is a

"pupil" and results of test performance do not meet criteria Tl = Tmax and O = Omax,

the system stops. If the operator is a "pupil" and meet criteria or he is already not a

"pupil", than transition to processing results is made.

Step 3. The results’ analysis of current test performances consists in calculation of

characteristic of TPT density, TPT oscillatory characteristics, performance reliability.

If mean time and number of errors are in allowable limits Tl ≤ Tдоп и O ≤ Oдоп these

values are displayed on the screen. Otherwise, the operator is offered to pass repeatedly

a session of test performance when will correct his/her state.

Step 4. Check if the assessment and the prognosis of fitness-for-work can be calcu-

lated. If number of sessions of successful test performance is less than the system

needed, the test session is finished.

Step 5. Construction of the test performance rating on the basis of comparison to

the data saved up over the previous period.

Step 6. A choice from the obtained results a subset of the parameters included into

the model. The volume and structure of a subset depends on characteristics of the in-

formation processing and an estimation of real professional fitness-for-work. Selection

criterion is a stability of model according to F-statistics.

Step 7. Construction of the multi-regression prediction model for dependent varia-

bles describing professional fitness-for-work of the operator. If the steady model may

be constructed, its parameters are recorded into DB.

Step 8. Prognosis construction by substitution of results of test performance in the

current model.

Step 9. Correction of the operator information model in DB by removing the out-

of-date information according to the prescribed criterion.

Thus, after results of test performance (levels, rates and oscillatory structure of ac-

tivity) the structure and parameters of prediction models, a choice of activated subsys-

tems and a mode of its work vary. In other words, not a particular model for the de-

scription of operator fitness-for-work is putting into SPAOF, but the algorithm of model

construction, algorithm of the system work control is incorporated on the basis of use

of the parameters estimating regulation of the operator functional state (varying in con-

ditions of non-steady-state of the control object, perturbation influences).

In practice, the system work is put into effect as follows.

Sequence of test tasks modeling process of the operator work is presented on the

screen of the display in the operator workplace at a predetermined time. Task perfor-

mance time and accuracy are measured. This information is collected in individual data-

bank (IDB) of the operator. Test performance characteristics of the operator activity

including oscillatory ones are calculated by results of the whole session. Thus a pattern

of the test performance dynamics is formed and is used to correct the operator model.

On the basis of the model the operator state mark is computed. When external parame-

ters of his professional work (for example, technical and/or economical parameters) are

available, which are to be predicted, the forecast of the operator fitness-for-work is

calculated in units of this parameter. At the end of a working day a real value of the

parameter are register in IDB for use at the following test performances. The model is

updated after each session of test performance, "tracing" thus changes of the operator

current state. Thus, the system provides an opportunity of a choice of optimum model

(in sense of the maximal relationship of parameters of functional system of activity with

the technological parameter).

4 Results and Industrial Check of the System’s Capacity for

Work

The purpose of industrial research was to test the system for operator fitness-for-work

by way:

Assessment of its work stability under operating conditions by operation per-

sonnel without experts-psychologists and developers.

Revealing parameters of SPAOF test performance the most informative in re-

lation to the efficiency of professional operator work, i.e. the most connected to tech-

nical and economic parameters of operators work.

Assessment of the operators’ attitude to such kind of the control of their state.

38 operators of the thermal power plant who performed the SPAOF test at the be-

ginning of each working day (daily monitoring) have taken part in research if it was

allowed by the industrial situation objectively. During a session of test performance it

was necessary for operators to execute the same test what was carried out by volunteers

in laboratory daily research.

It is known that any kind of activity demands the certain training to develop neces-

sary psychomotor skills. It is right and in relation to test performance, especial for use

of cognitive tests. To reduce time of operator adaptation to the test all of them passed

oral instructing and preliminary training up to full understanding of feature of work

with the system. After that they have to train with the training subsystem and started

day-to-day work with SPAOF.

The data collection obtained during 9 months of regular testing has been analyzed

in similar manner as in the laboratory research. Results of comparison of one technical

and economic parameter (fuel’s “over-burning”, i.e. fuel consumption over the norma-

tive amount) are given in Table 1, registered by the technological system of the power

plant with parameters of work in the cognitive test for 14 operators, whose number of

test performance sessions (days of the system use) was not less than 70. Theoretical

number of sessions could be significantly more than 70, but a shift-work scheme used

at the plant made it less, as well as holidays, days of illness, technological operations

at the beginning of the working day etc.

Table 1. Correlation of TEP parameters and parameters of task performance time

Individual code

of the operator

% of the TEP prediction

error

Multiple correlation

factor R

2413 11,1 0,75 **

2584 15,8 0,67 *

2411 18,5 0,73 *

2613 11,0 0,71 *

2634 7,9 0,91 **

2424 9,2 0,87 **

2662 13,1 0,77 **

2425 13,2 0,82 **

2568 11,9 0,75 **

2593 9,9 0,83 **

2434 10,7 0,57 *

2629 5,8 0,88 **

2439 9,8 0,84 **

2500 13,9 0,82 **

Notes: * significance (p < 0.05), ** significance (p < 0.01).

The average percent of TEP prediction error is specified in a column 2 by results of

test performance by each operator within one year. The column 3 characterizes rela-

tionship R between TEPs with 5 operator work parameters chosen by means of step-

wise algorithm from full set of variables (number of errors; test task performance indi-

ces - mean time, standard deviation, factors of asymmetry and excess, 16 spectral char-

acteristics, time and day of test performance).

It is possible to see the level and reliability of relationship between values of effi-

ciency of an operator work and parameters of dynamics of task performance. Thus cor-

relation relationship runs up to a level "high" and practically is always authentic.

For the practice, of course, the accuracy of prediction (per cent of the TEP predic-

tion error) is more useful rather than statistical evaluation of relationship. From the data

published in scientific and technique reports and journals it is known that the prediction

accuracy level 70% is quite acceptable and even some of authors believed that it was

difficult and impossible to achieve from a practical standpoint [6; 7]. But data obtained

have demonstrated more higher level of accuracy (mean value of errors is 11,4 % that

corresponds the accuracy level 89%). In other words, the mean error was 2 times less

than in the best developments by other authors.

We should note that the more an operator worked with SPAOF, the higher was the

accuracy of prediction, because the system was tuning in his features of the cognitive

task performance by means of adjusting of the model parameters used in the system.

In addition to objective parameters of quality of an operator professional work his

supervisor assessed his work as well. The operator himself estimated accuracy of his

state assessment. As a whole, 35 persons from 38 have specified objectivity of the es-

timations exposed by system, and 15 were initiators of continuation of use of system

even after stop of the research, having noted not only accuracy of estimations, but mo-

bilizing and training meaning.

On the basis of operating experience of SPAOF it was made its adaptation to oper-

ators of hydroelectric power stations (HEPS) and dispatchers of power grids (DPG).

Data obtained at two HEPS (SSh and Kr, 94 operators) and DPG (17 operators) during

industrial implementation of the system have been compared with data of operators of

the fossil power plant (FPP). Average time of tasks performance (TPT) and average

number of errors (O) are different. But difference is reliable for different types of en-

terprises and is not significant for HEPS, as it follows from Table 2.

Table 2. Indices of SPAOF tests’ performance by operators

Indices of test

performance

FPP SSH Kr DPG

TPT, s 4.33 2.61 2.41 3.09

O, # per 64

tasks

1.63

2.39

2.46

2.14

Prediction of technological parameters at HEPS and DPG was not carried out owing

to absence of the appropriate technological subsystem for registration objective tech-

nical and economic parameters. However, active use of results of test performance by

operators with the purpose to optimize their state and fitness-for-duty allows proposing

the system for wide application.

5 Conclusions and Outlooks

The systems use at power plants has shown:

The operator functional state needs to be assessed on the basis of 3-level model

which describes operator psychophysiological features and whether or not they meet

the occupational requirements.

The systems can be equally effective in use at power industry enterprises of all

types (nuclear, fossil, hydroelectric plants, power grids), space and aviation, as well as

for regular computer users.

Prognosis of professional fitness-for-duty is enough accurate (the forecast accu-

racy is 85-90 % for the day-to-day check.

To identify and prevent undesirable events caused by human performance prob-

lems the systems use permits to ensure the psychophysiological support over the oper-

ator professional biography.

To ensure the system efficiency it was realized organizational (performance, pre-

diction, recommendations for managers), informational (valuation of state and progno-

sis of fitness-for-work) and technical (software) support. Development of psychophys-

iological indicators used in the systems is based on the recognized importance of man-

agement and organizational processes in influencing human performance. They have

demonstrated that potentially usable indicators can be developed from psychophysio-

logical models of management and organizational performance by rational process, and

that these indicators can be applied in a power plant setting.

The systems have demonstrated hands-on experience using the psychophysiologi-

cal indicators, and preventive steps must be developed to respond to adverse changes

predicted by them.

References

1. Bedny, G. Z., Karwowski, W.: Meaning and sense in activity theory and their role in study of

human performance. Ergonomia, Vol. 26, No. 2, 121-140 (2004).

2. Burov, А.: Evaluation of operators’ functional state on parameters of mental service-

ability. Human Physiology, 2, 281-288. (1986).

3. Burov, O.: Day-to-day monitoring of an operator’s functional state and fitness-for-

work: a psychophysiological and engineering approach. Ergon. Psychol. Dev. Theor

Pract. 1, 89 (2008).

4. Burov, O.Yu.: Individual Abilities and Lifelong Learning. Information Technologies and

Learning Tools. Vol.5, # 55, pp. 1-11 (2016), http://journal.iitta.gov.ua/index.php/itlt/arti-

cle/view/1274

5. Gerasimov, A.V.: Psychophysiological factors of predisposition erogenous deci-

sions and acts. Problems of personnel training in power industry. Book of abstracts,

Kiev, 106-109 (1986)

6. Hockey, G.R.J, Gaillard, A.W.K., Burov, O. (eds.): Operator functional state: The

assessment and prediction of human performance degradation in complex tasks. Am-

sterdam: IOS Press. (2003).

7. Jones, B., Kennedy, R.S., Turnage, J.J.: Performance Tests and Fitness for Duty:

Some Strategic Considerations. Advances in Industrial Ergonomics and Safety VII,

Edited by A.C. Bittner and P.C. Champney. Taylor & Francis, London, pp. 477-484

(1995)

8. Kennedy, R.S., Jones, M.B., Drexler, J.M., Rugotzke, G.G.: Comparison of Two

Batteries for Fitness-for-Duty Testing. In: We Have a Solution! Proceedings of the

Human Factors and Ergonomics Society 43rd Annual Meeting, Houston, Texas,

September 27-October 1, 1999. The Human Factors and Ergonomics Society, Santa

Monica, California. 1, 607-611. (1999).

9. Mulder, L.J.M. et al.: How to use cardiovascular state changes in adaptive automa-

tion / L.J.M.Mulder, A. Van Roon, H. Veldman, K. Laumann, A. Burov, L. Quispel,

P.J. Hoogeboom. In: Hockey, G.R.J., Gaillard, A.W.K., Burov, O. (Eds.), Operator

Functional State. The Assessment and Prediction of Human Performance Degrada-

tion in Complex Tasks. NATO Science Series. IOS Press, Amsterdam, pp. 260–272

(2003)

10. Parsons, S.O., O’Hara, J.M., Seminara, J.L.: History of Human Factors/Ergonomics

in Power Systems. International Encyclopedia of Ergonomics and Human factors.

2nd edn.. Edited by Waldemar Karwowsky. Tailor&Frances, pp. 84-97 (2006).

11. Veltman, H., Wilson, G., Burov, O.: Operator functional state assessment. Cognitive

load. NATO Science Series RTO-TR-HFM-104. Brussels, pp. 97–112 (2004).