evaluating road safety audit proceduressites.poli.usp.br/p/hugo.pietrantonio/ersastudy.pdf ·...

EVALUATING ROAD SAFETY AUDIT PROCEDURES: 1

SOME QUESTIONS AND A NEW METHOD OF STUDY 2

Hugo Pietrantonio, D.Sc. (Corresponding Author) 3

Lecturer, Traffic Engineering 4

Department of Transportation Engineering 5

Escola Politécnica – University of São Paulo, Brazil 6

Edifício de Engenharia Civil, Cidade Universitária 7

São Paulo/SP, CEP 05508-900, Brazil 8

Email: [email protected] 9

Phone: +55-11-3091-5492; Fax: +55-11-3091-5716 10

11

Lili L.Bornsztein, M.Sc. 12

Traffic Analyst (Superintendence of Traffic Engineering) 13

Traffic Operations Division 14

Traffic Engineering Company, Prefecture of São Paulo/SP, Brazil 15

Alameda Ministro Rocha Azevedo, 976–ap.72, Cerqueira Cezar 16

São Paulo/SP, CEP 01410-003, Brazil 17

Email: [email protected] 18

Phone: +55-11-3081-9838 19

20

Abstract: The paper discusses current views on the application of road safety audits (RSA) as a tool for 21

the improvement of road safety and proposes a new method of study for detailed evaluation of some 22

features that can influence their effectiveness, which is demonstrated in the practical setting of road safety 23

audits applied to existing roads review or inspection. Starting from a review of published references on 24

RSA in different countries, the identified a set of questions and undertook a case study of the impact of 25

alternative RSA procedures on its final effectiveness. The RSA effectiveness is measured both as 26

agreement to a safety expert assessment and to an accident-based study. Main features analyzed in the 27

case study included the type of observer used for data collection and the type of checklist used as guide to 28

field work. Based on an exploratory study, the performance of RSA application is evaluated using 29

weighted indexes of concordance and disagreement, and the rating of detection or omission for the 30

observations gathered in the accident diagnosis of the safety problems at the intersection studied, 31

supplemented by a statistical analysis of the influence of selected covariates on these scores. The main 32

results of the case study can be summarized as clear support to team work for field observation and 33

support to employing less experienced personnel for field observation. The study also stresses the need of 34

procedures/criteria for priority setting. These conclusions are useful to the selection of alternative RSA 35

procedures in agencies responsible for promoting or enforcing RSA and in professional teams carrying-out 36

RSA tasks. Several features deserving further study are also identified. 37

38

Key words: road safety audits, road safety reviews, statistical analysis of safety measures. 39

40

41

mailto:[email protected]

mailto:[email protected]

Pietrantonio, Bornsztein 1

1

EVALUATING ROAD SAFETY AUDIT PROCEDURES: 2

SOME QUESTIONS AND A NEW METHOD OF STUDY 3

Hugo Pietrantonio and Lili L. Bornsztein 4

5

1. INTRODUCTION 6

The use of Road Safety Audits (RSA) as a tool for improving traffic safety has received 7

increasing attention in several countries as a result of its development and dissemination from the UK to 8

Australia, New Zealand, to continental Europe, Canada, USA, and also to developing countries such as 9

Brazil and Chile. Examples of recommendations that are widely known and recognized in several of the 10

leading countries can be seen in IHT (1991, 1996, 2008), DoT (1994a, 1994b, 2003), and Austroads 11

(1994, 2002, 2006). Similar recommendations, but with relevant changes, were adopted by late-comers as 12

can also be seen in references INST (1995), Road Directorate (1997), TAC (2001, 2004), CONASET 13

(2003), and FHWA (2006), as examples. 14

There is a favorable position among the general public, politicians and technicians supporting 15

RSA application as a preventive measure that tries to avoid accidents through a (hopefully low cost and 16

effective) revision process and also saving money in future (more expensive) road changes. As usual when 17

reviewing subjects covered by favorable views, the “previous” general agreement on the merit and the 18

purposeful defense of the positions do the bad job of hiding some points that vary among different 19

concepts and practices, features that can change its results and its value. 20

The careful analysis of concepts and practices for RSA application is interesting as it stresses 21

some overlooked questions and clarifies the need of more detailed methods for studying RSAs. Despite 22

the widespread acceptance of the general idea, one can observe that differences in proposals for the RSA 23

technique are not stressed or at least noticed by scholars or practitioners. Also, it is important to 24

disentangle features that are judged to be essential from other features that are a matter of detail and 25

should be tailored to the needs and means of each application. In our view, some of these differences and 26

details can compromise the effectiveness of RSA application and should be carefully studied. 27

In the following sections, the paper tries to carry-out three tasks. First, it discusses the main 28

features that vary among different concepts and practices for RSA (in sections 2 and 3). Then, it presents a 29

method of study that can be used in the comparative effectiveness of alternative procedures for field work 30

and reports the results of its use in the application of RSA procedures at a signalized intersection (in 31


section 4 and 5). The final section summarizes the main conclusions that can be delivered by the research 1

on the effectiveness of alternative RSA procedures and on the validity of the method of study. 2

From the outset, one should note that the results in this paper are limited to the RSA application to 3

existing sites (being only partially applicable to road projects) and that the method should be extended and 4

improved before using it for the study of RSA application in the design phase of road projects (usually its 5

main scope). Some authors or institutions distinguish the applications of RSA principles to existing roads 6

and treat it as different technique (called inspection or review, instead of audit, as in EC, 2008, even if 7

considering the monitoring of their first years as in DoT, 2003). However, application to existing roads is 8

acknowledged and described as RSA in several of the original guidelines (e.g. IHT, 1991) and current 9

ones (mainly, Austroads, 1994, 2002, 2006). The vision here is that the techniques, and their problems, are 10

very similar and that the major part of the discussion should apply to both applications. 11

Also, one should note that the problems encountered in the evaluation of RSA procedures are also 12

relevant for other studies (e.g. the evaluation of other surrogate measures of safety, such as traffic 13

conflicts, conflict opportunities and exposure measures). Nevertheless, each technique can have features 14

that can not be evaluated with the methods proposed. This limitation applies even to RSA as some general 15

questions (e.g. on where and when to use RSAs) are outside the realm of this work and its methods (which 16

are more useful in the study of procedural alternatives for RSA work and its tasks). 17

18

2. ALTERNATIVE VIEWS AND CONCEPTS OF ROAD SAFETY AUDITS 19

The very idea of an audit on road safety can be surely traced back to the independent (third party) 20

evaluation of safety by a qualified expert (the way it was originally applied on Surrey County assessments, 21

as reported in Bullpit, 1996). Nevertheless, as the original RSA concept stresses (e.g. IHT, 1991 or 22

Bullpit, 1996), it is not a matter of checking. Instead, RSA must add the view on road safety of a qualified 23

professional. The exclusive focus on traffic safety is another specific point, as a mean for warranting 24

attention to its specific concern (road safety), also a feature universally acknowledged in RSA definitions. 25

Departing from audits in other areas, such as accounting and management, the ex-ante (preventing 26

bad results) feature is stressed in RSA (instead of the ex-post checking of practices). For some authors, 27

RSA when applied to the design stage could be considered more similar to the preventive procedure 28

applied in other areas. However, the justification for taking the application to existing roads as another 29

technique, Road Safety Inspections-RSI, seems to be weak as it can be preventively applied to existing 30

roads as well (the major problem, especially in less developed countries). Otherwise, the word assessment 31

would be better than audit in conveying this peculiar meaning of the RSA concept, similarly to areas such 32

as environmental impact assessment (sharing the same preventive and complementary concerns), but 33


distinctively from its general use in accounting and management. Despite agreeing with the latter view, we 1

keep the usual denomination used for the technique, as it is widely adopted nowadays. 2

After this basic agreement, we identified seven features on which the final concept varies: 3

- the essential qualification of experts (especially about accident investigation); 4

- the coverage of RSA application (wideness of the scope for preventive action); 5

- the role of systematic checking (and the importance of checklists); 6

- the requirement for formal reporting (from auditors and from designers); 7

- the focus on human factors and/or all users (from pedestrians with disabilities to truck drivers); 8

- the focus on special road conditions (such as day/night, road works, adverse weather); 9

- the role of multidisciplinary or multirole teams (and their composition). 10

Some of the variations can be seen as inheritance of old questions (traces of the evolution in the 11

concept of RSA) but others are actual and relevant questions meriting careful consideration and study. 12

Among these features, coverage and scope have shown a clear path of evolution. For example, the 13

possibility of applying RSA procedures to existing roads (not only to projects), to maintenance work, to 14

construction planning and execution, to operating procedures, all were finally recognized in a progressive 15

trend, unfolding several fields in which the technique is potentially effective (because they identified areas 16

that bear some burden in road safety and accidents). 17

Another example of evolution is related to the special focus on the needs of all users and on the 18

requirements of all road conditions, just reminding features that are usually present in the causation of 19

road accidents and that supposedly are topics have important shortcomings in current design guidelines (a 20

feature identifying an area of special concern or value for RSA application today). However, these critical 21

concerns can change along time as better guidelines are developed or as new areas of concern appear 22

(maybe due to changes in urban patterns or automation tools, for example). These features can thus be 23

taken as episodic (instead of essential) in defining the concept of RSA. 24

Sometimes it is difficult to distinguish historical questions that evolved during the maturation of 25

the concept and technique (and that were “solved” or “settled” in some sense) from current questions that 26

should be discussed (which leaves some room for personal judgment, of course). 27

For example, in the original sources of the RSA concept, the main requirement for an independent 28

and qualified auditor was expertise on accident investigation. This feature was stressed as a mean to bring 29

the specific knowledge gained in the analysis of traffic accidents to the design process, based on the 30

judgment that design guidelines were insufficient to fully warrant traffic safety (because they stressed 31

other goals or because they use to neglect the importance of some details that matter to safety). 32


This view was clearly stated in the historical review by Bullpit (1996, p.211) when the author 1

notes that the basis of RSA are both the principle “prevention is better than cure” and the use of 2

accumulated knowledge in accident investigation and prevention. The same point is highlighted in the first 3

version of the U.K.IHT Guidelines for Road Safety Audits (IHT, 1991, p.9). Nevertheless, the withdrawal 4

of this highlight from the second edition of the U.K.IHT Guidelines must also be noted, where one can 5

find stronger emphasis on the need of improved and updated knowledge for the experts (IHT, 1996, p.20). 6

What about the change? Of course, RSA knowledge can be a source for the development of road 7

design guidelines and the confidence in the accident investigation and prevention expert has to be 8

conditional. Despite recognizing the importance of the specific knowledge gathered in accident 9

investigation, one can expect that its content should be progressively subjected to scientific investigation 10

and integrated in the evolution of design guidelines sooner or later. One must also recognizes that the 11

knowledge of accident investigation experts is subject to idiosyncrasies and changes (and that there are 12

good and bad experts, even among those with experience on accident investigation). 13

Changes in concept can be the result of changes in the specific or historic settings but can also 14

sign changes in points of view concerning the technique or its practice (changes that can perhaps 15

compromise its final efficiency and effectiveness). For example, the pressure for “certified auditors” can 16

also explain the change in the view about qualified experts. 17

For better, the evaluation of current design guidelines and of RSA criteria should be an empirical 18

question as should be the evaluation of actual experts´ competence. The adoption of one or other view on 19

the concept would be a matter of faith unless one can evaluate the practical effectiveness of each option. 20

Another example: the diffusion of RSA procedures in the mid 1990´s brought an over stressed 21

attention to checklists as a tool for applying RSA (and also for communicating its principles and criteria 22

on practical grounds, of course). Some authors mentioned its use as an essential feature of RSA (e.g. 23

Proctor, Belcher, 1993). The original view (IHT, 1991, Bullpit, 1996) took its use as a mean for reducing 24

conflicts between the RSA and design teams and for coding RSA knowledge (to boost its adoption by 25

designers also). Nowadays, several RSA guides prefer to recommend the use of generic prompt-lists (e.g. 26

TAC, 2001, and FHWA, 2006, the more recent sources for Canada and USA), as opposed to checklists, 27

preserving (or restricting) their use by experienced observers. 28

Objectively, the development of lists has been carried-out by every RSA team and there are 29

several examples of real implementations. Today, one can find checklists or prompt-lists with variations in 30

content and organization, in the type of questions and also in their coverage, even in the form and mode of 31

application, but all these features following personal taste rather than application tests. 32


Unfortunately, the development of these lists did not track well-defined and certified paths, 1

despite requiring considerable work. Then, one can not expect that their use would surely contribute to 2

deliver better results per se. A critical assessment of their content (against each other or against 3

conventional knowledge) gathers really disappointing results. The analysis carried-out by Bornsztein 4

(2001, chapters 3 and 4), that examined checklists for signalized intersections only, identified several 5

questions that were set on excessively generic terms, failing to encode the available safety knowledge as 6

represented in similar guidance used for conventional road safety analysis (e.g. TRRL, 1991, Gold, 1998). 7

For example, the analysis of road surface on a very detailed Canadian checklist (Hildebrand, 8

Wilson, 1999, A-20 or B-12) asks for checking skid resistance along curves, intersection approaches and 9

steep grades (where we know it is critical), then providing valid hints. Nevertheless, the questions on 10

pavement defects and depression areas are generic. Despite being one of the best checklists available in 11

matter of coverage and detail, it is easy to find similar deficiencies in several subjects regularly considered 12

by road safety analysts (a must for auditors too). 13

Another case: the adequate treatment of pedestrians in traffic signals should avoid usual 14

operational traps (vehicle flows starting/stopping at different stages in the same approach, exposition to 15

permitted left turns or leading protected left turns, etc.) and avoid intended/unintended violations (because 16

of excessive delay to pedestrians, of large gaps in priority vehicular flows having right of way, especially 17

at high speed sites, with large crossing distances or binding sight constraints), all these features are 18

unnoticed in the Canadian checklists quoted (see Hildebrand, Wilson, 1999, A-16, 26 or B-10, 11, 16, 17). 19

In some other areas, even the accumulated knowledge of safety analysts is really vague (e.g. on 20

the effect of signs in accidents or on when they are really critical to their occurrence). Use of checklists 21

can ask for improvement of (current) views, and permit their evaluation, as opposed to prompt-lists. The 22

introduction of another phase for the monitoring of the RSA process (Stage 4 in DoT, 2003) after the first 23

12 and 36 months of project opening, should be welcome for providing a toll for filling this need (even if 24

yet to be fully integrated into standard practice; see IHT, 2008; Belcher et al., 2008). 25

Of course, lists serve more than one function (from a formalizing tool that favors an objective and 26

fair evaluation to an aide-mémoire guide for field work) but its main use is instrumental and should be 27

judged by its effectiveness in contributing to the RSA goal (i.e. the prevention of accidents). Again, their 28

adequacy must be investigated on empiric grounds in the task of identifying accident contributory factors. 29

Some other features can, perhaps, be taken as simple matters of concept and practice. For 30

example, formalization of reports (by the auditor and by the designer) has a clear role and function: 31

warranting explicit consideration to all and every question identified as potentially dangerous and prone to 32

cause accidents. There is no informal RSA (an informal procedure would constitute another practice). This 33


feature can be taken as definition, building a practice devoted to warrant that due attention is devoted to a 1

public matter (so the “project owner”, as the sponsor, is also subject to the requirement). 2

Finally, for other questions, the answer can depend on the nature or scale of work (for example the 3

composition of the RSA team and the procedures to be followed on the field tasks). Therefore, concepts 4

and practices tailored to different types of work should be sought. 5

For every advocate or contender of RSA, the precise view on the concept and its essential features 6

should be clearly stated. Our working position is the following one: 7

- the essential qualification of the expert is to have good knowledge about accident causation and 8

correction (not only road design), acquired through experience or at least specific training for the subject; 9

- the preventive action of RSA covers all stages and activities that bear a relevant effect on road safety (i.e. 10

RSA applies to different stages of design and construction, to pre-opening and initial operation, to existing 11

roads and operational procedures, and so on); 12

- systematic checking (and the use of checklists) in RSA procedures is a means (not an essential feature) 13

for achieving a complete evaluation of features, establishing valid and fair criteria, minimizing 14

subjectivity in results and avoiding conflicts with design experts (relevant points, for sure); 15

- formal reporting is a need for warranting real attention to all safety concerns through the documentation 16

of statements and for delimiting responsibilities of designers (the final decision-makers) and auditors (in 17

their advisory role); 18

- the focus on all users and conditions is a technical requirement for RSA, due to their importance in the 19

realm of road safety and the lack of consistent consideration of several needs and limitations (mainly for 20

vulnerable users) in current design guidelines; 21

- the need for multidisciplinary or multirole team, or any other procedural RSA feature, is a practical 22

matter that should justify for itself in each RSA type (multidisciplinary background is a conventional 23

requirement for a traffic safety engineer or expert). 24

Before leaving these conceptual remarks, a final comment should be directed to the role of 25

checklists. Despite over stressed, checklists are the current tools for consolidation of the RSA knowledge 26

base. In our view, their content should receive greater attention than usual and be submitted to systematic 27

and critical evaluation. The lack of studies and methods for justifying the content of proposed checklists as 28

well as the vagueness (more than openness) of most of the questions can undermine their knowledge role. 29

The wide variation in presentation (mostly the level of detail) is also a matter of study (in our study, we 30

related it to the kind of professional performing the extensive field observation). Last, but not least, the 31

need of objective observational hints for identifying a feature as defective or the need of criteria for 32

weighting its relevance to safety is also something to be noted. 33


The points made above can be summarized by stressing that most questions should be the subject 1

of careful empirical studies, maintaining a clear concern in warranting or preserving the potential 2

effectiveness and efficiency of the RSA application practice, instead of taking them as a matter of faith or 3

preference only. Our method of study was conceived to contribute to this effort. 4

5

3. ALTERNATIVE PRACTICES FOR ROAD SAFETY AUDITS 6

As regards practice, two conventional sets of questions should be recognized: practice in RSA use 7

(when and where) and practice in the RSA process (who and how). Other more detailed (and disregarded) 8

questions are related to another set: the procedures applied on RSA work. 9

From the RSA guidelines, a clear trade-off can be observed in setting a rigorous procedure or 10

keeping it a simple and straight task that can be applied with small burden on project costs and schedules. 11

This goal stresses the importance of identifying practices that can deliver a reward for their burden. 12

In the first set of questions (on the use of RSA) are the requirements that elevate it to more than a 13

recommended technique for a class of projects, either as a fully mandatory procedure for the overall set 14

(as for trunk road schemes in the UK) or for a sampling rate of the undertaken projects (perhaps 15

depending on their scale or nature, as in Australia and New Zealand, for small scale projects or works). 16

Despite recognizing that the kind of requirement or motivation for undertaking RSAs could be a factor in 17

determining its final effectiveness, these issues are not of main interest here. 18

We concentrated the investigation on the second and third set of questions, viewed as interlinked 19

ones. Among these questions, we identified six main features on which the practice varies: 20

- the degree of formalization in the process and the allocation of roles (for auditors and others); 21

- the composition of the RSA team (if more than the expert is involved in it); 22

- the identification of tasks (and their allocation among team members); 23

- the procedure for field work (observation team, number of visits, periods/tasks of work); 24

- the type of supporting tools used for RSAs (as measuring or registering devices); 25

- the type of checklists used for RSAs, in special (and its relation to qualification of team members). 26

Some of these questions are really unnoticed by current guidelines, despite its apparent 27

importance for the potential effectiveness of RSA application. Some of them may have effects that depend 28

on the particular use of RSA as they can set the context for small or large scale applications. This 29

dependence could be complex and it was a strong motivation for this research. 30

For example, despite being optional instead of mandatory, the most widely known applications of 31

RSA in Brazil are large scale ones. The first application dealt with 1507 signalized intersections in the city 32

of São Paulo, those that were converted to real time control after 1994 (Scaringela et al., 1999). After, 33


RSA applications moved to overall networks of highways under concession, also voluntarily applied by 1

some of the new infrastructure operators in the State of São Paulo (following current trends of 2

privatization of road administration based on tolling schemes), covering some hundreds of kilometers, all 3

at once (Martinez, 2000). 4

This practice weakens the requirements for formalization (because the application of RSA were 5

motivated by the contractors) but enforces the interest in efficient processes (combining expert role in 6

judgment matters with clerical roles in field observation) and quality assurance in RSA (including 7

training, checklists, supervision, rechecking and so on). Concerns with the multi-role composition of 8

teams is markedly different from the usual recommendation of multidisciplinary composition by 9

emphasizing the combination of functions (rather than competences in complementary subjects). 10

Furthermore, the adoption of a mandatory requirement for RSA in a country with “soft” legal 11

commitment to tort liability, such as Brazil, means a danger of a “bureaucratic” action (i.e. safe on paper, 12

but not on road). The legal apparatus is not the weak point in Brazil, as codes are very similar to European 13

ones, but the lack of commitment to tort liability comes from a fragile institutional framework that 14

undermines the enforcement of law. Calling for attention to RSA use/guidance should be rooted in greater 15

efficiency and effectiveness than in the conventional institutional framework of a developed country. 16

The importance of the practical issues on RSA application raised above (in any context) seems to 17

be obvious and the main goal of this discussion is to stress the need of informed evaluations for several 18

components of practical RSA procedures. 19

Without being exhaustive, one can suggest that at least the following points should be addressed: 20

- the degree of dependence on a safety expert, as it could be a constraint for application of RSA at large if 21

it is not possible to warrant an efficient work routine, combining clerical personnel (or junior professionals 22

at least) and expert supervision but without compromising effectiveness; 23

- the composition of teams, with evaluation of the comparative performance of personnel with different 24

qualifications in alternative tasks as: 25

. field observation, 26

. specialized evaluations, 27

. rating of risk or danger; 28

- the type of supporting tools used, as guide to work (especially, the type of checklist), studying the 29

importance of tailoring them to each work routine for preserving the efficiency of the RSA process, at 30

least in large scale applications (including RSA preparation, e.g. training tasks); 31

- the degree of integration and enforcement of RSA activities and the commitment to tort liability in 32

agencies or regions, features that should be relevant for the design of RSA process, taking into account: 33


. mandatory or recommended RSA application, 1

. level of rigor in calling for attention of RSA use/guidance; 2

- the organization of the RSA work, including field and office activities and the commonly stressed 3

relationship between auditors and designers, their specific requirements and responsibilities, dealing with: 4

. degree of interaction in the RSA team, 5

. procedures for analysis and prioritization, 6

. interaction between RSA team and decision teams. 7

Concluding the theoretical review, our point must be made clearly: the efficiency and 8

effectiveness of RSA must be carefully studied on the field. All effort should be directed to selecting 9

appropriate methods of organization and work that can warrant good results with compatible costs (money 10

and time). Also, recalling what was said at the beginning of this section, rigorous procedures should be 11

kept at a minimum to avoid compromising RSA viability itself. 12

13

4. A NEW METHOD FOR EVALUATING THE EFFECTIVENESS OF ROAD SAFETY 14

AUDITS 15

The previous discussion set forth some questions and the motivation for better evaluation of RSA 16

concepts and their procedures. It remains to deliver a methodological approach that can overcome most of 17

the difficulties one would face in RSA evaluation studies. 18

The study of RSA application has a lot of specific problems coming from its own features (and 19

that should be added to the usual difficulties of traffic safety studies and the analysis traffic accident data). 20

Most data on the effectiveness of RSA application are based on efforts that aimed at evaluating 21

the cost/benefit ratio attained by specific audit trials. For example, based on studies carried-out in Scotland 22

and New Zealand, cost/benefit ratios of 1:15 and 1:20 are usually quoted in several references (Bullpit, 23

1996, pp.413-4, Road Directorate, 1997, p.6-2, CONASET, 2003, p.10). These studies are not clearly 24

presented or discussed in the references but one can grasp that these methods are not prone to the analysis 25

of alternative safety audit procedures in detail. 26

Therefore, a more detailed method of study is needed for evaluating the kind of questions set 27

previously. Nevertheless, difficulties are pervasive. 28

It is usually quoted that, as a preventive action that aims at avoiding traffic accidents, the 29

application of a RSA destroys by itself some of the evidence needed for certifying its value (the accidents 30

that would be caused by the “potential” defects present in the original situation or proposal). Of course, 31

the use of RSA can be regularly evaluated after the implementation of its recommendations, against the 32

alternative scenario of a do-nothing that can be described by the before situation. This procedure will 33


evaluate the accepted recommendations from RSA (of RSA in relation to observed accidents if all 1

recommendations were accepted) and will be open to all criticism driven to before and after studies 2

(mainly the presence of selectivity bias and influence of exogenous factors). 3

Either if data on accidents in the before period is unavailable (as when considering a new road), if 4

it can not be used as the base for evaluation (because of differences between before and after scenarios) or 5

if the effects of RSA recommendations were not observed, this procedure can not be applied. This 6

difficulty is especially relevant for evaluating RSA applied to the design stages or applied as preventive 7

action for existing sites (as no before data can make evident the accident risk), at least without confidence 8

on techniques that can be applied for predicting what would be the accident frequency without the 9

interventions recommended by the RSA application (see Macaulay, McInerney, 2002, McInerney, 2003, 10

and Montella, 2005). 11

Of course, one can try to verify the comparative performance of audited projects against control 12

projects but, in a field in which the effect of random and exogenous factors are so important (as traffic 13

safety), the exercise could be subjected to weaknesses that will shed relevant doubts on results (or at least 14

weakens its statistical significance). This kind of with/without method is a much needed approach for 15

evaluating RSA application, through the comparison of the safety performance of audited or non-audited 16

projects, but would be probably insufficient for elucidating all the set of questions previously discussed. 17

The need (and cost) of subsequent corrections is also a clue on the effectiveness of RSA 18

application to projects, yet to be better explored. The responsibility for keeping the road safe should be 19

less burdensome in audited projects (or in the years after auditing roads). Data on these features could be 20

easier to recover but they will leave aside the evaluation of final safety or would have to compensate for 21

different levels of attained safety in projects or works. Ideally both should be studied together (the final 22

level of safety achieved and the need/cost of interventions after project implementation) and will require a 23

long observation time to provide data required for the evaluation. 24

This view suggests a better approach in which the use of RSA is compared to a do-minimum 25

scenario with monitoring of interventions (instead of a do-nothing scenario without interventions). 26

All these approaches (i.e. the monitoring of accidents after changes, the comparison of accidents 27

against control sites and the follow on of corrections after changes) are mandatory (as they can deliver 28

strong evidence of the effects of RSA on the road) and should be improved. But all of them are coarse for 29

the use intended in this study. 30

This paper presents an alternative approach for studying RSA application that is weaker than 31

certifying its final effect based on accident and cost records but that could be potentially richer for 32

evaluating the effects of detailed procedures and tools used on the field. Complementary methods are, 33


naturally, desirable. The proposed method aims at scientifically evaluating the promises of RSA 1

applications (that could affirm their potential) against some gold standard of diagnosis. 2

This new method has two tools: an evaluation of alternative RSA results and an evaluation of 3

comparative results to AAD (Accident Analysis Diagnostic). Both tools are based on the availability of 4

some Gold Standard (i.e. an accepted method). These components can be taken almost independently. 5

Both analyses are carried-out in the context of simulated RSA applications and they can not evaluate 6

features that appear on real applications only (as those questions related to commitment or implementation 7

of recommendations). 8

The analysis of alternative RSA results can be used to evaluate the application of RSA in all 9

stages (as it relates RSAs) but the analysis of comparative results to AAD can not evaluate the design of 10

all-new projects or even the design of corrective action without further adaptation (as it partially depends 11

on previous accident records). Each tool is distinguished by the evaluation that is taken as the Gold 12

Standard. 13

The evaluation of alternative RSA results asks for the comparison of each alternative procedure to 14

some standard way of doing a RSA. The evaluation is intermediate in the sense that it compares 15

observations gathered with alternative RSA procedures against the agreed standard and not the quality of 16

the results by themselves. Here, the fundamental hypothesis of evaluation is simple: similar results favor 17

the more cost effective procedure or can permit the selection based on the elicited cost/quality trade-off. 18

The evaluation of comparative results to AAD judges the effectiveness of RSA application by 19

comparing the diagnostic of safety problems based on RSA for real sites against equivalent results based 20

on traditional accident studies. It is also intermediate in the sense that it compares results from the 21

diagnostic of problems (instead of trying to verify the real effect on accident and cost records). But now 22

the fundamental hypothesis is that the validity of the RSA application can be demonstrated by showing 23

that it can reach similar conclusions about the safety problems of the studied sites to those that would be 24

gathered from detailed accident data. So, it will decide corrections that would also be similar to the ones 25

decided by accident studies (usually with reduced costs for the traffic agency and for the road users). At 26

least, this comparison can demonstrate that RSA can reach similar results to accident analysis even if 27

accident data are unavailable (even more if accident data are available). 28

The comparison against a Gold Standard is the conventional method of evaluation for diagnostic 29

methods in health studies (see Zhou et al., 2002). In our case, the Gold Standard of each tool is the 30

diagnosis using the standard way of doing RSA for the evaluation of alternative RSA or the diagnosis 31

using traditional accident analysis in the evaluation comparative to AAD. It is surprising that these 32


methods were not previously suggested as a main alternative in the road safety area or, at least, were not 1

similarly conceived. 2

Our study also felt the need to complement these “objective” evaluations with a “subjective” 3

rating (based on technical judgment) about the correctness and importance of further concerns identified 4

by RSA (but not related to the available sample of accident records). Similar approaches were used by 5

other studies. For example, the evaluation of a Road Safety Risk Index by de Leur, Sayed (2002) also felt 6

the need of using an evaluation (partially subjective) of the proneness to accident and its consequences, 7

that was used to weight observations and then called for an objective comparison with accident 8

occurrences (delivering an accident-based measure for validation of the proposed index and procedure). In 9

Australia and New Zealand, this point was developed further and incorporated in a software system (the 10

Road Safety Risk Manager) that can be used in setting priorities to recommendations emanated from 11

RSAs (McInerney, 2003). 12

This effort corresponds to the need for evaluating the consequences of defects. Ideally, measures 13

of the impact on welfare or cost to road users and other roles in society would be required. A large body of 14

literature on the evaluation of diagnostic tests in health studies also deals with the assessment of their 15

economic effectiveness in a wide sense (i.e. considering social costs or welfare, as in Felder, Mayrhofer, 16

2011), by embedding the effects on a framework for assessing/ordering decisions under risk and/or 17

uncertainty. But this further step would require some knowledge about the final effectiveness of RSA 18

recommendations, as in our previous study on the traffic conflict analysis (Pietrantonio, Tourinho, 2006). 19

Our results can be analyzed with simple cross-tabulation methods and subjective evaluation or can 20

be analyzed with conventional statistical modeling methods and inference criteria. Both approaches will 21

be exemplified in the following case study and can assess a wider range of questions on RSA procedures 22

than other methods. Statistical modeling is preferable for evaluating the effect of the procedures 23

controlling for several confounding factors that can affect the results of application work, despite the 24

recognized lack of robustness of techniques based on correlational data (such as regression analysis). 25

Of course, the proposed study approach is not free of criticisms. The case study will demonstrate 26

several shortcomings that can found in the application and the analysis of results as well. The final section 27

will also stress some general points on methodological strengths and weaknesses of the approach. 28

29

5. CASE STUDY OF ALTERNATIVE ROAD SAFETY AUDITS PROCEDURES FOR 30

SIGNALIZED INTERSECTIONS IN THE CITY OF SÃO PAULO/SP, BRAZIL. 31

The city of São Paulo has some 10 million inhabitants and heads a metropolitan region with about 32

18 million inhabitants. It has a very trafficked road network. The congestion level is noticeable for any 33


visitor of the city. The freeway network is comparatively small (around 40 km of freeways that almost 1

circulate the main consolidated area of the city) and the overall road network is heavily dependent on the 2

arterial corridors. Signalized intersections with vehicular and pedestrian peak hourly flows reaching the 3

tens of thousands are usual in the city. The sites selected for study are examples of these usual signalized 4

intersections, located in the consolidated part of the city (the fringe area has smaller junctions). 5

The case study selected four observer types and developed three types of checklists for field work. 6

These are the main features of RSA evaluated in this research. All observations were carried-out as 7

simulated RSAs following common required routines that can be summarized as a first visit to the site 8

during the afternoon peak, extended to the subsequent (night) off-peak period, and a second visit at the 9

morning peak, extended to the subsequent inter-peak period, complemented by subsequent visits in special 10

periods as selected by the observer (nevertheless, real routines varied a lot around the requested one). 11

The different observers were classified as Safety Engineering Expert (Exp, with experience in 12

accident investigation and knowledge of RSA application), Senior Traffic Engineer (Des, with experience 13

in road and signal design), a Road Safety Engineer (SAn, with more than 5 years experience in accident 14

analysis) and a Junior Traffic Engineer (Jun, with basic professional training in traffic engineering, 15

covering design and safety). Checklists were developed, after reviewing several sources (IHT, 1991, 1996, 16

DoT, 1994, Austroads, 1994, INST, 1995, , Hildebrand, Wilson, 1999), keeping in mind these types of 17

observers. A Generic Checklist (G) was developed for the Safety Expert (Exp). A Detailed Checklist (D) 18

was developed for the Design Engineer (Des) and for the Safety Analyst (SAn). A Detailed Checklist with 19

Instructions (I) was developed for the Junior Engineer (Jun). 20

The basic applications are named ExpG, DesD, SAnD, JunI, combining both features, an 21

observer type and a checklist type. Some crossed applications were also carried-out for comparison with 22

the same observer (applied with increasing detail) and a Detailed Checklist of the Safety Expert (ExpD) 23

was always gathered after the regular ExpG task, as it is the standard of most comparisons. Table 1 24

presents the general structure of G and D checklists (I checklists are similar in structure to D checklists 25

with added observation instructions). 26

The research team was composed by the RSA supervisor, who carried-out all the research tasks 27

involved in the submission and collection of RSA checklists, and an independent AAD engineer, who 28

collected accident records for both intersections and did all the research tasks involved in the accident 29

analysis diagnostic. Both worked together in evaluating the results. 30

The practical work applied the proposed RSAs at four signalized intersections but, because of the 31

high number of hours required for application of the field work and comparison of observations in 32

checklists, the application of the analysis procedure were tried in two intersections. Furthermore, given the 33


lack of confidence on traffic accident records (especially regarding the exact location of accident 1

occurrences on the intersection), full data was considered more adequate for one intersection only and its 2

results are reported here. 3

Note that the large number of hours required for field word and comparison of observations is 4

greatly determined by the large number of questions involved in our checklist (43 to 226 per RSA, as 5

shown in Table 1). From a methodological point of view, the method can be better applied to a smaller 6

number of questions whose status is surely known (a clearly deficient and dangerous feature, a clearly 7

deficient but almost safe feature, a clearly sufficient but critical design, and so on), delivering a more 8

controlled application. Reducing the time required for data acquisition and processing would permit large 9

scale studies on alternative RSA methods. Nevertheless, as the evaluation comparative to AAD depends 10

on accident data, this would be a constraint to any application of its further evaluation. 11

12


Table 1. General Features of Generic and Detailed Checklists. 1

Topic Item No.Generic

Questions

No.Detailed

Questions

1. CHOICE OF TYPE AND GENERAL LAYOUT OF INTERSECTION

1.1. INTERSECTION

LOCATION

A. LOCATION IN THE ROAD NETWORK

a. consistency (road and surroundings)

b. functional hierarchy

c. speed approaching the intersection

B. LOCATION IN THE URBAN TISSUE

C. LOCATION OF PEDESTRIAN PATHS

1

1

2

2

1

7

3

11

9

5

1.2. TRAFFIC

CONTROL TYPE

A. SELECTION OF SIGNAL CONTROL 1 5

1.3. INTERSECTION

LAYOUT

A. GENERAL GUIDELINES OF LAYOUT

a. restrictions to movements

b. geometric constraints

c. layout comprehension

d. attention to pedestrians

B. OTHER GEOMETRIC DESIGN ELEMENTS

a. general design

b. physical features of pedestrian crossings

2

1

1

1

3

1

6

5

5

8

12

7

1.4. VISIBILITY IN

THE INTERSECTION

A. VISIBILITY FOR STOPPING IN APPROACHES TO

THE INTERSECTION

B. CONSPICUITY OF INTERSECTION

C. VISIBILITY OF PEDESTRIAN PATHS

1

1

1

6

2

5

2. INTERSECTION

PAVEMENT

A. PAVEMENT FRICTION

B. PAVEMENT DISTRESSES

C. PAVEMENT DRAINAGE

1

1

1

4

8

5

3. INTERSECTION

LIGHTINING

A. VISIBILITY OF THE INTERSECTION AT NIGHT

B. CONTRIBUTION OF LIGHTING TO VISIBILITY OF

SIGNS

1

1

10

9

4. SIGNING A. ROAD MARKINGS

B. VERTICAL SIGNS

C. PEDESTRIAN CROSSINGS

1

1

1

9

9

3

5. TRAFFIC SIGNALS

5.1. VISIBILITY IN

SIGNALS

A. VISIBILITY AT A DISTANCE

a. stopping sight distance

b. conspicuity of signal heads

c. positioning and understanding of signal heads

d. obstructions and interferences

B. COUNFOUNDING AT A DISTANCE

a. vision of signal heads of conflicting movements

b. signal heads of adjacent stop lines sited in the field

of view of approaching drivers

C. VISIBILITY AT THE STOP LINE

a. field of view at the stop line

b. vision of signal heads of conflicting movements

c. intervisibility between conflicting movements

1

1

1

1

1

1

2

1

1

5

7

3

5

3

2

6

3

6

5.2. SIGNAL

TIMING

A. TIMING OF PHASES

B. PHASING PLAN

C. TIMING OF INTERGREENS

D. TIMING FOR PEDESTRIANS

1

2

1

1

9

11

4

9

No.QUESTIONS 43 226


A. Data Gathering and Processing 1

The Co-CP intersection, shown in Figure 1, is at the border of the old CBD (well inside the 2

consolidated area of the city), linking some major radial arterials to the old CBD and to a major arterial 3

ring (East-West) road through the Consolação (Co) avenue. Pedestrian flows are heavy and mainly related 4

to local land uses and to a major university campus located after the intersection, in the Maria Antonia 5

(MA) street (the other street, Cesário Mota, CM, is a minor street with predominantly local traffic). 6

Crossing roads are one way from the Caio Prado (CP) road to the Maria Antonia (MA) and Cesário Mota 7

(CM) streets. Left turn movements are forbidden in the two-way road (Co) as usual in the city. The 8

Consolação (Co) radial arterial carries a heavier traffic in the inbound direction all along the day (due to 9

the connection function to the arterial ring road). Nevertheless, the Consolação (Co) road also has heavy 10

flow on the outbound direction especially during the afternoon peak. The Caio Prado (CP) road has one-11

way operation, combining the access function to the arterial ring road with the function of carrying 12

vehicular flows crossing the arterial (including left turn and return flows to the outbound direction). 13

The intersection is operated under the real time control system of the central area (São Paulo has 14

five real time control areas). Traffic signal plans for the intersection are also shown in Figure 1. The Co-15

CP intersection signal operates with three stages just to handle pedestrians in the crossing of the outbound 16

direction (after the intersection) in the Co avenue (stage E1 in Figure 1). The cycle times are around 120s 17

in the peaks (pedestrian delays are high for some movements, mainly during stage E1) and the two-way 18

avenue receives the major share of green. Pedestrian signal heads are used at all painted pedestrian 19

crossings and the number of pedestrians outside these crossings is small. Nevertheless, the number of 20

pedestrian crossings against their red display is significant (mainly pedestrians movements allocated to 21

stage E1 but crossing in other stages, due to their great delay and the availability of gaps in the traffic). 22


1

Figure 1. Study Intersections: Consolação x Caio Prado (Co-CP) 2

3

4


A representative sample of accident records for pedestrian accidents and vehicular injury 1

accidents was recovered. For PDO accidents, the fraction and number of recovered accident records was 2

smaller and disregarded for the rest of analysis. The AAD engineer carried-out his field observation 3

routines as usual in accident investigations, without having access to data gathered by the RSA observers 4

(the RSA observers had no access to accident data either). Accident records were processed and a list of 5

problems that were related to the recovered sample of accident records was prepared for the intersection. 6

The quality of accident data for the Co-CP intersection for two years (1996, 1997) was better because the 7

accidents could be located precisely in the intersection (as streets change their name before and after the 8

main avenue and pedestrian movements are limited to one side of the main road). This feature is important 9

because precision of the accident analysis diagnostic was felt to be a critical point for further analysis with 10

the proposed method of study (as it is the yardstick for evaluating the final effectiveness of alternative 11

RSA procedures). 12

Our data sample from RSA applications consists of results for each individual question of each 13

subject area. The comparisons were carried-out by the RSA supervisor and submitted to the AAD 14

engineer for discussion only during the research tasks involved in the comparison of results. General 15

criteria of comparison are represented in Figure 2. After RSA observation and data processing, a 16

preliminary comparison of results was subjected to all the field observers. Their comments were used to 17

correct the initial AAD (after a careful discussion between observers and researchers). 18

19


Figure 2. Criteria for Comparing Alternative RSAs and for Comparing RSAs and AAD Results. 1

2

For each quotation of a RSA checklist collected in the field observation, a comparison of 3

agreement was done, taking the results from the Detailed Checklist of the Safety Expert (ExpD) as a gold 4

standard. Quotations in common questions were rated for agreement or disagreement. Additions were also 5

rated for value. Figure 2a graphically sketches the criteria used for the comparison of RSAs. Omissions 6

were not rated as it was felt that their importance would appear in the subsequent comparison. One should 7

note that several agreements/disagreements and additions/omissions were partial (i.e. there could be a 8

partial agreement and a simultaneous addition in the comparison for the same item). 9

In the comparison to the accident analysis diagnostic (AAD), the RSA quotations were classified 10

as coincidences (if supported by the AAD results) or precautions (if not) and rated for relevance. Figure 11

2b graphically sketches the criteria used for the comparison with AAD. 12

The rating for value used general criteria studied for prioritizing deficiencies (Navin et al., 1999, 13

TAC, 2001, de Leur, Sayed, 2002, McInerney, 2002), taking into consideration the predicted danger of 14

accident occurrence (considering frequency of risk and proneness to accident), the expected accident 15

severity (in the event of a probable accident) and also the viability of correction (with measures usually 16

applied in São Paulo). A deficiency was rated into high, medium and low degrees for each feature. Any 17

a. comparison between road safety audits (RSAs) quotations

b. comparison of road safety audits (RSAs) and accident analysis diagnostic (AAD) quotations.


feature rated as low was taken as a sign of a less relevant quotation or secondary deficiency. Otherwise, 1

the deficiency was taken as a main (relevant) one. 2

Of course, there is some degree of subjectivity in the comparison and rating of several quotations. 3

Most of all, the comparison exercise was shown to be burdensome as there is a need to reach a revised 4

reporting of answers to compensate for errors in the place of quotation (keeping the content of the answer 5

itself as the real point, even when the same observation was made when answering to different questions 6

of the RSA). The complexity of the checklists and of the intersections increased the difficulty in filling the 7

forms. Overall, it was felt that the results were robust to the degree of subjectivity. 8

In the comparison to the AAD, the aggregate analysis was supplemented by a finer analysis, 9

classifying questions by type, based on three subject matters (functional design, geometric design, signs 10

and signals), Other segmentations were considered (for example based on the existence of observational 11

clues in checklists for identification or judgment, among other features) but were not analyzed. 12

13

B. Presentation and Analysis of Results 14

The following results are exploratory and intend to discuss the kind of conclusions that can be 15

drawn from the qualitative analysis of the comparisons or on the statistical analysis of their data. Table 2 16

presents the general results of the comparison between RSAs and to the AAD for the main study in the 17

Co-CP intersection. Table 3 presents the results about the average success of each observer in detecting 18

the features listed in the AAD items (called coincidences) or in a superset of non-AAD items (called 19

precautions), with data on each of the four observers on three subject areas (Functional Design, Geometric 20

Design and Design of Signs and Signals), that gave us a sample of 12 measurements. 21

22


1

Table 2. Results on the Comparison of RSAs and the Comparison of RSAs and AAD. 2

3

A qualitative analysis of the results from the comparative performance of alternative RSA 4

procedures (Table 2) can be summarized as: 5

- the degree of concordance achieved was high in all cases; only one average score for agreements was 6

smaller than 9,00 (for the Safety Analyst, mainly for additions of less relevant quotations); 7

- the Safety Expert use of the Generic checklist was comparable to the Detailed one (given the general 8

agreement of observations) but the number of relevant additions was also noticeable; 9

- the number of omissions was high for less qualified personnel (that directly applied the detailed 10

checklist); all of the observers had difficulties in answering questions about lighting and drainage; 11

- the number of disagreements was also high for less qualified personnel (the Safety Analyst disagreed in 12

23,5% of the common quotations with the Safety Expert; nevertheless, the average score for 13

disagreements of 6,60 reveals that some are relevant but most are less important); 14

- the small number of annotations errors and the average score for annotations errors also show a pattern 15

(occurring only for the Junior Professional); 16

Comparison between RSAs ExpDxExpG DesDxExpD SAnDxExpD JunIxExpD

Number of strong agreements 218 189 143 162

Number of weak agreements 1 5 10 2

Average score for agreement 9,91 9,39 8,80 9,40

Number of disagreements 0 25 47 30

Average score for disagreement 0,00 -8,10 -6,60 -8,10

Number of relevant additions 11 10 21 10

Number of secondary additions 6 6 50 38

Average score for additions 4,50 4,90 3,70 3,60

Number of misannotations 0 0 0 4

Average score for misannotations 0,00 0,00 0,00 -10,0

Number of common omissions - 7 7 7

Total number of omissions (7 in ExpD) - 7 24 32

Comparison of RSAs and AAD ExpDxAAD DesDxAAD SAnDxAAD JunIxAAD

Main Coincidences (of 24): 4 3 8 8

Main Precautions: 2 1 7 6

Secondary Coincidences (of 8): 1 0 3 3

Secondary Precautions: 9 9 17 9

Notes: 1- Analysis of quotations of the Road Safety Audits (RSA) and features identified on the Accident Analysis

Diagnostic (AAD); 2- Observers classified as Safety Experts (Exp), Design Engineer (Des), Safety Analyst (SAn) and Junior

Engineer (Jun) and checklists classified as Generic (G), Detailed (D) and Detailed with Instructions (I).


- the number of additions also increased for the less qualified personnel but the average score for additions 1

and the proportion of relevant additions decreased (the Senior Designer had the smaller number of 2

additions, probably by knowing that some features are hard to fix by designing countermeasures). 3

A qualitative analysis of the results from the performance obtained by comparison of the RSAs to 4

the AAD and accidents (Table 3), can be summarized as: 5

- the Safety Analyst and also the Junior Professional had the largest number of detections and precautions 6

(they were also the observers with the highest number of quotations); 7

- no single observer was able to reach a high level of detection; nevertheless, combining all the 8

observations, the “team” was able to detect 20 out of 32 deficiencies (with 15 out of 24 main detections). 9

10


Table 3. Average Observer Performance by Subject Area as %Hit in Identifying Features of the Global Diagnostic

Observed RSA Expert (Exp) Senior Designer (Des) Safety Analyst (SAn) Junior Engineer (Jun)

No.Cases No.Hits %Cases No.Hits %Cases No.Hits %Cases No.Hits %Cases

Main Accident Functional Design 12 2 16.67% 1 8.33% 7 58.33% 4 33.33%

Factors Geometric Design 4 0 0.00% 1 25.00% 0 0.00% 2 50.00%

Signs and Signals 8 2 25.00% 1 12.50% 1 12.50% 2 25.00%

All Subjects 24 4 16.67% 3 12.50% 8 33.33% 8 33.33%

All Accident Functional Design 18 3 16.67% 1 5.56% 9 50.00% 7 38.89%



All Subjects 32 5 15.63% 3 9.38% 11 34.38% 11 34.38%

Main Preventive Functional Design 9 2 22.22% 1 11.11% 4 44.44% 4 44.44%



All Subjects 14 2 14.29% 1 7.14% 7 50.00% 5 35.71%

All Preventive Functional Design 22 8 36.36% 4 18.18% 12 54.55% 8 36.36%



All Subjects 47 8 17.02% 10 21.28% 24 51.06% 15 31.91%

Both Main Factors All Subjects 38 6 15.79% 4 10.53% 15 39.47% 13 34.21%

All Factors All Subjects 79 13 16.46% 13 16.46% 35 44.30% 26 32.91%


For analyzing these results, it was important to keep in mind some peculiarities of each observer. For 1

the study on the Co-CP intersection, all observers had good previous knowledge about the site (as it is a well 2

known intersection in the city) but the Senior Designer and the Safety Analyst had worked previously on the 3

design of the current signal scheme in the past (the Safety Analyst moved from this traffic management area 4

some 5 years ago and the Senior Design remains in charge in this area). The Junior Professional also had some 5

knowledge about the current design and its problems from previous studies. This fact could explain, at least in 6

part, the large number of additions of the Safety Analyst and also the small number of additions of the Senior 7

Designer (who knows and weights most of the constraints to action). Also the performance of the Junior 8

Professional could benefit from his previous knowledge. 9

Some of these features are non-ideal (when considering the proposal for RSAs) and there must be a 10

method to measure and correct their influence. Generally, confounding factors are always present in practical 11

studies and there must be a method for identifying and measuring their effects, as done in our next step. 12

The statistical analysis of the data on the relative performance of each observer was carried-out using 13

logistic regression against a set of explanatory variables. Dependent variables were the relative performance of 14

observers in each of the subject areas (functional design, geometric design and design of signs and signals), 15

measured by the percentage of detections achieved on the features identified by the accident analysis diagnostic 16

(called coincidences) and on the superset of features that were not related to the accident analysis diagnostic but 17

that were judged relevant as preventive actions based on our technical investigation at the site (called 18

precautions). Subject areas were used to allow a more robust aggregate analysis based on the average rating of a 19

group of questions. Analyses could be carried-out using other aggregate groups as types of questions (for 20

example, general or specific questions, questions with or without hints for identification of settings where the 21

deficiency is critical, and so on). Of course, it is possible to carry-out a disaggregate analysis as well. 22

The estimation of logistic regression models is widely discussed in the literature on the analysis of 23

discrete data or choice modeling and is supported by the used of several statistical packages. Our illustrative 24

results were based on the analysis of average detection percentage in each subject area using the nonlinear 25

regression model option (NLSQ) of LIMDEP version 7 (see Greene, 1998, Chapter 19). This option was chosen 26

because the NLSQ procedure permits a better control of model specification and seems to be more robust 27

numerically (also providing a simple tool for evaluating elasticities, as reported ahead). Separate analyses were 28

carried-out for main coincidences, total coincidences, main precautions and total precautions, adding 29

explanatory variables for features of observers: their technical training, their professional experience and by 30

their previous knowledge of the site and their participation in the design of the current project operating at the 31

site for each subject area, represented as scores based on a relative scale, as shown on Table 4. 32

33


Table 4. Further Data for Evaluating the Effect of Selected Features on Observer Performance 1

2

Several specifications were tried, searching for a meaning-full and robust set of models. The individual 3

effects of training and experience were hard to distinguish as the variables were naturally correlated. We tried 4

combinations of training and experience scores as we did with site knowledge and current design scores. 5

Combinations by averages and maximum scores were rationalized as complementarity or substitutability of 6

attributes. We also tried generic intercepts, observer specific dummies or subject area specific dummies. The 7

more meaning-full and robust models used the maximum of training and experience scores, individual scores for 8

previous knowledge and current design, also including a set of observer specific dummies in the model. The 9

resulting model coefficients are presented in Table 5. As can be seen, model estimation shows that: 10

- the inclusion of observer features always increase the adjusted explained variance ratio (even if marginally as 11

in the case of main coincidences) but the Akaike information criteria delivers opposite results (an improvement 12

is indicated only for main precautions); 13

- the coefficients of observer specific dummies are statistically significant in the estimation of the simplest 14

model (the dummy-only specification) but their values are not robust to the inclusion of observer attributes in 15

the more complex model; 16

- in general, the coefficients of observer attributes variables are not statistically significant in the estimation of 17

the more complex model but the parameters values are consistent 18

Observer Subject Technical Training (T)

Professional Experience (E)

Knowledge Of Site (K)

Participation In Project (P)

RSA Expert Functional Design 3 3 2 0

Geometric Design 3 3 1 0

Signs and Signals 3 2 2 0

Senior Designer Functional Design 3 2 3 2



Safety Analyst Functional Design 3 2 1 1



Junior Engineer Functional Design 1 0 3 0



Note: Scores were based on the following qualitative scale

Formal Training: 0=no special training, 1=quick courses, 2=specialized training, 3=expert;

Professional Experience: 0=small experience; 1=1-2 years; 2=2-5 years; 3=more than 5 years

Previous Knowledge: 0=no knowledge; 1=site info; 2= accident info; 3=diagnostic info;

Participation in the Project: 0=no participation; 1=shared information; 2=partial tasks; 3=project leader.


. all signs are as expected, except for the coefficient of training and experience in the 1

explanation of detection of all precautions; 2

. in all the other cases, trained and experienced observers were more effective and the use of 3

junior professionals is justified based on cost considerations only; 4

. in all the cases, participation in the design of the project operating on the site were correlated 5

with less effective performance (there is thus a commitment effect); 6

. in all the cases, previous knowledge of the site were correlated with more effective 7

performance; 8

- observer specific dummies are different and relevant in value even in the more complex model, remaining the 9

more statistically significant coefficients even in this more complex model (signing the presence of important 10

omitted variables, other that the observer´s attributes considered). 11

Such kind of model can be useful for predicting the performance of some standard observer of each type 12

(for example, a observer without previous knowledge or participation on the design) or measuring the effect of 13

changing some variable (for example, the requirement of training or experience for observers). Table 5 shows 14

elasticities to training and experience (MaxTE). The sensitivity is more relevant for coincidences than 15

precautions (increasing the detection, around 20%, by 7% for main coincidences and 5% for all coincidences). 16

The estimated models are suggestive and would certainly be unsound for use in developing guidelines, giving 17

our limited sample. Nevertheless, it is interesting to note that the results are reasonable even in an illustrative 18

application and to emphasize the usefulness of the method of analysis. 19

However, some conclusions can be drawn from the consistency of these results, mainly the adequacy of 20

using less experienced professionals for field observation (with appropriate checklists, given their positive 21

contribution) and the importance of teamwork in field tasks (as no single observer approached the full set of 22

diagnostic features gathered by the AAD). The importance of expert supervision for evaluating quotations 23

seems also clearly supported (as the reported features included several non-critical problems and even 24

misleading points). The sufficiency of the Safety Expert to the final effectiveness of the RSA is not supported 25

(despite the large experience of the observers of this type that collaborated in the research, the main 26

coincidences to the AAD surpassed the 60% only summing all observers quotations). Nevertheless, the role of 27

the expert supervision in prioritizing quotations and achieving a better final evaluation should be stressed. 28

29

30


1

Table 5. Results on the Logistic Regression for Selected %Hit Models (LIMDEP output with NLSQ procedure). 2

3

4

Overall, this research seems to show that the proposed method of study is a valuable alternative. It is 5

important to note that the same approach can also be used to analyzing several finer questions about procedures 6

or tools. For example, our study showed that checklists vary a lot in the way they put questions to the RSA 7

observer and the proposed method can be used to evaluate the comparative success of tort detection in 8

alternative options. Also, our exercise showed that several observations were hard to fill at the sites (especially 9

Accident Factors-Main Coincidences

Model Dummies only dExp dDes dSAn dJun

AdjR2 32.06% Coeff. -1.82455 -1.71298 -1.17412 -0.570545

AIC -0.295 t-ratio -2.52666 -2.56728 -2.45202 -1.52412

Model D&Ob.attributes dExp dDes dSAn dJun MaxTE SKnow ProjP

AdjR2 34.17% Coeff. -3.0216 -2.28376 -1.84587 -1.4823 0.310598 0.183978 -0.38196

AIC 0.173 t-ratio -1.11798 -0.519037 -0.680754 -0.705371 0.45234 0.321319 -0.243038

sensitivity to training or experience at 20% 7.24% at 50% 11.31%

Accident Factors-All Coincidences


AdjR2 39.52% Coeff. -1.97155 -1.85587 -1.18958 -0.562527

AIC -0.511 t-ratio -2.72814 -2.79849 -2.74474 -1.67762


AdjR2 44.58% Coeff. -3.29174 -2.66948 -1.8678 -2.10597 0.193672 0.486311 -0.522563

AIC -0.098 t-ratio -1.31521 -5.96E-01 -7.04E-01 -1.08747 0.310301 0.912817 -3.18E-01


Preventive Factors-Main Precautions


AdjR2 62.41% Coeff. -2.52573 -3.2581 0.374693 -1.04982

AIC 0.112 t-ratio -1.63671 -1.09787 0.854678 -1.90484


AdjR2 97.12% Coeff. -31.8606 -6.55861 3.92506 -43.4787 0.470002 14.1856 -19.7438

AIC -1.959 t-ratio -0.000352 -1.07E-07 1.28E-07 -0.00032 0.16522 0.000313 -6.45E-07


Preventive Factors-All Precautions


AdjR2 67.27% Coeff. -1.5384 -1.38119 -0.034636 -0.671579

AIC -1.233 t-ratio -4.14289 -4.10214 -0.160192 -2.78144


AdjR2 79.88% Coeff. -2.27385 -3.88083 -1.12265 -2.12014 -0.043931 0.563382 0.455656

AIC -1.220 t-ratio -2.19421 -2.26368 -1.13405 -2.5613 -0.155113 2.24287 0.848456

sensitivity to training or experience at 20% -2.48% at 50% -3.88%

Note: MaxTE=maximum of training and experience scores; SKnow=previous knowledge about the intersection;

ProjP=participation in design projects on the intersection.


those related to lighting and drainage) and the effectiveness of engineering measures as supporting data for 1

further analysis can be used as an aid and be evaluated in the same way. These more detailed analyses were not 2

carried-out in this research but seems to be worth of investigation. 3

Note that, comparing to studies in the health area, the confidence on the application of the Gold 4

Standard of diagnosis that is embodied in their terminology (that classify the results of the proposed alternative 5

as True positives, False positives, True negatives, False negatives) we avoided. A similar problem appears on 6

the literature on health studies when recognizing that an action based on the Gold Standard can not be fully 7

effective but this effect is usually attributed to the fail of the remedial action (as therapeutic uncertainty) instead 8

of the accepted diagnosis. There are techniques suggested to correct for imperfect Gold Standards in health 9

studies (see Zhou et al., 2002, for examples) but it is felt that the alternative approach usually applied for 10

evaluating the effect of therapeutic uncertainty is more meaningful also for weighting the diagnostic uncertainty 11

of the Gold Standard and the proposed alternative, similarly. This is our current line of investigation. 12

13

6. CONCLUSIONS. 14

This paper discussed Road Safety Audits (RSA) as a safety improvement technique with the aim at 15

identifying alternative conceptual or practical features that can contribute to the effectiveness of the procedure. 16

A new method of study that can be used to evaluate detailed procedures was proposed. Several features of RSAs 17

were identified and a case study was designed for evaluating three main points: the degree of dependence on a 18

safety expert, the comparative performance of personnel with different qualification for field observation and 19

the adequacy of different types of checklists. Four types of observers and three types of checklists were used for 20

the field study. The new method of study was capable of analyzing these points. 21

After reviewing some methodological questions about the evaluation of RSA applications, the new 22

method proposed two analyses: a comparative analysis of quotations gathered with alternative RSA procedures 23

against a standard (view as an intermediate evaluation task) and a comparative analysis of the RSA results 24

against an accident analysis diagnosis (view as a better evaluation task). Both analyses were revealing in 25

comparing the relative performance of each procedural option. The same procedure can also be applied using 26

any other Gold Standard (e.g. results from the monitoring stage or from an extended RSA process involving 27

supplementary analyses not usually carried-out or further advice from experts not usually participating). 28

The main drawback of the method is the level of effort required for its application. Despite applying the 29

alternative RSA procedures in four intersections, the study was tried in two intersections and finally applied to 30

one of them, due to the large volume of data processing and handling tasks (a feature that can be attributed to 31

the large number of questions of the RSA) and the need of a precise accident data for the comparative diagnostic 32

(needed for the final evaluation task). The comparison of RSAs rated quotations as agreements or disagreements 33


with the standard application (a safety expert with a detailed checklist). The comparison to the accident 1

diagnostic rated the quotations as coincidences with the accident analysis diagnosis or as further precautions. 2

Each quotation was also rated as main or secondary based on the predicted danger of accident occurrence 3

(considering frequency of risk and proneness to accident), the expected accident severity (in the event of a 4

probable accident) and also the viability of correction (with measures usually applied in the city of São Paulo). 5

Also, the dependence on accident data is a major problem, at least for developing countries such as Brazil, and 6

the use of alternative standards for comparisons (as a super-RSA, based on all evidences) is worth studying. The 7

time required for data collection and analysis can be reduced by using a smaller set of well defined questions in 8

a simulated RSA (a practice that can potentially deliver also more confident results). 9

The methods used for analyzing the results are widely accessible. Two options were applied: simple 10

cross-tabulation methods and conventional statistical modeling methods using logistic regression. The general 11

evaluation of results based on cross-tabulation and the inference about the influence of covariates based on 12

logistic regression, both delivered consistent results. More advanced methods are also applicable. 13

Despite the need of subjective rating of quotations, it was felt that the method produced useful results on 14

the global performance of each RSA procedure that supports the use of less experienced personnel for field 15

observation (as they can reach a good agreement with expert observations, using detailed checklists) but with 16

recourse to team work (combining professionals with different degrees of qualification and background 17

experience) and to the supervision of a qualified expert (a clear trend for less relevant quotations and even some 18

misleading observations by less experienced professionals was found). Therefore, the proposed method is 19

thought to be a valuable option for further study, being capable of finer analyses (comparing types of questions, 20

for example) and additional investigations (analyzing observation routines or the use of other data or 21

measurements, for example). Additionally, one can note that the same kind of method can be applied to other 22

safety evaluation procedure based on surrogate measures. 23

Last but not least, one should stress the need of complementary methods of study that can be applied to 24

grasp the value of RSA on real applications based on a direct evaluation of its real final effects on the safety 25

improvement and/or reduction of future corrective actions. 26

27

ACKNOWLEDGEMENTS 28

We would like to thank to the Traffic Engineering Company of the City of São Paulo for providing data and support for 29

most of the work, to the professionals who collaborate as observers in the field study, to FAPESP (the Foundation for 30

Research Aid of the State of São Paulo) that funded the project that supplied the accident data, and also especially to 31

Denise Lima Lopes and Veridiana Maria Armond Vasconcelos (who helped us in the data processing tasks) and to 32

Profa.Cristina Borba (who help us in the revision of a previous version of this paper). The remaining errors are ours. 33

34


1

REFERENCES 2

Austroads (1994) – Road Safety Audits, Austroads, Australia. 3

Austroads (2002) – Road Safety Audits, Second Edition, Austroads, Australia. 4

Austroads (2006) – Road Safety Audit, Guide to Road Safety, Part 6, Austroads, Australia. 5

Belcher, M.; Proctor, S.; Cook, P. (2008) –Practical Road Safety Auditing, Second Edition, Thomas 6

Telford Publishing, UK. 7

Bornsztein, L.L. (2001) – Road Safety Audits Applied to Signalized Intersections, Master of Science 8

Dissertation, Polytechnic School, University of São Paulo, Brazil (in Portuguese) 9

Bullpit, M. (1996) – “Safety audit - an overview”. In Proceedings of the Institution of Civil 10

Engineers, Transport, v. 117, pp.211-215. 11

CONASET (2003) – Guide for Undertaking a Road Safety Audit, National Comission for Traffic 12

Safety, Chile (in Spanish). 13

de Leur, P; Sayed, T. (2002) – “Development of a Road Safety Index”. In Transportation Research 14

Record 1784, Transportation Research Board, USA, pp.33-42. 15

DoT (1994a) – Road Safety Audits (HD 42/94), Department of Transport, UK. 16

DoT (1994b) – Road Safety Audits (HA 19/94), Department of Transport, UK. 17

DoT (2003) – Road Safety Audits (HD 19/03), Department of Transport, UK. 18

EC (2008) – On Road Infrastructure Safety Management, Directive 2008/96/EC, adopted by The 19

European Parliament/The Council of the European Union. 20

Felder, S.; Mayrhofer, T. (2011) – Medical Decision Making – a Health Economic Primer, Springer. 21

FHWA (2006) – FHWA Road Safety Audit Guidelines, Federal Highway Administration, 22

Publication No. FHWA-SA-06-06, Department of Transport, USA. 23

Gold, P.A. (1998) – Traffic Safety – Using Engineering Measures to Reduce Accidents, Inter-24

American Development Bank. 25

Greene, W.H. (1998) – Limdep Version 7.0 – User’s Manual, Econometric Software Inc.. 26

Hildebrand, E.; Wilson, F. ; Editors (1999) – Road Safety Audit Guidelines, Transportation Group, 27

University of New Brunswick, Canada. 28

IHT (1991) – Guidelines of Safety Audit of Highways, Institution of Highways and Transportation, 29

UK. 30

IHT (1996) – Guidelines of Safety Audit of Highways, Second Edition, Institution of Highways and 31

Transportation, UK. 32


IHT (2008) – Road Safety Audits, Institution of Highways and Transportation, UK (Guidelines of 1

Safety Audit of Highways, Third Edition). 2

INST (1995) – Road Safety Audits, Boletim Técnico No.1, National Institute for Traffic Safety, Brazil 3

(in Portuguese). 4

Macaulay, J.; McInerney, R. (2002) – Evaluation of Proposed Actions Emanating from Road Safety 5

Audits, Publication AP-R209/02, Austroads, Australia. 6

Martinez, A (2000) – Road Safety Audit – SP310: km153 to km227; SP225: km91 to km225, 7

CENTROVIAS Highway Systems S/A (in Portuguese). 8

McInerney, R. (2003) – The Road Safety Risk Manager Software Tool: Background Research, 9

Publication AP-R222/03, Austroads, Australia. 10

Montella (2005) - “Safety Review of Existing Roads: Quantitative Safety Assessment Methodology”. In 11

Transportation Research Record 1922, Transportation Research Board, USA, pp.62-72. 12

Navin, F; Ho, T.; Zein, S. (1999) – “Road Safety Audit in Canada Early Success Stories”. In Road 13

Safety Audit Seminar, Institute of Transportation Engineers, USA, pp.101-109. 14

Pietrantonio, H.; Tourinho, L.F.B. (2006) - “A Decision-based Criterion for Selecting Parameters in the 15

Evaluation of Pedestrian Safety Problems with the Traffic Conflict Analysis Technique”. In Transportation 16

Planning and Technology, vol.29, no.3, pp.183-216. 17

Proctor, S.; Belcher, M. (1993) – “The Use of Road Safety Audits in Great Britain”. In Traffic 18

Engineering and Control, v.34, no.2, pp.61-65. 19

Road Directorate (1997) – Manual of Road Safety Audits, Road Directorate, Ministry of Transport, 20

Denmark (available in English). 21

Scaringela, R.S.; Martinez, A.; Rodolpho, A.J. (1996) – “Safety Audit of São Paulo’s City Signaling 22

Projects”. In Eighth International Conference on Road Traffic Monitoring and Control, Conference 23

Publication No.422, Institution of Electrical Engineers, UK, pp.131-135. 24

TAC (2001) – The Canadian Road Safety Audit Guide, Transportation Association of Canada, 25

Canada. 26

TAC (2004) – The Canadian Guide to In-service Road Safety Reviews, Transportation Association 27

of Canada, Canada. 28

TRRL (1991) – Towards Safer Roads in Developing Countries, Transport and Road Research 29

Laboratory, Overseas Development Unit, UK. 30

Zhou, X.-H.; Obuchowski, N.A.; McClish, D.K. (2002) – Statistical Methods in Diagnostic Medicine, 31

Wiley Interscience (Second Edition, 2011, Wiley). 32

33

evaluating road safety audit proceduressites.poli.usp.br/p/hugo.pietrantonio/ersastudy.pdf ·...

Documents