Safety Science 50 (2012) 1627–1635

Contents lists available at SciVerse ScienceDirect

Safety Science

journal homepage: www.elsevier .com/locate /ssc i

Readability of OHS documents – A comparison of surface characteristics of OHStext between some languages

Geoff Taylor ⇑Department of Civil Engineering, Curtin University, Perth, Australia

a r t i c l e i n f o a b s t r a c t

Article history:Received 2 December 2010Accepted 28 January 2012Available online 11 April 2012

Keywords:OHS documentsReadabilityEight languages

0925-7535/$ - see front matter � 2012 Elsevier Ltd. Adoi:10.1016/j.ssci.2012.01.016

⇑ Tel./fax: +61 8 94576487.E-mail address: wsha@iinet.net.au

Readability of documents, including occupational health and safety (OHS) documents, is a key factor intheir adoption and application. The author uses two measures of English readability favoured by theUS for health communication, and in defense communications respectively. Also included is the use ofmeasures for Chinese, Japanese, Malay/Indonesian, and Spanish. One measure, McLaughlin’s (1969)SMOG formula, was also trialed on documents in five languages other than English. The tests includedparallel texts from China, Europe, Malaysia and the US, and also parallel texts in English, Chinese andSpanish, and on like topics (except for Turkish where available choices were limited). On the dataobtained, it is suggested that the SMOG test with modified criteria may be applicable to Finnish, Hungar-ian, Malay, Spanish and possibly Turkish. Readability of the OHS documents on the measures used variedquite widely, both within and between languages. To put this in context, some internationally compara-ble literacy results for mid high school students are presented.

� 2012 Elsevier Ltd. All rights reserved.

1. Introduction

In a previous paper the author (Taylor, 2010) examined thereadability of a range of government OHS documents used in Aus-tralia to provide advice to employees. Readability of the documentsis very important, not just because they are used by workplaces toproduce safety and health procedures and to support training, butbecause they are important in ensuring observance of the legalduty of care.

In the earlier paper, the author used the SMOG measure of read-ability developed by McLaughlin (1969). In that paper, it was sug-gested that readability could be linked with literacy level, becausea SMOG grade of 10–11 can be linked to the mid high school pop-ulation used in the Program for International Student Assessment(PISA) studies (Lokan et al., 2000, OECD, 2006, 2010). From this itis possible to estimate the percentage of mid-high school studentswho would have difficulty with the access and retrieval, integra-tion and interpretation, and reflection and evaluation require-ments at Level 3 or above on a six level scale.

By requiring a minimum number of sentences, McLaughlin as-serted that using the number of polysyllabic words as a parameterwould avert the need to measure the actual number of syllables.The method found favour with Harvard University in the area ofhealth documents (Rudd, n.d.). Another readability measure, Flesch

ll rights reserved.

Reading Ease (Flesch, 1948, 1949), uses syllables per word, andnumber of words per sentence.

The US Office of Cancer Communications uses the SMOG for-mula for testing public and patient health communications (DHHS,1982). The US military has taken a particular interest in the field inrelation to training and operating manuals. This resulted in theFORCAST formula (Caylor et al., 1973). The US Navy has also devel-oped an index (Kincaid et al., 1975). Some other US governmentagencies use the Flesch–Kincaid Grade Level measure in develop-ing documents for public reading.

The links between comprehension, the cognitive basis for read-ing and readability have been considered extensively (Balota et al.,1990; Flood, 1984; Gilliland, 1972; Israel and Duffy, 2009; Kucer,2009).

Website readability in particular is also receiving attention(Sato et al., 2008).

2. Literacy levels

The PISA studies relate to mid-high school students. Those sur-veyed in 2000 are now in the workforce, although some presum-ably with significantly greater literacy arising from furthereducation.

Literacy measures may also incorporate mathematical, scien-tific, and graphical document literacy not just competence withprose. Readability is only one measure of the demands placed ona reader’s literacy competency by text material.

Table 1PISA 2009 selected data (OECD, 2010).

PISA 2009readingdomain score

PISA 2009 percent not able toeffectively handle materialLevel 3 and above

Australia 515 34.1Canada 524 29.6China PRC–Hong Kong 533 23.9China PRC–Macao 487 45.4China PRC–Shanghai 556 16.4China–Taipei 495 40.1Finland 536 23.7Hungary 494 41.1Japan 520 30.8Korea 539 20.7Mexico 425 72.1New Zealand 521 32.5Spain 481 46.2Turkey 464 56.6UK 494 43.3US–English 500 41.3US–Spanish Not surveyed

OECD average 493 42.4

1628 G. Taylor / Safety Science 50 (2012) 1627–1635

PISA was rerun in 2006 and 2009 with mid high school stu-dents, in 2009 including among others Shanghai students. See Ta-ble 1. Female students outperformed male students in everyparticipating country. Finnish students scored best in the 2006 sur-vey. China – Shanghai students did best in 2009.

Reading comprehension of sixth graders and middle high schoolstudents in Finland was measured by Lehto et al. (2001). That re-search examined the high-level cognitive text processing of pas-sages of text. It found girls outperformed boys regardless of thecomprehension measure. This tends to support the conclusions ofthe PISA studies.

Korea performed well in PISA, above the average for OECDcountries, and has a unique writing system which while ideo-graphic also contains pronunciation syllable by syllable, and sepa-rates words, while by contrast only some characters in Chineseprovide clues to pronunciation (in the right hand part), and wordsare not separated. A sole Chinese character may form a word, orwords can be formed from two or more characters, each of whichmay relate to the word’s meaning, or the word may have an en-tirely separate meaning. Japan also performed well, despite a writ-ing system with one ideographic component and two phoneticones.

While they are not directly comparable with the PISA results,the Australian results in the International Adult Literacy and Lifes-kills Survey (ALLS) conducted in 12 countries suggest that 46% ofAustralians aged 15–74 would have trouble with prose materialwith a literacy demand level of 3 and above on a 5 level scale(ABS, 2006, and see Kirsch, 2001).

3. Language issues

Finland has an enviable history of providing OHS expertise todeveloping countries. Finnish or to give it its correct name,Suomeksia, is an agglutinative language in which added meaningsuch as tense, case and position is achieved by adding morphemesbefore and after a word root. It shares this feature with, amongother languages, Hungarian (correctly called Magyar), Estonianand Turkish. This results in a high proportion of polysyllabic words.To some extent Japanese and Korean can also be included in thisgroup. Malay and Bahasa Indonesia also have many polysyllabicwords, but not to indicate for example tense or case. As Hungaryis a country relatively new to the EU’s overarching health and

safety directives it seemed opportune to include some EuropeanAgency for Safety and Health (EASH) documents in Hungarian inthis further study, as well as some in English, Finnish and Spanish.

Turkey has also sought to join the EU and hosted the 2010 Inter-national Social Security Association (ISSA) World OHS Congress,yet performed relatively poorly in the PISA study (Cinoglu, 2009),so some Turkish government OHS documents have been included.Ulusoy (2006) noted that readability scores need to be validatedfor Turkish.

Some government occupational health and safety documentsfrom the US are also included. This is because the US National Insti-tute of Occupational Safety and Health (NIOSH) and OccupationalSafety and Health Administration (OSHA) attempt to address tosome extent the 40 million Hispanic workers with publicationsin Spanish, so both English and Spanish US documents are in-cluded. Many of the longer words in English share the polysyllabicLatin roots underlying much of Spanish. For example, Spanishworkers will recognise as an adverb a word formed by the finalmorpheme ‘‘mente’’ added to the shared word root, as Englishworkers do with ‘‘ly’’, e.g. rápidamente.

In addition, some documents from the Spanish OHS body Insti-tuto Nacional de Seguridad e Higiene en el Trabajo (INSHT) are in-cluded. Work on readability scores for Spanish has been done(Fernández Huerta, 1959; Parker et al., 2001; Legibilidad, 2007;Barrio-Cantalejo et al., 2008).

By contrast with some of the languages above, Chinese has alower proportion of polysyllabic words, and uses ideographs (eacha single syllable) not a phonetic system. There are also no breaksbetween written words. In the spoken language, the tone not justthe sound of a syllable is crucial to its meaning, as is syntax. Thereare no inflections of words to show tense, but there are tense indi-cators, and like e.g. Japanese there are no plurals. This has advan-tages and disadvantages but one character can replace what inEnglish or Spanish for example may be a fifteen letter word (e.g.qiao – coincidental). Conversely English ‘‘mash’’ is Chinese‘‘daochánghúzhuàng’’. Japanese uses Chinese characters (but notthe simplified PRC form), but unlike Chinese a character may havea polysyllabic Japanese pronunciation or monosyllabic Chinese-de-rived pronunciation (e.g. murasaki, shi – purple). Modern Manda-rin ‘‘tianqì’’– weather is Japanese ‘‘[o]tenki’’. Of course readersdon’t generally consider word sounds if they read silently, but justword meaning. However the inflections in Japanese, e.g. tense,which Chinese doesn’t have, are indicated with a series of phoneticsymbols, each a representing a syllable (e.g. the root sirabe – inves-tigate, can become sirabesaserareru (8 syllables, represented by acharacter for the root, ‘‘investigate’’ and five characters from a syl-labary following it) in the passive of causative tense). Chinese hasno definite article, and Japanese no articles. Note also that China–Taipei and Japan use traditional characters, in which the strokecount is often markedly higher than in simplified PRC characters.

Here are some examples of longer polysyllabic words found inthe OHS documents surveyed: zàimouzhongchéngdùshàng(Chinese – 6 syllables), työturvallisuustoimenpiteissa (Finnish –10 syllables), idegöesszeroppanas (Hungarian – 7 syllables), mem-pertimbangkan (Malay – 5 syllables), musculoesqueleticos (Span-ish – 8 syllables), görevlendirilenlere (Turkish – 8 syllables).

In the author’s previous work there was no specific focus on theparticular problems involving guest workers in Australia who camefrom a number of linguistic backgrounds. It is generally recognisedthough that literacy competency in their first language is variablein this group.

However, recently in a landmark case, a court in New SouthWales ordered a construction company employing many workersof Asian background to provide health and safety information infive languages, including Chinese and Korean (Tooma, 2010). Soaccurate translation and good readability are important.

G. Taylor / Safety Science 50 (2012) 1627–1635 1629

In this context, the Occupational Safety and Health Council(OSHC) in China–Hong Kong produces publications aimed at guestworkers in languages aimed at certain occupational areas, e.g.cleaning (Tagalog, Bahasa Indonesia, Thai) and construction (Hindi,Nepali, Urdu). The Malaysian Department of Occupational Safetyand Health (DOSH) produces some guidance in English, Chinese,Malay and Tamil.

So, OHS documents from Malaysia and Hong Kong are includedin this study. Note however that Malaysia has not participated inthe PISA survey of literacy.

4. Readability

Readability is the starting point for comprehension of a docu-ment. The UK Health and Safety Executive has examined how con-sistent conceptual organisation of OHS documents aids readerunderstanding (Healey and Greaves, 2007).

Readability involves surface attributes. Apart from typographyand layout (Itkonen, n.d.), and how these are perceived (Kucer,qv. Ch. 5, these attributes include word, phrase and sentence struc-ture, perhaps even paragraph structure (Gilliland, qv. Ch. 7). How-ever readability also involves the deeper lexical content of the text,and the cognitive challenge to the reader. It involves the reader’sschemata (Kemper, 1983), that is the reader’s knowledge of ob-jects, situations and events as well as processes (Kucer, qv. at p.135), and in addition the complexity of ideas, the familiarity ofthe subject to the reader and the motivation to read that subject.For example, in the case of OHS the reader’s schema of risk heldin long term memory may be called up, awakening connotations.Readability also includes deeper syntactic structures and concep-tual load and organisation (Valdes et al., 1984; Agnihotri and Khan-na, 1988, working in Hindi), as well as the level of abstraction.(Vachon and Haney, 1991). Reading also probably involves ‘‘chunk-ing’’, which is a general feature of cognitive processing as some-one’s expertise in a field increases. (Ellis, 2001, citing Reason,1979).

DuBay (2004) provides a convenient review of the field of read-ability. He cites early work (Lively and Pressey, 1923; McCall andCrabbs, 1926 et seq., Gray and Leary, 1935; Lorge, 1938), use ofreadability formulas in 14 subject areas including child safety re-straints, and notes that they have been developed for 11 languages(Malay/Bahasa Indonesia is a twelfth).

The review, citing 16 studies, notes that tone, content, organisa-tion, coherence and design of text lie outside readability per sealthough these are important in the cognitive theory of reading.However ways to measure them have proved elusive. Vocabularyand sentence length were found to be the strongest predictors ofdifficulty (Kintsch and Miller, 1981). Syllable count partly reflectsvocabulary. Bormuth (1966) was to identify other key factors.

The review notes that some writers favour morphemes per 100words as a good guide to meaning difficulty. Other measures men-tioned include Yngve word depth (Yngve, 1960) and the LexileFramework (Stenner et al., 1988), which uses average sentencelength and average word frequency referenced from a word corpus.

DuBay gives examples of textual modification resulting fromreadability analysis. Klare (1976) examined whether rewritingwas effective. The Taylor Cloze test introduced in 1953 (Taylor,1953) was an important new approach, based on correctly insert-ing words deleted from a text. This test can be used as a basis tocompare other formulas, and now exists in multichoice and nomultichoice variants.

In the 1960s the plain language movement affected legal, legis-lative and commercial documents, DuBay notes.

There have been attacks on readability formulas, Du Bay’s re-view notes, but several writers emphasise that they should not

be used in isolation. The review cites research which shows thatlimitations aside, the measurable surface features of text have re-mained the best predictors of readability as measured by compre-hension tests.

Different criterion scores are another aspect when comparingformulas, says DuBay. The percentage of correct answers in a read-ing test is a crucial factor. Where one formula may predict the levelof reading skill required to answer 75% of the questions correctlyafter a reading test based on a criterion passage, another mightonly require 50%. The McLaughlin SMOG grading for example, usesa 100% criterion score and so it predicts higher required gradescores for a reader to be successful than Dale–Chall (1948), whichuses 50% comprehension (Gilliland, qv. at p. 95).

DuBay makes the point that motivation to read a text and astrong interest in a subject area do play a part in readability. Meyer(1982) and Armbruster and Anderson (1984) are cited in the re-view as pointing out the benefits of a planned approach to writingtext.

The review also notes that computerised writing aids to assistproduction of more easily readable text have been produced.

Other English readability research includes that of Chall andDale (2000), and Klare (1974). Tekfi (1993) and Ackerman (n.d.)have also provided an overview of readability formulas.

Readability as an issue has featured prominently in health infor-mation materials, not just in English (DHHS, 1982) but in other lan-guages too, e.g. Spanish (Pérez and Couto, 2002; Gröne, 2009;Hernández, 2009), so that consideration in relation to OHS docu-ments is a logical extension of this concern.

Puurtinen (1998) discusses a range of features affecting transla-tion into Finnish and readability of Finnish. Compound construc-tions in Finnish which are lexically more dense may still befound in direct translations, even though the use of more conjunc-tions would make the translation easier to read. Here is an exampleof lexical density from the language of Tierra del Fuego: mamihlap-inatapai – looking at each other hoping that either will offer to dosomething which both parties desire but are unwilling to do. How-ever longer word length does not always imply semantic complex-ity (Kirchner et al., 2009), e.g. Malay mempertimbangkan – ponder.One issue Puurtinen raises is left facing expressions where thereader finds the subject at the end of a descriptive or adjectivalphrase, but such constructions are also common in Chinese andJapanese (e.g. gareeji ni haitteiru roudousha, the worker who isentering the workshop, where the subject, ‘‘worker’’ is the lastword). A similar construction in Chinese would be dài hùmùjìng hétóuk�ui de gongrén (the worker – gongrén- who is wearing gogglesand helmet).

For Spanish, Barrio-Cantalejo et al. (qv.) working in the healthinformation field have developed the Inflesz readability criterionwhich they believe is superior to the Flesch–Huerta (FH) andFlesch–Szigriszt–Pazos (FZP) criteria (Fernandez-Huerta qv., Legi-bilidad, qv.). The Inflesz criterion can be assessed with a computerprogram.

Other readability criteria have also been automated to checkwebsite readability, for example Japanese (Sato et al., qv.) as notedearlier. Microsoft Word � also includes software to perform this inEnglish and Spanish. Smileycat (2007) gives details of a number ofcomputer programs to assess readability. More complex methodsof readability analysis are based on comparisons with a selectedbody of texts (Kirchner et al., qv.). Both Lin et al. (2009) and Lau(2006) have used computer programs to measure readability ofChinese. Li (2000) has published a survey of studies in China ofreadability of Chinese and English, and Yan and Huang (2005) ofprogramming to measure English readability. Hue (n.d.) has alsopublished extensively in this area.

Yunus (1982) has published a formula for estimating the read-ability of Malay, a language virtually identical to Bahasa Indonesia.

1630 G. Taylor / Safety Science 50 (2012) 1627–1635

Courtis and Hassan (2002) compared the readability of com-pany annual reports in Chinese and English and Malay and Englishrespectively. They used the Flesch (qv.) Reading Ease, Yang (seeLau, 2006) and Yunus (qv.) formulas. For the Chinese check anew list of 4645 basic words was compiled. Courtis and Hassanfound that in both cases the English version was less readable.

The Yang (1970) formula for readability of Chinese is given inLau (qv.). It includes a consideration of the number of characterswith 5, 12, 22 and 23 strokes, and the number of complete sen-tences, as well as the average stroke count per character.

Yang used the concept of pseudowords to examine readabilityof Chinese so that one character counted as a word, thus avoidingthe task, for a computer program, of parsing the sentences. Lauinvestigated some characteristics of Chinese which affect readabil-ity. These include the radical (a clue to meaning), symmetry orasymmetry of characters, number of strokes in a character, com-mon and uncommon characters, characters per word, words persentence, number of full sentences, and use of idiom.

Contreras et al. (1999) took the view that Spanish and French,based on surface characteristics, are not inherently less readablethan English, even though there is a relatively higher proportionof Latinate words, and the possessive case demands extra words.Therefore variations in SMOG grades in the three languages donot necessarily reflect changes in reading difficulty. So Contreraset al. offered formulas for conversion of SMOG grades betweenthe three languages, after proposing that there was a systematicbias. Contreras et al. were firmly of the view that while readabilityformulas have limitations, they do provide objective and standard-ised assessment of reading difficulty. There are also cultural factorswhich intrude when trying to measure readability for people read-ing in a second language (Carrell, 1987). Gilliam et al. (1980) exam-ined syllable count and sentence length in comparing readability inEnglish and Spanish.

A perfectly bilingual person who creates text with the sameinformation in both languages should produce text with readabil-ity that is nearly the same in both languages. So even though Span-ish for example tends to use longer sentences and because of itsLatin origins, some longer words, this should be of equal readabil-ity. However where material is originally written in e.g. English, aswith the EASH Factsheets examined in this study, and then trans-lated into e.g. Hungarian, it may reflect the sentence structure andeven some of the phrasing of the English version, rather than thesentence breaks and structure of text originated in Hungarian.Even so, as Puurtinen (qv.) has noted, translation is ongoing andso one language will have evolving effects on the way another lan-guage is written. EASH Factsheets are originated in English by topiccentres and then translated into the other EU official languages.

The revised Tateisi formula for Japanese considers how manystraight runs of the three types of script and any Roman scriptthere are, the average sentence length, and the number of commasand fullstops, while the Yunus formula for Malay (and hence forBahasa Indonesia) considers the average sentence length, and theaverage number of syllables in a word.

Nguyen and Henkin (1985) investigated readability of Vietnam-ese. Das and Roychoudhury (2006) have examined readability inthe Bangla language.

This sets the scene for a preliminary examination using English,US (English and Spanish), Chinese, Finnish, Hungarian, Japanese,Malaysian, Spanish and Turkish occupational health and safetydocuments currently available online.

5. Method

Current online documents in the key OHS areas of manual han-dling, noise and stress were obtained from Finland’s Työtervalli-

suuskeskus (TTK), US NIOSH (in English and Spanish), EASH (inEnglish, Finnish, Hungarian and Spanish), the Hong Kong OSHC(in Chinese and English), the Malaysian DOSH in English and Malay,and the Spanish INSHT. Documents on manual handling and noisewere also obtained from the Japanese Ministry of Health Labourand Welfare (MHLW), but no document on stress could be found.As no documents on these topics could be accessed on Turkey’sISGGM site (ISGGM), regulations relating to construction safetywere selected as well as guidance on pneumoconiosis. All of thesedocuments were from government with the exception of TTK,which offered a preferable range of advice documents in Finnish.However one document from the Finnish government’s Työsuojelu(n.d.) was included (for all of the above, see references). Also thesame passages on manual handling, noise and stress in identicalparts of three books by the author and colleagues in English, Span-ish and Chinese (Taylor et al., 2004, 2006, 2007) were analysed.

This allowed for comparison of parallel texts which have beentranslated from an original language, as well as of the originaltexts. Although the US NIOSH text on manual handling was writtento the same headings in both English and Spanish, the texts are nototherwise parallel.

For consistency, strict attention was paid to diphthongs andother relevant pronunciation rules in syllable counts, even thoughoral readability was not being tested.

Generally, straight prose text was selected, but where bulletpoints were included, these were of a structure which could beread as a complete sentence. Also the emphasis was placed on textwhich gave practical advice. Such an approach to selection neces-sarily led to variable amounts of selected text. It is acknowledgedthat this did not always meet recommended minimum amountsof text specified by some developers of readability scores.

Although there are programs which purport to count syllablesin some languages and even compute a readability score, resultswith some tried previously were questionable. So all countingand computations were manual, except for stroke counts in Chi-nese characters (see below).

Some preliminary analysis using the Flesch–Kincaid system, aswell as the Fernandez Huerta system based on Flesch–Kincaiddid not produce very good agreement about readability betweenEnglish and Spanish. On the other hand, modifications of theMcLaughlin method achieved reasonable correlation of readabilitygrades between five languages, two of which were agglutinative.The revised Tateisi formula (Sato et al., qv.) was used for Japanese,the Yang formula for Chinese and the Yunus formula for Malay.

In using the Yang formula, one assumption made was that allwords were in the basic word list. However it was calculated thateven if 5% were not, this would not markedly affect the Yang score;e.g. in result no. 16 it would lower it by 3 (in 69). The MDBG Chi-nese Dictionary (MDBG, current date) was used to provide strokecounts of the number of strokes in Chinese characters. The Yangformula was developed using traditional Chinese characters, buthere was used for simplified Chinese characters. It has proved dif-ficult to find out from specialist sources in the PRC whether there isa revised formula for simplified Chinese.

Flesch and Flesch–Kincaid scores were not calculated for NIOSH,Malaysian DOSH and Hong Kong OSHC documents in English, asthe focus was on comparisons of SMOG grades and comparisonswith the Spanish, Malay and Chinese measures respectively.

For convenience, the revised Tateisi formula, and the Yang andYunus formulas are given before the Acknowledgments, as they areless accessible than the other formulas.

6. Results and discussion

The results are set out in Tables 2–5.

Table 2Readability of some samples of parallel text.

Averagesyllables perword

No. of polysyllables Average wordsper sentence

SMOG grade(see Table 6)

Flesch–Kincaid Grade Level Flesch reading ease (0–100:100 easiest)

EASH factsheetsEnglish 3 Syllable cutoff1. Manual

handling1.71 109 16.0 13 11 46

2. Noise 1.64 100 18.6 14 11 513. Stress 1.56 96 18.6 13 10 56Finnish 5 Syllable cutoff4. Manual

handling3.13 88 14.5 12

5. Noise 3.20 90 13.3 136. Stress 2.89 68 17.1 11Hungarian 5 Syllable cutoff7. Manual

handling2.78 84 13.7 12

8. Noise 2.83 107 15.2 149. Stress 2.63 86 13.1 12Spanish 4 Syllable cutoff Fernandez Huerta Reading Ease (0–100)a

and (grade level)Flesch Szigriszt Pazos score(0–100: 100 easiest)

10. Manualhandling

2.26 123 20.2 14 65 (8–9) 45

11. Noise 2.21 119 21.1 14 72(7) 4812. Stress 2.27 96 19.2 13 65(8–9) 46EOSH

English3 Syllable cutoff

13. Manualhandling

1.49 48 21.0 12 10 59 (fairly difficult)

14. Noise 1.64 44 17.6 13 11 50 (ditto)15. Stress 1.67 79 22.6 15 13 43 (difficult)EOSH

ChineseNo. of 3, 4 syllablewords

No. of 2syllable words

Average strokes per character, no. of 5, 12,22, 23 stroke characters per 100

Yang score (100 easiest)

16. Manualhandling

1.56 9, 0 (none greaterthan 3)

20.4 204 7.53, 6.88, 2.79, 0, 0 69

17. Noise 1.60 5, 0 (one greaterthan 3; actually 6)

19.3 194 7.81, 11.25, 4.72, 0.18, 0 66

18. Stress 1.68 14, 7 (none greaterthan 4)

19.2 214 7.48, 8.42, 4.21, 0, 0 69

No. of polysyllables SMOG grade(see Table 6)

Fernandez Huerta Reading Ease (0–100)a

(grade level)Flesch Szigriszt Pazos score(0–100: 100 easiest)

EOSHSpanish

4 Syllable cutoff

19. Manualhandling

2.03 48 23.3 12 80 (6) 57

20. Noise 2.06 37 22.1 12 78 (7) 5621. Stress 2.34 80 26.7 15 70 (7) 34

No. of words ranged from 282–706, syllables or characters from 463–1498 and sentences from 16–36.a 90–100 very easy 80–90 easy 70–80 fairly easy 60–70 normal 50–60 fairly difficult 30–50 difficult 0–30 very difficult.

G. Taylor / Safety Science 50 (2012) 1627–1635 1631

Considering readability of documents per se, Table 6 shows theSMOG grades and the corresponding US educational level that aperson must have reached if they are to fully understand a text as-sessed as corresponding to that level. The standard error is 1.5grades.

It is important to bear in mind that Table 6 refers to US educa-tional levels, and that the educational criteria set in those levelsmay have changed over some 40 years. The literacy demands ofthe examples cited may also have varied. However they providea useful indication.

SMOG grades 10–11 as noted earlier may be compared with themid high school population used in the PISA studies (Lokan et al.,2000; OECD, 2006, 2010).

Looking firstly at the English results (nos. 1–3, 13–15, 29–31,35–37 and 44–46), SMOG grades vary from 9 (Grade 9) to 19 (post-graduate), with stress as a topic generally making the highest de-mand on the reader. SMOG uses three syllables or more as acriterion for polysyllables in English. It was decided to use thesame formula but with a four or more syllable criterion for Spanishand Malay and a five or more syllable criterion for Finnish andHungarian. This was based on the idea that due to the nature of

their languages, such readers are inured to words of higher syllablecount, i.e. these languages are not inherently less readable to a na-tive reader, in line with the view of Contreras (1999).

Given the number of factors involved in actually measuringreadability, there is relatively good agreement between results 1,4, 7 and 10 (mean 12.8; also between results 2, 5, 8 and 11 (mean13.8); and likewise between 3, 6, 9, and 12 (mean 12.2). A grade of12 equates to high school graduate, according to McLaughlin, and agrade of 13 to some college (university) education. Result 6 couldsuggest that to a Finnish reader, the subject of stress has beenmade a little easier to read.

In nos. 1–3, and 13–15 the Flesch–Kincaid grade level is lowerthan that indicated by the SMOG grade (see Table 6). This reflectsfindings by Hendrickson et al. (2006).

Results 10–12 include the grade level obtained by the Fernan-dez Huerta (FH) reading ease formula. FH appears to overestimatereadability by comparison with the SMOG grades (if the Contrerasconcept of equal readability of the same material across languagesis accepted). On the other hand the Flesch Szigriszt Pazos (FZP)score appears to assign a readability value higher than that ofSMOG. It is claimed that Inflesz (Barrio-Cantalejo et al., qv.), for

Table 3Readability of samples of some OHS documents from national sources on similar topics.

Averagesyllables perword

No. ofpolysyllables

Averagewords persentence

SMOG grade(see Table 6)

Finland TTK 5 Syllable cutoff22. Manual handling 2.86 34 6.9 823. Noise 2.93 57 14.3 1224. Stress 3.16 46 10.7 11Finland Työjuoselu25. Psychosocial workload 2.96 77 12.8 12Spain INSHT 4 Syllable cutoff Fernandez Huerta Reading Ease (0–

100)a (grade level)Flesch Szigriszt Pazosscore (0–100: 100easiest)

26. Manual handling 2.12 58 18.7 13 74 (7) 5627. Noise 2.15 80 40.0 18 75 (7) 2928. Mobbing 2.28 88 30.2 18 66 (8–9) 34US NIOSH English 3 Syllable cutoff29. Manual handling 1.48 44 16.4 10�

30. Noise 1.45 35 12.9 931. Stress 1.66 110 23.1 15US NIOSH Spanish 4 Syllable cutoff32. Manual handling 2.08 73 23.7 13� 78 (7) 5333. Noise 2.05 53 17.8 11 78 (7) 6134. Stress 2.13 134 24.2 15 75 (7) 50Malaysia DOSH English 3 Syllable cutoff35. Manual handling, in

Guidelines on OHS inAgriculture (GOA)

1.61 53 6.8 10

36. Noise, in GOA 1.63 47 15.4 1237. Guidance for the Prevention

of Stress and Violence1.88 82 29.0 19

Malaysia DOSH Malay 3, 4 Syllable cutoff Yunus grade level38. Manual handling, in GOA 2.57 164, 56 9.5 15# 10^ 5.639. Noise, in GOA 2.66 108, 72 17.0 16# 14^ 9.040. Guidance for the Prevention

of Stress and Violence2.70 167, 81 32.3 #Not calc.,

19^15.0

China – Hong Kong OSHCleaflets – Chinese

No. of 3, 4 syllablewords

No. of 2syllablewords

Avg. strokes per character, no. of 5,12, 22, 23 stroke characters per 100

Yang score (100easiest)

41. Manual handling 1.72 1, 0 (none greaterthan 3)

14.5 84 9.22 57

42. Noise 1.61 13, 8 (none greaterthan 4)

20.7 326 10.15 60

43. Stress 1.33 9, 1 (none greaterthan 4)

13.2 76 9.82 55

China – Hong Kong OSHC books– English

No. of 3 or morethan 3 syllablewords

SMOG grade

44. Occ. Musculosk. Injuries –Prevention

1.68 12 13.0 9

45. Construction – PreventingHearing Loss

1.47 17 11.1 10

46. Workplace Stress –Prevention for Employees

1.81 33 10.8 12

China–Hong Kong OSHC books– Chinese

No. of 3, 4 syllablewords

No. of 2syllablewords

Avg. strokes per character, no. of 5,12, 22, 23 stroke characters per 100

Yang score (100easiest)

47. Occ. Musculosk. Injuries –Prevention

1.63 3, 1 (none greaterthan 4)

23.0 93 10.16, 3.79, 8.33, 0.38, 0 62

48. Construction – PreventingHearing Loss

1.56 1, 0 (none greaterthan 3)

16.5 69 9.90, 8.08, 5.56, 1.01, 0.50 58

49. Workplace Stress –Prevention for Employees

1.75 2, 4 (none greaterthan 4)

9.7 79 9.62, 4.05, 6.76, 0, 0.90 58

� Only the headings are parallel text, # = 3 or more syllable criterion, ^ = 4 or more syllable criterion.a 90–100 very easy 80–90 easy 70–80 fairly easy 60–70 normal 50–60 fairly difficult 30–50 difficult 0–30 very difficult.

1632 G. Taylor / Safety Science 50 (2012) 1627–1635

which a computer program is available, gives values between thetwo. Like nos. 1–3 and 10–12, nos. 13–15 and 19–21 contain di-rectly equivalent text, and again show good correlation of theSMOG grades using the four or more syllable criterion for Spanish.

If we accept the idea that a four or more syllable criterion forSpanish gives reasonably valid SMOG grades, based on the earliercomparisons, then once again in nos. 26–28 and 32–34, FH appearsto overestimate readability, and this is supported by the FZP scoresfor nos. 27 and 28. The FZP scores align fairly well with FH for nos.

26, 32 and 34. As nos. 30 and 33 contain identical material, as do 31and 34, the SMOG criterion appears to be a reasonably valid mea-sure (grades of 9 vs. 11, 15 vs. 15). However if we accept that FHhas greater validity, then the NIOSH material is more readable inSpanish than in English. The scores of 9 and 11 in nos. 30 and 33certainly show the effect of punchy text using short words and sen-tences on measured readability.

We turn now to the comparisons between Malaysian OHS doc-uments. Comparing nos. 35–37 with nos. 38–40 respectively, the

Table 6SMOG grades and educational levels (McLaughlin, 1969).

SMOG grade Educational level Some examples

0–6 Low-literate7 Junior high school8 ‘‘Junior high school’’9 Some high school Readers Digest10 ‘‘Some high school’’11 ‘‘Some high school’’12 High school graduate Time Magazine13–15 Some college New York Times16 University degree17–18 Postgraduate studies Harvard Business Review19+ Postgraduate degree Tax law (US)

Table 4Possible SMOG readability grades for some Turkish government OHS documents.

Turkey ISGGM No. ofwords

No. ofsyllables

No. ofsentences

Average syllablesper word

No. of polysyllables – 4syllable cutoff

No. of polysyllables – 5syllable cutoff

Averege wordsper sentence

SMOG grade(see Table 6)

50. Construction safety 371 1128 16 3.04 125 43 23.2 19#, 12^51. Maintenance and repair 629 1884 22 3.00 222 75 28.6 21#, 14^52. Demolition 253 738 15 2.92 81 32 16.9 16#, 11^53. Pneumoconiosis 147 464 12 3.16 63 28 12.3 16#, 12^

# = 4 or more syllable criterion, ^ = 5 or more syllable criterion.

Table 5Tests of readability of some Japanese national OHS documents on two key topics.

Japan MHLW No. ofwords

No. of characters(Chinese, syllabaryand Roman)

No. ofsentences, fullstops, commas

No. ofcharacters, no.of characterruns

No. ofkatakana, no.of kataganaruns

No. ofhiragana, no.of hiraganaruns

No. of Roman andnumerals, no. ofruns of them

Revised Tateisiscore (0–100:100 easiest)

54. ManualHandling(preventionof lumbago)

Not counted 505 11, 11, 15 183, 82 111, 22 211, 102 0 39

55. Noise Not counted 649 10, 10, 16 325, 120 25, 7 263, 104 36, 26 35

G. Taylor / Safety Science 50 (2012) 1627–1635 1633

SMOG grade using a four or more syllable criterion for Malay pro-vides results reasonably comparable with the English results. How-ever the Yunus grade level estimate suggests only primary schoolcompetence is needed in Malay for two of these documents all ofwhich contain content directly comparable to the English versions.

Results 27 and 40 are notable for their comparatively high sen-tence length.

The next comparison involves nos. 16–18, 41–43 and 47–49.The Hong Kong OSHC leaflets on the three selected topics are notproduced in English, so there is no direct comparison with Englishversions. The Yang scores for nos. 16–18 indicate reasonable read-ability, but somewhat less for 41–43 and 47–49. The SMOG gradesfor the English versions corresponding to nos. 16–18 and 47–49,that is nos. 13–15 and 44–46, indicate grades of mid high schoolto early university in the first case, and from mid high school tohigh school graduate in the second case.

In results 13–15 and 16–18, the Yang scores indicate relativereading ease in Chinese versus SMOG grades equivalent to upperhigh school in English. This finding tends to agree with that ofCourtis and Hassan (qv.). Better agreement between Chinese andEnglish readability occurs for results 45–46 and 48–49.

We turn next to Table 4 on Turkish ISGGM documents. Here twopolysyllable criteria were tested. Levels of 11–14, i.e. upper highschool to some university from a five or more syllable criterion,are probably reasonable estimates for reading government regula-tions and advice on pneumoconiosis. Thus a common criterion fora SMOG grade may be shared with Finnish and Hungarian. Only

further work by native experts such as that called for by Ulusoy(qv.) will tell.

Finally, there is a brief exploration of Japanese OHS documents,as shown in Table 5. The indication from the revised Tateisi score isthat both documents are quite difficult to read.

7. Conclusion

There is quite a wide range of readability in the OHS documentssurveyed. Clearly careful thought needs to be given to who the tar-get audience is when creating such a document. Most guidesappearing on a government or key OHS organisation’s websiteshould aim at maximising audience reach. So the creators shouldensure that the readability of the document is appropriate to thetarget audience, before publication.

It is suggested that Finnish, Hungarian and Spanish readers arecomfortable with a higher proportion of polysyllabic words com-pared to native English readers because that is inherent in theirrespective languages. So one could hypothesise equal actual read-ability between languages for parallel texts on the same topicssuch as the EASH Factsheets, those from the three textbooks citedand two of those from NIOSH, and those from Malaysia. Applyingthe McLaughlin readability test with the varied polysyllable countcriteria would appear, on the limited sample undertaken, to sup-port this contention.

More data will need to be acquired to ensure validity of testingof readability of OHS documents, particularly in languages otherthan English and Spanish.

The SMOG grade using a five or more syllable criterion for Turk-ish suggests an undergraduate grade level for easy reading of theregulations. However further research in Turkey would be requiredto validate use of such an approach to the SMOG grading.

The results suggest that further work by native language read-ers in Finnish, Hungarian, Malay, Spanish and Turkish could beused to refine the coefficients and constant in the regression equa-tion used to obtain SMOG grades in those languages.

8. Formulas

Revised Tateisi:�(0.12 � avg. symbols of all types per sentence) � (1.37 � avg.

no. of Roman symbols per run) + (7.4 � avg. no. hiragana per

1634 G. Taylor / Safety Science 50 (2012) 1627–1635

run) � (23.18 � avg. no. kanji per run) � (5.4 � avg. no. katakanaper run) � (4.67 � no. commas/no. periods) + 115.79. Kanji arecharacters and hiragana and katakana are two types of syllabary.

Yang:

2� f13:90963þ 1:54461 Aþ 39:01497 B� 2:52206 C

� 0:29809 Dþ 0:36192 Eþ 0:99363 F� 0:64671 Gg

A is the proportion of words in a full sentence, B the proportion ofwords in Chinese basic word list, C the average no. of strokes incharacters, D the no. of characters with 5 strokes (in 100 charactersample), E the no. of characters with 12 strokes (in 100 charactersample), F the no. of characters with 22 strokes (in 100 charactersample), G the no. of characters with 23 strokes (in 100 charactersample).

Yunus:0.3793 � average sentence length + 0.0207 � no. of syllables in

300 words � 13.988 (select 300 words from beginning, middleand end of text).

Acknowledgements

Eila Leino of the Finnish Työsuojelu, Dr. Q. Zhang of BeijingInstitute of Technology, Prof. Fan. Y. Xiao of China University ofGeosciences for some book text in Word�, the Japanese Ministryof Health Labour and Welfare for supplying documents or URLswhere they could be found, to Yang Zhap for marking up some Chi-nese documents, to S.L. Chow of the Hong Kong OccupationalSafety and Health Council for some leaflet and booklet text inWord�, to Angela Barrau of EASH, to Cynthia Prentice for some ad-vice on Japanese, to Trish Taylor for some Chinese syllable counts,to Patrick Lau of CUHK and Prof. Yuka Tateisi for some information,to Dr. M. Ulusoy of Gazi University for his comments, and to MartaZimmermann Verdejo of INSHT for assistance in obtaining adocument.

The PLA Foreign Language University in Henan has been askedfor information on current methods of assessing readability of sim-plified Chinese.

The author was unable to obtain a response to his queries fromHungary’s OMMF, Korea’s KOSHA and Turkey’s ISGGM.

References

ABS, 2006. 4228.0 – Adult Literacy and Life Skills Survey, Summary Results,Australia, 2006 (Reissue). <http://www.abs.gov.au/ausstats/abs@.nsf/Latestproducts/4228.0Main%20Features22006%20(Reissue)?opendocument&tabname=Summary&prodno=4228.0&issue=2006%20(Reissue)&num=&view=>and <http://www.oecd.org/dataoecd/44/7/34867438.pdf> (accessed 16.02.10).

Ackerman, R., n.d. Ideosity. <http://www.ideosity.com/SEO/SEO-Readability-Tests.aspx> (accessed 17.08.10).

Agnihotri, R.K., Khanna, A.L., 1988. Evaluating the readability of school textbooks – aIndian study. Journal of Reading 35 (4), 282–288.

Armbruster, B.B., Anderson, T.H., 1984. Producing Considerate Expository Text: OrEasy Reading is Damned Hard Writing (Reading Education Report No. 46).Urbana, University of Illinois, Center for the Study of Reading.

Balota, D.A., Flores d’Arcais, G.B., Rayner, K. (Eds.), 1990. Comprehension Processesin Reading. Lawrence Erlbaum Associates, Hillsdale NJ.

Barrio-Cantalejo, I.M., Simón-Lorda, P., Melguizo, M.I., Escalona, I., Marijuán, M.I.,Hernando, P., 2008. Validation of the Inflesz scale to evaluate readability of textsaimed at the patient. Anales Sis San Navarra (online) 31 (2), 135–152.

Bormuth, J.R., 1966. Readability – a new approach. Reading Research Quarterly 1(3), 79–132.

Carrell, P.L., 1987. Content and Formal Schemata in ESL Reading. TESOL Quarterly 21(3), 461–481.

Caylor, J.S., Sticht, T.G., Fox, L.C., Ford, J.P., 1973. Methodologies for DeterminingReading Requirements of Military Occupational Specialties: Technical ReportNo. 73-5. Human Resources Research Organization, Alexandria, VA.

Chall, J.S., Dale, E., 2000. Readability Revisited: The New Dale–Chall ReadabilityFormula. Brookline Books, Inc., Brookline, MA.

Cinoglu, M., 2009. What does PISA 2003 mean for Turkey? Firat University Journalof Social Science 19(1), 43–50. <http://web.firat.edu.tr/sosyalbil/dergi/arsiv/cilt19/sayi1/043-050.pdf> (accessed 05.09.10).

Contreras, A., Garcia-Alonso, R., Daye-Contreras, F., 1999. The SOL formulas forconverting SMOG readability scores between health education materialswritten in Spanish, French and English. Journal of Health Communication 4,21–29.

Courtis, J.K., Hassan, S., 2002. Reading ease of bilingual annual reports. Journal ofBusiness Communication 39 (4), 394–413.

Dale, E., Chall, J.S., 1948. A formula for predicting readability. Educational ResearchBulletin 27 (1), 11–20 and 28.

Das, S., Roychoudhury, R., 2006. Readability modelling and comparison of one andtwo parametric fit: a case study in Bangla. Journal of Quantitative Linguistics 13,17–34.

DHHS, 1982. Readability Testing. In Pretesting in Health Communications. Methods,Examples, and Resources for Improving Health Messages and Materials (pp. 10–11, 43–47). US DHHS, Public Health Service, National Institutes of Health,National Cancer Institute, Bethesda, MD.

DOSH (Malaysia), 2000. Guidelines on Occupational Safety and Health inAgriculture (Garis Panduan Keselamatan dan Kesihatan Pekerjaan dalamPertanian). <http://www.dosh.gov.my> (accessed 04.10.10).

DOSH (Malaysia), 2001. Guidance for the Prevention of Stress and Violence at theWorkplace (Panduan Bagi Pencegahan Tekanan dan Keganasan di TempatKerja). <http://www.dosh.gov.my> (accessed 20.10.10).

DuBay, W.H., 2004. The Principles of Readability. Costa Mesa, California, ImpactInformation. <http://www.impact-information.com/impactinfo/readability02.pdf> (accessed 29.08.10).

DuBay, W.H., n.d. Costa Mesa California, Impact Information. <http://www.impact-information.com/impactinfo/klarepubs.htm> (accessed 23.09.10).

EASH (various dates). <http://osha.europa.eu/en/publications/factsheets/> (accessed11–12.09.10).

Ellis, N., 2001. Constructions, Chunking and Connectionism. The Emergence ofSecond Language Structure (revised chapter for Doughty and Long (Eds.)),Handbook of Second Language Acquisition. Blackwell, Oxford. <http://web.mac.com/ncellis/Nick_Ellis/Publications_files/DoughtyLongall.pdf>(accessed 03.10.10).

Fernández Huerta, J., 1959. Medida Sencillas de Lecturabilidad. Consigna 214, 29–32.

Flesch, R., 1948. A new readability yardstick. Journal of Applied Psychology 32, 221–233.

Flesch, R., 1949. The Art of Readable Writing. Harper, New York.Flood, J. (Ed.), 1984. Understanding Reading Comprehension. International Reading

Association, Newark, DE.Gilliam, B., Pefia, S.C., Mountain, L., 1980. The fry graph applied to Spanish

readability. The Reading Teacher 33, 426–430.Gilliland, J., 1972. Readability. University of London Press, London.Gray, W.S., Leary, B., 1935. What Makes a Book Readable. Chicago University Press,

Chicago.Gröne, O., 2009. Inventario de Instrumentos para Medir la Legibilidad de un

Texto en Castellano. HPH Catalunya. http://www.hospitalclinic.org/Portals/0/hospital%20clinnic/corporaci%C3%B3/20091210Inventario_instrumentos_textoescrito.pdf (accessed 03.11.10).

Healey, N., Greaves, D., 2007: A Review of Consistency of References to RiskManagement Frameworks in HSE Guidance HSL/2007/13. HSE, Buxton. <http://www.hse.gov.uk/research/hsl_pdf/2007/hsl0713.pdf> (accessed 16.02.10).

Hendrickson, R.L., Huebner, C.E., Riedy, C.A., 2006. Readability of pediatric healthmaterials for preventive dental care. BMC Oral Health 6, 14. <http://www.ncbi.nlm.nih.gov/pubmed/17109743> (accessed 28.11.10).

Hernández, I.R., 2009. Influencias del Nivel Académico de un Texto y el Lugar deOrigen de un Lector en la Comprensión de Lectura de Mensajes de Comunicaciónen Salud. Observatorio (OBS⁄) Journal 8, 122–137, http://www.obercom.pt/ojs/index.php/obs/article/download/232/253 (accessed 20.11.10).

Hue, C., n.d. Curriculum Vitae. <http://140.112.62.203/main2008.html> (accessed26.10.10).

INSHT (various dates). <http://www.insht.es/portal/site/Insht/menuitem. 1f1a3bc79ab34c578c2e8884060961ca/?vgnextoid=d8388dd6caa62110Vgn VCM100000dc0ca8c0RCRD&vgnextchannel=75164a7f8a651110VgnVCM100000dc0ca8c0RCRD> (accessed 17.09.10, 21.09.10).

ISGGM (various dates). <http://www.csgb.gov.tr/csgbPortal/isggm.portal> (accessed1–02.09.10).

Israel, S.E., Duffy, G.G. (Eds.), 2009. Handbook of Research on ReadingComprehension. Routledge, New York.

Itkonen, M., n.d. Typography and Readability. <http://papunet.net/selkokeskus/fileadmin/tiedostot/muut/Itkonen.pdf> (accessed 03.10.10).

Kemper, S., 1983. Measuring the inference load of a text. Journal of EducationalPsychology 75, 391–401.

Kincaid, J.P., et al., 1975. Derivation of New Readability Formulas (AutomatedReadability Index, Fog Count and Flesch Reading Ease Formula) for NavyEnlisted Personnel. Springfield, [US] National Technical Information ServiceReport AD-A006 655/5GA. For abstract see <http://eric.ed.gov/ERICWebPortal/custom/portlets/recordDetails/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValue_0=ED108134&ERICExtSearch_SearchType_0=no&accno=ED108134>(accessed 17.02.10).

Kintsch, W., Miller, J.R., 1981. Readability: A View from Cognitive Psychology. In:Teaching: Research Reviews. International Reading Association, Newark, DE.

Kirchner, J.S., Nuger, J., Zhang, Y., 2009. An extensible crosslinguistic readabilityframework. In: Proceedings of the 2nd Workshop on Building and UsingComparable Corpora, ACL-IJCNLP 2009, 11–18, Suntec, Singapore, 6 August.<http://www.mt-archive.info/BUCC-2009-Kirchner.pdf> (accessed 26.09.10).

G. Taylor / Safety Science 50 (2012) 1627–1635 1635

Kirsch, I., 2001. The International Adult Literacy Survey (IALS): Understanding whatwas Measured, ETS Research Report RR-01-25, Princeton. <http://www.ets.org/Media/Research/pdf/RR-01-25-Kirsch.pdf> (accessed 16.02.10).

Klare, G.R., 1974. Assessing readability. Reading Research Quarterly 10 (1), 62–102.Klare, G.R., 1976. A second look at the validity of the readability formulas. Journal of

Reading Behavior 8, 129–152.Kucer, S., 2009. Dimensions of Literacy, third ed. Routledge, New York.Lau, T.P., 2006. Chinese Readability Analysis and its Applications on the Internet.

M. Phil. Thesis. The Chinese University of Hong Kong. <http://www.cse.cuhk.edu.hk/~king/PUB/thesis_tplau.pdf> (accessed 25.10.10).

Legibilidad.com, 2007. <http://www.legibilidad.com/home/index.html> (accessed23.08.10).

Lehto, J.E., Scheinin, P., Kupiainen, S., Hautamäki, J., 2001. National survey of readingcomprehension in Finland. Journal of Research in Reading 24 (1), 99–110.

Li, S., 2000. A Survey of Studies on Readability. Journal of PLA University ofForeign Languages, April. <http://en.cnki.com.cn/Article_en/CJFDTOTAL-JFJW200004000.htm> (accessed 15.10.10).

Lin, S., Su, C., Lai, Y., Yang, L., Hsieh, S., 2009. Assessing text readability usinghierarchical lexical relations retrieved from word net. ComputationalLinguistics and Chinese Language Processing 14, 1, 45–84. <http://www.aclweb.org/anthology/O/O09/O09-3003.pdf> (accessed 05.10.10).

Lively, B.A., Pressey, S.L., 1923. A method for measuring the vocabulary burden oftextbooks. Educational Administration and Supervision 9, 389–398.

Lokan, J., Greenwood, L., Creswell, J., 2000. The PISA 2000 Survey of Students’Reading, Mathematical and Scientific Literacy Skills (Paris, OECD). <http://www.oecd.org/dataoecd/44/63/33692793.pdf> (accessed 17.02.10).

Lorge, I., 1938. The Semantic Count of the 570 Commonest English Words. Bureau ofPublications, Teachers College, Columbia University, New York.

McCall, W.A., Crabbs, L.M., 1926, 1950, 1961, 1979. Standard Test Lessons inReading. Teachers College, Columbia University Press, New York.

McLaughlin, G., 1969. SMOG grading: a new readability formula. Journal of Reading12 (8), 639–646. <http://www.harrymclaughlin.com/SMOG.htm> (accessed16.02.10).

MDBG (current date). <http://www.mdbg.net/chindict/chindict.php?page=chardict>.Meyer, B.J.F., 1982. Reading research and the composition teacher: the importance

of plans. College Composition and Communication 33, 1, 37–49.MHLW (various dates). <http://www.mhlw.go.jp/new-info/kobetu/roudou/gyousei/

anzen/index.html> (accessed 10.09.10).Nguyen, L.T., Henkin, A.B., 1985. A second generation readability formula for

Vietnamese. Journal of Reading, 29(3), 219–225. <http://www.jstor.org/pss/40029662> (accessed 27.08.10).

NIOSH. <http://www.cdc.gov/niosh/pubs/all_date_desc_nopubnumbers.html>(accessed 25.08.10).

OECD, 2006. PISA 2006 Science Competencies for Tomorrow’s World. Chapter 6Tables. <http://www.oecd.org/dataoecd/31/0/39704446.xls> (accessed 18.09.10).

OECD, 2010. PISA 2009 Results: What Students Know and Can Do. Vol. 1, p. 196.<http://www.oecd.org/document/61/0,3746,en_32252351_32235731_46567613_1_1_1_1,00.html> (accessed 09.06.11).

OSHC (various dates). <http://www.oshc.org.hk/eng/media/communicate.asp>(accessed 16.09.10).

Parker, R.I., Hasbrouck, J.E., Weaver, L., 2001. Spanish readability formulas forelementary level texts: a validation study. Reading and Writing Quarterly 17,307–322.

Pérez, A.B., Couto, U.G., 2002. Legibilidad de Las Páginas Web Sobre Salud Dirigidasa Pacientes y Lectores de la Población General. Revista Española de Salud

Pública 76 (4). <http://www.scielosp.org/scielo.php?script=sci_arttext&pid=S1135-57272002000400007&lng=en&nrm=iso&tlng=es> (accessed 30.10.10).

Puurtinen, T., 1998. Syntax, readability and ideology in children’s literature. Meta:Translators’ Journal 43(4), 524–533. <http://id.erudit.org/iderudit/003879ar>(accessed 08.09.10).

Reason, J.T., 1979. Actions not as planned. The price of automatisation. In:Underwood, G., Stevens, R. (Eds.), Aspects of Consciousness: PsychologicalIssues, vol. 1. Academic Press, London.

Rudd, R.E., n.d. Health Literacy Studies. <http://www.hsph.harvard.edu/healthliteracy/materials.html> (accessed 18.02.10).

Sato, S., Matsuyoshi, S., Kondoh, Y., 2008. Automatic assessment of japanese textreadability based on a textbook corpus. In: Proceedings of the 6th LREC(European Language Resources Association).

Smileycat, author unknown, 2007. Write Better: Online Testing Tools Compared.<http://www.smileycat.com/miaow/archives/000875.php> (accessed 15.10.10).

Stenner, A.J., Horabin, I., Smith, D.R., Smith, R., 1988. The Lexile Framework.Metametrics, Inc., Durham, NC.

Taylor, W.L., 1953. Cloze procedure: a new tool for measuring readability.Journalism Quarterly 30, 415–433.

Taylor, G.A., 2010. Read it and heed it –readability of OHS documents. Journal ofHealth, Safety and Environment 26 (5), 471–483.

Taylor, G.A., Easter, K.M., Hegney, R.P., 2004. Enhancing Occupational Safety andHealth. Elsevier, Oxford (also e-book 2010, Perth, Enhancing Safety Pty. Ltd.).

Taylor, G.A., Easter, K.M., Hegney, R.P., 2006. Mejora de Seguridad y Salud en elTrabajo. Elsevier Espana, Madrid.

Taylor, G.A., Easter, K.M., Hegney, R.P., 2007. Zhiye Anquan yu Jiankang(Occupational Safety and Health). Chemical Industry Press, Beijing.

Tekfi, C., 1993. Readability formulas: an overview. Journal of Documentation 43 (3),261–273.

Tooma, M., 2010. Danger! Peligro! Publicity Order in Five Languages. http://www.safetysolutions.net.au/articles/42890-Danger-Peligro-Publicity-order-in-five-languages (accessed 04.10.10).

TTK (various dates). <http://www.tyoturva.fi/> (accessed 01.09.10).Työsuojelu (n.d.). <http://www.tyoturva.fi/tyosuojelu_tyopaikalla/psykososiaalinen_

tyokuormitus> (accessed 20.10.10).Ulusoy, M., 2006. Readability approaches: implications for Turkey. International

Education Journal 7 (3), 323–332.Vachon, M.K., Haney, R.E., 1991. A procedure for determining the level of

abstraction of science reading material. Journal of Research in ScienceTeaching 28, 343–352.

Valdes, G., Barrera, R., Cardenas, M., 1984. Constructing matching texts in twolanguages: the application of propositional analysis. The Journal for theNational Association of Bilingual Education 9, 3–19.

Yan, S., Huang, L., 2005. English readability measurement programming. Journal ofChongqing University (Social Sciences Edition) 2. <http://en.cnki.com.cn/Article_en/CJFDTOTAL-CDSK20050200N.htm> (accessed 05.11.10).

Yang, S.J., 1970. A Readability Formula for Chinese Language. University ofWisconsin, unpublished doctoral dissertation.

Yngve, V.H., 1960. A model and hypothesis for language structure. Proceedings ofthe American Philosophical Association 404, 444–466.

Yunus, K.R., 1982. An Assessment of Structural Variables in Malay: A ReadabilityFormula. Unpublished doctoral dissertation. Coral Gables, University of Miami,p. 118.