mapping workshop on universal reporting parameters for the speech of individuals with cleft palate...

Mapping

Workshop on Universal Reporting Parameters for the Speech

of Individuals with Cleft Palate

Washington DC, April/May 2004

Tara Whitehill, Hong Kong

Proposed Three-Stage Plan

• Evaluating: The usual speech evaluation by the local speech specialist in cleft palate.

• Mapping: Conversion from the local speech evaluation to the universal reporting measures.

• Reporting: Universal parameters that describe the cardinal features of speech in individuals with cleft lip/palate and related craniofacial anomalies regardless of the individual’s primary language.

Already covered

• Evaluation (range of protocols and procedures; key issues)

• Parameters (proposal; key issues)

• Focus: mapping

Fanny LAW Sze Kuen

International Parameters for Cleft Palate Speech: A Pilot Study

Honours dissertation

Bachelor of Science (Speech & Hearing Science)

University of Hong Kong

May 2004

Supervisor: Dr. T. L. Whitehill

DRAFT - in progress

• 6 sets of client forms from 5 centres

• 5 clients each set = 30 client forms

• Original forms only (no recordings, no additional data such as transcriptions, unless provided on original form)

• Two groups of “mappers”– 5 “experts” - members of WG who supplied

forms

– 5 students - little/no previous experience cleft

• Used “current” version of parameters and guidelines

• Students given session to familiarize/

clarify terminology• Mapped from client forms onto parameters

form• Identifying information masked (client, examiner,

centre)

• Provided feedback on ease/difficulty of mapping each parameter– Either for client form or by set (centre)

• Weighted scores - not included in this study

“Outcome measures”

• Agreement– Within expert group– Within student group– Across all mappers

• Feedback (ease/difficulty)

Why use students?

• In future, unlikely mappers will be naïve

• However, “expert” group was restricted to individuals who had developed and debated the parameters

Agreement

• A big problem was “missing data”• OK if client form stated MD (or WNL or NAD),

but what if didn’t? - assume OK? tick MD?

• Calculated

(a) absolute agreement (Condition 1)

(b) disregarding “missing data” (Condition 2)

Trial Results amongexperts

Results amongstudents

Agreement and explanation

1 1 1 1 1 1 1 1 1 1 Agreed among experts, students and allmappers in both Condition 1 and 2.

2 1 1 1 1 2 2 2 2 2 Agreed among experts, and students in bothCondition 1 and 2

Not agreed among all mappers in bothconditions

3 1 MD 2 1 1 2 1 1 1 Not agreed among any groups in eitherconditions

4 1 MD 1 MD 1 1 1 1 1 Agreed among students in Condition 1 Agreed among experts in Condition 2 Agreed among all mappers in Condition 2

5 MD MD NA 2 MD 2 2 2 MD Trial for experts was deleted as there would be

no agreement when the MD and NA wereexcluded.

Agreed among students in Condition 2 Agreed among all mappers in Condition 2

Key: MD = missing data, NA = no response, 1 = WNL/Absent, 2 = Present

Table 1. Examples to illustrate how agreement was defined

Primary Parameters

Hypernasality

Agreement• Experts Condition 1 - 60%

Condition 2 - 63%• Students Condition 1 - 73%

Condition 2 - 73%• All Condition 1 - 57%

Condition 2 - 60%

Not much missing data

Primary Parameters

HypernasalityFeedback/Issues to consider

• Some difficulties when no. scale points differed (e.g., client form - 7 pt. scale; parameters form - 4 pt. scale)

• In such cases, mappers reported difficulty, and agreement decreased

• Implication: explicit instructions?• Another difficulty: “mild to moderate” or “moderate to

severe” (students selected less severe and experts more severe!)

• “Set B” - definition of “mild” = “evident but acceptable”. One expert rater mapped onto WNL.

Primary Parameters

Hyponasality




Condition 2 - 97%

Excellent, when missing data is accounted for

Primary Parameters

HyponasalityFeedback/Issues to consider

• Missing data

• Otherwise, excellent

• (binary scale)

Primary Parameters

Audible Nasal Emissionwith/without Nasal Turbulence

Level One (absent, present, MD)

• Experts Condition 1 - 63%

Condition 2 - 87%

• Students Condition 1 - 77%

Condition 2 - 80%

• All Condition 1 - 57%

Condition 2 - 70%

Level Two (subcategories)


Condition 2 - 60%


Condition 2 - 38%


Condition 2 - 27%

Primary Parameters

Audible Nasal Emissionwith/without Nasal Turbulence

Feedback/Issues to consider• Low agreement for subcategories• “phoneme specific” - e.g. “audible for plosive and /s/”• Confusion about pervasiveness e.g. “mild but

consistent”, “phoneme specific and frequent”• Results of mirror test (if not audible in speech)• (NE w/wout nasal turbulence - not raised here)

Primary Parameters

Weak Oral Pressures




Condition 2 - 46%

Primary Parameters

Weak Oral Pressures

Feedback/Issues to consider• All 9 mappers found this parameter difficult to map• Primarily because of lack of data on client forms - no

form had specific category for this• Mappers needed to derive the information from other

information; particularly difficult for students• Also, students unclear about definition

Primary Parameters

Substitution ErrorsLevel One (absent, present, MD)


Condition 2 - 73%


Condition 2 - 85%


Condition 2 - 70%

Level Two (subcategories)


Condition 2 - 30%


Condition 2 - 15%


Condition 2 - 20%

• Agreement not bad for level one, but very poor for subcategories; missing data was not the main problem

Primary Parameters

Substitution Errors

Feedback/Issues to consider• All forms included level one (present/absent/MD)• Subcategories: most forms had patterns, but few

included frequency information • Some patterns which expert listeners considered

‘related to cleft palate’ not included here (e.g. nasal substitution/realization’, ‘double articulation’)

• Confusion between this parameter and ‘Developmental Delay or Other Articulation/Phonological Errors’ (even for experts) e.g. omission, backing /t/ -> [k].

Secondary Parameters

Errors related to dentition, occlusion, palatal vault configuration




Condition 2 - 53%

Better when missing data taken into consideration, but still pretty poor


Errors related to dentition, occlusion, palatal vault configuration

Feedback/Issues to consider• No specific category for this on client forms;

information had to be extracted from other information

• Some forms (Sets B, E, F) included categories such as ‘lateralization’, ‘palatalization’, ‘dentalization’ - but still not clear whether these due to organic factors or “Other Articulation/Phonological Errors” - even for experts

• Errors related to hearing problem? fistula?


Developmental Delay or Other Articulation/Phonological Errors




Condition 2 - 43%

Again, better when missing data taken into account, but still pretty poor


Developmental Delay or Other Articulation/Phonological Errors

Feedback/Issues to consider• Mappers needed to make judgement regarding

whether errors were related to cleft or not • Confusion with both “Substitution Errors” and “Errors

related to dentition..”• Client forms rarely specified cause or explanation of

error patterns - mappers needed to judge• Special cases e.g. “no oral consonants”


Voice/Laryngeal Disorder




Condition 2 - 96%

Agreement was good, especially when missing data was taken into account


Voice/Laryngeal Disorder

Feedback/Issues to consider• Similar to hyponasality - mappers were familiar with

term• When data was provided on client form, it was clear• Binary choice; no subcategories

Global RatingSeverity/Intelligibility

• Not decided at time of this study

• Only 2 of the 6 sets included such data

Conclusions

• Reliability lower among students than experts

• Experts were those who developed form

• Who will be mapping in the future?

mapping workshop on universal reporting parameters for the speech of individuals with cleft palate...

Documents

students condition

missing data condition

md md na

absolute agreement condition

speech of individuals

tick md

conditions31 md

universal parameters