A standardization and validity study of a speech and language screening toolA look at a Canadian linguistic minority outside Québec

Michèle Minor-Corriveau, M.Sc.S. Speech-Language Pathologist

Associate ProfessorLaurentian University

Sudbury, Ontario Canada

Abstract : The purpose of this longitudinal and cross-sectional study is to standardize and validate the Speech and Language Profile, a speech and language screening tool developed by and for educational Speech-Language Pathologists. Reliability and validity measures obtained will draw upon data taken from administering the screening tool, which will be measured against each individual child’s academic performance through teacher report and standardized testing, including numeracy and literacy scores obtained by the Education Quality and Accountability Office’s (EQAO) provincial mandatory standardized testing. This study hopes to demonstrate predictive, concurrent and criterion-referenced validity of this tool, as well as interrater agreement and test-retest reliability. Favorable results could support the use of this tool by educational Speech-Language Pathologists to help them determine, at the earliest possible moment in the child’s academic trajectory, which services would be required in order to maximize his/her learning potential. A sufficiently high level of reliability and validity could also support the importance for standardizing this tool with other populations in minority and majority language settings at an international level. Statistical analysis will speak to the measures of internal consistency, content validity and face validity.

Context Methods

Data and results

Conclusions

References

2001 : SLPs from CSCNO school board design the Profile de la langue, du langage et de la parole (PLLP) a tool that would help them identify which children should receive community-based speech and language services upon entry into Junior Kindergarten (from 46 to 58 months)

2007 : PhD research aimed at validating and standardizing the PLLP begins

2008 : an internet poll aimed at French-speaking S-LPs confirms : no gold standard is used to identify speech or language delays in francophone children attending French-language schools

Of 74 respondants (49% response rate) :• 39% = no protocol in place for a universal screening in JK in their school boards• 61% = a protocol is in place for screening in JK their school boards

IS THERE A UNIVERSAL SYSTEMATIC PROTOCOL IN PLACE USED TO HELP DETECT SPEECH AND LANGUAGE DELAY IN YOUR SCHOOL BOARD ? IF SO, WHAT ARE S-LPs USING ?

• Results from ongoing validity and reliability measures will be published as part of the author’s doctoral thesis.• Development of an iPad application currently in the works to enable S-LPs to administer the PLLP electronically and enter data online.

Normative sample and Item analysisChi-square analysis (p<0,05) indicate that on the whole, test items were consistent with existing theory.

in the results for 2004, 2005, 2009 and 2010 ≠ significant difference linked to sex

linked to language (French ; English). Test-retest reliability results show universal agreement among 19/48 items ≠ significant difference on 25/48 items significant difference for 4/48 items (English /l/, s, cup, why).

Interrater reliability results show high interrater agreement, ranging from 0.90 to 0.98.

Criterion-referenced validityAlongside the PLLP, the following tests were administered. Preliminary results yield high correlation between the PLLP vocabulary subtest and the Leiter-R:

• Carrow-Woolfolk (1999)• Concepts and directions (CELF-CDN-FR)• Échelle de vocabulaire en images – Peabody (ÉVIP)• Edmonton Narrative Norms Instrument (ENNI)• Leiter International Performance Scale• Non-word repetition test (Courcy, 2000)• Number repetition (CELF-CDN-FR)• Rapid automatic naming• Sentence repetition (CELF-P)

Predictive validity (preliminary results)The results of 2251 PLLP scores were measured against their report card grades and IEPs from grade 1 to grade 6 as well as on EQAO provincial testing. Preliminary predictive validity results indicate a weak to moderate correlation, depending on the area. Correlation was also weak when measuring academic achievement against EQAO provincial testing, even within the same school year. Correlation of PLLP scores and academic achievement is comparable to that between academic achievement and EQAO provincial testing.

Internal consistency will be measured using the split-half reliability method and Cronbach’s Alpha. Data analysis is ongoing. Preliminary results yield promising findings.

There is sufficient evidence to suggest that the PLLP should be used by S-LPs to help identify which JK students would benefit from further speech and

language assessment.

• Validity measures demonstrate the PLLP’s stability across time (test-retest reliability), and across examiners (interrater agreement).• Preliminary results indicate a high correlation between PLLP and some subtests belonging to the Leiter-R, weak to moderate correlation between the PLLP test results and academic achievement. (grades obtained on report cards in grades 1 to 4, and IEPs up to grade 6). By the same token, data obtained for the same sample indicate a weak correlation between academic achievement in grades 1 to 4 and EQAO provincial test results. Correlation to IEP is pending.

This is an observational and cross-sectional study aimed at demonstrating reliability and validity of the PLLP .

OBSERVATIONAL STUDY

Normative sample (n=2251) : Data obtained through administration of the PLLP in 2004, 2005, 2008, 2009 and 2010. Analysis of data will determine if there is a difference in results obtained during these years, and whether this difference can be attributed to sex or language of administration (French or English).

Test-retest reliability (n=24) is used to measure the correlation between thescores obtained by the same persons on 2 administrations of the test. The PLLP was administered once in October 2008 and again in January 2009. The goal is to establish test stability over time.

Interrater reliability (n=75) refers to the degree to which the same individual would receive the same score, even if the test was administered by different examiners. The PLLP was administered as part of the interrater reliability study. All sessions were videotaped. 5 SLPs who were blind to the original test results rated the children’s test scores using the same test form as usedby the original examiner, while watching the videotaped session. The goal is to establish the transparency of the test.

Criterion-related validity (n=26) refers to the effectiveness with which thetest is able to correlate to an individual’s performance in specified activities or on other tests administered at the same time or within a short interval. The PLLP was administered alongside other assessment batteries normed inQuébec. This data collection is part of another doctoral study (see Mayer-Crittenden, PhD candidate, Human Studies, Laurentian University).

CROSS-SECTIONAL STUDY

Predictive validity refers to the effectiveness with which a test is able to predict future performance in specified activities or on like tests administeredafter a longer interval : Test scores obtained on the PLLP will be measured against the students’ academic achievements (report card results and IEPs) ingrades 1 – 6, as well as provincial test results (EQAO) in reading, writing andmathematics.

TEST ANALYSIS

Item analysis will measure success and failure rates of each item to determine degree of difficulty of items and whether items should be added, substituted or deleted. Internal consistency will measure whether test items belonging to the same subset of scores are correlated and whether they belong to the same construct.Face validity will determine whether the test items are appropriate and to what degree they reliably represent the targeted items. Content validity will link the theory in support of the test items used, those added, substituted or deleted.

• American Educational Research Association, American Psychological Association & National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. USA : American Educational Research Association.

• Anastasi A., & Urbina, S. (I1997). Psychological testing (7th ed). New Jersey : Prentice Hall.• Brown, L., & Bryant, b. (1984). The why and how of special norms. Remedial and Special Education, 5(4), 52-61.

11% - observational checklist14% - parent questionnaire4% - non-word repetition test4% - Fluharty Preschool Screening Test*19% OEW (French equivalent of WBTT)7% PLLP4% TPLS*4% TSRI** unstandardized, unadapted measures

NO33% YES

67%

Wh-Q

uestions

Spati

cal C

oncepts

Vocabular

y

2-Step Dire

ctions

Body Part

s

Articu

lation

8486889092949698

100

% in

terr

ater

agr

eem

ent

Future directions

INTERRATER RELIABILITY

This initiative has been made possible thanks to funding fromLaurentian University/Université Laurentienne and Health Canada (CNFS).

X5

Oct 2008 Jan 2009

IEPA+

EQAO

To S-LPs and School Boards • Lists compiled help prioritize

service delivery and help manage wait lists efficiently.

BENEFITSTo Parents

• Goals are targeted early onto help remediate minor speech

or language issues in theshort term.

To Children• All children entering JK at CSCNO between 2001 and

2010 have been seen and/or flaged as requiring follow-up.

• PLLP Available in French and English• Average administration time = 12 minutes

• Excellent platform for S-LPs to identify students in need of service or follow-up

• See Direct Benefits (below)

Carrow-Woolfolk

Baccalauréat (B.Sc.S.) et maîtrise (M.Sc.S.) ès sciences de la santé (orthophonie)

Michèle Minor-Corriveau, M.Sc.S.Speech-Language PathologistAssociate ProfessorLaurentian UniversitySudbury, OntarioCanada

• PLLP data entered online• Accompanying report provides

recommendations based on results (cut-off scores)

ATTRIBUTES