+

English Writing Placement Test Research Feb. 20, 2018 Vanessa Vega, Ph.D., External Evaluator & Education Researcher

+Agenda

n Context of the work

n Disproportionate Impact Analyses

n Subsequent Validity Survey

n Subsequent Validity Analysis, Karen Engel, CoA & PCCD IR

n Q&A

+Context for the research Title V requires CCCCO to ensure placement tests:

n  minimize or eliminate cultural or linguistic bias n  yield valid and reliable information for placement purposes.

(Specifically, for writing tests, the grading process, which is more subjective, must yield evidence of inter-grader reliability)

n  Are used in the manner for which they were approved (e.g., standardized implementation)

n  meaning of scores are clearly communicated to students

PCCD’s application to use a locally constructed writing test for English placement was approved by CCCCO in Nov 2016, pending additional required evidence:

n  Disproportionate Impact Analysis n  Subsequent Validity survey

PCCD’s Chancellor’s directive: Consistency in student experiences across colleges

+Subsequent Validity Survey: PCCD students in Fall 2017 English courses who took Writing Test during April – Sept 2017.

n  34% Response Rate

n  40% of students who took the writing test had high school measures

+Subsequent Validity Survey: Student survey

Source: PCCD Subsequent Validity Student Survey (Administered Nov. 2017)

+Subsequent Validity Survey: Student survey

Source: PCCD Subsequent Validity Student Survey (Administered Nov. 2017)

+Subsequent Validity Survey: Student survey

Source: PCCD Subsequent Validity Student Survey (Administered Nov. 2017)

+Subsequent Validity Survey: PCCD students in Fall 2017 English courses who took Writing Test during April – Sept 2017.

Source: PCCD Subsequent Validity Student Survey (Administered Nov. 2017)

+Subsequent Validity Survey: CoA Student Survey

Source: CoA Subsequent Validity Student Survey (Oct. 2017 Courtesy of Karen Engel, PCCD Research)

+Subsequent Validity Survey: Faculty Survey

n  40% Response Rate

n  32 faculty rated 421 of their currently enrolled students who took the writing test between Apr-Sept 2017

Source: PCCD Subsequent Validity Student Survey (Administered Dec. 2017)

+Disproportionate Impact Analysis

n  No evidence of any serious disproportionate impact associated with the Writing Assessment Test

PCCD Demographic Group

NMean

Writing Test Score

80% of Dominant Group Performance

Males 522 5.42 > Females 523 5.59 4.47

+Disproportionate Impact Analysis

n  No evidence of any serious disproportionate impact associated with the Writing Assessment Test

PCCD Demographic Group

NMean

Writing Test Score

80% of Dominant Group Performance

African-American 241 5.26 > 

Latino-Hispanic 360 5.39 >  Asian 210 5.5 >  White 145 6.1 4.88 Multiracial 82 5.68 >  All Other 24 5.42 > 

+Disproportionate Impact Analysis

n  No evidence of any serious disproportionate impact associated with the Writing Assessment Test

PCCD Demographic Age Group

NMean

Writing Test Score

80% of Dominant Group Performance

13-17 128 6.09 4.87

18 422 5.54 >  19 180 5.29 >  20-29 255 5.35 >  30-39 47 5.53 >  40 or older 30 5 > 

+ Writing Test Scores showed no evidence of disproportionate impact, but enrollment rates suggest disproportionate access to transfer-level courses (<60%) among non-dominant groups

75%

48% 43% 34%

51%

17%

43%

100%

46%

16%

40% 38% 39%

32%

50% 57%

0%

35%

9% 12% 20%

27% 17%

33%

0% 0%

19%

0%

20%

40%

60%

80%

100%

Transfer Level One-level below 2+Levels Below

Fall 2017 Eng. Course Enrollments Among Various Ethnicity Groups of Writing Test-Takers July-Sept. 2017

+

Data Source: PCCD Institutional Research, Writing Assessment Scores by test site, (July- Sept., 2017).

26%

69%

56% 53% 55%

15% 8% 11% 12% 11%

59%

23% 33% 35% 34%

0%

20%

40%

60%

80%

100%

CoA BCC Laney Merritt PCCD

Writing Test Scores & Meaning at Each College (Jul.- Sept 2017, n=649)

Transfer Level English (1A)

One Level Below Transfer

Two or More Levels Below

Scores: 6, 7, 8 = Transfer-level; 5 = 201B; 4 = 201A; 2, 3 = 2 + levels below

+

Data Source: PCCD Institutional Research, Fall 2017 Course Enrollments (Actual Placements) Among Writing Test Takers between July - Sept., 2017, n=612. Note: Merritt did not place students into 201B; Laney did. Sample and percentages do not include courses such as 208, “14,” “210” “5,” and “70A.”

38%

68%

32%

41% 46%

6%

32%

5%

42%

21%

55%

1%

63%

18%

33%

0%

20%

40%

60%

80%

100%

COA BCC Laney Merritt PCCD

Fall 2017 Eng. Actual Course Enrollments Among English Writing Test Takers July-Sept. 2017 (n=612)

Transfer Level 201B or 264 201A, 269AB, 252AB

+

26%

69% 56% 53% 55%

15% 8% 11% 12% 11%

59%

23% 33% 35% 34%

0%

20%

40%

60%

80%

100%

CoA BCC Laney Merritt PCCD

Writing Test Scores & Meaning at Each College (Jul.- Sept 2017, n=649)

Transfer Level English (1A)

One Level Below Transfer

Two or More Levels Below

38%

68%

32% 41%

46%

6%

32%

5%

42%

21%

55%

1%

63%

18%

33%

0%

20%

40%

60%

80%

100%

COA BCC Laney Merritt PCCD

Fall 2017 Eng. Actual Course Enrollments Among English Writing Test Takers July-Sept. 2017 (n=612)

*Note: 201B =1 level below, 201A = 2 below

+

Data Source: PCCD Institutional Research, Fall 2017 Course Enrollments (Actual Placements) Among Writing Test Takers between July - Sept., 2017, n=615. Note: Sample and percentages do not include courses such as 208, “14,” “210” “5,” and “70A.”

38%

68%

32% 41% 46%

33% 32% 34% 42% 35% 29%

0%

34% 18% 19%

0%

20%

40%

60%

80%

100%

CoA BCC Laney Merritt PCCD

Fall 2017 Eng. Actual Course Enrollments Among English Writing Test Takers July-Sept. 2017

26%

69% 56% 53% 55%

15% 8% 11% 12% 11%

59%

23% 33% 35% 34%

0%

20%

40%

60%

80%

100%

CoA BCC Laney Merritt PCCD

Writing Test Scores & Meaning at Each College (Jul.- Sept 2017, n=649)

Transfer Level English (1A) One Level Below Transfer Two or More Levels Below

*Note: 201AB =1 level below

+Future Implementation Issues

n  Most students were satisfied with course placements (78%); although, satisfaction varied. English 264 had the highest level of student satisfaction with placement (91%), Eng 252/269 had the lowest level (67%).

n  English Faculty viewed approximately 5 out of 10 of their students as appropriately placed, 3 out of ten were under-prepared, and 2 out of ten were over-prepared.

n  The writing test scores showed no evidence of disproportionate impact; however, enrollment rates in transfer level English (1A) were disproportionately lower among non-dominant racial-ethnic groups.

n  The meaning of a test score and placement into 201A vs. 201B should be clarified across colleges: Does placement into 201A constitute placement into 2 levels below transfer? What percentage of students who complete 201A go to 1A? And for 201B?

n  Communication & sharing best practices within & across colleges is essential to support consistency in student placement experiences

Ensuring Future Consistency: Confirming & Communicating the Meaning of Writing Assessment Scores

2 levels of 201 B = “1 level below”

A = “2 levels below”?

+Comments & Questions: n  Request for information about the learners in English 1A who

reported that the course too difficult – are they multilingual? Meryl Alper has an interest in this line of research, specifically: “how ELLs (multilingual students) get placed because we see that they usually come in with high HS GPAs, and being placed in a regular 1A might not be the most helpful for them”

n  Interest in Disaggregated data about students who found courses difficult – Also, request for raw data from this survey. Interest in continuing this validation work – replicating student and faculty survey responses about their satisfaction with placement.

n  Request for studies on the effect of tutoring.

+Comments & Questions: n  Need for Further Clarification of Policy & Discussion Among

English Faculty & Counseling Team at Laney: n  Laney only places students into 201A. Yet, the data from PCCD IR

show that among students who took the placement test at Laney College between July-Sept. 2017 (approx. 182), 7 students enrolled in 201B in Fall 2017, suggesting they were placed into 201B.

n  201AB are two “levels” of the same course.

n  The language used to refer to this is “Accelerated” -- Students can go from 201A to 1A *if the instructor identifies the student & the student/instructor complete the petition and students takes it to the counselor or A&R. Otherwise, they go to 201B.

n  The term “accelerated” is used differently at BCC where accelerated learning in 264 means that the faculty expectation, and the curriculum is designed so that ALL students are expected to go from 264 to 1A (no petition process required)

+ Comments & Questions:

n  Need for Further Clarification of Policy & Discussion Among English Faculty & Counseling Team at Laney: n  Why do some students have to take 1A if they are pursuing an AA

degree?

n  As of 2009 it’s required by state of CA. (Meryl can provide documentation).

n  Are there ESL classes that are also valid (instead of 1A) -- Transfer-level ESL classes ? This needs discussion to develop shared understanding across counseling/English Faculty at Laney:

n  At BCC, Gabe Wiener might have taught a transfer level ESL class.

n  Faculty at Laney: there are not any current transfer-level ESL courses.

+ Comments & Questions: n  What have you heard about this idea that we do not need the Writing

Assessment? n  When English Faculty created the writing test, the context was to improve upon the

standardized test as a placement tool, which everyone agreed was NOT working well to place students. The Writing Test DEFINITELY improved upon the standardized test system that was being used.

n  Alongside English Faculty efforts to implement the writing test, a lot of research has been done on how well high school grades and courses predict course success in college. This research (which includes PCCD) has shown that high school grades and courses ARE good predictors of course success in college in English and Math, and approved as multiple measures for placement. AB 705 (Oct. 2017) requires high school measures to be used for placement if they are available.

n  At PCCD, approximately 92% of writing test takers were 28 years or younger, suggesting they could provide their unofficial high school transcript for placement purposes

n  The open question posed for English Faculty to consider: What is the value of using the writing test for placement in addition to the high school measures? Does the writing test provide value above and beyond the high school measures? Should students be required to put forth the time and effort to take the writing test for placement if they have high school measures?

n  PCCD has developed data sharing agreements with local high schools to facilitate placement based on high school transcripts. HS data will be readily available to PCCD in the near future.

n  Research about how well the Writing Test, High School Measures, and the combination thereof predict English Course Success, (and/or Student satisfaction), can help to provide evidence that will help faculty to make a better decision about what added value the writing test provides and whether they want to use the writing test or not.

+Age of Writing Test Takers

Data Source: PCCD Institutional Research, Writing Assessment Results, (April - Sept., 2017)

92% of Writing Test Takers were age 28 or younger, between April - Sept., 2017. We assume these students would likely be able to provide an unofficial HS transcript, which is valid for placement purposes

+Followup Question: n  Request for disaggregated information about students who

are experiencing difficulties

+Followup Question: n  Request for disaggregated information about students who

are experiencing difficulties

Thank you for participating!