effective early college programs - gssaweb.org › ... › 2015 › 04 ›...

1750 H Street NW, 2nd Floor, Washington, DC 20006 P 202.756.2971 F 866.808.6585 hanoverresearch.com

MARKET EVALUATION SURVEYING DATA ANALYSIS BENCHMARKING LITERATURE REVIEW

Effective Early College Programs Hanover Research presents an examination of effective early college programs and the methods used to evaluate their efficacy.

http://www.hanoverresearch.com/

2

HANOVER RESEARCH AUGUST 2012

© 2012 Hanover Research – District Administration Practice

Executive Summary

This report presents an examination of several exemplary programs and methods used to evaluate their efficacy. The study is organized into two main sections: Section I: Case Studies of Effective Programs – In this section, Hanover

Research reviews the programmatic structure and characteristics of effective early college programs throughout the country.

Section II: Gauging Effectiveness – In this section, we examine how educational institutions, interested non-profit organizations, and government agencies have measured and evaluated the effectiveness of early college high school programs. Where available, we include information specifically focused on ongoing progress monitoring.

The report Appendix reproduces two survey templates produced by the National Association of Concurrent Enrollment Programs (NACEP) – one for students and one for teachers – designed to aid in the assessment of program effectiveness.

Key Findings Key findings of our research include the following: Early college programs are typically established with the aim of

accommodating high achievers and/or motivating higher levels of achievement among underperforming students. In both cases, programs recognized for effectiveness share a commitment to providing rigorous and diversified educational offerings and easing the transition between high school and college. With regard to the latter objective, such programs typically offer supplementary support structures (e.g., after-school assistance) or skills-based courses designed to familiarize students with the academic and social expectations associated with postsecondary education.

The Early College High School Initiative (ECHSI) has evaluated several early

college high schools praised as “innovator” programs by the North Carolina New Schools Project, identifying the following five principles integral to success:

o “Each staff member embraces responsibility for preparing every student for college success.

o Teachers use a consistent set of instructional strategies proven to accelerate learning. o Students receive intensive and individualized supports to overcome academic barriers. o Students are coached to take full ownership of their learning over time. o Staff collaboration extends beyond institutional borders.”

3



STAR Early College School in Brooklyn, New York – another early college offering recognized for its effectiveness – employs several features clearly related to the principles outlined above. The school requires “early immersion” activities and coursework, along with additional resources, including dedicated educational assistants, to help students successfully complete college-level coursework.

Studies designed to evaluate the effectiveness of early college programs display

several commonalities in design and method. Such studies typically evaluate effectiveness through a multiple-measure approach, including evaluation of instructors and assessments of student achievement, both pre- and post-program completion. Though quantitative data offer the most direct indicator of program effectiveness, many successful programs also consider qualitative indicators (e.g., student attitudes toward continuing postsecondary coursework) in assessing program effectiveness.

Our review of relevant research suggests that there is not a clear or universal set of best practices with regard to ongoing progress monitoring, though several common data points collected in effectiveness studies lend themselves to this type of analysis—namely, attendance rates; course grades; and formative assessments. On the qualitative side, surveys designed to gauge students’ educational aspirations or perceptions of program effectiveness may also offer useful indicators of program impact.

While graduation rates are perhaps the most commonly-utilized basic indicator

of program effectiveness, many early college programs also seek to obtain more detailed data on early college alumni. Such studies may take the form of surveys administered at set intervals post-graduation, designed to collect basic descriptive statistics (number of post-secondary units completed; degree(s) earned; college GPA) and gauge program impact through attitudinal measures (e.g., asking alumni to describe how early college affected their later educational experience).

4



Section I: Case Studies of Effective Programs

In this section, Hanover Research reviews the structures and characteristics of effective early college programs throughout the country. Exposure to college-level education and institutions is widely perceived to be beneficial for high school students; a variety of programs have thus arisen to facilitate these opportunities. As far back as the 2002-03 academic year, the National Center for Education Statistics (NCES) found that more than half of U.S. Title IV postsecondary institutions had high school students pursuing courses for credit, and that nearly half of these institutions had formal dual enrollment programs in place—defined loosely as “an organized system with special guidelines that allows high school students to take college-level courses.”1

The NCES study’s broad definition of dual enrollment largely fails to distinguish between the diverse structures currently employed by early college programs. In a more recent report, researcher Drew Allen, writing for the City University of New York, identified more specific distinctions useful in examining early college programs. Allen noted that “dual credit and concurrent enrollment typically refer to programs in which students earn high school and college credit simultaneously.”2 These programs may differ structurally, as “partnering institutions must decide the location of the course (high school or college), who will teach the course (college faculty or high school faculty certified as college adjuncts), what the student mix will be (high school students only, or high school students mixed with college students), how the courses will be financed (who will pay tuition?), and which students will be permitted to enroll.”3 According to Allen’s research, highly effective dual credit programs have a number of features in common, namely “a well-designed, coherent sequence of courses” as opposed to the “cafeteria-style” options typically offered. Allen notes that programs should also focus on “real college-level material’ directly aligned with college course standards.4 It is important to note, however, that dual enrollment or dual credit programs are just one of several types of early college options. Advanced Placement, Tech Prep, International Baccalaureate, and early college high schools all constitute opportunities for high school students to gain exposure to (and earn credits toward)

1 Kleiner, B. and L. Lewis. “Dual Enrollment of High School Students at Postsecondary Institutions, 2002-03,”

pp. 1-2. April 2005. U.S. Department of Education. National Center for Education Statistics. P.1-2. http://nces.ed.gov/pubs2005/2005008.pdf

2 Allen, Drew. “Dual Enrollment: A Comprehensive Literature Review & Bibliography,” p. 2. August 2010. http://www.cuny.edu/academics/k-to-12/databook.html

3 Ibid., 3. 4 Ibid.

In order to be highly effective, early college programs should offer a clear, “well-designed, coherent sequence” of

courses, as opposed to “cafeteria-style” options.

5



college-level coursework.5 Our examination of effective programs focuses primarily on early college high schools, though we also touch briefly upon programs that fall into the broader category of ‘dual enrollment.’ Allen describes early college high schools as “small schools in which students earn both a high school diploma and two years of college credit.”6 The creation of early college programs is typically motivated by a desire to provide higher-level courses for high-achieving students; by a desire to assist under-performing students in obtaining a post-secondary degree; or both. As such, the targeted student populations associated with an early college high school’s objectives often contribute to differences in program structure and focus.

The Early College High School Initiative (ECHSI) is a platform for institutions concerned primarily with the latter motivation for early college programs. Since its founding in 2002, the Institute has partnered with more than 240 schools in the United States that are “designed so that low-income youth, first-generation college goers, English language learners, students of color, and other young people underrepresented in higher education can simultaneously earn a high school diploma and an associate’s degree or up to two year of credit toward a bachelor’s degree—tuition free.”7 Since its founding, ECHSI has been a platform for the development of broadly applicable best practices in early college administration.

The ECHSI is coordinated by Jobs for the Future and supported, in part, by the Bill and Melinda Gates Foundation. Part of its directive involves collecting data from participating schools and conducting studies to determine what makes effective schools successful; as such, the ECHSI serves as a highly useful source of exemplary practices. In a 2009 ECHSI report, Thad Nodine noted that three focus areas are key in building college readiness:8

The structure of the student experience (from orientation to college classes) should be structured to raise expectations and increase knowledge about college.

Aligned sequences of rigorous high school classes should build to college coursework.

Comprehensive academic and social support services should be offered.

These principles can take different forms at individual schools, based on local needs and circumstances, but are generally evident among effective early college high schools. To more fully examine the structure and characteristics of effective early

5 Ibid., 4. 6 Ibid., 5. 7 “Welcome to Early College High School.” Early College High School Initiative.

http://www.earlycolleges.org/ 8 Bullet points quoted, with some alterations, from: Nodine, Thad. “Innovations in College Readiness: How

Early College Schools Are Preparing Students Underrepresented in Higher Education for College Success,” pp. 15-16. Jobs for the Future. October 2009. http://www.earlycolleges.org/Innovations_in_College_Readiness_PDFVersion_102909.pdf

6



college programs, we provide several case studies in the subsections that follow. Each of the programs reviewed below have been recognized for effective or innovative practices.

North Carolina New Schools Project In March 2011, ECHSI produced a report based on the experiences of the top five “innovator” early college high schools in North Carolina. The schools were selected by the North Carolina New Schools Project, a supporting public-private organization that was launched in 2003 and is responsible for assisting the state in developing 71 early college high schools.9 The “innovator” schools – Anson County Early College, Buncombe County Early College, Davidson County Early College, Vance County Early College and Warren Early College – were selected “based on their highly effective strategies to prepare all students for postsecondary education.”10 The report indicated that each of these successful early college high schools has prioritized the implementation of the following five principles:11 Each staff member embraces responsibility for preparing every student for

college success. Teachers use a consistent set of instructional strategies proven to accelerate

learning. Students receive intensive and individualized supports to overcome academic

barriers. Students are coached to take full ownership of their learning over time. Staff collaboration extends beyond institutional borders.

Fostering College-Readiness The ECHSI report identified two characteristics that all five “innovator” schools have in common that encourage college-readiness: an emphasis on the development of student-staff relationships (faculty and administrations) and an avoidance of student tracking based on past performance. The development of relationships between staff and students not only helps teachers provide individualized education, but also provides a basis for staff to encourage higher levels of achievement. On this point, the report noted that “…staff members are close to students, and they are familiar with the academic strengths and weaknesses of each student as well as their personal histories and interests. They also regularly push students to work harder and aim higher, using the foundation of respect and caring

9 “About NCNSP.” North Carolina New Schools Project. http://newschoolsproject.org/about-us/ 10 Le, Cecilia and Jill Frankfort. “Accelerating College Readiness: Lessons from North Carolina’s Innovator

Early Colleges,” p. 1. Early College High School Initiative. March 2011. http://www.jff.org/sites/default/files/Accelerating_College_032011.pdf

11 Ibid. Bullet points quoted from source.

7



between them.”12 Each school provides different structures to support and encourage the development of these relationships; however, a shared emphasis on prioritizing these relationships – and leveraging them to support student achievement – is key. The schools discussed in the report also avoid tracking students based on their past performance. One school initially tracked its students based on their English language proficiency level, but soon found that “students in the lower-level class were not ‘getting the extra oomph’ they could get from learning with the higher-performing students.” Integrating all of the students into the same classes demonstrated positive results—by the end of their sophomore year, all but three students were eligible to begin taking college-level courses.13 Learning Framework The five North Carolina “innovator” schools all utilize a learning framework developed by Jobs for the Future, in conjunction with the University Park College School. The framework involves six strategies that emphasize student participation, in an attempt to motivate students to take ownership of their education. These strategies are listed in Figure 1.1, on the next page. ECHSI noted three characteristics related to the way schools implemented the framework. The first is the creation of a collaborative culture of learning, in which students take responsibility for the learning of their group. The report describes a typical 9th grade English class, in which students are split into groups, and a book is assigned to the group to read. Each student is assigned a particular “role,” such as “question corrector,” “vocabulary master,” or “summarizer,” and the group discusses the book together.14 The second characteristic is an emphasis on pushing students to take intellectual risks, using Socratic techniques to lead students to confidently make conclusions about what they learn. The final characteristic identified by ECHSI, which specifically assists in accelerated learning, is the integration of writing into daily classroom routines, a strategy that aids students to developing and expressing ideas.

12 Ibid., 7. 13 Ibid., 8. 14 Ibid., 10.

Exemplary “innovator” schools exhibit two characteristics that effectively promote

college readiness:

Emphasis on student-staff relationships

Avoid student tracking based on past performance

8



Figure 1.1: Six Strategies to Accelerate Learning Strategy Description

Collaborative Group Work

Collaborative group work creates an engaging classroom culture in which students with diverse skill levels are supported and challenged by their peers. Students are grouped intentionally and each student is

accountable for his or her contribution.

Writing to Learn

Writing to learn helps students, including English language learners, develop their ideas, critical thinking, and fluency of education in all subjects. Students experiment with written language in class every

day.

Literacy Groups Literacy groups help build comprehension, fluency, and higher-level discourse across a variety of texts in different disciplines by assigning each student a role to play and structured guidelines for participation.

Questioning Questioning challenges students and teachers to use deep, probing

questions to foster purposeful conversations and stimulate intellectual inquiry.

Classroom Talk

Classroom talk encourages all students to develop their thinking, listening, and speaking skills, and promotes active learning.

Classroom talk takes place in pairs, in group work, and as whole class.

Scaffolding

Scaffolding encompasses a broad range of techniques such as pre-reading activities and graphic organizers that help students connect prior knowledge—from an earlier grade, different content area, or

personal experience—to challenge new concepts. Source: Early College High School Initiative Personalized Instruction All of the innovator schools employ flexible schedules to accommodate the extra support required by individual students’ instructional needs. For example, one school does not begin regular classes until 11 a.m., to allow time for additional classes in the morning for students that require more instructional time in particular subjects. In addition to flexible schedules, administrators regularly collect and assess student progress data (including grades and assessment scores) and target deficiencies on a student-by-student basis. Typical indicators include missed assignments, poor attendance, or low grades throughout the year; interventions occur once a pattern in such indicators has been identified.15 Finally, school administrators attempt to personalize education by allowing greater independence and autonomy as individual students demonstrate greater competency and motivation. Some schools allow capable students to opt for online courses or to take college courses elsewhere. All five “innovator” schools attempt to maintain some connection with students in these cases, to ensure that they receive adequate support, if necessary.16 15 Ibid., 12. 16 Ibid., 13.

9



Student Accountability The recognized “innovator” schools also attempt to motivate students to take ownership of their college-readiness, by encouraging them to evaluate their own progress and identify areas in which they may need additional work. The schools have also attempted to create a culture in which students are expected to take responsibility for their peers’ progress, as well. Encouraging a culture of shared concern can help to harness peer pressure and social support systems to promote student progress. Another feature of “innovator” early college high schools is a large term project, designed to engage students in independent work. Projects are typically developed by the student and presented for faculty approval. Independent projects fit into the broader goal of preparing students for the college environment (e.g., exposure to administrative structures and cultural expectations). Students are encouraged to meet with their instructors during office hours, and principles of time management, study skills, and note-taking are incorporated into the curriculum.17 ‘Purposeful’ Design The final characteristics exhibited by each of North Carolina’s early college “innovator” high schools is “purposeful design.” ECHSI noted two characteristics of the five innovator schools that matched this strategy: regular reviews of teaching practices and effective working relationships between high school instructors and professors at partnered colleges. The former element (regular reviews) typically takes the form of professional development, in which teachers observe one another and offer feedback based on their observations. The latter element (building effective secondary-postsecondary instructor relationships) is primarily oriented toward easing a student’s transition to the college level—each teacher knows the other’s expectations and pedagogical methods. Some early college high school teachers reported attending the relevant college courses and offering additional class periods to support students struggling with the material.18

STAR Early College School, Brooklyn, New York The STAR Early College School is the result of collaboration between the Woodrow Wilson National Fellowship Foundation and the City University of New York. The school is focused on science, technology, and research education.19 Like many new early college high schools, STAR serves a historically low-achieving population: its

17 Ibid., 15. 18 Ibid., 19. 19 Nodine, Thad. “Innovations in College Readiness: How Early College Schools Are Preparing Students

Underrepresented in Higher Education for College Success,” Op. cit., 20.

10



student body is 80 percent African-American and 62 percent low-income. More than 25 percent of students enroll with low reading and math skills. STAR was also recognized as an “innovator” by the ECHSI for its promotion of “early immersion” activities and curricula. Incoming ninth grade students, for example, participate in an “intensive summer bridge academy at Brooklyn College” focused on English, math, and college study skills. During their first semester, ninth graders are introduced to various college departments and facilities on a weekly basis, and are provided exposure to scientific studies at the college. In the spring semester, students take additional intensive seminars in subjects such as anatomy, archaeology, and law. Finally, all ninth and tenth graders complete university library-based research projects in groups.20

The collegiate and academic training continues in the last two years of high school. Students take summer courses as necessary to “fill in skill and knowledge gaps,” or immersion courses to gain college credit for “experiential learning.” All

students have access to courses taught by college faculty, and to integrated classes “which are traditional college courses … open to anyone.”21 Many of STAR’s support features, which are designed to assist achievement in the accelerated learning environment, are similar to the features in place at other early college schools. STAR offers extra after-school support, including “homework help, exam preparation classes, and individual interventions.” STAR also offers a “College 101” class, which covers issues such as time management and plagiarism; CUNY additionally provides in-class undergraduate tutors for STAR students.22

Hidalgo Independent School District, Texas In its efforts to become a comprehensive early college district, Hidalgo ISD has leveraged relationships with several local colleges. Hidalgo has successfully fostered partnerships with three nearby institutions: the University of Texas-Pan American, South Texas College, and Texas State Technical College.23 Given Hildalgo’s relatively long-standing focus on early college, the district provides an exemplar of effective postsecondary institutional partnerships. Since each institution employed its own “course structures, funding streams, 20 Ibid. 21 Ibid. 22 Ibid. 23 Nodine, Thad. “College Success for All: How the Hidalgo Independent School District Is Adopting Early

College as a District-wide Strategy,” p. 17. Jobs for the Future. October 2010. http://www.jff.org/sites/default/files/college_success_for_all.pdf

STAR offers an immersive curriculum supplemented by skills-based courses

and extra after-school support.

11



accountability, professional expectations, and governance,” both the district and its partner institutions enacted several adjustments to facilitate effective coordination of early college offerings. To promote student achievement, partner postsecondary institutions made several pedagogical adjustments—UT-Pan American, for instance, “was very deliberate in selecting faculty members who had been effective with underprepared students, but even these professors found that they had to adjust their teaching styles, shortening their lectures and expanding their engagement strategies.”24 Hidalgo’s teachers noted that they needed to make corollary adjustments, such as making their curricular goals more ambitious.25 The district also found that they had to plan their institutional partnerships in accordance with their students’ interests. A 2010 report on the effort noted that “one of the first major challenges that … Hidalgo and UT-Pan Am faced in creating an early college district was developing a range of postsecondary options for students who initially were not interested in pursuing a four-year degree.” In response to student interests, Hidalgo solidified its relationship with South Texas College (a two-year institution) and Texas State Technical College.26 Finally, Hidalgo sought to improve educational quality by instituting a variety of incentive programs for teachers. Realizing that the more-than-20 minute commute from Hidalgo’s schools to the nearest partner university would become increasingly burdensome to students and staff, Hidalgo incentivized its teachers to earn their master’s degree and become adjuncts at partner colleges. In exchange for obtaining their degree and teaching college-credit courses at Hidalgo’s high schools, teachers earn a higher salary and receive small bonuses for each college-credit course taught. These incentives not only reduced the logistical burden for students traveling to class, but also increased the educational quality and culture of the early college program.27 24 Ibid. 25 Ibid., 18. 26 Ibid., 19. 27 Ibid., 20.

12



Section II: Gauging Effectiveness

In this section, Hanover Research examines how educational institutions and affiliated organizations measure and evaluate the effectiveness of early college high school programs, with a specific focus on measures that may be employed for the purposes of ongoing progress monitoring. While differences in research design are evident among such studies, key commonalities are exhibited in the types of data considered. Typically such data fall into two broad categories: instructional effectiveness (teacher evaluations) and student achievement, which may encompass quantitative and qualitative indicators assessed throughout the early college experience and post-graduation. Assessing Instructional Effectiveness Evaluation of a program’s instructors is occasionally employed to help gauge the effectiveness of an early college program. Typically, instructor assessment is utilized in the review of dual-enrollment programs—those early college programs in which high school teachers may instruct courses for college credit. The National Alliance of Concurrent Enrollment Partnerships (NACEP) has been a significant proponent of standards of instructional quality. The NACEP is “a professional organization for high schools and colleges that fosters and supports rigorous concurrent enrollment….[the] NACEP serves as a national accrediting body and supports all members by providing standards of excellence, research, communication, and advocacy.”28 The organization offers “the only national set of quality standards applicable to concurrent enrollment partnerships” – standards that “have been adapted or incorporated into state policy in 15 states.”29 Adherence to standards is a necessary prerequisite for accreditation with the NACEP.30 Selected standards related to instructional effectiveness are listed in Figure 2.1, on the following page.

28 “National Concurrent Enrollment Partnership Standards,” p. 2. January 1, 2011. National Alliance of

Concurrent Enrollment Partnerships. http://nacep.org/docs/standards/NACEP-Standards-2011.pdf 29 “Standards.” National Alliance of Concurrent Enrollment Partnerships. http://nacep.org/standards/ 30 Ibid.

Early college program evaluations typically consider multiple indicators

related to student success and instructional effectiveness.

13



Figure 2.1: Selected NACEP Concurrent Enrollment Partnership (CEP) Standards Related to Instructional Effectiveness31

Standard Dimension & Number

Description

Faculty 1 CEP instructors are approved by the respective college/university academic

department and meet the academic department’s requirements for teaching the college/university courses.

Faculty 2

The college/university provides new CEP instructors with discipline-specific training and orientation regarding, but not limited to, course curriculum, assessment criteria,

pedagogy, course philosophy and administrative responsibilities and procedures prior to the instructor teaching the course.

Faculty 3

The CEP provides annual discipline-specific professional development activities and ongoing collegial interaction to address course content, course delivery, assessment, evaluation, and/or research and development in the field. The CEP ensures CEP

instructor participation.

Faculty 4 CEP procedures address instructor non-compliance with the college/university’s

expectations for courses offered through the CEP (for example, non-participation in CEP training and/or activities).

Evaluation 1

The CEP conducts end-of-term student university/college course evaluation for each course section offered through the CEP.

Evaluation 4

The CEP conducts surveys of participating high school instructors, principals, and guidance counselors at least once every three years. Surveys include NACEP essential questions (additional questions may be used). Methodology includes one follow-up

contact with non-respondents. Qualified institutional evaluator/researcher collaborates with CEP to develop the survey and analyze the data.

Source: National Concurrent Enrollment Partnership Standards The faculty standards stipulate basic quality checks, designed to ensure and maintain the rigor of college-credit courses. The two evaluation standards listed in Figure 2.1 stipulate periodic surveys of students and teachers. The responses to these surveys assist administrators with evaluation of courses, curriculum, student attitudes, and instructional quality over the course of the program.32 Survey-based evaluations have similarly been utilized outside the NACEP context; in a 2009 study of an early college high school in its third year of operation, University of Texas researcher Melinda Martin Valdez, for instance, utilized a combination of quantitative data and qualitative interviews with students to gauge “perception[s] of … operational effectiveness.”33

Student Achievement Studies on the effectiveness of early college programs tend also to include assessments of student achievement. Typically, these evaluations encompass multiple quantitative (and in some cases, qualitative) measures, with data collected on an

31 “National Concurrent Enrollment Partnership Standards,” Op. cit., 3. 32 “Survey Templates.” National Alliance of Concurrent Enrollment Partnerships.

http://nacep.org/accreditation/survey-templates/ 33 “The Role of Early College High School in Success: A Case Study of Students’ Perceptions of Mission Early

College High School Effectiveness.” 2009. http://repositories.lib.utexas.edu/handle/2152/6902

14



ongoing basis throughout a student’s educational experience. Teachers and school administrators may evaluate these data on a continuous basis to identify educational deficiencies and diagnose problem behaviors quickly. On a broader level, data may be used to track an instructor’s or school’s effectiveness. While the specific data points collected and evaluated by individual early college programs may differ by school, several appear common in studies of program effectiveness: Graduation/drop-out rates Total number of college credits earned Daily attendance rate Rate of passage on state or district assessments Aspirations to attend college or university

As noted in the previous section, many highly-effective early college programs collect and monitor data that tracks a student’s ongoing academic experience—such as grades, attendance on a per-class basis, and homework assignment completion—in order to ascertain when intervention may be necessary. As the list of commonly-collected data above demonstrates, studies of program effectiveness typically rely on a combination of quantitative and qualitative data. This trend reflects an important principle regarding early college programs and their assessment. The Early College High School Initiative (ECHSI) emphasizes that early college high schools are not designed only to bolster student achievement, but also to encourage students to begin to identify as college-goers.34 Since programs aim to foster qualitative changes of this sort, they must rely to some extent on qualitative methods of analysis to gauge effectiveness. The following subsections provide an overview of program effectiveness measures employed by ECHSI and others, which are useful in conceptualizing broad, summative indicators, as well as school-level progress monitoring on a more frequent basis. ECHSI ECHSI regularly collects data from its partner schools and draws upon the data, in aggregate, to create studies and reports designed to both advocate for the early college institutional model and assess areas for improvement. In 2011, ECHSI produced a report entitled Early College High School: A Portrait in Numbers, in which it updated the statistics it had periodically been releasing for the past several years.35 34 “How Do Early College High Schools Organize to Promote Student Success? Overview & FAQ.” The Early

College High School Initiative. http://earlycolleges.org/overview.html 35 Nodine, T. “Innovations in College Readiness: How Early College Schools Are Preparing Students

Underrepresented in Higher Education for College Success,” Op. cit.; “Six Years and Counting: The

15



The data featured in the periodic reports were taken from the ECHSI’s Student Information System (SIS), a database of information collected from early college high schools, school districts, and post-secondary institutions designed to “provid[e] districts, early college schools, and sponsors with an integrated set of data to support continuous improvement and advocacy.”36 To affiliated districts and interested parties, “the SIS provides comparative information, including longitudinal data on postsecondary degree attainment, on the impact of district early college school designs on students’ secondary and postsecondary success.”37 The ECHSI gathers and stores the following sorts of aggregated data from partner early college high schools in the SIS: Demographics (e.g., race, age, gender, free/reduced lunch) Standardized test scores (including two to four years before enrolling in early

college high school) Student attendance and persistence Rates of attendance Courses taken and grades (including two to four years before enrolling in early

college high school) SAT/PSAT and ACT scores Date high school diploma awarded GPA

The statistics the ECHSI reports rely on most regularly include the average daily attendance rate, the grade-to-grade promotion rate, standardized test scores (particularly state assessments), and student GPA. These statistics are typically assessed against those of comparable public high schools, controlling for characteristics like student demographics and past performance/behavior, to isolate the effectiveness of the early college curricula at various points throughout a student’s educational experience. North Carolina New Schools Project The North Carolina New Schools Project has commissioned a five-year study through the SERVE Center at the University of North Carolina at Greensboro and several other research partners. The study is updated on an annual basis, reflecting

ECHSI Matures.” American Institutes for Research. SRI International. August 2009. http://www.earlycolleges.org/Downloads/AIR.SixYearsAndCounting.pdf; Hoffman, N. and Michael Webb; “Early College High School: Modest Experiment or National Movement?” Education Week. June 11, 2009. http://www.earlycolleges.org/Downloads/Early-CollegeHighSchoolModestExperimentorNationalMovement.pdf

36 “Overview.” Student Information System. Early College High School Initiative. https://echs-sis.org/PublicPages/OverViewFAQ.aspx

37 “Student Information System: Frequently Asked Questions.” Early College High School Initiative. http://www.earlycolleges.org/Downloads/SISQ&A.pdf

16



new data from more than 1,000 students (including a sample and control group) as they progress through the educational system.38 The study is designed to “evaluate the impact of the ECHS [early college high school] model on student outcomes including: attitudes, attendance, achievement, course-taking, and school leaving/dropout rates.”39 Students included in the study were were randomly assigned to the treatment and control groups—the early college high schools and regular public schools, respectively—at the beginning of the study, and their performance and attitudes are recorded and compared on an annual basis.40 The study examines the following indicators, encompassing both large-scale, summative assessments as well as more informal progress monitoring. College preparatory courses and success (measures whether students took the course,

the pass rate, and the number of students that took and passed the course): o English I o Algebra I o Geometry o Algebra II o Biology o Civic and Economics

Daily attendance Suspensions Dropouts Continued enrollment (as a buttress for the dropout data) Aspirations to attend a four-year college or university

They study’s authors report on the collected data in a variety of ways depending on the nature of the data:41

The impacts of ECHS on these outcomes are estimated within an Intent-To-Treat (ITT) framework, in which a students’ initial experimental status as a treatment or control student, rather than actual participation in an ECHS, serves as our measure of treatment. We report unadjusted means for the two groups. We also report adjusted impacts using logistic regression to analyze binary outcomes (e.g., coursetaking, dropouts, etc.) and linear regression for

38 Edmunds, J., Lawrence Bernstein, Fatih Unlu, Elizabeth Glennie, and Nina Arshavsky. “A Better 9th Grade:

Early Results from an Experimental Study of the Early College High School Model,” pp. 2-4. SERVECenter. The University of North Carolina at Greensboro. http://www.earlycolleges.org/Downloads/Abetter9thgrade.uploads_publications_SERVE_A-1.pdf

39 Ibid., 4. 40 Edmunds, J., Lawrence Bernstein, Fatih Unlu, Elizabeth Glennie, and Nina Arshavsky. “The Impact of the

Early College High School Model on Core 9th and 10th Grade Student Outcomes.” 2011 SREE Conference Abstract Template. http://www.eric.ed.gov/PDFS/ED518187.pdf

41 Ibid., 4.

17



continuous outcomes, such as days absent. In our regression-based models, we employ baseline student characteristics and site-level indicators (or site fixed effects) as covariates.

The SERVE Center evaluations thus differ from the ECHSI studies in terms of the indicators used to assess college preparation: SERVE measures students’ grades in particular preparatory courses, while the ECHSI relies primarily on standardized testing scores. SERVE is also a more narrowly-defined and controlled study than the ECHSI’s ongoing analytics; however, both approach the measurement and evaluation of educational efficacy in the same way, and utilize similar sets of data to accomplish their examination.

Post-Completion Measures The evaluation of student success post-graduation is perhaps the most commonly-utilized measure of program effectiveness. Studies of this type typically utilize surveys of program students, administered at set increments pre- and post-completion. Such surveys enable program administrators to develop comparative data on student attitudes during and after a student’s enrollment in an early college program. In addition to offering basic descriptive statistics on early college students, the results of attitudinal surveys are often aggregated and compared against the responses of students who did not participate in an early college program. Data commonly measured by these surveys include: Progress on post-secondary degree (how many post-secondary units the

student has completed; whether students earn a post-secondary degree after completion of the early college program)

College GPA Most studies pair responses to post-graduate surveys to the data collected earlier in a student’s academic career, controlling for characteristics such as past performance, ethnicity, and other significant features of the response group. In the following subsections, we examine several instances of this approach. NACEP The NACEP standards list fairly detailed procedures for collecting post-graduate survey data. Selected relevant standards are shown in Figure 2.2, on the next page. NACEP requires accredited partner programs to survey alumni one year after graduation and again three years later. The “essential questions” for both surveys referenced in the standards are provided in Appendix A.

18



The surveys gather a wide range of information: the school(s) the alumnus has attended; degrees earned; whether the alumnus transferred credits from their early college program; how the early college experience improved the respondent’s college experience; and various demographic characteristics.

Figure 2.2: Selected NACEP Concurrent Enrollment Partnership (CEP) Standards – Program Evaluation42

Standard Dimension & Number

Description

Evaluation 2

The CEP conducts an annual survey of CEP alumni who are one year out of high school. Survey includes NACEP essential questions (additional questions may be

used). Methodology includes one follow-up contact with non-respondents.

Evaluation 3

The CEP conducts a survey of CEP alumni who are four years out of high school at least once every three years. Survey includes NACEP essential questions (additional

questions may be used). Methodology includes one follow-up contact with non-respondents.

Source: National Concurrent Enrollment Partnership Standards ECHSI As mentioned above, the ECHSI’s SIS retains data on early college students not only while they are enrolled in an early college high school, but also post-graduation. Unlike the NACEP, ECHSI does not rely on surveys to collect its post-graduate information, although the organization does occasionally conduct survey-based studies.43 Instead, the ECHSI relies on information from the National Student Clearinghouse (NSC) to track the achievements of their early college high school graduates.44 Among other services, the NSC “offers FERPA-compliant access to a nationwide coverage of postsecondary enrollment and degree records” and “serves as a valuable source for longitudinal and other studies on educational progress.”45 NCHSI articulates a few caveats to using the NSC data, noting that “not all postsecondary institutions participate in the clearinghouse. Students may opt out of making their enrollment record available. And it may not be possible to access the records of students whose names have changed.”46 Despite these caveats, however, NSC offers a valuable means of accessing college enrollment information for the graduates of early college programs.

42 “National Concurrent Enrollment Partnership Standards,” Op. cit., 3. 43 “Early College Graduates: Adapting, Thriving, and Leading in College.” Early College High School Initiative.

2007. http://www.earlycolleges.org/Downloads/EarlyCollegeGraduates.pdf 44 “Student Information System: Frequently Asked Questions,” Op. cit. 45 “What We Do.” About the Clearinghouse. National Student Clearinghouse.

http://www.studentclearinghouse.org/about/what_we_do.php 46 Webb, M. and Lia Mayka. “Unconventional Wisdom: A Profile of the Graduates of Early College High

School,” p. 12. Jobs for the Future. March 2011. http://www.eric.ed.gov/PDFS/ED519999.pdf

19



Appendix: NACEP Surveys

Below, we reproduce the one-year post-graduation survey produced by NACEP for use by its accredited programs, along with an accompanying teacher survey also utilized to assess program impact.47

Figure A.1: Graduate Survey (One Year Post-Completion)48

47 “Survey Templates,” Op. cit. 48 Note that the survey for students four years post-completion is nearly uniform in terms of question content;

as a result, it is not reproduced here.

20



21



Figure A.2: Teacher Survey

22



23



24



Project Evaluation Form Hanover Research is committed to providing a work product that meets or exceeds member expectations. In keeping with that goal, we would like to hear your opinions regarding our reports. Feedback is critically important and serves as the strongest mechanism by which we tailor our research to your organization. When you have had a chance to evaluate this report, please take a moment to fill out the following questionnaire. http://www.hanoverresearch.com/evaluation/index.php Caveat The publisher and authors have used their best efforts in preparing this brief. The publisher and authors make no representations or warranties with respect to the accuracy or completeness of the contents of this brief and specifically disclaim any implied warranties of fitness for a particular purpose. There are no warranties which extend beyond the descriptions contained in this paragraph. No warranty may be created or extended by representatives of Hanover Research or its marketing materials. The accuracy and completeness of the information provided herein and the opinions stated herein are not guaranteed or warranted to produce any particular results, and the advice and strategies contained herein may not be suitable for every member. Neither the publisher nor the authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Moreover, Hanover Research is not engaged in rendering legal, accounting, or other professional services. Members requiring such services are advised to consult an appropriate professional.

http://www.hanoverresearch.com/evaluation/index.php

effective early college programs - gssaweb.org › ... › 2015 › 04 ›...

Documents