1400 10th Street P.O. Box 3044 Sacramento, California 95812-3044 (916) 322-2318 FAX (916) 324-9936 www.opr.ca.gov

State of California

Governor’s Office Office of Planning And Research

EDMUND G. BROWN JR KEN ALEX Governor Director

California Education Learning Lab

REQUEST FOR PROPOSALS 2018-19:

“Improving Equity, Accessibility and Outcomes for STEM Gateway Courses”

Revised on February 8, 2019, with Full Proposal Instructions in Section IV. F, pages 7-10. Other changes have been highlighted.

MILESTONES DATE

Request for Proposals Announced Wednesday, December 12, 2018

Letter of Intent to Submit a Proposal Due Monday, January 7, 2019

Concept Proposals Due Tuesday, January 22, 2019

Notification of Finalists Tuesday, February 5, 2019

Full Proposals Due Friday, March 15, 2019 Friday, March 22, 2019 (new date)

Selection Committee Meeting (brief public meeting, followed by closed session deliberation)

Monday, April 15, 2019 (venue TBD)

Awardees Announced Monday, April 8, 2019 (estimated) Wednesday, April 24, 2019 (new estimated date)

Projects Commence June 1 or July 1, 2019

Duration of Projects 36 months

Funding For 6-9 projects, approximately $1 million to $1.5 million total per project (including indirect costs1).

1 See Item VI.

2

I. California Education Learning Lab Assembly Bill 1809 (Chapter 33, Statutes of 2018) established the California Education Learning Lab (“Learning Lab”) as a competitive grantmaking program for intersegmental faculty teams2 to incorporate learning science and adaptive learning technology into their curriculum and pedagogy, with the express purpose of increasing learning outcomes and closing equity and achievement gaps in STEM and other disciplines. The Learning Lab is housed in the Governor’s Office of Planning and Research, with an annual budget of $10 million. Initial calls for proposals will focus on lower-division online and hybrid courses in STEM. In later years, other disciplines may compete for funds and funds may be used to support professional development and a curated resource library.

II. Learning Science and Adaptive Learning Technologies

“The goal of learning sciences is to better understand the cognitive and social processes that result in the most effective learning, and to use this knowledge to redesign classrooms and other learning environments so that people learn more deeply and more effectively.” -- R. Keith Sawyer, Washington University Learning science is the study of how human learning takes place. Interdisciplinary in nature, drawing from fields such as cognitive science, neuroscience, computer science, education, psychology, sociology, design studies and more,3 learning science strives to understand how people learn, how to support learning, discipline based learning, and the role of technology in enhancing learning and collaboration.4 Learning science can cover how people process, gather, and interpret information; how they develop knowledge, skills, and expertise; or the extent to which social and physical context and design environments influence cognition.5 Scaffolding, inquiry or problem-based learning, collaborative learning, game and simulation-based learning, as well as metacognition are all examples of how teaching methods and approaches to curriculum can be influenced by what we understand about learning. Additionally, strategies linked to social psychology and multicultural education emphasize the importance of attending to students’ identity and culture when addressing achievement gaps. One of the goals of learning science is to create a positive feedback/continuous improvement loop between theories of learning and practice, which results in improved student learning and advances the field of learning science.6 For the purposes of the Learning Lab, as public higher education strives

2 “Intersegmental faculty teams” refers to a team of faculty from more than one segment of public higher education, e.g., University of California, California State University, California Community Colleges. 3 Sawyer, R.K. (2006). The Cambridge Handbook of the Learning Sciences. Cambridge, U.K.: Cambridge University Press. 4 Sommerhoff, D., Szameitat, A., Vogel, F., Chernikova, O., Loderer, K. & Fischer, F. (2018). What Do We Teach When We Teach the Learning Sciences? A Document Analysis of 75 Graduate Programs. Journal of the Learning Sciences, 27:2, 319-351. https://doi.org/10.1080/10508406.2018.1440353. 5 Ibid. 6 The Simon Initiative Learning Engineering Ecosystem at Carnegie Mellon University emphasizes: 1)

3

to educate more students with diverse backgrounds in a rapidly changing world, leveraging, increasing and applying our knowledge of human learning is a challenge we must embrace. Adaptive learning is defined by statute to mean “a technology-mediated environment in which the learner’s experience is adapted to learner behavior and responses.” For the purposes of this RFP, adaptive learning technologies will be considered in the broad sense of deploying technology to better understand learner experience/learner gaps and assets, and to modify learning environments, pedagogical approaches and/or available resources to be more inclusive of students most likely to leave the sciences (such as first-generation college-going students and underrepresented students in the sciences) and produce better learning outcomes. The adaptive learning technology approach that is proposed will be considered in the context of all of the other elements in the proposal.

III. Demonstration Projects - Summary For this RFP, up to $9 million will be provided from the Learning Lab to fund six to nine demonstration projects to support curricular and pedagogical innovations that aim to increase learning outcomes, transform the culture of learning, and close equity and achievement gaps in online and hybrid learning environments within lower division STEM undergraduate curriculum. In order to have the potential for large scale impact, this call will be open to lower-division “gateway” courses in the following disciplines: biology, chemistry, physics, engineering and computational sciences, including computer science, mathematics and statistics. Within the available funds, approximately $1 million to $1.5 million will be available to each awarded demonstration project. Projects are encouraged to develop pedagogical innovations that promote students’ sense of belonging in science, students’ science identity and connections between science learning and students’ personal lives, career aspirations and home communities, leveraging affective components of learning to reduce achievement gaps. Projects must be co-hosted by a faculty team representing a minimum of two public higher education segments in California. (Example: a faculty member from the California Community Colleges must collaborate with a faculty member from the University of California OR the California State University. Faculty collaboration across all three segments is welcome and encouraged.) Other faculty from private independent/nonprofit institutions and nonfaculty (i.e., professionals operating in a nonfaculty role for the purposes of the project) may participate in the project as well. A strong project will engage many stakeholders iteratively and throughout the duration of the project, as well as lay the foundation for sustainability of innovations and institutional culture change. Demonstration projects will be selected through a three-stage process involving: (1) submission of letters of intent to submit concept proposals; (2) submission of concept proposals; and (3)

building and leveraging cognitive models of expertise to inform the design of effective student-centered instructional materials; 2) collecting rich data on student interactions and learning outcomes; 3) data analysis via state-of-the-art machine learning and analytic methods; 4) data-informed iterative improvement of the instructional materials; and 5) leveraging these assets to drive fresh insights in learning science. https://chronicle-assets.s3.amazonaws.com/5/items/biz/pdf/SimonLearningEngineeringEcosystem.pdf.

4

submission of full proposals, based on selected concept proposals, from which the final selection of awards will be made. A selection committee will make recommendations for final awards. After awards are announced, Learning Lab will work with awardees to establish an agreement governing the award period, including concrete metrics and goals to track the progress of the demonstration projects, and provide technical assistance.7

IV. Applications

A. Application process

Stage 1: Letter of intent to submit a concept proposal (DUE: Monday, Jan. 7, 2019) Applicants should submit a brief letter of intent. The letter should note the expected host institutions and co-principal investigators, provide a (tentative) title of the proposal and a tentative total budget. The letter should also include a brief description of the proposal and characterize the discipline-specific problem that co-PIs are trying to solve and/or investigate. Please provide institutional data disaggregated by course and student characteristics (e.g., ethnicity, gender, socio-economic status, first-generation college going) on existing campus-, school- or department-specific equity issues that your project is designed to address. Stage 2: Institutional cover letter and concept proposal (DUE: Tuesday, Jan. 22, 2019) Applicants should submit institutional cover letters and short concept proposals; see sections C and D below. Stage 3: Full proposal (DUE: Friday, March 15, 2019 March 22, 2019) The selection committee selected a subset of submitted concept proposals to move onto the full proposal stage. (21 proposals were invited to the full proposal stage.) For the finalists advancing to this next stage, instructions for submission of the full proposal is in Section F (beginning on page 7). The selection committee will recommend between six and nine final projects for this grant cycle. The Governor’s Office of Planning and Research (OPR) will approve and announce the final funding decisions. For questions, please see the FAQ document or contact learninglab@opr.ca.gov, or go to our webpage (opr.ca.gov/learninglab). Please join our email distribution list to recieve updates directly by sending an email to learninglab@opr.ca.gov.

B. Eligibility

1. Applicant teams must include faculty co-principal investigators (PIs) from at least two public higher education segments. Representation from all three public higher education segments is encouraged. Additional partnerships, such as with private independent/nonprofit institutions and/or industry partners, are also encouraged. All faculty teams must commit to teaching and evaluating the codeveloped or jointly redesigned curriculum or innovative pedagogy during the grant period.

7 Contracting entity will be the Governor’s Office of Planning and Research.

5

2. Demonstration projects should aim to improve learning outcomes and close equity/achievement gaps for STEM undergraduate students in lower division course series8 where the mode of learning is online or hybrid, i.e., makes use of both online and in-person interactions as part of the formal course environment or requirements.

C. Institutional Cover Letter (to be submitted with the Concept Proposal)

For each faculty team application, the relevant departments/schools/institutions should provide answers for Section C1, C2 & C3, in a brief (limit one page); minimum Arial 11 font; 0.5 inch margins; no appendices. Host institutions: Identify the institutions that are submitting the proposal and will be responsible for receipt/administration of the grant funds, if awarded. Institutional focus: Describe each department/school/institution’s commitment (e.g., faculty release time, funding, administrative support) to the proposed demonstration project. (Each participating institution should sign the cover letter. Additional demonstration of institutional commitment will be highlighted in the full proposals stage.) Principal investigators: Identify the investigators who will serve as faculty (co-)PIs. Please briefly describe each PI’s capacity, including any previous and/or current grant funding received, strength of faculty and student engagement activities, and history of successful intersegmental partnerships. Authorized submission: The Institutional Cover Letter (C1-C3) and the concept proposal (section D) should be submitted electronically to learninglab@opr.ca.gov by the signatories, which must include the department chair AND either the dean, vice chancellor/vice president of research or the provost or equivalent.

D. Concept Proposal

For each application, please provide answers for Section D in a short Concept Proposal: maximum two pages for questions 1-7; maximum 1 page for questions 8-10; minimum Arial 11 font; 0.5 inch margins; no appendices. 1. How will your proposal measure or define success?: Describe what problem you are trying to

solve. Please include data/metrics to highlight the problem and elaborate on the description and data provided in your letter of intent. Describe how your proposed project will improve understanding of learning science and/or assessments, and/or effectiveness of pedagogical methods and/or adaptive learning technologies. What will you measure? (For example: increased retention or increased proficiency and performance with STEM; increased conceptual understanding/higher order thinking or passion for STEM careers; increased communication skills, leadership, and teamwork capabilities of STEM students; increased self-efficacy/ability to learn independently; increased facility with the scientific method; increased faculty impact; or reduction

8 High school dual enrollees may also be captured as part of this population.

6

of a particular pain point experienced by faculty or students.) How will you evaluate students? How will you evaluate faculty?

2. Project plan: Describe the components and timeline of your proposed project (specific aims and research strategy).

3. Data and adaptive learning technologies: Each proposal should demonstrate its commitment to

the use of robust data and technology tools, including adaptive learning technology (see definition above). Please describe how your proposal will use real-time learning outcomes data and adaptive learning technology and other technology tools to improve the pedagogy and/or curriculum.

4. Learning science: Describe how you will use evidence-based pedagogical approaches supported by

research from a variety of disciplines. What is innovative about your approach? How will you take an existing approach and experiment with achieving broader scale?

5. Student engagement: Describe your approach to student engagement, potentially including

engagement of students who may not identify as STEM proficient. Examples: How might your approach increase students’ sense of belonging, and encourage students’ help-seeking behavior from faculty, teaching assistants, other students, technology resources, etc. Will your approach include engagement through active learning, applied learning through a career or workforce pathway lens, and/or highly contextualized learning? How might students drive their own learning and/or the learning of their peers? Will your proposal individualize learning or use metacognition? How often will students receive meaningful and timely feedback, whether through a technology-mediated environment or face-to-face?

6. Culture: How will your proposal impact traditional “classroom” and disciplinary culture? In

particular, how will your approach address aspects of classroom or disciplinary culture that are barriers to student learning and to their sense of belonging? How might it encourage a strengthening-assets or growth-oriented approach to student learning and how might it help establish a classroom context in which all students can succeed? How might your proposal take advantage of under-represented communities’ cultural strengths to increase their achievements in STEM?

7. Scalability and value analysis: Describe how your work could be scaled, afforded, replicated and/or

modified through an open educational resources model? What other dimensions of value can be evaluated in your project? With whom will you partner to do the analysis, what data will you analyze, etc.?

8. Project team: Provide a brief description of the co-PI(s), team, and key collaborators. Describe the

nature and strength of any existing collaborations among project team members, and how you will use the expertise of all involved to create a well-balanced collaboration. Describe also how the project team may use external expertise and/or stakeholder input to iterate over the course of the project.

9. Budget overview: Briefly outline how Learning Lab funds (approximately $1 million to $1.5 million)

7

will be used and how other resources may be leveraged including any outside funds or institutional funds. How will you maximize existing structures or resources? Will your innovations place any costs on users? If so, how will these be minimized?

Note: Learning Lab funds are intended to be used exclusively in California. If the project necessitates the use of Learning Lab funds outside of California, provide a brief justification and estimate of the funding that will leave the state. The amount of funds that can leave the state will be subject to the final award agreement.

10. Common data-sharing/technology platform: Please discuss the potential for using a common data-

sharing platform to deliver the course or course series.

Submission: Concept proposals, including the institutional cover letter, must be submitted electronically as a single PDF to learninglab@opr.ca.gov by 5:00pm PT on Tuesday, January 22, 2019.

E. Full Proposal-NEW

Of the 42 concept proposals that the Learning Lab received, 21 have been invited to submit full proposals. Please provide answers for Section F in your Full Proposal: maximum 15 pages total, not including appendices or institutional cover letters; minimum Arial 11 font; 0.5 inch margins.

Please note that the questions below are modified versions of the questions contained in the Concept Proposal section. Please read the questions below carefully, using the page length maximums (indicated in parentheses) to expand on your answers from the Concept Proposal and address any new requested or suggested content. Please include in your Full Proposal submission:

1) Institutional Cover Letter(s) included in your Concept Proposal, updated for content and/or signatories;

2) Full Proposal responses; 3) Appendices, as follows:

a. Information on additional team members, i.e., statement of qualifications, not covered under Question 8 (maximum 3 pages total);

b. Budget information (maximum 2 pages total); c. Bibliography of key sources (maximum one page total); d. Any other supporting documents (maximum 3 pages total); e. Any brief letters of support from additional faculty colleagues who are interested in

being part of the scaling efforts related to Question 7 below. (Maximum 5 pages for all additional indications of support. This can be a single letter with signatories or individual letters. Please identify name, title and contact information for signatories.)

Updated rubric and suggested templates for additional institutional cover letters (any added since the submission of your Concept Proposal) and Appendix B will be available on March 1, 2019, at

8

http://www.opr.ca.gov/learninglab/. All submissions are due in full by Friday, March 22nd, 2019, by 5pm. Please email your entire submission in a single PDF to learninglab@opr.ca.gov. If you have any questions, please contact learninglab@opr.ca.gov General Notes: When responding to the questions below, to the extent possible please describe students and faculty from an asset-based perspective (i.e., building on strengths), rather than a deficit-based perspective (i.e., cataloging what is “wrong” with learners or faculty that needs to be “fixed). Please be as clear as possible about what learners and faculty will do differently based on this project, in both academic and other domains (social, emotional, etc.). As stated in the “Demonstration Projects – Summary” (Section III), projects are encouraged to develop pedagogical innovations that promote students’ sense of belonging in science, students’ science identity and connections between science learning and students’ personal lives, career aspirations and home communities, leveraging affective components of learning to reduce achievement gaps. A strong project will engage many stakeholders iteratively and throughout the duration of the project, as well as lay the foundation for sustainability of innovations and institutional culture change. A strong proposal will describe the project as succinctly and clearly as possible, contrasting how it differs from the status quo, or what is currently the norm in the discipline or course. (½ page)

1. How will your proposal measure or define success?: Describe what problem you are trying to

solve. Please include data/metrics to highlight the problem. What will you measure? (For example: increased retention or increased proficiency and performance with STEM; increased conceptual understanding/higher order thinking or passion for STEM careers; increased communication skills, leadership, and teamwork capabilities of STEM students; increased self-efficacy/ability to learn independently; increased facility with the scientific method; increased faculty impact; or reduction of a particular pain point experienced by faculty or students.) How will you evaluate students? How will you evaluate faculty? Will your project improve understanding of science of learning and/or assessments, and/or effectiveness of pedagogical methods and/or adaptive learning technologies? A strong proposal will describe the learning outcomes to be measured, over what time period, and the validity of these outcome measures with clarity. (1-1½ pages)

2. Project plan: Describe the components and timeline of your proposed project (specific aims and research strategy). A strong proposal will describe in detail the steps to be undertaken and by whom. (1-1½ pages)

3. Data and adaptive learning technologies: Each proposal should demonstrate its commitment to

the use of robust data and technology tools, including adaptive learning technology (see definition above). Please describe how your proposal will use real-time learning outcomes data and adaptive learning technology and other technology tools to improve the pedagogy and/or curriculum. (1 page)

9

4. Science of learning: Describe how you will use evidence-based pedagogical approaches supported

by research from a variety of disciplines. What is innovative about your approach? How will you take an existing approach and experiment with achieving broader scale? A strong proposal will demonstrate knowledge of and grounding in the literature of the science of learning, and connect the different parts of the project/interventions to the research cited. If relevant, a strong proposal will describe how the project furthers existing research and/or addresses the gaps in our understanding of human learning, with an explicit hypothesis, analytic framework, research design and evidence gathering. (1 page)

5. Student engagement: Describe your approach to student engagement, potentially including

engagement of students who may not identify as STEM proficient. Examples: How might your approach increase students’ sense of belonging, and encourage students’ help-seeking behavior from faculty, teaching assistants, other students, technology resources, etc. Will your approach include engagement through active learning, applied learning through a career or workforce pathway lens, and/or highly contextualized learning? How might students drive their own learning and/or the learning of their peers? Will your proposal individualize learning or use metacognition? How often will students receive meaningful and timely feedback, whether through a technology-mediated environment or face-to-face? (1 page)

6. Culture: How will your proposal impact traditional “classroom” and disciplinary culture? In

particular, how will your approach address aspects of classroom or disciplinary culture that are barriers to student learning and to their sense of belonging? How might it encourage a strengthening-assets or growth-oriented approach to student learning and how might it help establish a classroom context in which all students can succeed? How might your proposal take advantage of under-represented communities’ cultural strengths to increase their achievements in STEM? (1 page)

7. Scalability and value analysis: Describe how your work could be scaled or replicated; made

affordable for users; and/or modified through an open educational resources model. What other dimensions of value can be evaluated in your project? With whom will you partner to do the analysis, what data will you analyze, etc.? A strong proposal will describe the depth and breadth of institutional support for making successful practices normative within the discipline(s), and how faculty will be encouraged or incentivized to adopt successful practices. A strong proposal will include a proposed plan for broad dissemination and lasting impact. (1–1½ pages)

8. Project team: Provide a brief statement of qualifications of the co-PI(s), team, and key

collaborators. Describe the nature and strength of any existing collaborations among project team members, and how you will use the expertise of all involved to create a well-balanced collaboration. Describe also how the project team may use external expertise and/or stakeholder input to iterate over the course of the project. A strong project will demonstrate collaboration with social scientists, behavioral scientists, instructional designers, and/or others with relevant expertise outside of the discipline to be impacted. A strong proposal will also demonstrate meaningful, balanced, near equivalent contributions across the segments represented in the proposal, from design to implementation to evaluation. (1-1½ pages)

10

9. Budget overview: Briefly outline how Learning Lab funds (approximately $1 million to $1.5 million) will be used and how other resources may be leveraged including any outside funds or institutional funds. How will you maximize existing structures or resources? Will your innovations place any costs on users? If so, how will these be minimized? (1 page, with more detail allowed as Appendix B, template to be provided by March 1. Please see http://www.opr.ca.gov/learninglab/) Note: Learning Lab funds are intended to be used exclusively in California. If the project necessitates the use of Learning Lab funds outside of California, provide a brief justification and estimate of the funding that will leave the state. The amount of funds that can leave the state will be subject to the final award agreement.

10. Common data-sharing/technology platform: Please discuss the potential for using a common

data-sharing platform to deliver the course or course series. A strong proposal will discuss the robustness of technology approach and interoperability with other systems. (1 page)

11. Information requested by the Selection Committee. Please respond to the request for information in the individualized summary feedback you received on February 8, 2019, from the Learning Lab. (1–1½ pages)

12. Accessibility. Please describe your plan for ensuring access for students with disabilities,

compliant with your institution’s policies. (½ page)

V. Selection Selection Committee: Learning Lab has recruited an advisory committee, which shall serve as the selection committee to recommend awards. External readers will be recruited to score proposals. Readers may be recommended by the Legislature, public solicitation or academic referral. Selection committee members shall not be deemed to be interested in any contract including any award of Learning Lab funds and will be screened for conflict of interest consistent with National Science Foundation procedures. The names of selection committee members will be provided on the Learning Lab webpage on OPR’s website (OPR.ca.gov). The selection committee will use a process consistent with National Science Foundation procedures for reviewing the proposals and making award recommendations. Learning Lab will use a process consistent with National Science Foundation practices to ensure proposals are evaluated in a manner that is fair, equitable, timely and free of bias. Selection criteria: Section 65059.1 of the Government Code sets forth the following selection rubric, which may be augmented by the Learning Lab and the selection committee:

• “The potential for reducing achievement and equity gaps in the particular discipline that is the subject of the call for proposals.”

• “The depth and breadth of expertise in the particular discipline and deployment of learning science or adaptive learning technologies across the proposal's team members.”

11

• “The prospects for increasing equity and accessibility in quality STEM education and other disciplines that show high initial failure or dropout rates, including scaling access to a newly developed or redesigned course or course series in the future.”

• “The potential to incorporate real-time learning outcome data to improve the curriculum.”

• “The potential to utilize a common technology platform to deliver the course or course series.”

• “The representation of all three public higher education segments on the proposal's faculty team.”9

• “The inclusion of career education and workforce pathways in the proposal.”

• “Opportunities to leverage nonstate funding.”

• “The quality of the concrete metrics and goals identified in the proposal.” The Selection Committee will also consider additional factors in reviewing the proposals,

such as:

• The degree of innovation in the concepts, approaches or methodologies, assessments, or interventions to improve learning outcomes or reduce equity/achievement gaps.

• The feasibility of the project (can the project plan be achieved within the proposed timeline).

• The quality and extent of student engagement and faculty engagement.

• Approaches to protect privacy and personal information.

• Robustness of technology approach and interoperability with other systems.

• Sharing data across institutions.

• Where the project is located in California in order to balance geographic equity of awards, and diversity of awarded institutions.

• Diverse expertise and background of team members, including complementary expertise from social or behavioral scientists that can contribute to design of the proposal and evaluation.

• The degree to which a clear path to broad dissemination and adoption is envisioned and planned.

• Overall impact to advance learning science and learning outcomes.

Results: Applicants that are selected for award will be notified in early to mid-April late April (estimated notification date is April 24). Applicants who are not selected for award will receive a summary statement with perceived strengths and weaknesses of the proposal to inform future submissions for subsequent requests for proposals.

VI. Post-Award Agreements. Applicants of proposals that are selected will be asked to enter into an agreement with the Governor’s Office of Planning and Research. The Learning Lab will administer the agreement,

9 The representation of all three public higher education segments is not an eligibility requirement, but the selection committee will weight proposals that span across all three segments, i.e., UC, CSU and community colleges.

12

which will address project implementation, including the following:

a) Indirect Costs: Up to 8 percent in indirect costs are allowed. Total costs (direct plus indirect) are to be within the $1 million to $1.5 million total per project.

b) Open Educational Resources: Agree to terms and conditions that require course and course series and technology/platforms enabled with Learning Lab funds to be available as open educational resources.

c) Start Date: Initiate work within 30 days of signing the agreement. d) Reporting: Submit progress reports at agreed-upon intervals, including tracking of

milestones and expenditures, participate in conference calls and convening activities, and seek technical assistance from the Learning Lab Advisory Committee or Learning Lab staff. All post-award expectations will be specified in award agreements.

e) Use of Data: Investigators and demonstration teams are expected to share data and research findings consistent with academic standards.

f) Protection of Privacy and Personal Information: Investigators and demonstration project teams are expected to follow state and federal law to protect privacy and personal information.