office of planning and assessment, devin dupree … · online senior assessment 2013: natural...

Online Senior Assessment 2013: Natural Sciences

Office of Planning and Assessment, Devin DuPree and Gail Alleyne Bayne, June 2013 of 13

INTRODUCTION The fourth component of the OSA contains questions for the Natural Sciences core. The first questions ask the participants about how they fulfilled their Natural Sciences core requirement.



The following table shows the number and percentage of participants who selected each response to the first question regarding where students took their core curriculum course. The number of participants selecting each response adds up to more than the 1,578 total participants because participants were instructed to select all that apply.

How did you complete your multicultural requirement?

Response n % of Sample

I took a multicultural class through dual credit in high school. 50 3.2%

I took an advanced placement multicultural class in high school.

59 3.7%

I took a CLEP exam for multicultural credit. 13 0.8%

I received transfer multicultural credit for a class that I took at another institution.

557 35.3%

I took my core curriculum class at Tech. 899 57.0%

For the analysis in this report the 1,578 participants will be divided into the “TTU” group and the “ELSE” group. The TTU group will represent the 899 participants (57.0% of the sample) who selected “I took my core curriculum class at Tech.” The participants who selected one or more of the other responses were asked if the class they took outside of Tech counted for their core curriculum credit. The ELSE group will represent the 606 participants (38.4% of the sample) who reported that a class taken outside of Tech did count for their Natural Sciences core curriculum credit. These 606 participants were asked which course counted. The following table shows the number and percentage of the sample who selected each response.

Which one?

Response n % of Sample

A dual credit class. 47 3.0%

An advanced placement class. 42 2.7%

A CLEP exam. 2 0.1% A class I took at another institution. 502 31.8% I don't know. 6 0.4%

[did not answer] 7 0.4%

Total 606 38.4%

73 participants (4.6% of the sample) reported that they did not know if a course taken outside of Tech counted for their Natural Sciences core curriculum credit. These 73 participants will be excluded from any TTU vs. ELSE group analyses.



The system stores data for each of the participants which makes it possible to identify the respondents who major in programs related to the Natural Sciences core. The following majors were identified and classified as Natural Sciences majors: Animal Sciences, Biochemistry, Biology, Cell and Molecular Biology, Chemistry, Environmental Crop and Soil Sciences, Geosciences, Horticultural Turf Grass Science, Microbiology, Physics, and Zoology. The following table shows that there were a total of 171 Natural Sciences majors in the OSA sample. It also displays how many participants were in each of the Natural Sciences majors.

Natural Sciences Majors

Major n % of sample

Animal Sciences 27 1.7% Biochemistry 23 1.5%

Biology 49 3.1% Cell and Molecular Biology 11 0.7% Chemistry 19 1.2% Environmental Crop and Soil Sciences 4 0.3% Geosciences 16 1.0% Horticultural and Turf Grass Sciences 7 0.4% Microbiology 7 0.4% Physics 6 0.4% Zoology 2 0.1%

Total 171 10.8%



RESULTS The student learning outcomes for the Natural Sciences are:

Demonstrate knowledge of the scientific method and to contrast it with other ways of understanding the world.

Demonstrate knowledge of the tools and methods used by scientists to study the natural world.

Explain some of the major theories in the Natural Sciences.

Describe how Natural Sciences research informs societal issues, including ethics.

The Natural Sciences section of the OSA contains six knowledge questions. The first learning outcome aligns with the fourth, fifth, and sixth question. The second learning outcome aligns with the fourth and fifth question. The third learning outcome aligns with the first, second, and third question. The fourth student learning outcome aligns with the second question. A screenshot and summary of responses for each question is shown on the following pages.



Natural Sciences Question 1:

The chart below shows the distributions of answers for the first question for participants who took their Natural Sciences course at TTU (red) and participants who took their Natural Sciences course elsewhere (blue). Answer 4 is the correct choice. It can be seen that almost half of both groups chose the correct answer. Overall, a few more people in the ELSE group chose the correct answer as compared to the TTU group (50.5% vs. 44.5%). This difference is not statistically significant at the 0.05 level. This means that on average students who took their Natural Sciences course elsewhere do not do significantly better with this question than students who took their course at TTU. Since the first question aligns with the third learning outcome, this suggests that on average students who took their Natural Sciences course at TTU meet this learning outcome similar to students who took their course elsewhere.

2.8%

50.5%

9.7%

19.8%

17.2%

5.9%

44.5%

10.6%

19.6%

19.5%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

Cellular organization

Movement

Homeostasis

Heredity

Metabolism

Percent of Sample

TTU

ELSE




The chart below shows the distributions of answers for the second question for participants who took their Natural Sciences course at TTU (red) and participants who took their Natural Sciences course elsewhere (blue). Answer 4 is the correct choice. It can be seen that less than half of both groups chose the correct answer. Overall, a few more people in the ELSE group chose the correct answer as compared to the TTU group (44.9% vs. 40.9%). This difference is not statistically significant at the 0.05 level. This means that on average students who took their Natural Sciences course elsewhere do not do significantly better with this question than students who took their course at TTU. Since the second question aligns with the third and fourth learning outcomes, this suggests that on average students who took their Natural Sciences course at TTU meet these learning outcomes similar to students who took their course elsewhere.

2.5%

44.9%

14.0%

5.4%

33.2%

1.7%

40.9%

13.9%

4.9%

38.6%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

Does not have an effect on the evolution ofbacteria

Selection for antibiotic-resistant bacteria

Selection against antibiotic-resistant bacteria

Decreased mutation rates in bacteria

Increased mutation rates in bacteria

Percent of Sample

TTU

ELSE




The chart below shows the distributions of answers for the third question for participants who took their Natural Sciences course at TTU (red) and participants who took their Natural Sciences course elsewhere (blue). Answer 1 is the correct choice. It can be seen that a majority of both groups chose the correct answer, with about one fourth of both groups choosing another answer. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (72.1% vs. 71.3%). This difference is not statistically significant at the 0.05 level. This means that on average students who took their Natural Sciences course at TTU do not do significantly better with this question than students who took their course elsewhere. Since the third question aligns with the third learning outcome, this suggests that on average students who took their Natural Sciences course at TTU meet this learning outcome similar to students who took their course elsewhere.

3.5%

12.0%

13.2%

71.3%

3.6%

10.7%

13.7%

72.1%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

There is no difference between oceanic andcontinental crust.

Older oceanic crust exists but has not beendated.

Oceanic crust first began to form 160 millionyears ago; before that no oceans existed.

Oceanic crust is continuously being destroyedby subduction.

Percent of Sample

TTU

ELSE




The chart below shows the distributions of answers for the fourth question for participants who took their Natural Sciences course at TTU (red) and participants who took their Natural Sciences course elsewhere (blue). Answer 4 is the correct choice. It can be seen that a large majority of both groups chose the correct answer. Overall, a few more people in the ELSE group chose the correct answer as compared to the TTU group (79.0% vs. 78.3%). This difference is not statistically significant at the 0.05 level. This means that on average students who took their Natural Sciences course elsewhere do not do significantly better with this question than students who took their Natural Sciences course at TTU. Since the fourth question aligns with the first and second learning outcomes, this suggests that on average students who took their Natural Sciences course at TTU meet these learning outcomes similar to students who took their course elsewhere.

79.0%

6.6%

9.4%

5.0%

78.3%

6.2%

11.0%

4.4%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

hypothesis

conclusion

control group

non-control group

Percent of Sample

TTU

ELSE




The chart below shows the distributions of answers for the fifth question for participants who took their Natural Sciences course at TTU (red) and participants who took their Natural Sciences course elsewhere (blue). Answer 2 is the correct choice. It can be seen that a majority of both groups chose the correct answer. Overall, more people in the ELSE group chose the correct answer as compared to the TTU group (76.9% vs. 73.6%). This difference is not statistically significant at the 0.05 level. This means that on average students who took their Natural Sciences course elsewhere do not do significantly better with this question than students who took their Natural Sciences course elsewhere. Since the fifth question aligns with the first and second learning outcomes, this suggests that on average students who took their Natural Sciences course at TTU meet these learning outcomes similar to students who took their course elsewhere.

0.5%

1.3%

76.9%

21.3%

1.7%

2.2%

73.6%

22.5%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

hypothesis

conclusion

control group

non-control group

Percent of Sample

TTU

ELSE




The chart below shows the distributions of answers for the sixth question for participants who took their Natural Sciences course at TTU (red) and participants who took their Natural Sciences course elsewhere (blue). Answer 2 is the correct choice. It can be seen that a large majority of both groups chose the correct answer. Overall, a few more people in the TTU group chose the correct answer as compared to the ELSE group (86.8% vs. 84.3%). This difference is not statistically significant at the 0.05 level. This means that on average students who took their Natural Sciences course at TTU do not do significantly better with this question than the students who took their Natural Sciences course elsewhere. Since the sixth question aligns with the first learning outcome, this suggests that on average students who took their Natural Sciences course at TTU meet this learning outcome similar to students who took their course elsewhere.

2.5%

5.3%

84.3%

7.9%

2.0%

6.3%

86.8%

4.9%

0.0% 20.0% 40.0% 60.0% 80.0% 100.0%

absolutely disproved

non-supported

supported

absolutely proved

Percent of Sample

TTU

ELSE



Natural Sciences Average: The table below compares the differences between TTU and ELSE when the results for all the questions are averaged (e.g., if a student got 3 out of the 6 questions correct, his score will be 3/6 = .50). The mean is almost the same, but slightly higher for students who took their core requirement for the Natural Sciences elsewhere. This difference is not statistically significant at the 0.05 level. This means that on average TTU scores similar to ELSE in the Natural Sciences.

Core at TTU Core Elsewhere

n Mean SD n Mean SD t-stat p-value

Natural Sciences Overall

899 66.0% 22.9% 606 67.8% 22.3% -1.50 0.0671

The chart below shows the distributions of scores for participants who took their class for the Natural Sciences core requirement at TTU (red) and participants who took their class for the Natural Sciences core requirement elsewhere (blue). The distributions are similar, but it looks like the ELSE group had a few more participants who answered all of the questions correctly.

1.0%

4.3%

10.8%

16.4%

27.9% 28.0%

11.6%

0.8%

3.6%

7.4%

19.3%

28.2%

25.9%

14.7%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

0% 17% 33% 50% 67% 83% 100%

Pe

rce

nt

of

Sam

ple

Percent Correct

Natural Sciences Questions: TTU vs. ELSE

TTU

ELSE



The table below shows a comparison of the average scores for the participants selecting each course option within the ELSE group.

n Mean SD

A dual credit class. 47 65.2% 20.2% An advancement placement class. 42 78.2% 22.5% A CLEP exam. 2 75.0% 35.4% A class I took at another institution. 502 67.2% 22.2% I don’t know. 6 50.0% 23.6% [did not answer] 7 65.4% 2.1%

Total 606 67.8% 22.3% Based on the mean, students who took their Natural Sciences course through advanced placement are the highest scoring group and students who took a dual credit class are the lowest scoring group, excluding the students that did not know which course counted for their core credit. The following table shows a summary of the correlations between the percentage of Natural Sciences questions answered correctly and the following variables of interest: gender, age, SAT score, ACT score, transfer credit hours, total credit hours, cumulative GPA, Humanities major, Mathematics major, Natural Sciences major, and Social and Behavioral Sciences major.

Correlation p-value n

Gender -0.05 0.0321 1,575 Age -0.01 0.6938 1,578 SAT score 0.39 <.0001 897

ACT score 0.34 <.0001 743 Transfer hours -0.03 0.2143 1,578 Total credit hours 0.06 0.0165 1,578

Cumulative GPA 0.15 <.0001 1,576 Humanities major 0.10 0.0001 1,578 Mathematics major 0.02 0.3292 1,578 Natural Sciences major 0.15 <.0001 1,578 Social/Behavioral major -0.02 0.5289 1,578



The following tables summarize the regression model using these same variables of interest with Natural Sciences scores on the OSA as the outcome variable. Note that the variables SAT score and ACT score were combined into one variable, SAT/ACT score, to include more students in the one model. This variable was created by using a conversion table from the ACT website to convert ACT scores to the SAT score range (see http://www.act.org/solutions/college-career-readiness/compare-act-sat/).

Regression model predicting Natural Sciences score

n F-value p-value R2

1,183 26.08 < .0001 0.1820

Variable B SE β

Intercept -0.1896 0.0818 - Gender -0.0093 0.0131 -0.02 Age 0.0063 0.0024 0.08** SAT/ACT score 0.0005 0.0000 0.38*** Transfer credit hours 0.0000 0.0002 0.01 Total credit hours 0.0002 0.0003 0.02 Cumulative GPA 0.0186 0.0112 0.05 Humanities major 0.0724 0.0293 0.07* Mathematics major -0.0137 0.0191 -0.02

Natural Sciences major 0.1088 0.0207 0.15*** Social and Behavioral Sciences major 0.0558 0.0152 0.11***

Note: * p < 0.05; ** p < 0.01; *** p < 0.001; gender: 0 = male, 1 = female The model overall is significant at the 0.05 level (R2 = 0.1820). For this model age, SAT/ACT score, Humanities major, Natural Sciences major, and Social and Behavioral Sciences major were significant predictors at the 0.05 level for Natural Sciences scores on the OSA. The model suggests that when controlling for the other variables in the model older students, students with higher SAT/ACT scores, students with a Humanities related major, students with a Natural Sciences related major, and students with a Social and Behavioral Sciences related major, on average, score higher on the Natural Sciences section of the OSA.

http://www.act.org/solutions/college-career-readiness/compare-act-sat/

http://www.act.org/solutions/college-career-readiness/compare-act-sat/

office of planning and assessment, devin dupree … · online senior assessment 2013: natural...

Documents