hkasme science assessment test 2017€¦ · b. it is a saturated solution. (41%) c. it contains...
TRANSCRIPT
2017/7/6
1
HKASME Science Assessment Test 2017
DATE: 6 JULY 2017
(Debriefing Seminar)
HKASME Science Assessment
Test Purpose:
• To develop and to implement a science assessment test for S2&3 students in Hong Kong
• To analyse the test results and to provide feedback to schools/the administration on the strengths and weaknesses of students in learning science
2017/7/6
2
HKASME Science Assessment
Test
• Duration: 1 hour
• Paper Structure:
Section A: 24 MCQs
Section B: 2 Short-response questions
• Coverage: Science knowledge in
topics 1 to 11 of CDC Science Curriculum
S1 to S3
2017 Science Assessment Test
• Bronze Award based on a criteria-based model
• Introducing a Diamond Award (top 5%)
• Expert judgment for Gold, Silver and Bronze Award
• QP and students’ performance on individual questions available to participating schools
• More informative school reports
2017/7/6
3
Participations in 2017 SAT
No. of Participating Schools: 88
Participants S1 S2 S3 Total
Boys 14 622 654 1290
Girls 6 574 532 1112
Total 20 1196 1186 2402
Science Process Skills
• Observing, measuring and recording data (SP1)
• Comparing and classifying (SP2) • Planning and designing (SP3) • Experimenting (SP4) • Interpreting data / Inferring (SP5) • Communicating (SP6)
2017/7/6
4
Weaknesses of Participants as
Revealed from Their
Performance
Section A
Discrimination index (DI)
describes the ability of an item to distinguish between high and low scorers
ranges between -1.00 and +1.00
expected that the high-performing students select the correct answer for each item more often than the low-performing students
DI = (H-L) / ½ N
2017/7/6
5
items with p-value between 30 - 70 and DI > 0.24
“appropriate”
Understanding and Interpreting
Graphical Information
The graph below shows the variation of solubility of a
salt X with temperature:
Q.12
2017/7/6
6
In an experiment, 40 g of X is added to 100 g of water kept at
60oC and the mixture is stirred until there is no further change.
Based on the above information, which of the following
statements about the resulting mixture is correct?
*A. It is an unsaturated solution. (42%)
B. It is a saturated solution. (41%)
C. It contains undissolved X. (13%)
D. It has a mass of 100 g. (4%)
Less than half of the participants were able to choose the key
showing that they were weak in the interpretation of graphs.
Discrimination index = 0.56
Q.19
An experiment using the set-up
shown on the right can be used to
study the pressure-volume
relationship of a fixed mass of air at
room temperature.
Pressures (P) of different
magnitudes are applied onto the
plunger and the corresponding
volumes of the air inside the syringe
(V) are measured. Which of the
following graphs correctly
represents the experimental results
obtained?
2017/7/6
7
V
P
V
P
V
P
V
P
A majority of the participants knew that, under constant
temperature, the volume of a fixed mass of gas decreases with its
pressure. However, most of them did not know that the volume-
pressure relationship is non-linear.
Discrimination index = 0.22
Understanding of Abstract Science Concepts
Q.7 Toby throws a ball in air at an angle. The diagram
below shows the path of the ball and four positions, P,
Q, R and S, of the ball along the path:
In which of these positions, does the arrow represent the
direction of the force acting on the ball? (Assume that there is
no air resistance.)
2017/7/6
8
A. P (17%)
B. Q (54%)
C. R (12%)
*D. S (17%)
More than half of the participants wrongly thought that the
direction of the force is tangential to the trajectory of the ball.
They did not realize that when the ball had left Toby’s hand,
the pushing force (a contact force) would cease to act on it,
and the only force acting on the ball was the gravitational
attraction (a non-contact force).
Discrimination index = 0.58
Q.15 Peter used the set-up shown below to study the image
formed by a convex lens:
He moved the lens to and fro until a sharp inverted
image was formed on the screen. If Peter repeats the
experiment using a thicker convex lens, which of the
following combinations correctly describes the position of
the lens and the appearance of the sharp image formed?
2017/7/6
9
Position of the lens Appearance of the
image
*A. closer to the screen remains inverted (42%)
B. closer to the screen becomes upright (13%)
C. further away from the
screen remains inverted (36%)
D. further away from the
screen becomes upright (9%)
Nearly 80% of the participants knew that, under the
experimental conditions, an inverted image would be
formed. However, quite a number of them failed to realize
that a thicker convex lens bends a beam of light to a
greater extent and forms an image closer to the screen.
Discrimination index = 0.37
Classification of Animals
Q.14 Consider the animals shown below:
Which of the following features of these animals can be used
to classify them into two groups?
(1) whether or not they live in water
(2) whether or not they have antennae
(3) whether or not they have wings
2017/7/6
10
A. (1) only (12%)
B. (2) only (5%)
C. (1) and (3) only (56%)
*D. (2) and (3) only (27%)
A majority of the participants recognized that ducks
and flies have wings while elephants and frogs do not.
However, quite a number of them did not know that
frogs are amphibians, and some failed to realize that
flies do have antennae.
Discrimination index = 0.35
Weaknesses of Participants as
Revealed from Their
Performance
Section B
2017/7/6
11
Question No.
Mean S.D.
1 3.0
(30%) 2.13
(21.3%)
2 4.6
(58%) 1.79
(22.4%)
Overall 7.7
(43%) 3.28
(18.2%)
• Q.1 - to test participants’ understanding of
experimental procedures, communicative and
graph-plotting skills, and interpretation of
experimental data
• Q.2 – to test participants’ understanding of scientific
methods, the design of fair tests and the various
factors that affect a fair comparison
B1 Question on Data-handling
2017/7/6
12
(b) Word equation: chips + oxygen → carbon dioxide + water + energy
No. of
response
s
%
10 All correct 313 13.0
11 Replacing oxygen with air 2 0.1
12 Replacing energy with heat (熱/熱能) 252 10.5
13 Replacing oxygen with air and energy with
heat 5 0.2
90 Failure to give both (answers other than
oxygen/air & energy/heat) 590 24.6
91 Giving oxygen/air but failing to give
energy/heat 138 5.7
92 Failing to give oxygen/air but correctly
giving energy/heat 1053 43.8
93 Using O2 for oxygen & E for energy 2 0.1
99 Nil response 47 2.0
2017/7/6
13
90
92
(c)(i) Plot a graph to show the relationship
between ‘the mass of potato chip burnt’
and ‘the temperature rise of the water in the
boiling tube’.
Three aspects were looked for in marking the graph:
• correct labeling of axes (code: A1x, A9x);
• appropriate choice of scale (code: S1x, S9x); and
• joining the points with a straight line (code: G1x,
G9x).
2017/7/6
14
No. of responses %
A10 Plotting ‘temperature rise of water’ against ‘mass of potato chips burnt’, and both axes are correctly labeled with correct units
547 22.8
A11 Plotting ‘temperature rise of water’ against ‘mass of potato chips burnt’, and both axes are correctly labeled with some units (g/oC) missing
40 1.7
A12 Plotting ‘mass of potato chips burnt’ against ‘temperature rise of water’, and both axes are correctly labeled with correct units
240 10.0
A13 Plotting ‘mass of potato chips burnt’ against ‘temperature rise of water’, and both axes are correctly labeled with some units (g/oC) missing
22 0.9
A91
Plotting ‘temperature rise of water’ against ‘mass of potato chips burnt’, and both axes are correctly labeled with no/incorrect units (e.g. writing kg instead of g)
88 3.7
A92 Plotting ‘mass of potato chips burnt’ against ‘temperature rise of water’, and both axes are correctly labeled with no/incorrect units
40 1.7
A93 Plotting ‘final temperature of water’ against ‘mass of potato chips burnt’ or vice versa
1021 42.5
A94 Graph with one of the axes unlabeled 71 3.0
A98 Other mistakes related to axes (e.g. double axis) 33 1.4
A99 Graph without showing labelled axes/ Nil response 300 12.5
No. of
responses %
S10 Graph with appropriate scale in both x- and y-axes 831 34.6
S91 Graph with the scale in one of the axes not equally spaced
213 8.9
S92 Graph with a poor scale (e.g. the graph is highly compressed or falls out of the graph paper) in either axis
456 19.0
S93 Scale NOT at origin 128 5.3
S94 Scale using direct entry in either axis 502 20.9
S98 Other mistakes related to scale 94 3.9
S99 Graph without scale in both axes/ Nil response 178 7.4
2017/7/6
15
(Code: A94, S92, G13) (Code: A93, S94, G91)
(Code: A93, S93, G11) (Code: A94, S94, G92)
2017/7/6
16
No. of
responses %
G10 A straight line graph passing through all points and the origin
268 11.2
G11 A straight line graph passing through all points but not the origin
484 20.1
G12 A graph with line segments passing though all points and the origin
31 1.3
G13 A graph with line segments passing though all points but not the origin
153 6.4
G90 No line shown on the data points 163 6.8
G91 All or some of the points are incorrectly plotted 881 36.7
G92 A graph showing a histogram/bar chart/pie chart instead of a straight line
63 2.6
G98 Other mistakes related to graph (e.g. two lines/curves) 139 5.8
G99 Nil response 220 9.2
Mean score for plotting graph = 1.1 (out of 3)
Some participants were able to estimate the temperature rise to be
41oC, but failed to give the final temperature as 66oC.
No. of
responses %
20 No. The heat absorbed by the water in the boiling tube depends on its temperature rise. It is not related to the initial temperature of the water.
348 14.5
10 No. It is not possible to set all initial temperature at 25.0oC. 6 0.2 91 Yes. To ensure the experiment is a fair test, the initial
temperature should be kept the same. 988 41.1
92 Yes. The initial temperature is the controlled variable 221 9.2 93 Yes. Different initial temperatures will affect experimental
results/ lead to inaccurate results 333 13.9
98 Yes/No. Other irrelevant reasons (e.g. easier to compare) 439 18.3 99 Nil response 67 2.8
2017/7/6
17
91
92
93
No. of
responses %
2 Two correct reasons given 89 3.7 1 One correct reason given 744 31.0 0 Incorrect reasons/ nil response 1569 65.3
About 65% of the participants failed to give the following two reasons for
the difference in energy content:
• heat loss to surrounding, and
• incomplete burning of potato chips.
Some strange reasons such as ‘energy given out by digestion/body
consumption is different from the energy measured by the burning of
potato chips’, and ‘other substances present in the package may
increase the energy of potato chips’ were given.
2017/7/6
18
1 correct reason given:
Incorrect responses:
Q2
2017/7/6
19
Q.2(a)(i) Marking Scheme (Double-digit Coding)
Mean mark = 0.5 (out of 1)
Q.2(a) (i) Samples of Participants’ Work
93
10
92
2017/7/6
20
Q.2(a)(ii) Marking Scheme and Samples
(Similar marking scheme for dependent and controlled variables)
Mean mark for Q.2(a)(ii) = 2.4 (out of 3)
Q.2(a)(iii) Marking Scheme
Mean mark = 0.3 (out of 2)
2017/7/6
21
Q.2(a)(iii)Samples of Participants’ Work
Q.2(b) Marking Scheme
Mean mark = 1.4 (out of 2)
2017/7/6
22
Q.2(b) Samples of Participants’ Work
Analysis of 2017 SAT Results
Mean SD
Section A 14.7 (61%) 3.47 (14.4%)
B1 3.0 (30%) 2.13 (21.3%)
B2 4.6 (58%) 1.79 (22.4%)
Section B 7.7 (43%) 3.28 (18.2%)
2017/7/6
23
Comparison of Performance of
S2 and S3 Students
Section A Section B
Mean S.D. Mean S.D.
S2 14.0
(59%) 3.33
(13.9%) 7.1
(39%) 3.04
(16.9%)
S3 15.3
(64%) 3.49
(14.5%) 8.2
(46%) 3.42
(19.0%)
Comparison of the Performance
of Boys and Girls in Section A
Median (correct to the unit digit) = 15
15 correct
S2 S3 Whole group
Boys 53% 64% 59%
Girls 38% 45% 47%
2017/7/6
24
Performance of S2 Students in Section A
Performance of S3 Students in Section A
2017/7/6
25
Performance of Whole Group in Section A
2017 SAT Award Scheme
Bronze Silver Gold Diamond
Paper score / marks
24.5 – 31.5 32.0 – 36.0 36.5 – 41.5 >42.0
Section A score / MCQs
> 10 > 10 > 10 > 10
Section B
score / marks
> 4 > 4 > 4 > 4
(Paper score = 1.5 × Mark in Section A + Mark in Section B)
HKASME set up an expert panel to determine the
cut-off based on the performance of the
participants.
2017/7/6
26
2017 SAT Awards
Award type Number
Diamond 120
(5%)
Gold 351
(15%)
Silver 492
(20%)
Bronze 873
(36%)
Appreciation 566
(24%)
School-based Report –Appendix I
Analysis of Participants’ Responses to the Multiple-choice Questions (MCQs)
Q. No Skills assessed Key
% correct
Strength / weakness WG School
1 Comparing & Classifying; Understanding
B 76.6 78.8 -
2 Laboratory safety; Understanding
B 56.3 39.4 W
3 Inferring C 71.6 93.9 S
4 Planning & Designing; Fair tests
C 56.5 69.7 -
5 Inferring; Understanding D 63.0 93.9 S
6 Applying; Mathematics skills
D 85.9 97.0 -
7 Understanding; Inferring D 17.1 24.2 D
8 Measuring; Choice of apparatus
C 74.8 90.9 S
9 Understanding; Predicting
B 64.1 78.8 -
10 Inferring; Understanding A 54.2 63.6 -
2017/7/6
27
Strength in a question (indicated by “S”) means that
the school got a higher correct % by 15% or more on
that question compared to the whole group in 2016
SAT.
Weakness in a question (indicated by “W”) means
that the school got a lower correct % by 15% or
more on that question compared to the whole
group in 2016 SAT.
Questions that are poorly answered by the whole
group (1/3 correct) are represented by “D”
Question(s) showing “weaker” performance as compared to WG
WG School
Q. No. Key Most Popular
Distractor Key Most Popular
Distractor
9 B
(64.1%)
C
(26.0%)
B
(46.7%)
C
(36.7%)
12 A
(42.0%)
B
(40.5%)
A
(26.7%)
B
(40.0%)
20 D
(54.0%)
B
(27.5%)
D
(33.3%)
B
(40.0%)
2017/7/6
28
Analyzing student responses & identifying their problems
Q. No. A B C D Right
Ans.
Correct
%
Wrong
%
Diff.
Index
Disc.
Index
14 11.53
%
4.75
%
56.16
%
27.48
% D
27.48
% 72.44% 32.45% 0.35
Analysis of Participants’ Performance in
Short-response Questions – Appendix II
Mean and standard deviation
Whole Group (WG) School
Mean SD Mean SD
Question 1
(out of 10)
3.0
(30%)
2.13
(21.3%)
4.6
(46%)
1.82
(18.2%)
Question 2
(out of 8)
4.6
(58%)
1.79
(22.4%)
5.4
(68%)
1.06
(13.3%)
Section B
(out of 18)
7.7
(43%)
3.28
(18.2%)
10.0
(56%)
2.08
(11.6%)
2017/7/6
29
Double-digit coding system for Q1(c)(i) Plot a graph to show the relationship between ‘the mass of
potato chip burnt’ and ‘the temperature rise of the water in the boiling tube’
CODE ITEM Number of Response
WG School
Correct Responses : 831
(34.6%) 26
(78.8%)
S10 Graph with appropriate scale in both x- and y-axes
831 (34.6%)
26 (78.8%)
Incorrect Responses/Nil Response 1571
(65.4%) 7
(21.2%)
S91 Graph with the scale in one of the axes not equally spaced
213 (8.9%)
1 (3.0%)
S92 Graph with a poor scale (e.g. the graph is highly compressed or falls out of the graph paper) in either axis
456 (19.0%)
3 (9.1%)
S93 Scale NOT at origin 128
(5.3%) 0
(0.0%)
S94 Scale using direct entry in either axis 502
(20.9%) 3
(9.1%)
S98 Other mistakes related to scale 94
(3.9%) 0
(0.0%)
S99 Graph without scale in both axes/ Nil response 178
(7.4%) 0
(0.0%)
School-based Report – Appendix IV
(1) Diamond, Gold, Silver and Bronze Awards in 2017
(2) Overall Performance of School
Award type Number
Diamond 120 (5%)
Gold 351 (15%)
Silver 492 (20%)
Bronze 873 (36%)
Appreciation 566 (24%)
Whole Group
(WG)
School
Mean score:
(out of 54) 29.7 (55%) 34.7 (64%)
Standard
deviation: 7.61 (14.1%) 6.92 (12.8%)
2017/7/6
30
Student Performance Report (Sample)
Professional Learning Community
on Science Assessment
• Collaborative culture in the development of quality
assessment items/tasks among science teachers
• MCQs, short-response questions, double digit
coding marking
• Talks, group discussion, moderation exercise and
assignments
• TWO 3-hour Saturday meetings (19/8, 2/9)
• Meeting fee: $100
(Full refund if all 2 meetings are attended and
assignments completed.)