evaluation report on the implementation of the incas ... report on the... · scope of the report...

1

Evaluation Report on the Implementation of the InCAS

computer-based assessments – September 2010

Copyright 2010 CCEA. All rights reserved.

This report is available at: www.nicurriculum.org.uk

This evaluation was conducted by the CCEA Research and Statistics Unit from

September 2009 through to March 2010. For further information about this

evaluation please contact Karen Wiltshire:

CCEA Research & Statistics Unit

29 Clarendon Road

Clarendon Dock, Belfast

BT1 3BG, or alternatively email

[email protected]

2

Scope of the Report

This is the final evaluation report on the phased implementation of the InCAS

computer based assessments in Years 4 to 7. Based on feedback from

primary principals, teachers, pupils and parents this report sets out the scope

for improvement. As with findings from previous research reports, outcomes of

this evaluation will be used to inform future planning and development. This

report is also intended to provide stakeholders with an overview of the overall

implementation of InCAS across the primary sector in Northern Ireland.

A full breakdown of the evaluation process is detailed in the method section of

this report.

3

Acknowledgements

CCEA would like to thank all principals, teachers, pupils and parents who

contributed to this evaluation. We are extremely grateful to all participants for

taking the time to share their experiences of the InCAS computer based

assessments. This cooperation and involvement allows CCEA to have a

detailed and collective understanding of the implementation of the new

assessment arrangements across primary schools in Northern Ireland.

Furthermore, this evaluation has provided schools, parents and pupils with an

opportunity to inform the future planning and development of InCAS.

4

Contents

Section Page

1.0 Executive Summary 5

2.0 Introduction 12

3.0 Method 14

4.0 Results 16

4.1 Principal Feedback 16

4.2 Teacher Feedback 28

4.3 Parent Feedback 48

4.4 Pupil Feedback 59

5.0 Conclusion 66

6.0 Recommendations 68

5

1.0 Summary of Findings Rationale This is the final evaluation report on the phased implementation of the InCAS

computer based assessment. In the 2009 autumn term, InCAS became the

statutory form of assessment for pupils in Years 4 to 7. This evaluation

provided all stakeholders (principals, teachers, parents and pupils) involved in

the final phase of implementation with an opportunity to present their views

and opinions on the new primary assessment arrangements. A total of 182

principals, 469 teachers, 15,838 parents and 11,117 pupils completed

questionnaires. Information collected from this evaluation has been used to

measure the success of the implementation to date and inform future planning

and development.

Key Findings Administration of InCAS in schools Overall, feedback from principals, teachers and pupils on the administration of

InCAS was positive. 93.4% of principals (n = 170) found the implementation of

InCAS manageable. Furthermore, feedback from the majority of teachers

(93.4%, n = 438) and pupils (76.4%, n = 8,484) indicates that pupils found

InCAS user-friendly. Comments from pupils who did not find InCAS easy to

complete suggest that these pupils found the subjects (Reading or General

Maths) difficult as opposed to the usability of the assessment tool.

Despite such positive feedback, over a third of principals (45.9%, n = 83) and

teachers (37.1%, n = 174) encountered challenges administering InCAS. Only

a small number of respondents highlighted particular challenges they

encountered. The three most common challenges commented upon

concerned computers freezing (25 comments), logging pupils on

(24 comments) and insufficient computer capacity (15 comments). Such

findings are consistent to those collected in the previous year’s evaluation.

However, despite such challenges, the majority of teachers who encountered

difficulties administering InCAS indicated that they felt confident at the

prospect of using this assessment tool in the future (92.5%, n = 209).

6

One such teacher stated that “the guidance is clear and if I have a problem I

know how it can be addressed”.

It is important to note that over the last three years a small, but sizeable

number of principals and teachers have highlighted that they have inadequate

computer capacity to administer InCAS. Such respondents have found

administering InCAS challenging.

Support CCEA provides schools with a variety of support and guidance in the form of;

training materials, telephone helpdesk, a dedicated section on the Northern

Ireland Curriculum website, online tutorials and an InCAS pupil demonstration.

Both principals and teachers rated all aspects of support highly. Pupils found

Inky particularly useful, with 84.5% of pupils (n = 9,390) indicating that Inky

showed them how to operate InCAS.

However, just over a third of principals (38.1%, n = 69) felt they would benefit

from further support to assist with the administration of InCAS. Principals

requested funded substitute cover (18 comments) and additional computers

(17 comments). 62.6% of principals (n = 114) revealed that classroom

assistants support schools with the administration of InCAS. A large number

of principals (79.7%, n = 145) and teachers (71.7%, n = 170) felt classroom

assistants would benefit from training on InCAS.

InCAS Outcomes This academic year (2009/10) errors affecting the General Maths age-

equivalent scores and the standardised scores were detected in some of the

outcomes given to schools. Findings from this evaluation strongly suggest that

such errors reduced stakeholders’ confidence with the reliability of InCAS

outcomes. 42.3% of principals (n = 77) did not believe InCAS outcomes

reflected pupils’ level of ability. Furthermore, only 73.9% of teachers (n = 345)

deemed the outcomes from InCAS to be consistent with their professional

judgement. This represents nearly a 20% reduction from the 2009 evaluation

responses (93.5%, n = 314). Closer analysis of feedback from principals,

7

teachers and parents by year group suggests that stakeholders principally

questioned the outcomes produced for Year 4 pupils. A number of principals

(n = 22) and parents (n = 380) felt that InCAS outcomes for Year 4 pupils were

lower than pupils’ actual level of ability.

The 2009/10 evaluation shows teachers’ confidence levels in reporting

assessment outcomes to parents was 79.5% (n = 373) this is a proportional

drop from the 2008/09 responses (91.6%, n = 308). 56 teachers stated that

they were not confident reporting outcomes to parents as they questioned their

reliability.

A sizeable number of teachers (32.2%, n = 151) and principals (28.6%, n = 52)

did not believe the outcomes from InCAS enhanced the information shared at

the parents meeting. Furthermore, 25.8% of parents (n = 3,882) did not

believe InCAS outcomes accurately reflected their child’s level of ability.

Despite such concerns, 80% of teachers who responded to the evaluation felt

that InCAS outcomes were helpful in informing the learning and teaching of

individual pupils (80.6%, n = 378) as well as overall classes (79.3%, n = 372).

Furthermore, the majority of parents responding (91.5%, n = 13,749) found the

outcomes easy to understand and felt they informed them of their child’s

strengths (85.3%, n = 13,133) and areas for development (82.3%, n= 12, 556).

Despite being covered in the teacher training sessions and the InCAS manual,

19 teachers commented that they had difficulties interpreting outcomes

whenever pupils received a * or # symbol. Teachers requested clarification on

how to interpret these scores.

Just over half of pupils (51.6%, n = 5,731) indicated that they were unsure why

they completed InCAS.

School Management Feedback from this evaluation suggests that senior management have started

to establish InCAS into schools. Nearly 90% of principals held staff meetings

8

to determine how to share InCAS outcomes with parents (89.5%, n = 162) and

to determine how InCAS outcomes could be best used in their school

(89%, n = 161). Furthermore, a number of principals indicated that they used

InCAS outcomes for target setting (73.1%, n = 133) and also within their

school development plan (60.4%, n = 110).

Statutory Requirements Findings from this evaluation reveal that a small, but noteworthy number of

schools did not meet all the statutory assessment requirements. 14.8% of

principals (n = 27) did not report InCAS outcomes from previous years.

Furthermore, 13.5% of parents (n = 2,114) did not receive their child’s

outcomes from InCAS in writing.

9

Conclusion

This evaluation reveals that the errors in the InCAS outcomes during the

2009/10 autumn term have reduced stakeholders’ confidence with the

reliability of this assessment. Feedback from respondents indicated that

respondents particularly queried the reliability of outcomes for Year 4 pupils,

with many commenting that scores were lower than pupils’ actual level of

ability. It is imperative that all educational bodies (The Department of

Education, CCEA, and CEM) provide reassurance to schools by outlining the

procedures taken to ensure the reliability of InCAS in the future.

Feedback from this evaluation also indicates that schools encountered

challenges administrating InCAS. However, only a small number of

respondents specified challenges they encountered. The manner in which

InCAS is administered varies greatly between schools, depending upon their

individual setting. This evaluation suggests that schools may benefit from

guidance outlining the ways in which schools can best manage the

administration of InCAS. Furthermore, a review of ICT provision should be

carried out to identify schools in need of additional ICT resources. Feedback

from principals and teachers also suggests that training classroom assistants

on the administration of InCAS would further support schools.

Over 50% of pupils (51.6%, n = 5,731) indicated that they were not sure why

they completed InCAS. In keeping with the aim of the “Big Picture” of the

Northern Ireland Curriculum at Key Stages 1 & 21 which “aims to empower

young people to achieve their potential and to make informed decisions

throughout their lives” pupils should be aware of why they are completing this

assessment. Furthermore, if pupils’ outcomes from InCAS are explained in a

way which is meaningful to them, they should have an understanding of their

strengths and weaknesses. Such information should enable them, in

collaboration with their teacher or parent, to formulate achievable targets for

themselves for the year ahead.

1 http://www.nicurriculum.org.uk/docs/key_stages_1_and_2/Big-PicturePrimary-KS12.pdf

10

Despite the areas for improvement highlighted above, this evaluation has

indicated that schools have found InCAS manageable to deliver; pupils have

found this assessment user-friendly and InCAS outcomes are useful to schools

and parents. Moreover, principals and teachers rated the support provided by

CCEA very highly. CCEA should continue to offer such support to ensure

schools are adequately supported and empowered to manage these new

assessment arrangements.

11

Recommendations

Prior to the administration of InCAS in the 2010/11 autumn term there are a

number of issues documented within this report that should be addressed with

a matter of urgency.

This evaluation suggests that CCEA should:

in partnership with other educational bodies put a communication strategy

in place at the start of the autumn term to rebuild confidence throughout the

primary sector on the reliability of InCAS;

continue to provide schools with guidance on the ways in which schools

can best manage the administration of InCAS;

continue to empower schools to embrace InCAS and develop expertise and

confidence in administering, analysing and making use of the outcomes

produced from InCAS;

continue to work in partnership with C2K and CEM to prevent technical

difficulties occurring during the administration of InCAS;

provide classroom assistants with training on the administration of InCAS;

liaise with schools to ensure that the purpose of the InCAS assessments is

clear to pupils;

encourage teachers to share outcomes from the InCAS assessments with

pupils and set targets with them for the year ahead;

advise the Department of Education to carry out a review of ICT resources

to identity schools with inadequate ICT provision for the administration of

InCAS;

in partnership with the Department of Education, continue to communicate

with parents on the InCAS assessments; and

provide further advice to teachers on the meaning of the * or # symbols

allocated to some pupils’ outcomes.

12

2.0 Introduction

Purpose of InCAS Assessment

InCAS is a diagnostic assessment tool in Reading and General Maths and

optional associated assessments. InCAS is designed to support schools in

identifying pupils’ strengths and areas where they are having difficulties. The

outcomes from the InCAS assessments should therefore help teachers plan

their teaching during the school year to meet the needs of pupils in their

classrooms. The assessments also provide schools with useful information for

monitoring individual pupil progress to ensure that every pupil is reaching his

or her full potential and generally, to inform self-evaluation, development

planning and target-setting.

Requirement to Carryout Diagnostic InCAS Assessments

The Interactive Computerised Assessment System (InCAS) has been

introduced to primary schools in Northern Ireland on a phased basis over three

school years, from 2007/08. In 2007/08 InCAS applied to Year 5 pupils, in

2008/09 it was extended to cover Year 5 and 6 pupils, and from the 2009/10

school year it applied to pupils in Years 4 to 7.

This academic year (2009/10) under the Education (Assessment

Arrangements) (Foundation to Key Stage 3) Order (NI) 2007 (the 2007 Order)

the Department of Education formally specified in circular 2009/142 that InCAS

was the computer based assessment method to be used for carrying out the

statutory diagnostic assessment of reading and mathematics for pupils in

Years 4 to 7 in the autumn term.

The 2007 Order stipulates that the outcomes of the specified computer based

assessments are to be recorded and notified to parents by the end of that

2 http://www.deni.gov.uk/incas_circular_to_schools_-_september_2009-2.pdf

13

term. In accordance with this Order, the Department of Education stipulated

that from the 2009 autumn term, schools are required to:

provide age equivalent InCAS outcomes for Reading and General Maths to

parents in writing during the autumn term for the 2009/10 school year, and

where available the 2008/09 school year. If they wish, schools may also

provide outcomes for the 2007/08 school year and, if parents request

outcomes for 2007/08, this information should be provided;

provide information in a graphical format where two years of InCAS

outcomes are available; and

offer a parents meeting before the end of the autumn term to set the InCAS

outcomes in an educational context as a basis for explaining to parents

their child’s strengths, and any difficulties the child may be experiencing,

and the actions the school intends to take to address these. The meeting

also provides an opportunity to help parents understand what they can do

to support their child’s learning.

Evaluation of the Implementation of InCAS

CCEA’s Research & Statistics (R&S) Unit has been responsible for the

evaluation of the implementation of InCAS. Throughout each roll out year, the

R&S Unit has provided all stakeholders involved in this process with an

opportunity to provide feedback on this assessment. CCEA has used the

feedback collected throughout the evaluation process to determine how to best

support schools to implement InCAS and further develop this assessment tool.

14

3.0 Method

The overall aim of this evaluation was to conduct a robust and reliable

evaluation of the final implementation phase of the new InCAS computer

based assessments.

In order for this evaluation to be as inclusive as possible, information was

collected from all stakeholders involved in this year’s implementation of the

InCAS assessments. This encompassed principals, teachers, parents and

pupils. Excluding special schools, there are a total of 872 primary schools in

Northern Ireland (including preparatory and independent) with an approximate

population of 91,130 pupils in Years 4 to 7 (DE statistics 2008 – 2009).

A separate evaluation has been conducted with special schools, to further

explore the manageability of InCAS for pupils with special educational needs.

Summary of Participants and Research Tools

Questionnaires were used to collect feedback from stakeholders. Principals,

teachers and pupils were asked to complete an online questionnaire.

Additional hard copies of the questionnaires were available to all on request.

Hard copies of the parent questionnaire were posted to schools for distribution.

Questions within all questionnaires were twofold. Scales were used to obtain

quantitative evidence from respondents. Comment boxes were then used to

capture the reasoning behind respondents’ answers to all dichotomous and

scaled questions.

Feedback from Principals

All primary principals (n = 870) in Northern Ireland were invited to complete an

evaluation questionnaire on InCAS. A total of 182 principals completed a

questionnaire, which represents a response rate of 20.9%. Responses were

representative of the five Education and Library Boards.

15

Feedback from Teachers

All Year 4, 5, 6 and 7 teachers required to complete InCAS (Years 4 to 7) were

invited to complete an evaluation questionnaire on InCAS. A total of 469

teachers completed questionnaires, which represents a response rate of

approximately 13.4%. Responses were representative of the five Education

and Library Boards. Feedback was collected fairly evenly across the four year

groups.

Feedback from Parents

There are approximately 91,130 (2008/09 DE statistics) parents with a child in

Years 4 to 7 in Northern Ireland, all of whom were invited to complete a

questionnaire on InCAS. A total of 15,838 completed questionnaires were

returned which represents a response rate of approximately 17.4%. Parental

feedback was evenly received from parents with children in the four year

groups required to complete InCAS and from the five Education and Library

Boards.

Feedback from Pupils

A total of 11,117 pupils across Years 4 to 7 completed a questionnaire about

the InCAS assessments. Based on these figures this evaluation has obtained

a response rate of 12.2% of all Year 4 to 7 pupils in Northern Ireland.

Feedback was collected fairly evenly across the four year groups. Feedback

was representative of pupils in Years 4, 5, 6 and 7.

16

4.0 Results collected from Principal Questionnaires

This section presents the results for each question from the Principal

questionnaire. Results are based on completed questionnaires from 182

respondents. Percentages relate to the number of principals who answered

each question, which may be fewer than the overall total of 182*. A number of

principals reiterated comments throughout their questionnaire; as a result

some comments are repeated throughout this section of the report.

Findings from this year’s evaluation were compared to that collected from the

previous year. No noteworthy differences were noted.

* It should be noted that throughout the report, figures are rounded to one decimal place. Consequently

cumulative percentages may not total 100%.

17

20.9

72.5

60.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Verymanageable

Manageable Notmanageable

Not at allmanageable

The implementation of InCAS

Figure 1: Manageability of this year’s implementation of InCAS

The majority of

principals

(93.4%, n=170)

found the

implementation

of InCAS

across Years 4

to 7 to be

manageable.

Chart does not precisely reflect the data. For exact figures, see percentage above individual bars.

Despite such positive feedback, a small number of principals expressed

concerns regarding the manageability of the InCAS assessments. 16

respondents felt their school did not have a sufficient number of computers to

administer the assessments across Years 4 to 7.

Furthermore, 13 respondents stated that schools require funded substitute

cover to release teachers to administer InCAS. 12 principals indicated that

they encountered challenges organising the administration of InCAS.

18

Figure 2: School management decisions to support the

completion of InCAS

Figure 2

provides a

breakdown of

the managerial

decisions

principals made

for the

completion of

InCAS in their

school.


Figure 2 above reveals that the majority of principals (85.2%, n = 155) made

timetabling decisions for the administration of InCAS. A large number of

respondents also made decisions regarding the distribution of resources

(73.1%, n = 133) and supervision of classes (68.7%, n = 125). Over half of

respondents (51.6%, n = 94) made managerial decisions regarding the

deployment of staff.

Principals were asked to record any procedures they implemented to support

the administration of InCAS. A small number of principals indicated that they:

arranged substitute cover for the supervision of classes (20 comments);

organised the availability of the computer suite (13 comments); and

used non contact days to supervise pupils completing InCAS

(6 comments).

51.6

73.1 68.7

85.2

2.2

0

20

40

60

80

100%

of

Res

po

nd

ents

Deployment of staff

Resources

Supervision of classes

Timetabling

Other

19

45.9

54.1

0

20

40

60

80

100%

of

Res

po

nd

ents

Yes No

Figure 3: Challenges administering InCAS

Figure 3 illustrates

that 45.9% of

respondents (n=83)

encountered

challenges

administering the

InCAS

assessments.


Respondents were asked to highlight any particular difficulties they

encountered. Responses are summarised below:

insufficient computer capacity within school (15 comments);

computers / laptops froze during the assessments (8 comments);

challenges logging pupils onto InCAS (7 comments); and

unreliable internet access in school (7 comments).

20

CCEA support

Table 1: Use of CCEA support

Responses Support Number %

Training Manual 173 95.6 InCAS preparation letter 167 91.8 Resources folder 163 90.1 Inky pupil demo 160 87.9 On-line materials on www.nicurriculum.org.uk

144

79.6

CCEA helpdesk 127 70.2

Table 1 above provides a breakdown of the various modes of support

principals and their schools made use of. The majority of principals made use

of the Training Manual (95.6%, n = 173), InCAS preparation letter

(91.8%, n = 167) and Resources folder (90.1%, n = 163).

Principals had an opportunity to comment on their answer. A small number of

principals commended the usefulness of the Inky Pupil Demo (n = 14) and the

InCAS Preparation Letter 09/10 (n = 8).

Figure 4: Support provided by CCEA

Principals were

asked to rate the

support provided

by CCEA. The

response was very

positive with 94.5%

of respondents

(n = 171) rating this

support highly.


57.5

37

50.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very good Good Poor Very poor

21

28.637.4 34.6

38.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Attitudes DevelopedAbility

MentalArithmetic

Spelling

Although principals rated the support provided by CCEA highly, just over a

third of respondents (38.1%, n = 69) felt they would benefit from further

support to assist with the administration of InCAS.

Principals recommended the following:

further funded substitute cover (18 comments);

additional computers (17 comments); and

more examples of InCAS questions (9 comments).

Non statutory assessments

51.6% (n = 94) of respondents administered the non statutory assessments to

their pupils.

Figure 5: Use of non statutory assessments

Figure 5

provides a

breakdown of

the non

statutory

assessments

that principals

administered

within their

schools.


Over a third of principals administered the Spelling assessment

(38.5%, n = 70), the Developed Ability assessment (37.4%, n = 68), and the

Mental Arithmetic assessment (34.6%, n = 63). 28.6% of principals (n = 52)

also made use of the Attitude assessments.

22

57.7

42.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Irish Medium Assessments

Five principals from Irish Medium schools or Units responded to the evaluation.

These respondents indicated that they administered the following non statutory

assessments to their pupils: Spelling (n = 3); Mental Arithmetic (n = 2) and

Attitudes (n = 2).

Sharing InCAS outcomes with parents Figure 6: Do InCAS outcomes reflect pupils’ level of ability?

57.7% of

principals

(n=105) agreed

that the

outcomes from

InCAS reflected

their pupils’ level

of ability.


Of the 42.3% (n = 77) who disagreed, a small number of principals expressed

concerns regarding the reliability of InCAS outcomes. Such concerns are

summarised below:

discrepancies were apparent between InCAS outcomes and pupils’ level of

ability (38 comments);

year 4 InCAS outcomes were lower than pupils’ level of ability

(22 comments);

InCAS outcomes were inconsistent with other standardised assessments

(20 comments);

pupils scored higher than expected (16 comments); and

pupils scored lower than expected (12 comments).

23

71.4

28.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Two examples of such comments are provided below.

“There were occasions when the pupils’ scores did not compare with the

teacher’s own judgement or indeed other standardised assessments.”

“Some [pupils] achieved much higher than expected and some much

lower.”

Figure 7: Do you believe reporting outcomes to parents enhanced the

information shared at the parent meeting?

71.4% of

respondents

(n=130) felt that

reporting InCAS

outcomes

enhanced the

information

shared at the

parent meeting.


However, 28.6% of Principals (n = 52) disagreed. 25 of these principals

highlighted that parents became anxious whenever InCAS outcomes where

inconsistent with their child’s level of ability. One such respondent commented:

“For some parents the outcomes did not reflect the pupil’s capabilities or

work standards in class and so we felt they were misleading and hence

confusing for all concerned.”

In addition to reporting InCAS outcomes from this academic year, 85.2% of

principals (n = 155) provided parents with their child’s age equivalent

outcomes to InCAS from previous years. Five respondents felt that reporting

24

this information provided parents with a more detailed representation of pupils’

progress across time.

Figure 8: Charts and tables used to share InCAS outcomes with

parents

45.6

20.9 20.9

3.8

17.6 15.9

54.4

13.76.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Standard feedback table

Scores chart

Longitude chart

Difference table

Age comparison chart

Age standardised scores

Pupil progress chart

Pupil scores table

Developed own format


Figure 8 outlines the charts and tables that principals used to share InCAS

outcomes with parents.

Feedback demonstrates that the Pupil Progress Chart (54.4%, n = 99) and

Standard Feedback Table (45.6%, n = 83) were the most popular graphical

and tabular formats of sharing information with parents.

25

30.8

60.4

5.5 3.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very effective Effective Not effective Not at alleffective

Figure 9: How effective was your chosen format in sharing

information with parents?

91.2% of

respondents

(n=166) felt that

their chosen

format of sharing

information with

parents was

effective.


Principals were asked if they felt any other format of InCAS outcomes would

be useful for parents. Six respondents felt that a standardised score alongside

an age equivalent score would be useful for parents.

Use of InCAS outcomes by school management

The majority of principals took time to decide how to best use feedback from

InCAS within their school. This is evident from the large number of principals

who held staff meetings to:

discuss how InCAS outcomes would be shared with parents

(89.5%, n = 162); and

determine the use of InCAS feedback within their school (89%, n = 161).

A large number of principals indicated that they used the feedback from InCAS

to inform target setting (73.1%, n = 133) and their school’s development plan

(60.4%, n = 110).

26

79.7

20.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Despite a large number of principals indicating that outcomes were used at a

school management level, a small number of respondents (n = 15) questioned

the accuracy of InCAS outcomes. 10 respondents were of the opinion that data

from other standardised assessments is more reliable than outcomes from

InCAS. One such principal stated that:

“Parents and teachers need to feel more confident in the results before

using them for target setting.”

Future training

Over 60% of principals (62.6%, n = 114) indicated that classroom assistants

supported schools with the administration of InCAS.

Figure 10: Should classroom assistants be trained on InCAS?

79.7% of

principals

(n = 145) believed

classroom

assistants should

be trained in the

use of InCAS.


Seven respondents commented that training would be especially beneficial for

classroom assistants who support pupils with Special Educational Needs.

27

Additional comments

Principals were given the opportunity to make any additional comments at the

end of the questionnaire.

A small number of respondents took this opportunity to express concerns

regarding the accuracy of InCAS outcomes. Such comments are summarised

below:

query the accuracy of InCAS outcomes (10 comments);

other standardised assessments are more accurate than InCAS

(8 comments); and

InCAS data errors have led to a lack of confidence in the reliability of InCAS

outcomes (8 comments).

“Found feedback difficult to locate and understand and the difficulties with

program errors this year did not help.”

“I believe there to be great value [in] the use of InCAS but the severe

problems encountered this year will have to be overcome if their potential

value is to be realised.”

A small number of respondents (n = 10) commented that schools are

inadequately resourced for the administration of InCAS. Such respondents felt

that schools require more computers (n = 6) and further funded substitute

cover (n = 4).

28

Results collected from Teacher Questionnaires (Years 4-7)

This section presents the results for each question from the teacher

questionnaire. Results are based on completed questionnaires from 469

respondents. Percentages relate to the number of teachers who answered

each question, which may be fewer than the overall total of 469. A number of

teachers reiterated comments throughout their questionnaire; as a result some

comments are repeated throughout this section of the report.

Findings from this year’s teacher evaluation were compared to that collected

from the previous year. Feedback to each question within the teacher

questionnaire have also been analysed by teacher group (Years 4 to 7). Any

noteworthy differences have been highlighted throughout this section of the

report.

29

89.7

10.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Previous experience of InCAS Just over half of teachers (51.4%, n = 241) indicated that they had

administered InCAS to their pupils in the previous academic year (2008/09).

Figure 11: Confidence administering InCAS for the second year


The majority of teachers who have previously administered InCAS

(89.7%, n = 227) indicated that they felt more confident administering InCAS

for the second time. One such teacher stated that:

“Having completed the assessment last year I was familiar with the

system and how it operates. Also [teachers] were prepared for any

problems that would arise.”

30

CCEA support

Table 2: Use of CCEA support


Inky pupil demo 444 94.9 Training Manual 393 84.0 On-line materials on www.nicurriculum.org.uk

369

78.8

Resources folder 332 70.9 InCAS preparation letter 306 65.7 CCEA helpdesk 182 39.0

Table 2 above outlines the various forms of support respondents used in

preparation for InCAS. The majority of teachers used the Inky Pupil Demo

(94.9%, n = 444), Training Manual (84.0%, n = 393) and online materials

(78.8%, n = 369).

31

User issues (challenges and / or difficulties)

Over a third of respondents (37.1%, n = 174) indicated that they experienced

challenges administering the InCAS assessments.

Figure 12: Challenges experienced by teachers

Figure 12

illustrates the types

of challenges

teachers

experienced

administering

InCAS.


Figure 12 indicates that the largest percentage of respondents

(24.3%, n = 114) encountered technical difficulties administering InCAS. 78

teachers commented on the technical difficulties they experienced:

encountered problems logging pupils into InCAS (17 comments);

computers / laptops froze (17 comments);

sound failed to work (14 comments);

computers / laptops crashed (11 comments); and

unreliable internet access (10 comments).

The other main challenge highlighted by respondents was regarding

insufficient computer capacity within schools (19 comments).

24.3

5.512.4

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Technical failure Unfamiliarity withthe InCAS software

Other

32

27.9

64.6

7.10.4

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very confident Confident Not confident Not at allconfident

Figure 13: How was the problem resolved?

9.2 4.9 4.1

23

5.3 1.5 2.8

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Contacting CCEA helpdesk

Contacting C2K

Using the Training Manual

Yourself and/or colleagues

Using the online resources

Not yet resolved

Other


Respondents who experienced problems administering InCAS were asked to

indicate how their problem was resolved. The highest number of teachers

(23%, n = 108) indicated that any problems experienced were resolved either

by themselves or with the assistance of colleagues.

Figure 14: Confidence using the InCAS technology

92.5% of

respondents

(n = 209) who

experienced

challenges

indicated that they

feel confident

about using

InCAS in the

future.


33

29

64.4

5.80.9

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very user-friendly

User-friendly Not user-friendly

Not at all user-friendly

Completing the InCAS assessments

Figure 15: How user-friendly was InCAS for pupils?

The majority of

teachers

(93.4%, n = 438)

believed their

pupils found the

InCAS

assessments

user-friendly.


A small number of teachers (n = 29) stated that pupils did not encounter any

difficulties completing InCAS.

“[Pupils] coped well with the whole process.”

Despite this positive feedback, 14 teachers felt that pupils with limited

experience of computers had difficulty with InCAS.

34

Nearly half of respondents (49.3%, n = 231) administered the non statutory

InCAS assessments to their pupils.

Figure 16: Use of non statutory assessments

Figure 16

provides a

breakdown of the

non statutory

assessments

teachers

administered to

pupils.


35.6% of teachers (n = 167) administered the Spelling assessment to their

pupils, 34.8% of respondents (n = 163) administered the Mental Arithmetic

assessment, 29.6% of respondents (n = 139) administered Developed Ability

and 21.5% of teachers (n = 101) administered the Attitudes assessment.

Six teachers from Irish Medium schools or Units responded to the evaluation.

These respondents administered the Spelling (n = 2) and Mental Arithmetic

assessments (n = 2) to their pupils.

21.529.6

34.8 35.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Attitudes DevelopedAbility

MentalArithmetic

Spelling

35

31.2

58.1

91.7

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very useful Useful Not useful Not at all useful

Figure 17: Usefulness of outcomes obtained from non statutory

assessments

89.3% of teachers

(n = 209) found the

outcomes obtained

from non statutory

assessments to be

useful.


A small number of teachers specified how they used the outcomes from the

non statutory assessments. 45 teachers used the outcomes to inform target

setting. 33 respondents indicated that they reported the outcomes from the

non statutory assessments to parents. Teachers also used the outcomes for

comparative purposes (n = 23). Specifically, respondents compared the

outcomes from the Developed Ability assessment with Reading and General

Maths outcomes (n = 4) and with pupils’ IQ scores (n = 3).

However 11 respondents expressed concerns regarding the reliability of

outcomes from the non statutory assessments.

“There was some discrepancy between expected performance and

achieved performance.”

36

18.6

62.7

16

2.8

0

20

40

60

80

100

% o

f R

esp

on

den

ts


Interpreting InCAS outcomes

Figure 18: Teachers confidence accessing InCAS outcomes

81.3% of teachers

(n = 381) indicated

that they were

confident accessing

InCAS outcomes.


Respondents were asked to provide reasons for their answer. 23 teachers

commented that they were confident accessing InCAS outcomes as they had

the support of their colleagues.

17 respondents felt that accessing InCAS outcomes was straightforward, and

14 teachers stated that they were familiar accessing outcomes from previous

years.

Closer analysis of results by year group indicates that Year 4 (76.7%, n = 89)

and Year 5 (70.9%, n = 56) teachers expressed the lowest levels of confidence

accessing InCAS outcomes. This academic year (2009/10) was the first year

that Year 4 teachers were required to administer the InCAS assessments and

may explain their lower levels of confidence in accessing outcomes. However,

this is the third year that Year 5 teachers have administered InCAS and

accessed outcomes from this assessment. Year 5 teachers did not provide

specific reasons for their lower levels of confidence.

37

19.8

67.6

11.1

1.5

0

20

40

60

80

100%

of

Res

po

nd

ents


7.9

66

21.6

4.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Veryconsistent

Consistent Not consistent Not at allconsistent

Figure 19: Teachers confidence interpreting InCAS outcomes?

87.4% of teachers

(n = 410) indicated

that they were

confident

interpreting InCAS

outcomes.


A small number of respondents (n = 25) felt that the graphs and tables were

easy to interpret.

Nevertheless, 19 teachers commented that they had difficulties interpreting

outcomes whenever pupils received a * or # symbol. Teachers felt they

required clarification on how to interpret these scores.

Figure 20: Consistency of InCAS outcomes with teachers’ professional

judgement

73.9% of

teachers

(n = 345) felt

that InCAS

outcomes were

consistent with

their

professional

judgement.


38

Such findings represent nearly a 20% drop from the previous year, as 93.5%

of teachers (n = 314) deemed InCAS outcomes to be consistent with their

professional judgement in the 2008/09 autumn term.

This year a large number of respondents (n = 66) noted discrepancies

between pupils’ outcomes and the teacher’s judgement of their level of ability.

Two such teachers commented:

“[Pupils’] results contradicted their performance in class. Very

demoralising for teachers, parents and pupils; and

“Whilst some feedback was consistent, there were more inconsistencies

than not”.

40 respondents felt that pupils received higher outcomes than expected. In

contrast, 34 teachers commented that pupils obtained lower outcomes than

expected. A small number of respondents (n = 34) also believed that InCAS

outcomes were inconsistent with feedback obtained from other standardised

assessments.

Use of InCAS outcomes

80% of respondents felt that InCAS outcomes were helpful in informing the

learning and teaching of individual pupils (80.6%, n = 378) as well as overall

classes (79.3%, n = 372). 36 respondents commented that outcomes

informed target setting, particularly highlighting areas for improvement for

individual pupils (n = 24).

However, 15 teachers expressed lack of confidence in the reliability of InCAS

outcomes. Consequently teachers noted that they did not find the feedback

helpful in informing learning and teaching.

39

Figure 21: Charts and tables used by teachers to share InCAS

outcomes with parents

49.7

23.519.8

8.3

31.823.9

45

20

5.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Standard feedback table

Scores chart

Longitude chart

Difference table

Age comparison chart

Age standardised scores

Pupil progress chart

Pupil scores table

Developed own form

at


Respondents were asked to indicate which charts and tables they used to

share InCAS outcomes with parents. It should be noted that respondents were

given the option of selecting more than one chart or table.

Figure 21 demonstrates that the Standard Feedback Table (49.7%, n = 233)

and Pupil Progress Chart (45%, n = 211) were the most popular formats for

sharing InCAS outcomes with parents.

Just over 63% of respondents indicated that they drew year on year

comparisons on the InCAS outcomes at an individual pupil level (63.2%, n =

295). 15 respondents commented that they used the outcomes to track pupil

progress. A large number of Year 4 and 5 teachers (n = 58) were unable to

draw comparisons at an individual pupil level as this was the first year that their

pupils completed InCAS.

A smaller number of teachers drew comparisons on the InCAS outcomes at

class level (43.4%, n = 202).

40

16.2

63.3

16.2

4.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts


Figure 22: Teachers confidence in reporting outcomes to parents


that 79.5% of teachers

(n = 373) were

confident reporting

assessment outcomes

to parents. This

represents a 12% dip

in confidence levels

from the previous year

(91.6%, n = 308).


Data analysis of this year’s teacher feedback indicates that Year 5 teachers

had the lowest levels of confidence. Just over two thirds of Year 5 teachers

(69.6%, n = 55) indicated that they were confident reporting assessment

outcomes to parents.

A sizeable number of teachers (n = 56) across all year groups stated that they

were not confident reporting InCAS outcomes as they felt they were

inconsistent with pupils’ level of ability.

16 teachers questioned the reliability of InCAS outcomes and consequently

were not comfortable reporting outcomes to parents. One such teacher stated

that:

“As we felt the scores were inaccurate we were not confident

reporting them”; and

“The InCAS data was inaccurate. We informed parents that we have

to report the data but we have no faith in it”.

Ten respondents also stated that they were less confident reporting outcomes

due to the InCAS data errors.

41

67.8

32.2

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Figure 23: Did the InCAS outcomes enhance the information shared at

the parent meeting?

67.8% of

respondents

(n = 318) felt the

InCAS outcomes

enhanced the

information

shared at the

parent meeting.


23 teachers commented that the InCAS outcomes provided hard evidence that

supported pupil progress.

However, a small number of teachers (n = 26) reiterated the concern that there

were disparities in InCAS outcomes and pupils’ level of ability. 15 respondents

noted that such inconsistencies caused concerns for parents at the parent

meeting:

“[Inconsistencies] seemed to confuse parents and led them to worry

about their child.”

Over 60% of respondents (61.9%, n = 289) agreed that the InCAS outcomes

helped teachers inform parents of how to support their child’s learning at

home.

Whilst 43 teachers believed that the InCAS outcomes raised parents’

awareness of areas to be supported at home, 11 respondents felt it was not

specific enough to inform parents of how to support their child’s learning.

42

Figure 24: Are age equivalent outcomes meaningful to parents?

81.6% of

teachers

(n = 381) felt that

the age

equivalent

outcomes from

InCAS were

meaningful to

parents.


Despite this positive feedback, a small number of respondents (n = 36)

commented that age equivalent outcomes are not meaningful for parents as

they were inconsistent with their child’s level of ability.

19 respondents reiterated the anxiety caused to parents whose child obtained

a low age equivalent outcome. One such teacher commented:

“It is meaningful but it can be demoralising for some parents / pupils

especially where [pupils] have difficulties but are making good

progress.”

Two thirds of teachers (67%, n = 313) felt that the graphical representation of

the outcomes were meaningful to parents. Closer analysis of teacher feedback

revealed that slightly less Year 4 (56%, n = 65) and Year 5 (58.4%, n = 45)

teachers felt the graphical representation was meaningful.

81.6

18.4

0

20

40

60

80

100%

of

Res

po

nd

ents

Yes No

43

78.7

21.3

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Across all year groups, teacher comments were contradictory. While 21

respondents felt the graphical representation provided a clear depiction of

pupil progress, other respondents (n = 21) felt that graphs required a teacher

explanation. A further 20 teachers believed graphical illustrations of InCAS

data were difficult for parents to understand.

Using InCAS with pupils with Special Educational Needs (SEN)

Figure 25: Did you administer InCAS to any pupils on the SEN register?

78.7% of

respondents

(n = 369)

administered

InCAS to pupils

on the SEN

register.


68% of such teachers (n = 247) also indicated that they administered InCAS to

pupils with a statement of special educational need.

44

10.9

58.5

23

7.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Verymanageable

Manageable Notmanageable

Not at allmanageable

Table 3: Category(s) of special educational need


Cognitive and Learning 184 49.9 Social, Emotional and Behavioural 107 29.0 Medical conditions / syndromes 71 19.2 Communication and Interaction 58 15.7 Physical 22 6.0 Sensory 16 4.3 Other 6 1.6

Table 3 provides a breakdown of the categories of special educational need

teachers administered InCAS to. Nearly 50% of teachers (49.9%, n = 184)

indicated that they administered InCAS to pupils with cognitive and learning

difficulties.

Figure 26: How manageable did your pupils with a special educational

need find the assessments?

69.4% of teachers

(n = 248) believed

that pupils with

special

educational needs

found the InCAS

assessments

manageable.


45

However, almost a third of teachers (30.6%, n = 109) believed that the InCAS

assessments were not manageable for pupils with special educational needs.

Teachers commented that such pupils:

randomly clicked on answers (25 comments);

had poor concentration (18 comments);

found it difficult to understand the task involved (14 comments); and

sought assistance while completing InCAS (14 comments).

Table 4: Number of teachers who deem InCAS manageable for pupils

with special educational needs

Manageability Please indicate which year group you teach: Number % Year 4 50 58.1 Year 5 45 67.2 Year 6 39 69.6 Year 7 64 92.8 Composite classes 50 63.3

Table 4 above suggests that familiarity with InCAS may have had an impact on

the manageability of the assessments for pupils with special educational

needs. Feedback from teachers reveals that pupils find InCAS more

manageable as they progress up the school.

Table 4 demonstrates that Year 4 pupils found InCAS to be less manageable

(58.1%, n = 50) than pupils in other year groups. A much higher percentage of

Year 7 teachers (92.8%, n = 64) indicated that their pupils found InCAS

manageable. Three Year 7 teachers commented that their pupils were familiar

with InCAS from previous years, which made the assessments more

manageable for these pupils.

Over a third of respondents (34.1%, n = 124) agreed that pupils encountered

difficulties with particular InCAS modules. A large number of teachers

commented that pupils had particular difficulties with the Reading Module

46

(50 comments). 20 of such teachers stated the Comprehension module was

particularly difficult for pupils with special educational needs.

57 teachers who have previously administered InCAS highlighted that they

recognised developments made to the assessment from last year. Such

teachers believed the changes have made InCAS more manageable for pupils

with special educational needs.

Just under half of the respondents (46.7%, n = 168) indicated that pupils with

special educational needs who completed InCAS have a classroom assistant

specifically assigned to support them. 71.7% of teachers (n = 170) felt that

classroom assistants should be trained in the use of InCAS.

Figure 27: Do you require any additional support in order to assist pupils

with a special educational need in the use of InCAS?

A small

percentage of

respondents

(24%, n = 81)

agreed that they

require further

support to assist

pupils with a SEN

in the use of

InCAS.


Eight teachers felt that further guidance is required on administering InCAS to

pupils with special educational needs.

24

76

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

47

Additional Comments

Teachers provided additional comments at the end of the questionnaire.

A number of respondents expressed concerns regarding the accuracy of

InCAS outcomes. 26 teachers commented that the data errors that occurred

this year have reduced their confidence with the outcomes from this

assessment. Respondents (n = 10) also commented that data errors caused

extra work for schools to rectify. One teacher responded:

“The fact that there were errors in the calculation of the Maths

results was most inconvenient and time consuming.”

Teachers (n = 23) were concerned that InCAS outcomes are inaccurate and

that feedback is inconsistent with pupils’ ability levels (n = 18).

Teachers (n = 10) also expressed confusion regarding the number of pupils

who received a # or * symbol. Teachers stated that clarification is required on

such symbols.

12 teachers took the opportunity at the end of the questionnaire to highlight

their support of InCAS. One such respondent commented:

“InCAS is a useful diagnostic tool for confirming professional

judgement [and] arranging groups within class.”

Eight respondents also stated that InCAS is a user-friendly assessment, while

seven teachers commented on their pupils’ enjoyment of InCAS.

48

Results collected from mainstream Parent Questionnaires

This section presents the results for each question from the mainstream parent

questionnaire. Results are based on completed questionnaires from 15,838

respondents. Percentages relate to the number of parents who answered each

question, which may be fewer than the overall total of 15,838. A number of

parents reiterated comments throughout their questionnaire; as a result some


Findings from this year’s parent evaluation were compared to that collected

from the previous year. Feedback to each question within the parent

questionnaire has also been analysed by the year group in which the parents’

child is in (Years 4 to 7). Any noteworthy differences have been highlighted

throughout this section of the report.

49

Background Information

Figure 28: Breakdown of respondents by year group


that parental

feedback was

collected evenly

across the four year

groups.


The majority of parents indicated that they have access to a computer

(94.8%, n = 14,962) and the internet (95.8%, n = 14,592).

98% of respondents stated that English was their child’s first language

(98%, n = 15,491). Of the 2% who indicated that English was not their child’s

first language, Irish (63 comments) and Polish (52 comments) were the most

common additional languages recorded.

Just over 7% of parents indicated that their child has a special educational

need (7.7%, n = 1,209). The most common special educational needs

identified by parents included, Dyslexia, (n = 338) Attention-Deficit

Hyperactivity Disorder (n = 92), Autism (n = 78) and Asperger’s Syndrome

(n = 74).

50

Awareness of InCAS

Figure 29: Did your child sit the InCAS assessments during the autumn

term?

Figure 29 shows the

percentage of parents

who were aware that their

child sat the InCAS

assessment in the autumn

term.


Over 96% of parents indicated that their child sat the InCAS assessment in the

autumn term (96.3%, n = 15,202). 459 parents commented that they were

made aware of this at a parents meeting, while 426 parents indicated that their

child told them this information. However, 332 respondents stated that they

were not made aware until after their child had completed InCAS.

Figure 30: Are you aware of the purpose of InCAS?

Figure 30 illustrates that the

majority of parents were

aware of the purpose of the

InCAS assessments

(87.6%, n = 13,763).


51

Such parents stated that:

their child’s teacher/school informed them of the purpose of InCAS

(534 comments);

they were already aware of the purpose of InCAS (273 comments); and

they received an information leaflet on the purpose of InCAS

(162 comments).

Despite such positive feedback, a large number of parents stated they were

unaware of the purpose of InCAS (364 comments). 238 respondents felt that

there was not enough information given to parents. One such parent

commented that there was, “little to no communication from the school as to its

contents, its purpose etc.”

Figure 31: Did you receive a CCEA "Information for Parents" leaflet

about InCAS?

Figure 31 illustrates the

percentage of parents who

received a CCEA

“Information for Parents”

leaflet about InCAS.


Figure 31 indicates that the majority of respondents received a CCEA

“Information for Parents”, leaflet about InCAS (87.7%, n = 13,366). Of those

respondents who received the leaflet, 323 commented that it was informative

and clearly explained the InCAS assessment.

52

InCAS Feedback

Figure 32: Did you attend a parent meeting about your child in the

autumn term?

Figure 32 displays the

percentage of

respondents who

attended a parent

meeting about their

child in the autumn

term.


Over 97% of respondents attended a parent meeting during the autumn term

(97.2%, n = 15,330).

Figure 33: Did you receive your child's Reading and General Maths age

equivalent outcomes in writing?


that the majority of

parents received their

child’s Reading and

General Maths age

equivalent outcomes in

writing

(86.5%, n = 13,596)


82.4% of respondents also indicated that they received a chart presenting

these outcomes (n=12,891).

53

Of those parents who received the outcomes in writing or a chart, 91.5% found

the information easy to understand (n = 13,749). 281 parents commented that

the information was explained well to them and 269 stated that the information

was straightforward to understand.

Despite such positive feedback, 205 parents indicated that they found the

information difficult to understand and 213 respondents felt the information

was too basic and that they required a further breakdown. One such parent

commented that there was, “very little information – not very helpful”.

Figure 34: Did the InCAS outcomes reflect your child’s

level of ability?


percentage of

respondents who felt

their child’s InCAS

outcomes reflected their

level of ability.


Over 74% of respondents believed their child’s InCAS outcomes reflected their

level of ability (74.2%, n = 11,146). 177 parents stated that the outcomes

highlighted their child’s strengths and weaknesses accurately.

25.8% of respondents (n = 3,882) did not believe their child’s InCAS outcomes

reflected their level of ability. Closer analysis of such findings indicates that a

higher proportion of parents with a child in Year 4 (37.3%) did not feel InCAS

was reflective of their child’s level of ability. Parents of Year 4 pupils

commented that they felt the InCAS outcomes were too low and consequently

did not reflect the ability of their child (380 comments). 146 respondents felt

54

the InCAS outcomes did not reflect the teacher’s judgement. Two parents

stated:

“As discussed and agreed with by our child’s teacher, the test results

did not reflect his reading ability”, and

“My child’s teacher and I were appalled at these results, we both

agreed that these did not give an accurate account of my child’s

ability at all. Very disappointed”.

Over 82% of respondents felt the InCAS outcomes helped to inform them of

their child’s: strengths (85.3%, n = 13133); and areas for development

(82.3%, n = 12,556). 235 parents commented that the InCAS outcomes

informed them of what to concentrate on at home.

Respondents who did not feel the InCAS outcomes informed them of their

child’s strengths and areas for development stated that:

the outcomes were inaccurate (435 comments);

the outcomes were not specific enough (318 comments);

outcomes did not inform them of anything new (314 comments); and

they preferred the teacher’s assessment of class work (216 comments).

Figure 35: Based on the outcomes from the InCAS assessments and

discussion with the teacher, have you undertaken any activities

to support your child’s learning?

Figure 35 illustrates the


who undertook activities

to support their child’s

learning at home.


55

Having received their child’s outcomes from the InCAS assessments, just over

half of the respondents indicated that they have undertaken activities to

support their child’s learning (55%, n = 8,354). Types of activities that parents

have undertaken include:

additional reading (2,382 comments);

extra maths (1,567 comments);

completing more work at home (363 comments); and

arranging for their child to work with a tutor (320 comments).

45% of respondents (n = 6,845) did not undertake any activities to support

their child’s learning. A large number of such parents indicated that they have

not undertaken any additional activities with their child having received the

InCAS outcomes as:

they were already supporting learning at home (739 comments);

no areas for development were highlighted (299 comments);

they deem teacher advice and/or other assessments more reliable than

InCAS (223 comments); and

they did not receive guidance on how to support their child’s learning

(130 comments).

56

Figure 36: Have the outcomes from the InCAS assessments enhanced

the information shared at the parent meeting?

Figure 36 shows the

percentage of

respondents who felt

the outcomes from the

InCAS assessments

enhanced the

information shared at

the parent meeting.


The majority of respondents felt the InCAS outcomes enhanced the

information shared at the parent meeting (71.5%, n = 10,804). Such parents

commented that the InCAS assessment provided:

a clear way to inform parents of their child’s ability (247 comments);

an objective source of information (165 comments);

a focused discussion at the parent teacher meeting (150 comments); and

highlighted information on the child’s strengths and weaknesses (139

comments).

28.5% of respondents (n = 4,297) did not feel the outcomes from the InCAS

assessments enhanced the information shared at the parent meeting. A closer

study of the findings revealed that a higher proportion of parents with a child in

Year 4 (34.7%, n = 1,323) did not feel the results from the InCAS assessments

enhanced the information shared at the parent meeting. Such respondents felt

that the InCAS assessments provided no additional information (86 comments)

and that they preferred the teacher’s assessment of their child (85 comments).

65 respondents commented that they felt the InCAS outcomes were inaccurate

or unreliable. One such respondent commented that they “don’t think it gives a

true reflection on the child’s ability.”

57

Figure 37: Are you aware if your child encountered any difficulties

completing InCAS?



who were aware their

child encountered

difficulties completing

InCAS.


Nearly 24% of respondents stated that their child encountered difficulties

completing InCAS (23.8%, n = 3667). Parents indicated that their child

encountered the following problems:

ran out of time (227 comments);

left out / skipped questions (159 comment);

did not understand what was expected of them as they had no practice

(120 comments); and

found the General Maths assessment difficult (119 comments).

58

Additional Comments

Parents were presented with the opportunity at the end of the questionnaire to

provide any additional comments they had on the InCAS assessments.

A large number of respondents had concerns regarding the InCAS

assessment and the outcomes it produces. Such parents highlighted that they

felt:

the outcomes were inaccurate/unreliable (963 comments);

parents were not told enough about the InCAS assessment process

(441 comments);

InCAS is unnecessary – it is a waste of money/time (244 comments);

teacher’s professional judgement and their ongoing assessment of class

work is more valuable (212 comments);

InCAS relies too heavily on ICT skills (206 comments); and

the outcomes from InCAS are not detailed enough to aid parents in

supporting their child’s learning (188 comments).

A sizeable number of parents also used this opportunity to voice their support

on InCAS. Such respondents commented that the InCAS assessment was a

useful tool (174 comments). One parent stated InCAS is “a good tool to

identify a child’s development and more importantly to identify any deficits that

need to be worked on.”

Other comments from parents are summarised below.

the results highlight strengths and weaknesses (117 comments);

InCAS should replace the transfer test (110 comments);

InCAS is a good indicator of ability (109 comments); and

it clearly shows the child’s progress (64 comments).

59

Results collected from mainstream Pupil Questionnaires

This section presents the results for each question from the mainstream pupil

questionnaire. Results are based on completed questionnaires from 11,117

respondents. Percentages relate to the number of pupils who answered each

question, which may be fewer than the overall total of 11,117. A number of

pupils reiterated comments throughout their questionnaire; as a result some


Findings from this year’s pupil evaluation were compared to that collected from

the previous year. Feedback to each question within the pupil questionnaire

have also been analysed by the year group pupils were in (Years 4 to 7). Any

noteworthy differences have been highlighted throughout this section of the

report.

60

18.1 23.2 26.232.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Year 4 Year 5 Year 6 Year 7

48.4 51.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Background Information

Figure 38: Breakdown of respondents by the year group


that pupils who

responded to the

questionnaire were

split fairly evenly

across Years 4 to 7.


The majority of pupils indicated that they have access to a computer at home

(92.5%, n = 10,281).

Awareness of InCAS

Figure 39: Do you know why you did InCAS?

Figure 39 shows

the percentage

of respondents

who were aware

of why they

completed

InCAS.


61

20.9

55.5

21

2.6

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Very Easy Easy Not Easy Not at all Easy

Just over half of respondents (51.6%, n = 5,731) indicated that they were not

sure why they completed InCAS. Of those pupils who were aware of why they

completed InCAS (48.4%, n = 5,386), the majority felt it was for teachers to

see what level their work was at (n = 2244) and to assist with their learning (n

= 947). Other respondents felt that InCAS feedback identifies areas in which

pupils require assistance (n = 252) and provides parents with information on

how their child is performing (n = 66). A smaller number of pupils believed the

assessments provided information to post-primary schools (n = 66).

Figure 40: Ease of completing InCAS Reading and General Maths?

Figure 40 shows


pupils

(76.4%, n = 8,484)

found the InCAS

Reading and

General Maths

assessments easy

to use.


Pupils were asked to comment on their response. A large number of

respondents (n = 502) indicated that topics explored in both assessments were

covered in class. Pupils commented that they found the General Maths

(n = 211) easy and a large proportion of these respondents (n = 160) stated

that Maths is one of their strongest subjects. Other respondents (n = 204)

found the Reading assessments easy and explained that they are skilled in

this subject (n = 133). Pupils indicated that they found Inky helpful for

preparing them for the assessments (n = 234) and found the instructions very

clear (n = 172). A sizeable number of pupils (n=203) said they possess good

ICT skills and have a personal computer at home (n = 37).

62

18.8

81.2

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Despite such positive feedback, a large number of respondents commented

that they found the General Maths (n = 360) and Reading (n = 240)

assessments difficult. A small number of these pupils felt that Maths (n = 36)

and English/Reading (n = 24) were not their strongest subjects. A sizeable

number of respondents (n = 183) indicated that some of the questions in the

assessments had not yet been covered in class. 124 pupils believed that they

did not have enough time to complete the assessments.

Figure 41: Did you have any problems completing InCAS?

Figure 41 displays

the percentage of

pupils who

encountered

problems

completing

InCAS.


The majority of respondents (81.2%, n = 9,024) indicated that they did not

encounter any problems completing InCAS. However, 18.8% (n = 2,091) of

respondents experienced problems completing InCAS. A sizeable number of

such pupils stated that they found the General Maths (n = 375) and Reading

(n = 307) assessments difficult. Pupils also complained that they had

insufficient time (n = 268) to complete both assessments. A smaller number of

respondents commented that the assessments covered areas not yet taught in

class (n = 27), disliked the voice used throughout InCAS (n = 20) and

complained that their test froze (n = 15).

63

31.2

68.8

0

20

40

60

80

100%

of

Res

po

nd

ents

Yes No

Figure 42: Do you think the InCAS assessments could be better?


the percentage of

pupils who felt the

InCAS assessments

could be better.


Over two thirds of pupils (68.8%, n = 7,645) did not think the InCAS

assessments could be better. Nevertheless, 31.2% (n = 3,465) of respondents

disagreed.

A large number of such pupils (n = 451) requested additional time to complete

both assessments. Respondents also recommended that a greater number of

questions should be included within assessments (n = 127). 87 respondents

requested a different audio voice to be used. 59 pupils recommended a more

difficult General Maths assessment whilst a similar number of respondents

took a different viewpoint and sought less difficult General Maths (n = 53) and

Reading (n = 50) questions.

A number of pupils took this opportunity to comment on the layout of InCAS

and recommended brighter colours (n = 79), bigger text throughout the

assessments (n = 52) and more pictures (n = 49). Other pupils (n = 40)

requested more subjects to be introduced to the InCAS software. 21

respondents would have liked to complete more practice questions before

each assessment and a further 21 requested a facility to change the volume

during the test.

64

84.5

15.5

0

20

40

60

80

100

% o

f R

esp

on

den

ts

Yes No

Figure 43: Did Inky show you how to do InCAS?



respondents

(84.5%, n = 9,390)

agreed that Inky

showed them how

to do InCAS.


The majority of pupils also deemed Inky to be helpful (88.5%, n = 8,272). A

sizeable number of pupils commented that Inky helped them learn (n = 195),

had a clear voice (n = 102) and said Inky was fun (n = 41).

Despite such positive feedback, a number of respondents indicated that they

did not like Inky (n = 182) with a small number stating they disliked the voice

(n = 10). Close analysis of pupil feedback across the year groups revealed

that the highest proportion of respondents who felt that Inky did not show them

how to complete InCAS were Year 6 (15.6%, n = 453) and Year 7 pupils

(21.8%, n = 786).

65

Additional Comments

Pupils had an opportunity at the end of the questionnaire to provide any

additional comments they had on the InCAS assessments.

A large number of respondents commented that they found completing InCAS

fun (n = 1,048), believed it assisted their learning (n = 306) and would like the

opportunity to complete the assessments again (n = 142). A large number of

pupils also indicated that they found Inky helpful (n = 236).

Despite such positive feedback pupils also highlighted a number of concerns.

Some pupils found the Reading (n = 97) and General Maths (n = 58)

assessments difficult and recommended making them easier. 72 respondents

took an opposing view and recommended that the assessments are made

more difficult. 24 respondents felt that certain parts of the assessments

covered areas they had not yet been taught in school.

Pupils also took the opportunity to make a number of recommendations.

Respondents requested more time to complete the assessments (n = 213) and

indicated they would like brighter colours (n = 96) and more pictures (n = 34).

66

5.0 Conclusion

This evaluation reveals that the errors in the InCAS outcomes during the

2009/10 autumn term have reduced stakeholders’ confidence with the

reliability of this assessment. Feedback from respondents indicated that

respondents particularly queried the reliability of outcomes for Year 4 pupils,

with many commenting that scores were lower than pupils’ actual level of

ability. It is imperative that all educational bodies (The Department of

Education, CCEA, and CEM) provide reassurance to schools by outlining the

procedures taken to ensure the reliability of InCAS in the future.

Feedback from this evaluation also indicates that schools encountered

challenges administrating InCAS. However, only a small number of

respondents specified challenges they encountered. The manner in which

InCAS is administered varies greatly between schools, depending upon their

individual setting. Nevertheless, schools may benefit from guidance outlining

the ways in which schools can best manage the administration of InCAS.

Furthermore, a review of ICT provision should be carried out to identify schools

in need of additional ICT resources. Feedback from principals and teachers

also suggests that training classroom assistants on the administration of

InCAS would further support schools.

Over 50% of pupils (51.6%, n = 5,731) indicated that they were not sure why

they completed InCAS. In keeping with the aim of the “Big Picture” of the

Northern Ireland Curriculum at Key Stages 1 & 23 which “aims to empower

young people to achieve their potential and to make informed decisions

throughout their lives” pupils should be aware of why they are completing this

assessment. Furthermore, if pupils’ outcomes from InCAS are explained in a

way which is meaningful to them, they should have an understanding of their

strengths and weaknesses. Such information should enable them, in

collaboration with their teacher or parent, to formulate achievable targets for

themselves for the year ahead.

3 http://www.nicurriculum.org.uk/docs/key_stages_1_and_2/Big-PicturePrimary-KS12.pdf

67

Despite the areas for improvement highlighted above, this evaluation has

indicated that schools have generally found InCAS manageable to deliver;

pupils have found this assessment user-friendly and InCAS outcomes are

useful to schools and parents. Moreover, principals and teachers rated the

support provided by CCEA very highly. CCEA should continue to offer such

support to ensure schools are adequately supported and empowered to

manage these new assessment arrangements.

68

6.0 Recommendations

Prior to the administration of InCAS in the 2010/11 autumn term there are a

number of issues documented within this report that should be addressed with

a matter of urgency.

This evaluation suggests that CCEA should:

in partnership with other educational bodies put a communication strategy

in place at the start of the autumn term to rebuild confidence throughout the

primary sector on the reliability of InCAS;

continue to provide schools with guidance on the ways in which schools

can best manage the administration of InCAS;

continue to empower schools to embrace InCAS and develop expertise and

confidence in administering, analysing and making use of the outcomes

produced from InCAS;

continue to work in partnership with C2K and CEM to prevent technical

difficulties occurring during the administration of InCAS;

provide classroom assistants with training on the administration of InCAS;

liaise with schools to ensure that the purpose of the InCAS assessments is

clear to pupils;

encourage teachers to share outcomes from the InCAS assessments with

pupils and set targets with them for the year ahead;

advise the Department of Education to carry out a review of ICT resources

to identity schools with inadequate ICT provision for the administration of

InCAS;

in partnership with the Department of Education, continue to communicate

with parents on the InCAS assessments; and

provide further advice to teachers on the meaning of the * or # symbols

allocated to some pupils’ outcomes.

evaluation report on the implementation of the incas ... report on the... · scope of the report...

Documents