central examination board report august 2015

7/23/2019 Central Examination Board Report August 2015

1/53

1

THE BAR STANDARDS BOARD

CENTRAL EXAMINATIONS BOARD

CHAIRS REPORTAUGUST 2015

First Sit 2014/15


2/53

2

Executive summary

The Central Examination Board (CEB) has now completed its fourth cycle ofoverseeing first sit assessments in the three knowledge areas of the Bar ProfessionalTraining Course (BPTC).

The confirmed post-scale outcomes of the 2014/2015 first sit centralised assessmentsfollowing review of cohort performance by the CEB are as follows:

2015 FirstSit *

2014 FirstSit *

2013 FirstSit *

2012 FirstSit *

Change2014 to

2015

ProfessionalEthics

Number ofCandidates 1572 1649 1722 1591 -77

Passing MCQ 91.5% 81.0% 94.3% 92.6% 10.5%

Passing SAQ 58.0% 65.6% 89.5% 88.5% -7.6%

Passing Overall 56.7% 59.6% 86.4% 84.9% -2.9%

CriminalLitigation,Evidence andSentencing

Number ofCandidates 1483 1586 1719 1569 -103

Passing MCQ 83.3% 84.1% 88.9% 88.7% -0.8%

Passing SAQ 64.2% 78.2% 69.9% 77.8% -14.0%

Passing Overall 62.5% 72.8% 68.2% 74.7% -10.3%

Civil Litigation,Evidence andSentencing

Number of

Candidates1595 1663 1768 1568 -68

Passing MCQ 71.3% 68.6% 73.2% 83.7% 2.7%

Passing SAQ 65.0% 67.8% 61.5% 73.5% -2.8%

Passing Overall 58.0% 57.4% 56.2% 68.0% 0.6%

(*Although the first sit assessment, a number of candidates will have undertaken theassessments on a deferred or referred basis.)

The first sit all-Provider post scale pass rates for 2014/15 present a mixed picturewhen compared with the previous year. Professional Ethics MCQ all-Provider pass

rates and Civil Litigation MCQ all-Provider pass rates are up compared to 2013/14,


3/53

3

but all other MCQ and SAQ all-Provider pass rates are down. In terms of combinedpass rates (MCQ and SAQ) for the three knowledge areas, all-Provider pass rates arelargely unchanged on the previous year in Professional Ethics and Civil Litigation, butare noticeably lower in respect of Criminal Litigation. A more detailed analysis iscontained in the main report.

A number of general points should be borne in mind when considering the results data:

Knowledge area assessment pass rates in the above table indicate theaggregated percentage of candidates passing an assessment across allProvider institutions.

All data set out in the body of the report has been presented so as to preservethe anonymity of Provider institutions.

There is no pre-scale data for 2010/11 as this was pre-CEB and all knowledgearea assessments were set and marked by individual Provider institutions.

Where Provider trend analysis data is used in this Report it will only includeProviders who have had cohorts across all four assessment cycles. Thisimpacts on trend analysis data for Kaplan (last full intake in 2013/14) and BPPManchester where the BPTC was offered for the first time in 2013/14. Wheresuch data is excluded the commentary on the data table will note this.

1. Context

1.1 The 2010/11 academic year saw the first round of assessments under the BPTCregime (replacing the BVC) in the wake of the Wood Report (July 2008). For 2010/11all Providers were required to assess candidates in Professional Ethics, Civil Litigation,

Remedies & Evidence, and Criminal Litigation, Evidence & Sentencing (often referredto as the knowledge areas) by means of MCQs and SAQs. Together these threemodules represent 25% of the BPTC (i.e. 30 credits out of 120). For 2010/11 theknowledge area assessments were set and marked by the Providers. Centralisingthese assessments was a key recommendation of the Wood Report, and the CentralExaminations Board (CEB)was established to oversee this change on behalf of theBar Standards Board (BSB). 2011/12 was the first year of operation for the system ofcentralised examinations for the knowledge areas on the BPTC. No changes weremade to the format of assessment, but the setting of the assessments was undertakenindependently of the Providers by a team of CEB examiners appointed by the BSB.

1.2 In each knowledge area candidates are required to attempt a multiple choicequestion (MCQ) test, and a short answer question (SAQ) test. In the Civil andCriminal papers the MCQ comprises 40 questions and the SAQ 5 questions. InProfessional Ethics the MCQ comprises 20 questions and the SAQ 3 questions. Allquestions are compulsory and the pass mark in each paper is 60%. The marks for theMCQ and SAQ papers are combined to provide a candidate with an overall mark ineach of the knowledge areas. Candidates have to achieve the pass mark of 60% inboth elements of assessment, there being no scope for the aggregation of marksbelow 60% between MCQ and SAQ scores to achieve the minimum 60% pass markoverall.


4/53

4

1.3 The assessment process is overseen by the CEB whose members are appointedby the BSB. The CEB comprises a Chair, teams of examiners (a Chief Examiner andtwo Assistant Examiners for each knowledge area), an independent observer, anindependent psychometrician and senior staff from the BSB. The Chair and theexaminers contribute a mix of both academic and practitioner experience.

1.4 From the academic year 2011/12 onwards the CEB has had responsibility forsetting the knowledge area assessments and confirming the knowledge area cohortmarks for each of the Provider institutions. In confirming marks for cohorts ofcandidates the CEB is concerned to ensure that a consistent measure of achievementhas been applied across all Providers, and that proper account has been taken of anyrelevant factors that may have had a bearing on the performance of a cohort ofcandidates. As a result the CEB has the discretion to scale cohort marks (upwards ordownwards) if it feels there are issues relating to all candidates, or a statisticallyrelevant sub-cohort of candidates, that justify such intervention. In confirming themarks for MCQ and SAQ papers the CEB does not address the combined effect of the

marks in terms of whether or not a candidate has passed a knowledge area overall,or where a candidate falls in terms of grade boundaries.

1.5 Once the CEB has confirmed the MCQ and SAQ marks for each cohort ofcandidates at each Provider the marks are distributed to the Providers where they feedinto the individual BPTC student profiles considered at the Provider award andprogression examination boards. It is at the Provider examination boards that issuesrelating to individual candidates such as extenuating circumstances or academicmisconduct are considered.

1.6 For 2014/15 there were 8 Provider institutions offering the BPTC across 11centres:

BPPLondon, Manchester and Leeds

Cardiff

City University Law School

University (formerly College) of LawLondon and Birmingham

Manchester Metropolitan University

Nottingham Trent Law School

UNN

UWE

BPP Manchester came on stream as an assessment centre for the first time in2013/14, and Kaplan recruited its last intake in 2013/14 (although it had a very smallnumber of referred and deferred candidates in the 2014/15 first sit cohort).

Provider cohort results contained in the report have been anonymised. Providers areidentified by numbers in the graphs of cohort performance featured in this report, andthe allocation to a Provider of a number will vary from one graph to another dependingon the baseline adopted for the comparison Provider cohort performance.


5/53

5

1.7 Terms used in this report:

All-Provider refers to the aggregated data bringing together cohortperformance across all Providers centres

By Provider refers to data showing the performance of each of the Providers

compared with each other First sit refers to the March/April exam cycle note that some candidates

undertaking these examinations may be doing so on a referred or deferredbasis

Resit refers to the Augustexam cyclesome candidates undertaking theseexaminations may be doing so on a deferred basis (i.e. as if for the first time)

Combined refers to the result following aggregation of the MCQ and SAQscores. Combined pass rates can be lower than either MCQ or SAQ pass ratesthat feed into them depending on the pattern of candidate failure across the twoelements).

2. The setting and conduct of assessments

2.1 All Provider institutions were required to supply their 2010/11 knowledge areaMCQ and SAQ papers (first sit and resit), along with solutions, to the BSB. Thesupplied material was used by the CEB examiner teams, with necessary amendmentsand additions, to devise the 2011/12 first sit and resit assessments and relatedsolutions. Additional material was supplied by the Provider institutions for the 2012/13and 2013/14 assessments, and this was further supplemented by questions andsolutions devised by members of the CEB examining teams and suitably qualifiedindividuals recruited specifically as question writers. Assessment scrutiny meetings

were attended by relevant CEB examiners, the CEB Chair and BSB staff.Assessments were also checked by independent technical readers and any issuesraised referred back to the scrutiny meetings.

2.2 Candidates at all Provider institutions attempted the assessments in each of theknowledge areas on the same dates.

Professional Ethics23 March 2015

Civil Litigation, Remedies & Evidence8 April 2015

Criminal Litigation, Evidence & Sentencing10 April 2015

2.3 All examinations had a two oclock start time. In any case where a Provideridentified candidates as having special assessment arrangements necessitating astart time earlier than that of the main cohort, the relevant candidates were not allowedto leave their assessment area until the commencement of the main cohortassessment, and secure delivery and collection arrangements were put in place for allexamination materials.

2.4 In exceptional circumstances candidates can be allowed to attempt theassessments at locations overseas. The onus is placed on the Provider institution toensure that a secure assessment centre is available in these cases, and the BSBrequires the start time of the examination at the overseas centre to be the same (or


6/53

6

subject to similar to arrangements in 2.3) as the UK start time (regardless of timedifferences). To ensure the complete security of the examination papers the BSBdispatches all examinations to the overseas contacts directly.

2.5 Provider institutions were given guidance on examination arrangements by the

BSB. Exam invigilation reports were submitted by Providers to the BSB, detailing anymajor issues Providers believed may have had a bearing on the conduct of theexamination itself at their assessment centres (for example, transport difficulties, bombhoaxes, fire alarms, building noise).

2.6 Providers were also required to complete a pro-forma on each MCQ and SAQquestion in each of the knowledge areas indicating their view as to the level of difficulty,relevance to the syllabus, whether a question was technically defective in some way,and the appropriateness of the solution. The pro-forma also provided an opportunityto comment on the paper as a whole. External examiners also submitted commentson the assessments.

CEB examiners were invited to respond to the pro-formas, as required, by issuingrevised post-exam marking guidance in relation to the SAQ papers. All Providercomments were taken into account at the CEB meetings held to review MCQ and SAQmarks in each knowledge area.

3. The marking process

3.1 Candidates attempting the MCQ papers in each of the knowledge areas markedtheir answers on machine-readable answer sheets. Provider institutions were requiredto take copies of the MCQ answer sheets and return the originals to the BSB. TheMCQ scanning was undertaken by BSB support staff, using Speedwell scanners andsoftware, who had undergone training provided by Speedwell.

3.2 SAQs were marked by staff at the Provider institutions following (revised) markingguidance provided by the CEB examiners. Provider marking teams were required toobserve annotation conventions when marking SAQ scripts to show the impact ofinternal marking and moderation. Each Provider was required to ensure that anappropriately robust system of internal SAQ moderation was in place with particularregard being given to the pass/fail borderline candidates. The external examiner forthe relevant knowledge area at each Provider was also required to sample SAQ

papers as appropriate and require knowledge area teams to remark where it was feltguidelines had not been adhered to. From 2013 an enhanced sampling system wasput in place whereby randomised samples in appropriate marking bands were selectedby Providers, as directed by the BSB, for review by external Examiners, with copies ofany scripts supplied to the External Examiners also being sent to the CEB examinersat the BSB. The range of scripts within the sampling bands was adjusted to reflect thefact that the effect of upward scaling by the CEB could result in candidates who, onfirst marking, appear to be clear fails in SAQ papers becoming borderline pass/failcandidates once the effect of scaling is applied.

3.3 Reports were provided by each Provider and the relevant BSB external examiner

assigned to the Provider, commenting on the SAQ marking process in each of the


7/53

7

knowledge areas, with a requirement to flag any issues that should be considered bythe CEB as possible grounds for intervention in determining the marks to be confirmed.

4. How the CEB reviewed the results

4.1 Two sub-boards were constituted to review the results in each of the threeknowledge areas, one to look at MCQ results and one to look at SAQ results. Eachsub-board was chaired by the CEB Chair, and attended by the relevant knowledgearea examiners, and an independent psychometrician.

4.2 At each sub-board the pre-scale MCQ and SAQ results for the current, andprevious first sit rounds were noted, in particular the aggregated all Provider passrate (i.e. the percentage of candidates passing across all Providers combined), andthe pass rates at each Provider.

4.3 Other key sources of information available to the sub-boards included:

Representations made by Providers in the MCQ/SAQ feedback pro-formas(see 3.3 above) and comments from CEB examiners.

A spreadsheet showing the pass rate for each MCQ, and each component ofeach SAQ, achieved at each of the Providers cross-referenced to therepresentations made in the assessment pro-formas returned by the Providers(see 3.3 above) thus flagging up any correlation of Provider criticisms andconcerns with systematic poor performance by candidates.

For both the MCQs and SAQs, Manhattan diagrams indicating the extent towhich each question discriminated effectively between strong and weak

candidates. For the SAQs, Provider feedback, BSB External Examiner comments, and anyamended marking guidance issued by the CEB examiner teams.

Evidence provided by routine random sampling any further requested samplingof SAQ scripts undertaken at the direction of the CEB

4.4 On the basis of the above evidence, and as advised by the independentpsychometrician, the CEB sub-boards formulated a series of proposals, whererequired, for the scaling of cohort marks to be considered at the final CEB meeting.The CEB recognises that there are a multiplicity of approaches to cohort scaling andoperates within the general principles set out in the scaling protocol adopted following

consultation with the Providers. Amongst the options open to the CEB are:

Crediting more than one correct answer to an MCQ Disregarding an MCQ or part of an SAQ entirely if defective or inappropriate

(e.g. no correct answer) no candidate credited and maximum scorerecalculated

Crediting all candidates with the correct answer if an MCQ or part of an SAQ isdefective or inappropriate

Crediting all candidates with a mark or marks in respect of a part of an SAQ ifthere is headroom to do so (i.e. provided a candidate has not already achievedthe maximum marks available)


8/53

8

Adjusting marks globally for an SAQ as a whole, an SAQ paper as a whole, oran MCQ paper as a whole to compensate for generic factors if there isheadroom to do so (i.e. provided a candidate has not already achieved themaximum marks available)

Adjusting marks for each candidate in a sub-cohort due to local assessment

issues (not arising from the delivery of the course by a Provider, or mattersrelated to the conduct of the assessment that can be dealt with through aProviders extenuation processes) provided the sub-cohort constitutes astatistically reliable sample for scaling purposes.

4.5 The final CEB Board meeting for the 2014/15 first sit examinations, held on 26June 2015, considered the recommendations of the sub-boards in respect of theknowledge area MCQ and SAQ cohort marks. The meeting was attended by therelevant examiners, key BSB officers, an independent psychometrician and anindependent observer. The function of the final CEB board is to test therecommendations of the sub-boards, consider whether further sampling of SAQs may

be necessary, and to confirm the MCQ/SAQ cohort marks subject to any outstandingQA issues related to on-going sampling. Once cohort marks are confirmed by the CEBthey cannot subsequently be altered by Provider institutions. The process forchallenging marks confirmed by the CEB is outlined in section 16 (below).

5. 2014/15 first sit results in Professional Ethics: MCQs

5.1 MCQ pre-scale first sit pass ratesall Providers 2011/12 to 2014/15

Professional Ethics 2014/15 2013/14 2012/13 2011/12

MCQ pre-scale All-Provider passrate

91.5% 81.0% 86.4% 92.6%

The pre-scale all-Provider pass rate for Professional Ethics MCQ is significantly up on2013/14 showing a rise in excess of 10%, the highest performance level since the2011/12 first sits.


9/53

9

5.2 MCQ pre-scale 2014/15 first sit histogram question by question

This table indicates the range of pre-scale pass rates for the Professional Ethics MCQsacross all Providers, showing no MCQs with a pre-scale pass rate of below 40%(compared with 3 in 2013/14).

5.3 Final exam board review of the Professional Ethics MCQ results

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Passrate%

MCQ item number

Professional Ethics 2014/15 First Sit Pre-Scale

MCQ All Provider question by question histogram

Question Interventionapplied

CEB rationale

MCQ 4Nointervention

48.4% all-Provider pass rate, very good discriminationbetween candidates. The feedback from Providers wasconsidered and the Board agreed no further action wasrequired.

The Board decided there was no reason for intervention.

MCQ 9Nointervention

40.5% all-Provider pass rate, with strong discrimination.It was noted that 6 Provider cohorts had pass ratesbelow 40% for this MCQ. The Provider feedback wasconsidered and addressed by the Chief Examiner, whoexplained that the question covered the ethicalprinciples according to the Code, rather than what theProviders noted to be common practice. The feedbackfrom Providers did not give grounds for intervention asstudents have not gained the experience in the field toenable them to be aware of these practices. The Boardagreed that distractor A worked well.



10/53

10

The Final Exam Board noted the careful consideration of the statistical data, the writtenfeedback from Providers, and the written feedback from the external examiners thathad taken place at the Professional Ethics MCQ exam board, and therecommendations therefrom. The comments from the Chief Examiner for ProfessionalEthics regarding the impact of the new handbook on the setting of MCQs were noted.Having considered this evidence and the deliberations of the MCQ Board the FinalBoard determined that no interventions were justified.

5.4 MCQ post-scale first sit pass ratesall Providers 2011/12 to 2014/15


MCQ post-scale All-Provider passrate

91.5% 81.0% 94.3% 92.6%

Given the absence of intervention, the post-scale MCQ first sit pass rate for 2014/15remains at 91.5%. As indicated above, this is a noticeably higher figure comparedwith the post-scale MCQ rates for the previous year, but in line with 2012/13 and2011/12.

5.5 MCQ first sit pass rates across all Providers 2014/15

70

75

80

85

90

95

100

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider

Professional Ethics First Sit 2014/15

MCQ pass rates per Provider

MCQ 2014/15 Pass rate

MCQ 16Nointervention

45.7% all Provider pass rate, very good discriminationbetween candidates. It was noted that Providers did notsubmit feedback on this question.



11/53

11

Providers are ranged in order of pass rates (high to low) according to theirperformance. Hence Provider 1 had the highest pass rate at 98%, Provider 11 thelowest at just under 79%, a range of 18% between top and bottom.

5.6 MCQ post-scale first sit pass rates by Provider 2011/12 to 2014/15

Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit ProfessionalEthics MCQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass ratescompare. Hence Provider 1 achieved a pass rate of over 98% in 2011/12, over 96%in 2012/13, dropped to 80% in 2013/14, and recovered with a pass rate of 95% in

2014/15. Provider 7 was the only Provider to record a lower cohort pass rate for2014/15 compared to 2013/14. Provider 10 recorded a notable 20% increase in passrates compared to 2013/14. Provider 5 has the best 4-year average pass rate at over93%, whilst the weakest, on average, over the same period is Provider 7 with anaverage pass rate just under 88%. Note that results for the BPP Manchester cohortare not reflected in this table as there is no data pre 2013/14 and results for Kaplanhave been excluded, as their last intake was in 2013/14.

6. 2014/15 first sit results in Professional Ethics: SAQs

6.1 SAQ pre-scale first sit pass ratesall Providers 2011/12 to 2014/15


SAQ pre-scale All-Provider passrate

58.0% 65.6% 86.7% 88.5%

Following a significant decline in Professional Ethics SAQ pre-scale pass rates in2013/14 compared to the two previous rounds of first sit assessments (over 20%), theall-Provider pass rate for 2014/15 is a further 7.6% down on 2013/14.

70.0

75.0

80.0

85.0

90.0

95.0

100.0

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Professional Ethics MCQ Post-Scale pass rates

First Sit 2011/12 to 2014/15

MCQ 2011/12 post-scale MCQ 2012/13 post-scale



12/53

12

6.2 SAQ pre-scale first sit histogram question by question (showing sub-parts)

This table indicates the range of pre-scale pass rates for the Professional Ethics SAQsacross all Providers, clearly showing the relatively poor cohort performance in respectof SAQ2. Compared to 2013/14, where three out of eleven SAQ sub-elementsreported a pre-scale pass rate of below 40%, the 2014/15 SAQ paper shows only onesub-element out of six with a pass rate below 40%. Sub-element SAQ2(b) clearlypresented a significant challenge to candidates as it is something of an outlier with anall-Provider pass rate of 30%, resulting in an all-Provider pass rate for SAQ2 as awhole of just over 50%, but this is not of itself remarkable. SAQ 1 in the 2013/14 first

sit produced a pre-scale pass rate of less than 40% overall and was not regarded bythe examination boards as warranting any further intervention.

6.3 Final exam board review of the Professional Ethics SAQ results

QuestionInterventionapplied

CEB rationale

SAQ 1 (a)Nointervention

SAQ1(a) had 6 marks available and the all-Provider passrate for the component was 49.0%. The Boardconsidered the comments from Providers and noted one

comment suggesting the word unregistered should begiven a mark. The Chief Examiner explained that this wasincorrect and should not warrant an additional mark. TheBoard also noted comments suggesting an extra markshould be awarded for candidates stating the 2 coreduties rather than only 1. The Board agreed only onemark was required for this question.

The Chief Examiner confirmed that the question waswritten in a very clear way. The range of performancesacross the question suggested good candidates did well

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

Passrate%

SAQ sub-parts and totals

Professional Ethics 2014/15 First Sit Pre-scale

SAQ All Provider question by question histogram


13/53

13


CEB rationale

and the question had good discrimination. All commentsfrom Providers were considered and the Board agreed

that no further action or interventions were required.

The Board decided that there was no reason forintervention.

SAQ 1 (b)Nointervention

SAQ1(b) had 4 marks available and the all-Provider passrate for the component was 69.8%. The comments fromProviders were considered and amendments were madeto the mark scheme to reflect this. The Board agreed nointerventions in respect of this question were warrantedand noted the good discrimination data for the question.



SAQ2(a) had 4 marks available and the all-Provider passrate for the component was 52.4%.

The Board considered comments from Providers anddecided that there was no reason for intervention.


SAQ2(b) had 6 marks available and the all-Provider pass

rate for the component was 29.5%.The Board noted the relatively low all-Provider pass rate,and the rate achieved by the cohort at the best performingProvider across the 2014/15 first sit assessments as awhole. The Board noted that amendments had beenmade to the mark scheme in light of Provider and EEfeedback. The Board considered whether the extent towhich reliance on knowledge from the academic stage(criminal law) required to answer this question was fairand agreed that it was. It was noted that the amendedmarking scheme permitted 6 ways to get the full 4 marksfor question 2(b). The Board considered the statisticalinformation and noted that this was a more challengingquestion that discriminated well between weaker andstronger candidates.



SAQ3(a) had 5 marks available and all-Provider passrate for the component was 71.3%. The Board noted that


14/53

14


CEB rationale

some comments from Providers had been reflected in theamendments to the final marking scheme.



SAQ3(b)had 5 marks available and the all-Provider passrate for the component was 60.6%.The Board noted that the marking scheme permitted 8possible ways to get the 5 available marks for SAQ3(b),and that the comments from Providers had beenconsidered and the final mark scheme amendedaccordingly.


There were no other global or generic issues relating to the SAQ assessment.

6.4 SAQ post-scale first sit pass ratesall Providers 2011/12 to 2014/15


SAQ all-Provider pass rate 58.0% 65.6% 86.7% 88.5%

Given the absence of intervention, the post-scale SAQ first sit all-Provider pass ratefor 2013/14 remains at 58%, 7.6% down on the previous year. The Final examinationboard endorsed the recommendations of the SAQ Board.

6.5 SAQ pre and post-scale first sit pass rates across all Providers 2014/15


15/53

15

Providers are ranged in order of pass rates (high to low) according to theirperformance. Hence Provider 1 had the highest pass rate at 81.6%, Provider 11 thelowest at just under 37%. There is clearly a very significant range in the performancefor Provider cohorts across this assessment (44% between best and worst), but thisis largely consistent with 2013/14 where the range was 39%, and suggests that theassessment is discriminating effectively between weak and strong cohorts. Thebiggest difference between the 2013/14 and 2014/15 cohort data sets is that in2014/15 there are three very weak cohorts compared to the rest as the table belowindicates.

Cohort pass raterange

2014/15(11 Providers)

2013/14(12 Providers)

80%+ 1 2

70% to 79% 2 3

60% to 69% 3 3

50% to 59% 1 4

40% to 49% 2 0

Below 40% 2 0

This suggests that performance at the higher end has held up and that the weakercohorts in 2014/15 have been significantly weaker than in 2013/14.

6.6 SAQ first sit pass rate by Provider 2011/12 to 2014/15

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider


SAQ pass rates per Provider

SAQ 2014/15 Pass rate


16/53

16

Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit ProfessionalEthics SAQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass ratescompare. Hence Provider 1 achieved a pass rate of 95.5% in 2011/12, 92.4% in2012/13, and 71% in both 2013/14 and 2014/15. Provider 2, by contrast, has seen avery worrying year-on-year decline in cohort performance from 93% in 20111/12, to83% in 2012/13, 59% in 2013/14 dropping even further to 37% in 2014/15. ThreeProviders recorded cohort pass rates for the Professional Ethics SAQs that werehigher than their 2013/14 pass rates Provider 2 recorded the biggest decline in pass

rates from 2013/14, a drop of over 22%. Provider 1 has the best 4-year average passrate at over 81%, whilst the weakest, on average, over the same period is Provider 7with an average pass rate just over 65%. Note that results for that BPP Manchestercohort are not reflected in this table as there is no data pre 2013/14, and results forKaplan have been excluded, as their last intake was in 2013/14

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Professional Ethics SAQ Post-scale

Pass rates for First Sit 2011/12 to 2014/15

SAQ 2011/12 post-scale SAQ 2012/13 post-scale

SAQ 2013/14 post-scale SAQ 2014/15 post-scale


17/53

17

6.7 MCQ and SAQ first sit post-scale pass rates by Provider 2014/15 high to low

This table compares 2014/15 Professional Ethics MCQ and SAQ all-Provider post-scale pass rates. The order of Providers is determined by the Providers combinedpost scale pass rate for the MCQ and SAQs taken together (i.e. the percentage ofcandidates at the Provider pass both sections). The data shows that cohorts typicallyperform better in the MCQ element compared to the SAQ element. As in previousyears there is some evidence to suggest that the gap in pass rates between the twoforms of assessment tends to widen where cohorts are weaker overall (this is very

much the case with Providers 9 to 11).

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider


MCQ and SAQ cohort performance comparison

MCQ 2014/15 post-scale

SAQ 2014/15 post-scale


18/53

18

7. 2014/15 first sit in Professional Ethics: combined MCQ and SAQ post-scaleresults

7.1 Combined MCQ and SAQ all-Provider first sit post-scale pass rates and gradeboundary distribution 2014/15


Combined (MCQ and SAQ) All Provider post-scale results

Total number sat (All Providers) 1572 %

Pass Combined 892 56.7%

Fail Combined 680 43.3%

Combined Grade Boundaries

Outstanding 139 8.8%

Very Competent 557 35.4%

Competent 196 12.5%

Not Competent 680 43.3%

It should be noted that the CEB does not review, adjust or confirm the combined passrate for any of the three knowledge areas. The combined results are simply the productof the relevant confirmed MCQ and SAQ marks which may, or may not, have beensubject to adjustment by the CEB at an earlier stage. No further scaling is applied onceMCQ and SAQ marks are combined to produce the pass rate for a knowledge area as

a whole. Hence the overall knowledge area marks for cohorts and individualcandidates, including grade boundaries, lie where they fall.

7.2 Combined MCQ and SAQ first sit pre and post-scale pass rates by Provider highto low

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider


Combined pass rates by Provider

Combined 2013/14 pass rate


19/53

19

Providers are ranged in order of pass rates (high to low). Cohort performance in theSAQ element of the assessment largely drives the outcome of the combined cohortresult, hence this performance profile is very similar to the SAQ table at 6.5.

7.3 Combined MCQ and SAQ first sit pass ratesall Providers 2011/12 to 2014/15


Combined post-scale All-Provider passrate

56.7% 65.6% 86.4% 85.0%

7.4 First sit 2014/15 post-scale grade boundaries by Provider

Providers are ranked according to their 2014/15 post-scale combined ProfessionalEthics first sit pass rate. Hence Provider 1 has the highest combined first sit pass rateand, predictably the lowest level of Not Competent candidates. The level of NotCompetent candidates rises in line with the drop in the Provider pass rate. By contrast,the number of Outstanding candidates presents a profile less closely linked to overallpass rates, with Provider 5 showing a higher proportion of Outstanding candidatesthan Provider 1.

0.0

20.0

40.0

60.0

80.0

100.0

1 2 3 4 5 6 7 8 9 10 11

%s

tudentpercategory

Provider


Post-scale grade boundaries by Provider

Not Competent Competent Very Competent Outstanding


20/53

20

7.5 Combined MCQ and SAQ all-Provider first sit post-scale pass rates trendanalysis 2011/12 to 2014/15

Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit combinedProfessional Ethics pass rate, showing how their 2012/13, 2013/14 and 2014/15 passrates compare. Hence Provider 1 achieved a 92% pass rate in 2011/12, 83% in2012/13. 54.7% in 2013/14, and 41.7% in 2014/15 and, along with Providers 4 and 8

records a drop in combined pass rates year on year across all 4 cycles of first sitassessments in Professional Ethics. Five Providers show a decline in their combinedpass rate compared to 2013/14 (Provider 8 showing the largest drop at over 18%),whilst 5 Providers improve their combined pass rates (Provider 3 up 8%). Provider 3has the best 4-year average pass rate at over 78%, whilst the weakest, on average,over the same period is Provider 4 with an average pass rate just over 63%. Note thatresults for that BPP Manchester cohort are not reflected in this table as there is nodata pre 2013/14, and results for Kaplan have been excluded, as their last intake wasin 2013/14.

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Professional Ethics First Sit Combined (MCQ & SAQ)

Post-scale pass rated by Provider 2011/12 to 2014/15

% Pass 2011/12 % Pass 2012/13 % Pass 2013/14 % Pass 2014/15


21/53

21

7.6 All-Provider first sit post-scale grade boundaries trend analysis 2011/12 to2014/15

No significant movement in classifications between 2013/14 and 2014/15.

8. 2014/15 first sit results in Criminal Litigation: MCQs


Criminal Litigation 2014/15 2013/14 2012/13 2011/12


80.4% 74.6% 85.3% 86.8%

A 6% rise in the pre-scale MCQ all-Provider pass rate compared to the previousyear.

0.00

10.00

20.00

30.00

40.00

50.00

60.00

Outstanding Very Competent Competent Not Competent

%s

tudents

Grade category

Professional Ethics First Sit post-scale

Grade boundaries trend analysis 2011/12 to 2014/15

2011/12 2012/13 2013/14 2014/15


22/53

22

8.2 MCQ pre-scale first sit histogram question by question

This table indicates the range of pre-scale pass rates for the Criminal Litigation MCQsacross all Providers, showing 2 MCQs with a pre-scale pass rate of below 40%(compared to 3 in 2013/14 first sit).

8.3 Final exam board review of the Criminal Litigation MCQ results

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Passrate%

MCQ item number

Criminal Litigation 2014/15 First Sit Pre-scale



CEB rationale

MCQ 2Nointervention

32.1% all-Provider pass rate with high discrimination.The Board noted the relatively low pass rate and the lackof comments from Providers and External Examiners. Itwas agreed this was a harder question but one that wasappropriate for the paper.


MCQ 9Nointervention

45.9% all-Provider pass rate with very gooddiscrimination. The Board noted the low pass rate andconsidered the comments from Providers, especiallyreferring to the query over the correct answer. TheBoard agreed there was no possibility distractor C wasthe correct answer and the correlation figure supportedthis. The Board considered the issue of syllabuscoverage and determined that there were no matters ofconcern.


23/53

23

There were no other global or generic issues relating to the MCQ assessment

The Board decided there was no reason forintervention.


48.9% all-Provider pass rate and good discrimination.That the pass rate was below 50% was noted andProvider comments were considered. The statistics

were positive and showed this was a harder question inthe paper. The distractors were working.



48.1% all-Provider pass rate and very strongdiscrimination. The pass rate below 50% was noted andcomments from Providers were considered. The Boardagreed the question was covered in the syllabus.

The Board decided there was no reason forintervention.

MCQ 33

Credit D aswell as B.

65.3% all-Provider pass rate and good discrimination.The comments from Providers were considered. Therewas a concern with distractor D being wrong which wouldmake it the correct answer. The Board discussed andagreed to intervene and credit those candidates who hadanswered D as well as those who had chosen B.

The Board decided to intervene and credit D as well asB.

MCQ 39

Removequestion 39from thepaper.

22.4% all-Provider pass rate with very poor

discrimination. The Board considered the commentsfrom Providers and External Examiners as well as thevery poor discrimination and low pass rate. The Boardagreed to remove the question, as the answers offereddid not have the required level of clarity to fall safely into either a right or wrong category. The Board agreedthat the rationale for removing an entire question wassatisfied: because of the defect in the question, therewas no correct answer hence no candidate lostanything by virtue of the questions removal. Furtherthere was statistical evidence that candidates may have

been guessing which answer was correct (i.e. weakcorrelation with strong candidate scores)

The Board decided to remove question 39 from thepaper.


24/53

24



MCQ Post-scale All-Provider pass

rate83.3% 84.1% 88.9% 88.7%

The post-scale MCQ first sit pass rate for 2014/15 of 83.3% shows the impact of theCEB intervention in raising the pre-scale pass rate of 80.4% by 2.9. The trend data forCriminal Litigation MCQ national pass rates over these three cycles of first sitassessments indicates a broadly consistent pattern of all-Provider cohort performancewithin a 6% range.

8.5 MCQ pre and post-scale first sit pass rates across all Providers 2014/15

The graph shows the effect of the interventions in respect of MCQs 33 & 39 across allProviders. Providers are ranged in order of pass rates (high to low) according to their2014/15 first sit pre-scale cohort performance. Hence Provider 1 had the highest passrate both before and after CEB intervention. Generally the intervention favoured theweaker cohorts, particularly Provider 10, showing an uplift of 10%. The effect of the

intervention on the range of pass rates across Providers shows a compression from25% pre scale to 19% post-scale.

60

65

70

75

80

85

90

95

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider

Criminal Litigation First Sit 2014/15

MCQ pass rated per Provider

MCQ 2014/15 pre-scale



25/53

25

8.6 MCQ post-scale first sit pass rate by Provider 2011/12 to 2014/15

Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit CriminalLitigation MCQ pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass ratescompare. Hence Provider 1 achieved a 93% pass rate in 2011/12, 89% in 2012/13.89% in 2013/14, and 82% in 2014/15. Six Providers record a lower cohort pass ratefor 2014/15 compared to 2013/14. Provider 2 has the best 4-year average pass rateat over 90%, whilst the weakest, on average, over the same period is Provider 10 withan average pass rate just over 81%. Note that results for that BPP Manchester cohortare not reflected in this table as there is no data pre 2013/14, and results for Kaplan

have been excluded as their last intake was in 2013/14.

9. 2014/15 first sit results in Criminal Litigation: SAQs

9.1SAQ pre-scale all Provider first sit pass rate2011/12 to 2014/15



64.2% 75.7% 54.5% 73.9%

Data for 2014/15 shows an 11.5% drop in pre-scale all-Provider pass rates comparedto 2013/14 but sitting in the middle of the range for the 4 year cycle as a whole.

70.0

75.0

80.0

85.0

90.0

95.0

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Criminal Litigation MCQ Post-scale pass rates

First Sit 2011/12 to 2014/15




26/53

26

9.2 SAQ pre-scale first sit histogram question by question (showing sub-parts)

This table indicates the range of pre-scale pass rates for the Criminal Litigation SAQsacross all Providers, with a relatively strong cohort performance in respect of all theSAQs, with only one sub-element, SAQ2(c) recording an all-Provider pass rate below40% (compared to 4 in the 2013/14 SAQ first sits).

9.3 Final exam board review of the Criminal Litigation SAQ results

QuestionInterventionapplied CEB rationale

SAQ 2(c) No intervention SAQ2(c) had 2 marks available and the all-Provider pass rate for the component was 19.8%.There were no comments from Providers regardingthis question. The Board considered the relativelylow pass rate and considered how manycandidates only scored 1 mark. The IndependentPsychometrician confirmed the percentage ofcandidates scoring 0 was 27%, candidates scoring1 mark was 54% and scoring 2 marks was 20%.

The Board noted that the sampling of scriptssuggested that it was the second point thatcandidates did not pass and felt it was a harderquestion. The Board considered the operation ofthe marking scheme, in particular the fact thatcandidates only capable of securing 1 mark out of2 would not be achieving the 60% rate required topass the paper as a whole. The Board consideredthe options for crediting an extra mark or removingthe marks for the question, but concluded that theassessment had performed as it would have

0

10

20

30

40

50

60

70

8090

Passrate%


Criminal Litigation First Sit 2014/15 Pre-scale



27/53

27

wanted, and this conclusion was reinforced by theabsence of Provider comments on the question.


SAQ4(c) No intervention SAQ4c had 3 marks available and the all-Providerpass rate for the component was 47.1%. TheBoard noted that two Providers in particular hadnot performed well on this question. The commentsfrom Providers were considered and the Boardconcluded that no changes were required.


There were no other global or generic issues relating to the SAQ assessment

9.4 SAQ post-scale first sit pass rates all-Providers 2011/12 to 2014/15

CriminalLitigation

2014/15 2013/14 2012/13 2011/12

SAQ post-scale All-Providerpass rate

64.2% 78.2% 69.9% 77.8%

With no interventions in respect of the Criminal Litigation SAQ assessments the all-Provider pass rate remains unchanged at 64.2%, 14% down on the previous roundof first sit assessments, and the lowest level recorded across the four cycles of firstsit assessment.

9.5 SAQ first sit pass rates across all Providers 2014/15

45

55

65

75

85

95

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider



SAQ 2014/15 Pass rate


28/53

28

The graph shows the range of performance across all 11 Providers in respect of theCriminal Litigation SAQ assessment. As noted above, there were no interventionssanctioned by the examination boards, hence there is no pre and post scalecomparison. There is a 36% range between the top performing cohort at Provider 1and the weakest at Provider 11, an increase on the 27% post-scale range reported in

2103/14. The data for 2014/15 suggests a challenging assessment that the CriminalLitigation SAQ discriminated effectively across a range of cohort ability.


Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit CriminalLitigation SAQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 passrates compare. Hence Provider 1 achieved a pass rate of 91.1% in 2011/12, 89% in2012/13, and 92.9% in 2013/14 and 80.8% in 2014/15. Provider 2, by contrast, hasseen a very worrying year-on-year decline in cohort performance from 90% in20111/12, to 59.3% in 2014/15. None of the Providers reported in this graph recordedcohort pass rates for the Criminal Litigation SAQs that were higher than their 2013/14pass rates. Provider 10 recorded the biggest decline in pass rates from 2013/14, adrop of over 28.7%. Provider 1 has the best 4-year average pass rate at over 88%,

whilst the weakest, on average, over the same period is Provider 7 with an averagepass rate just over 67%. Note that results for that BPP Manchester cohort are notreflected in this table as there is no data pre 2013/14, and results for Kaplan havebeen excluded, as their last intake was in 2013/14.

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Criminal Litigation SAQ Post-scale

Pass rates for First Sit 2011/12 to 2014/15

SAQ 2011/12 post-scaleSAQ 2012/13 post-scaleSAQ 2013/14 post-scaleSAQ 2014/15 post-scale


29/53

29

9.7 MCQ and SAQ first sit post-scale pass rates by Provider 2014/15

This table compares 2014/15 Criminal Litigation MCQ and SAQ all-Provider post-scalepass rates. The order of Providers is determined by the Providers combined postscale pass rate for the MCQ and SAQs taken together (i.e. the percentage ofcandidates at the Provider passing both sections). The data shows that cohortstypically perform better in the MCQ element compared to the SAQ element (no cohortperforms better in the SAQ element). As in previous years there is some evidence tosuggest that the gap in pass rates between the two forms of assessment tends towiden where cohorts are weaker overall (this is very much the case with Provider 11,

where the differential is over 34%).

10. 2014/15 first sit post-scale results in Criminal Litigation: combined MCQ

and SAQ results

10.1 Combined MCQ and SAQ all Provider first sit post-scale pass rates and gradeboundary distribution


Combined (MCQ and SAQ) All Provider post-scale results

Total number sat (All Providers) 1483






Competent 170 11.5%


40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider






30/53

30

It should be noted that the CEB does not review, adjust or confirm the combined passrate for any of the three knowledge areas. The combined results are simply the productof the relevant confirmed MCQ and SAQ marks which may, or may not, have beensubject to adjustment by the CEB at an earlier stage. No further scaling is applied onceMCQ and SAQ marks are combined to produce the pass rates for a knowledge area

as a whole. Hence the combined knowledge area marks for cohorts and individualcandidates, including grade boundaries, lie where they fall.

10.2 Combined MCQ and SAQ first sit pre and post-scale pass rates by Provider2014/15

The graph shows the effect of the interventions in respect of the MCQs (there were noSAQ interventions) across all Providers. Providers are ranged in order of combinedpre-scale pass rates (high to low). Hence Provider 1 had both the highest pre-scaleand post-scale pass rates. Given the very limited level of intervention this year thereis very little difference in the pre and post scale result profiles.

10.3 Combined MCQ and SAQ post-scale first sit pass ratesall Providers 2011/12to 2014/15



62.5% 72.8% 68.2% 74.7%

The combined all-Provider pass rate is 14% down on the previous round of first sitassessments, and is the lowest level recorded across the four cycles of first sitassessment.

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider


Combined (MCQ and SAQ) Pre and Post scale pass rates

per Provider

Combined 2014/15 pre scale

Combined 2014/15 post scale


31/53

31


Providers are ranked according to their 2014/15 post-scale first sit pass rates inCriminal Litigation. Hence Provider 1 has the highest percentage of Outstandingcandidates and lowest percentage of Not Competent. The distribution of NotCompetent candidates follows the profile of declining cohort pass rates fairly closely,whilst the distribution of Outstanding candidates across Providers 2 to 9 seems lessclosely related to overall cohort performance. For example, Provider 10 is fifth best interms of the number of Outstanding candidates in its cohort.

10.5 Combined MCQ and SAQ all-Provider first sit post-scale pass rates trend

analysis 2011/12 to 2014/15

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10 11

%s

tudentspercategory

Provider


Post Scale grade boundaries by Provider

Not Competent Competent Very Competent Oustanding

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Criminal Litigation Combined (MCQ and SAQ)

Post-scale pass rates by Provider 2011/12 to 2014/15

% Pass 2011/12

%Pass 2012/13

% Pass 2013/14

% Pass 2014/15


32/53

32

Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit combinedCriminal Litigation pass rate, showing how their 2012/13, 2013/14 and 2014/15 passrates compare. Hence Provider 1 achieved an 89% pass rate in 2011/12, 86% in2012/13. 87% in 2013/14, and 76% in 2014/15. Provider 8 was the only Provider torecord a higher combined pass rate for 2014/15 as against 2013/14 (1.2%), with

Provider 9 showing a significant decline of over 28%. Provider 1 has the best 4-yearaverage pass rate at over 84%, whilst the weakest, on average, over the same periodis Provider 10 with an average pass rate just over 56%. Note that results for that BPPManchester cohort are not reflected in this table as there is no data pre 2013/14, andresults for Kaplan have been excluded, as their last intake was in 2013/14.


The three-year trend analysis shows a continued decline in the Very Competent witha corresponding increase in Not Competent, but little change in the other categories.

11. 2014/15 first sit results in Civil Litigation: MCQs


Civil Litigation 2014/15 2013/14 2012/13 2011/12


63.6% 60.9% 46.3% 67.8%

A slight improvement on the 2013/14 all-Provider pre-scale performance in the CivilLitigation MCQ assessment.

11.2 MCQ pre-scale first sit histogram question by question 2014/15

0.00

5.00

10.00

15.00

20.00

25.00

30.00

35.00

40.00

45.00


Passrate%

Provider

Criminal Litigation First Sit post-scale


2011/12

2012/13

2013/14

2014/15


33/53

33

This table indicates the range of pre-scale pass rates for the Civil Litigation MCQsacross all Providers, showing four MCQs with a pre-scale pass rate of below 40%,(there were seven in 2013/14) compared with two in the 2014/15 Criminal MCQassessment.

11.3 Final exam board review of the Civil Litigation MCQ results


CEB rationale

MCQ4Nointervention

MCQ 04 43.1 % all-Provider pass rate, and strongdiscrimination. The Board considered the commentsfrom Providers and External Examiners, discussing thepoint raised regarding the syllabus coverage of thequestion. The Board consulted the syllabus and agreedit was covered. It was recognised that this was achallenging question, however it was noted that thereshould be a mixture of difficult questions in the paper.


MCQ5

Remove thequestion fromtheassessment

MCQ 05 - 39.9% all-Provider pass rate, and very good

discrimination. It was noted that although the correctanswer for the question was D, more candidates choseanswer C. The Board considered if distractor C mightalso potentially be correct. The Chief Examinerhighlighted that there was a small technical inaccuracywith distractor C, however it was agreed that this mightnot be the reason why candidates chose C as the correctanswer. The Board discussed at length the optionsavailable, noting that in principle the rule for technicallydeficient questions should apply i.e. remove thequestion. Part of the rationale for removing an entire

question was that, because of the defect in the question,

0

10

20

30

40

50

60

70

80

90

100

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39

Passrate%

MCQ item number

Civil Litigation First Sit 2014/15 Pre-scale



34/53

34


CEB rationale

there was no correct answer, hence no candidate lostanything by virtue of its removal. To the extent it couldbe argued that there was a correct answer the statistics

strongly indicated that those choosing it may well havebeen guessing (i.e. weak correlation with strongcandidate scores).

The Board decided that question 5 would be removedfrom the paper and the pass rate adjusted accordingly.

MCQ 8Nointervention

MCQ 08 - 24.9% all-Provider pass rate and gooddiscrimination. The Board noted the pass rate was below25% and the comments from Providers were considered.It was agreed the question matter was covered in thesyllabus. The Board discussed if the question was too

difficult for candidates and if distractor B was unfair. Itwas agreed that it was a challenging question and theexamination paper needed a mixture of different difficultylevel questions. The Independent Psychometricianconfirmed the statistics showed this was a good questionand no concerns were raised.



MCQ 17 - 25.7% all-Provider pass rate and very gooddiscrimination. The Board considered the comments

from Providers and External Examiners suggestingproblems with identification of the correct answer. Thequestion was considered to be a negatively wordedquestion and there appeared to be some confusion withcandidates between selecting answers A and C. TheBoard agreed that distractor C remained incorrect andthere was no need to intervene on the question.


MCQ 20Disregard thequestion

MCQ 20 - 88.3% all-Provider pass rate with fair toadequate discrimination. The Board considered the

technical comments from the External Examiner. TheBoard agreed the question was technically deficientmeaning none of the answer options would have beencorrect and therefore to be consistent with MCQ 5 agreedto remove the question, with pass rates recalculatedaccordingly. As with MCQ 5, part of the rationale forremoving an entire question was that, because of thedefect in the question, there was no correct answerhence no candidate lost anything by virtue of its removal.To the extent it could be argued that there was a correctanswer the statistics strongly indicated that those


35/53

35


CEB rationale

choosing it may well have been guessing (i.e. weakcorrelation with strong candidate scores)

The Board decided that question 20 would be removedfrom the paper and the pass rate adjusted accordingly.

MCQ 34Credit optionsC and A

MCQ 34 - 34.4% all-Provider pass rate, with relativelypoor discrimination. The Board noted the low pass ratewith 50% of candidates opting for distractor A when thecorrect answer was C. The comments from Providersand External Examiners were considered and the Boardagreed this was a poorly worded question.

The Board therefore agreed to credit both answers A andC with a mark.

MCQ 35Credit optionsA and C

MCQ 35 - 48.2% all-Provider pass rate with very strongdiscrimination. Whilst the Board was content regardingsyllabus coverage, it accepted concerns that answer Amight not be unambiguously correct, resulting inunfairness to candidates who opted for distractor C.

The Board therefore agreed to credit both answers A andC with a mark.

There were no other global or generic issues relating to the MCQ assessment



MCQ Post-scale All-Provider pass rate 71.3% 68.6% 73.2% 83.7%

The post-scale MCQ first sit pass rate for 2014/15 of 71.3% shows the impact of theCEB intervention in raising the pre-scale pass rate of 63.6% by 7.7%, in effect a 12.1%increase as a proportion of the pre-scale pass rate.


36/53

36

11.5 MCQ pre and post-scale first sit pass rates all Providers 2014/15

The graph shows the combined effect of the interventions in respect of MCQs 5, 20,34 and 35 across all Providers. Providers are ranged in order of pass rates (high tolow) according to their pre-scale cohort performance. Hence Provider 1 had thehighest pre- and post-scale pass rate. The interventions had a differential impactacross the Providers in the sense that it was not connected with the level of cohortperformance. Hence, Provider 1 benefited the most (13.3%), followed by Provider 8(13%), and Provider 2 the least (3.9%).

11.6 MCQ post-scale first sit pass rate by Provider 2011/12 to 2014/15

Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit Civil LitigationMCQ pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass ratescompare. Hence Provider 1 achieved an 86.5% pass rate in 2011/12, 76% in 2012/13.67% in 2013/14, and 62.8% in 2014/15. Note Provider 1 has a year-on-year declineacross all four first sit cycles (the only Provider reported here for whom this is the

case). Four Providers record a lower cohort pass rate for 2014/15 compared to

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider

Civil Litigation First Sit 2014/15

MCQ pass rates per Provider

MCQ 2014/15 pre-scale


50.0

55.0

60.0

65.0

70.0

75.0

80.0

85.0

90.0

1 2 3 4 5 6 7 8 9 10

Passrates%

Provider

Civil Litigation MCQ Post-scale pass ratesFirst Sit 2011/12 to 2014/15






37/53

37

2013/14, with Provider 9 declining most significantly by over 17%. Provider 2, bycontrast, achieves an improvement of over 13%. Provider 4 has the best 4-yearaverage pass rate at over 76%, whilst the weakest, on average, over the same periodis Provider 10 with an average pass rate just over 64%. Note that results for the BPPManchester cohort are not reflected in this table as there is no data pre 2013/14, and

results for Kaplan have been excluded, as their last intake was in 2013/14.

12. 2014/15 first sit results in Civil Litigation: SAQs

12.1 SAQ all Provider pre- scale first sit pass rate 2011/12 to 2014/15



49.0% 52.7% 52.1% 61.4%

The Civil Litigation all-Provider SAQ pre-scale pass rate drops marginally by 3% fromthe previous 2 rounds of assessment, and is 15% lower than the corresponding pre-scale Criminal Litigation SAQ figure.

12.2 SAQ pre-scale 2014/15 first sit histogram question by question (showing sub-parts)

This table indicates the range of pre-scale pass rates for the Civil Litigation SAQsacross all Providers, clearly showing the relatively poor cohort performance in respectof SAQ 1(d), SAQ 2(c), SAQ 3 (a) and (b), SAQ4 (b) and (c), although there is no SAQwhere the overall pass rate dips below 40%. In total 6 SAQ sub-elements record anall-Provider pass rate below 40%, compared with 8 for the 2013/14 first sit cycle of thesame assessment.

0

10

20

30

40

50

60

70

80

Passrate%


Civil Litigation First Sit 2014/15 Pre-scale



38/53

38

12.3 Final exam board review of the Civil Litigation SAQ results

Question Intervention applied CEB rationale

SAQ1dCredit 0.5 mark where there isheadroom to do so

SAQ1(d) had 3 marks available and the all-Provider pass rate for the component was

21.4%. The Board noted the low pass rates atall Providers. The Provider comments wereconsidered. The Board agreed the question andanswers were technically correct but felt that themark scheme was not sufficiently flexible - thereshould have been 1 mark instead of half markavailable in 1(d) (i) as it was a key point in theanswer. As a consequence, the Boardrecommended all candidates to be awarded anadditional 0.5 where there was headroom to doso.

The Board decided to award an additional 0.5mark to all candidates where there washeadroom to do so for 1(d).

SAQ1Credit 0.5 mark where there isheadroom to do so

SAQ 1award additional 0.5 mark for SAQ1 asa whole (due to generic narrowness of the markscheme).

SAQ2cCredit 0.5 mark where there isheadroom to do so

SAQ2(c) had 3 marks available and an all-

Provider pass rate of 21.9%. The Boardconsidered the relatively low pass mark for allProviders for this question, with 4 Providersachieving pass rates below 11%. The Boardagreed they were happy that the question hadappropriate syllabus coverage and the answerwas technically correct. In relation to 2(c)however, the Board agreed to award allcandidates 1 additional mark where there washeadroom to do so with reference to thenarrowness of the mark scheme in relation to

the second bullet point on mark scheme,including reliability affected by situation re:Henrys wife.

The Board decided an additional 1 mark wouldbe award to all candidates in relation to 2c,where there was headroom to do so.

SAQ2dCredit 0.5 mark where there isheadroom to do so

SAQ2(d) had 2 marks available and an all-Provider pass rate of 44.9%. The Board notedthe weaker pass rates throughout the question.The mark scheme was comprised entirely of half


39/53

39

marks, thus candidates would need 3 half marksto pass. The Board agreed an additional 0.5mark should be given to all candidates, wherethere was headroom to do so, in respect ofbullet point 1 in the mark scheme, WRU can put

written questions about the experts report asthis point should have attracted one mark givenits centrality in the answer.

The Board decided to award a further 0.5 markto all candidates, where there is headroom to doso in relation to SAQ 2(d).

SAQ2Credit 0.5 mark where there isheadroom to do so

SAQ2award an additional 0.5 mark for SAQ2as a whole (due to narrowness of markscheme).

SAQ3aCredit 0.5 mark where there isheadroom to do so

SAQ3(a) had 2 marks available with an all-

Provider pass rate of 38.6%. The Board notedappropriate syllabus and the revisions to themark scheme after the consideration offeedback. On the basis that neither 3(a) or 3(b)operated as effective gateways to enablecandidates to demonstrate what they knew, theBoard agreed to award an additional 0.5 markfor all candidates in respect of 3(a) where therewas headroom to do so.

The Board decided to award an addition 0.5mark to all candidates in respect of 3(a) wherethere was headroom to do so.

SAQ3bCredit 1 mark where there isheadroom to do so

SAQ3(b) had 2 marks available with an all-Provider pass rate of 24.6%. The Board agreedto award an additional 1 mark, in respect on3(b), for all candidates where there washeadroom to do so for the same reasons asstated above in SAQ3(a).

The Board decided to award an additional 1

mark to all candidates in respect of 3b wherethere was headroom to do so

SAQ3eCredit 0.5 mark where there isheadroom to do so

SAQ3(e) had 2 marks available with an all-Provider pass rate of 42.7%. The Boarddiscussed the mark scheme and commentsfrom Providers. It was agreed that in respect of3(e), an additional 0.5 mark would be awardedto all candidates, where there was headroom todo so, for the point in the mark scheme relatingto point (i) Permission to make an additionalclaim is not required provided. On reflection the


40/53

40

Board agreed that this should have attracted 1mark given its centrality to the answer.

The Board decided that an additional 0.5 markwould be awarded to all candidates where there

was headroom to do so.

SAQ4aCredit 0.5 mark where there isheadroom to do so

SAQ4(a) had 2 marks available with an all-Provider pass rate of 42.1%. The Boarddiscussed the comments from Providers andthe mark scheme for the question. It was agreedthat an additional 0.5 mark would be awarded toall candidates where there was headroom to doso. The Board felt the inflation point shouldhave attracted one mark given its centrality tothe answer. This intervention brought the passrate from 42.1% pre scale to 57.6% post scale.

The Board decided to award an additional 0.5mark to all candidates where there is headroomto do so.

SAQ4b No intervention

SAQ4(b) had 2 marks available with an all-Provider pass rate of 36.7%. The Boarddiscussed at length the range of performanceacross all Providers, noted that there were noProvider comments for this question and agreedthe mark scheme was very generous in respectof 4(b), concluding that no interventions werewarranted.


SAQ4c No intervention

SAQ4c had 1 mark available with an all-Provider pass rate of 26.5%.


There were no other global or generic issues relating to the SAQ assessment.

12.4 SAQ post-scale first sit pass ratesall-Providers 2011/12 to 2014/15


SAQ post-scale All-Provider passrate

65.0% 67.8% 61.5% 73.5%

The post-scale SAQ first sit pass rate for 2014/15 of 65% shows the impact of the CEBintervention in raising the pre-scale pass rate of 49% by 16%, in effect a 32% increase

as a proportion of the pre-scale pass rate.


41/53

41

12.5 SAQ pre and post-scale first sit pass rates across all Providers 2014/15

The graph shows the effect of the nine SAQ interventions detailed above at 12.3.Providers are ranged in order of pass rates (high to low) pre-scale. Hence Provider 1had both the highest pre-scale SAQ pass rate, but Providers 9 to 11 benefitted morefrom the effects of the interventions with pass rates for their cohorts rising byapproximately 20%. Scaling had only a modest effect on the range of pass ratesacross Providers. It was 40% pre-scale and 35% post-scale.


25

35

45

55

65

75

85

95

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider



SAQ 2014/15 pre-scale


30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10

Passrate

%

Provider

Civil Litigation SAQ Post-scale

Pass rate for First Sit 2011/12 to 2014/15

% pass 2011/12

% pass 2012/13

% pass 2013/14

% pass 2014/15


42/53

42

Providers are ranked 1 to 10 based on their 2011/12 post-scale first sit Civil LitigationSAQ pass rates, showing how their 2012/13, 2013/14 and 2014/15 pass ratescompare. Hence Provider 1 achieved a pass rate of 81% in 2011/12, 53.7% in2012/13, and 74.6% in 2013/14 and 77.2% in 2014/15. Provider 2, by contrast, hasseen a very worrying year-on-year decline in cohort performance from 90% in

20111/12, to 59.3% in 2014/15. Four Providers reported in this graph recorded cohortpass rates for the Civil Litigation SAQs that were higher than their 2013/14 pass rates,with Provider 6 showing a 13% rise. Provider 5 recorded the biggest decline in passrates from 2013/14, a drop of over 24%. Provider 1 has the best 4-year average passrate at over 71%, whilst the weakest, on average, over the same period is Provider 10with an average pass rate just over 47%. Note that results for the BPP Manchestercohort are not reflected in this table as there is no data pre 2013/14, and results forKaplan have been excluded, as their last intake was in 2013/14.

12.7 MCQ and SAQ first sit post-scale pass rates by Provider 2014/15 high to low

This table compares 2014/15 Civil Litigation MCQ and SAQ all-Provider post-scalepass rates. The order of Providers is determined by a Providers combined post-scalepass rate for the MCQ and SAQs taken together (i.e. the percentage of candidates atthe Provider passing both sections). The data shows Provider 1 with the strongestperformances in both MCQs and SAQs. Provider 2 is the only Provider to have ahigher pass rate for the SAQ assessments compared with the MCQs (0.2% higher).Elsewhere the pattern is for a much stronger showing by cohorts in the MCQs, notablyfor Providers 5 and 6 where it is 17% and 13% respectively. In general the pattern ofSAQ performance tracks that for MCQs albeit at a lower level of pass rate.

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider






43/53

43

13. 2014/15 first sit post-scale results in Civil Litigation etc.: combined MCQ andSAQ results

13.1 Combined MCQ and SAQ all Provider first sit post-scale pass rates and gradeboundary distribution 2014/15


Combined (MCQ and SAQ) All Provider post-scaleresults

Total number sat (AllProviders)

1595 %






Competent 214 13.4%


It should be noted that the CEB does not review, adjust or confirm the combined pass

rate for any of the three knowledge areas. The combined results are simply the productof the relevant confirmed MCQ and SAQ marks, which may or may not have beensubject to adjustment by the CEB at an earlier stage. No further scaling is applied onceMCQ and SAQ marks are combined to produce the pass rates for a knowledge areaas a whole. Hence the combined knowledge area marks for cohorts and individualcandidates, including grade boundaries, lie where they fall.

13.2 Combined MCQ and SAQ first sit pre and post-scale pass rates by Provider2014/15

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider

Civil Litigation First Sit 2014/15 Combined(MCQ and SAQ) Pre and Post scale pass rates by Provider

Combined 2014/15 pre-scale

Combined 2014/15 post-scale


44/53

44

The graph shows the combined effect of the interventions in respect of both the MCQsand SAQs across all Providers. Providers are ranged in order of pre-scale combinedpass rates (high to low). Hence Provider 1 had the highest pre-scale and post-scalepass rates. The intervention typically resulted in an 15% uplift, but, somewhat counter-intuitively, the greatest impact was on Provider 1 where the increase was 20%), with

the result that the interventions actually increased the range between the strongestand weakest cohort pass rates from 37% pre-scale to 43% post-scale

13.3 Combined MCQ and SAQ post-scale first sit pass ratesall Providers 2011/12to 2014/15



58.0% 57.4% 56.2% 68.0%

The post-scale combined first sit pass rate for 2014/15 of 58% is largely in line withthe 2012/13 outcome for first sit candidates, 4% below that for Criminal Litigation2014/15.


Providers are ranked according to their combined 2014/15 post-scale first sit passrates in Civil Litigation. Hence Provider 1 has the highest post-scale first sit pass ratebut not the highest percentage of Outstanding candidates (see Provider 2 with22.3%). The distribution of Not Competent candidates does however follow the orderof Providers in the above chart.

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

1 2 3 4 5 6 7 8 9 10 11

%

studentspercategory

Provider

Civil Litigation First Sit 2014/15Post scale grade boundaries by Provider

Not Competent Competent Very Competent Oustanding


45/53

45

13.5 Combined MCQ and SAQ first sit post-scale pass rates trend analysis 2011/12to 2014/15

Providers are ranked 1 to 11 based on their 2011/12 post-scale first sit combined CivilLitigation pass rate, showing how their 2012/13, 2013/14 and 2014/15 pass ratescompare. Hence Provider 1 achieved a 71% pass rate in 2011/12, 50% in 2012/13,47% in 2013/14, and 54% in 2014/15. Five Providers recorded a higher combinedpass rate for 2014/15 as against 2013/14 (Provider 6 up by over 10%), whilst Provider8 recorded the sharpest decline at over 13%. Provider 9 is the only Provider to recorda year-on-year decline in combined pass rates across all 4 cycles of first sitassessment. Provider 2 has the best 4-year average pass rate at over 64%, whilst theweakest, on average, over the same period is Provider 10 with an average pass ratejust over 45%. Note that results for the BPP Manchester cohort are not reflected in thistable as there is no data pre 2013/14, and results for Kaplan have been excluded, astheir last intake was in 2013/14.

35.0

40.0

45.0

50.0

55.0

60.0

65.0

70.0

75.0

80.0

1 2 3 4 5 6 7 8 9 10

Passrate%

Provider

Civil Litigation First Sit Combined (MCQ and SAQ)

Post scale pass rates by Provider 2011/12 to 2014/15%pass 2011/12

% pass 2012/13

% pass 2013/14

% pass 2014/15


46/53

46


The four-year trend analysis shows a slight recovery in the level of Outstandingclassifications with a corresponding drop in the Competent grouping.

14 First sit modes of assessment and subject areas 2014/15

14.1 MCQ first sit post-scale pass rates for 3 CEB areas 2014/15 compared

This graph compares the post-scale first sit MCQ pass rates for the three knowledgeareas across all Providers 2014/15. Providers are ranked according to the average oftheir post-scale MCQ pass rates across the three knowledge areas. Hence Provider 1had the highest average cohort pass rate across all three MCQ assessments andProvider 11 the lowest average across all three. All Providers recorded their highest

0.0

10.0

20.0

30.0

40.0

50.0


%s

tudents

Category

Civil Litigation First Sit Post scale


2011/12

2012/13

2013/14

2014/15

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider

First Sit 2014/15 MCQ post scale

Pass rate for knowledge areas by Provider

Ethics MCQ 2014/15 post scale

Crime MCQ 2014/15 post scale

Civil MCQ 2014/15 post scale


47/53

47

MCQ pass rates in respect of Professional Ethics and all recorded their lowest MCQpass rates in respect of Civil Litigation. The gap between a Providers pass rate inProfessional Ethics and its pass rate in Civil Litigation tends to be more marked inrespect of the weaker cohorts. For Provider 1 the gap is 9%, whilst for Providers 8 and10 the gap is over 33%. Provider 2 slightly bucks this trend with a surprisingly poor

showing in Civil Litigation where the pass rate is 28% below that for ProfessionalEthics. Provider 1 has an average MCQ pass rate of over 93%, compared with 69.8%for Provider 11 showing a 23% range between the strongest MCQ Providerperformance and the weakest.

14.2 SAQ first sit post-scale pass rates for 3 CEB areas 2014/15 compared

This graph compares the post-scale first sit SAQ pass rates for the three knowledgeareas across all Providers 2014/15. Providers are ranked according to the average oftheir post-scale SAQ pass rates across the three knowledge areas. Hence Provider 1had the highest average cohort pass rate across all three SAQ assessments andProvider 11 the lowest average across all three. Unlike the MCQ data reported at 14.1there is a more complex picture in terms of Provider performance in respect of theSAQ assessments. For three Providers their strongest cohort performance was inrespect of the Professional Ethics SAQ, for five it was in respect of the Criminal

Litigation SAQ and for three the Civil Litigation SAQ. In terms of weakest SAQperformance, for five Providers this was in respect of Professional Ethics, for two itwas in respect of Criminal Litigation, and for four it was in respect of Civil Litigation.

It is also instructive to look at the average SAQ pass rate in the two litigation modulesfor each Provider and compare this with the Providers pass rate in respect of theProfessional Ethics SAQ. Provider 1saverage pass rate across the SAQs in the twolitigation modules is 85.4%, compared with a Professional Ethics SAQ pass rate of81.6%, hence a 3.8% negative variance in respect of the Professional Ethics SAQ.Across all 11 Providers, applying the same methodology, there is an average negativevariance of 5.3% between the average of their litigation module SAQ pass rates and

20

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

Passrate%

Provider

First Sit 2014/15 SAQ post scale

Pass rate for knowledge areas by Provider

Ethics SAQ 2014/15 post scale

Crime SAQ 2014/15 post scale

Civil SAQ 2014/15 post scale


48/53

48

their pass rates for the Professional Ethics SAQ. Significantly, however, there arethree Providers where the negative variance is very much higher: Provider 7 (-24.4%);Provider 8 (-24.7%) and Provider 10 (-21.7%). If these three outlier Providers areremoved from the equation, the remaining eight Providers actually record a positivevariance of 1.5% in respect of their pass rate in Professional Ethics SAQ compared

with the average of their litigation module SAQ pass rates, suggesting the overallpicture is being skewed by a very poor Professional Ethics SAQ cohort performanceat three Providers.Again the range between a Providers best SAQ pass rate and theirworst is typically 15%. For Providers 7, 8 and 10 the range is between 22% and 26%because of their poor performance in the Professional Ethics SAQs. In terms of therange of Provider cohort performance in SAQs as a whole it is striking that there is a34% gap between the average SAQ pass rate for Provider 1 and that for Provider 11.

14.3 Combined MCQ and SAQ 2014/15 first sit post-scale pass rates for 3 CEB areas2014/15 compared

This graph compares the post-scale first sit combined pass rates for the threeknowledge areas across all Providers for 2014/15. Providers are ranked according tothe average of their post-scale combined pass rates across the three knowledgeareas. Hence Provider 1 had the highest average pass rate across the three combined

assessments, with pass rates of around 82% for all three modules (the best cohortacross all three). By contrast Provider 11 has an average combined pass rate acrossthe three modules of 44.8%, some 37% below that of Provider 1. The graph shows areasonably clear trend of declining pass rates from Provider 1 to Provider 11, but thesignificantly poor performances by Providers 7, 8 and 10 in respect of ProfessionalEthics (effectively the SAQ element) is apparent from the graphs, as is the fact thatProviders 4, 6 and 9 recorded their highest combined pass rates in respect ofProfessional Ethics.

30

40

50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11

central examination board report august 2015

Documents