Summers are predictably busy times for exam boards. Teachers lend us their expertise to mark the 7 million exam papers we receive, and colleagues oversee the logistics so that everything is marked well ahead of results day. For the subject team, arguably our most important work all year is the grade awarding process.
This summer has seen a number of pieces written on the awarding. Schoolsweek ran two articles, one from our own Alex Scharaschkin and a nice piece on comparable outcomes by Phil Beech of Ofqual. The process we all follow for awarding hasn’t changed much since Will Woodward’s piece from 2003. We still blend the qualitative and quantitative evidence when making our judgements, and use tick charts to evaluate the boundary decisions. Thankfully, technological advances mean we can run through the impact of boundary changes in real time.
I thought it would be interesting to look more specifically at our GCSE maths awards. A couple of things stood out this year for us; a change in entry patterns, and a drop in a key foundation tier boundary.
Entry patterns are changing
The proportion of people taking GCSE over the age of 16 continues to rise. In part, this will be down to funding rules around post-16 study. I’ll leave the debate about whether GCSE is the right qualification for these students to others, but there’s no doubt that it’s having an impact.
Whilst the increase in 17 and 18 year olds was expected, we also saw an increase in the number of 19+ year olds taking our GCSE, as shown in Table 1. I’m not sure what’s driving this, and we’ll be working with FE institutions to understand more about this group, as we want to ensure anyone teaching the GCSE as a resit over 1 year is well supported.
Our high proportion of these older students skews some of the headline figures. Looking at the JCQ figures for all students, 63.3% of students gained A*– C (up from 62.4% the year before). If we compare this to our specifications (linear; 60.0%, unitised; 52.3%), the casual observer might think that it’s harder to get the all important “good pass” with AQA than with other boards.
Of course, this is where comparable outcomes come in, and is the reason boundaries are only set after the papers are marked and the performance of the cohort can be reviewed. As you’d imagine, students resitting GCSE after 16 are doing so because they weren’t satisfied with their grade the first time around. As such, these older students aren’t reflective of the full ability range, and cause the skew in the headline figures.
As part of the comparable outcomes process for awarding, we pay particular attention to students who are aged 16 (Year 11), and where we know something about their prior attainment. For this group, the proportion of students gaining A*-C on our linear specification was 69.0%, well above the published 60.0%.
Although the entry patterns are shifting, Ofqual’s research into variability at school level indicates that mathematics was relatively stable this year. This is perhaps not surprising, given that mathematics is a compulsory subject and so we have the whole cohort entering every year.
Grade boundary changes
Most teachers appreciate that grade boundaries will fluctuate from year to year, as setting examinations of exactly the same demand is incredibly difficult. That said, it’s a responsibility of exam boards to produce papers that will fairly differentiate between students of all abilities, and part of this is offering an approach (in terms of language, layout and overall demand) that students can become familiar with prior to their exam so that they know what to expect on the day.
Across the two papers of our higher tier, the demand of the qualification was very much in-line with the standard of 2014, as evidenced by Table 2.
Even though this appears stable, performance shifted across the papers, with a Grade A being awarded four marks higher than in 2014 on Paper 1, and 5 marks lower than in 2014 on Paper 2. I can remember speaking to a number of teachers (including Bruno Reddy, who I think had taken the paper himself) about the challenge in the 2014 Paper 1, so it was pleasing to see the boundary go up this year, indicating the paper hadn’t been quite as demanding.
On the foundation tier, the story is a little different. The Paper 1 boundaries were exactly the same as in 2014, (46/70 marks for a C, and 25/70 marks for an F), but we experienced a significant drop in performance on Paper 2, which had an impact in the grade boundaries, as shown in Table 3.
To quantify this, for Paper 2, the 2014 Grade C boundary was 76/105, which 43.77% of students secured. This year, if we’d taken forward the mark of 76, only 16.13% of students would have got a Grade C on that paper. The actual boundary mark of 62/105 was achieved by 40.67% of the cohort, much more in-line with the 2014 performance.
A boundary that moves down often causes less concern than one that goes up, but it’s something we’ll be reflecting on as that particular paper didn’t perform exactly how we intended it to. There could be a number of reasons for this, most likely a couple of parts of the specification that are poorly understood by candidates appeared, but we are keen to understand this in detail.
The onset of electronic marking of papers means we now have item level data (the performance of each question on the paper). We’ll use this to identify which questions didn’t perform as we expected them to, and build this in to the design of future papers. Whilst making awarding decisions on this paper was straightforward, and there’s no question that the award was robust, we’d still like to have more consistent boundaries year on year, in the interests of not phasing students with a particularly hard paper in any one year.
For those of you thinking of using this paper as a mock, it’s worth bearing in mind that this is the lowest we’ve ever set a boundary for this specification, so the paper may be a little tougher for students and, of course, in future the boundary marks could be higher.
I hope this summary has been of use. This is our second blog on maths assessment, following Andrew Taylor’s piece on balancing recall and reasoning. We’re always happy to try and explain more about our work, so if you have any suggestions for future posts, do please let us know.
Ben Stafford, Maths Qualifications Manager at AQA