Cinneadh
Mr X and the State Examinations Commission
Ó Oifig an Choimisinéara Faisnéise
Cásuimhir: 150363
Foilsithe
Teanga: Níl leagan Gaeilge den mhír seo ar fáil.
Ó Oifig an Choimisinéara Faisnéise
Cásuimhir: 150363
Foilsithe
Teanga: Níl leagan Gaeilge den mhír seo ar fáil.
Whether the SEC was justified in refusing access to records relating to the 2015 Leaving Certificate Mathematics Papers 1 and 2 at Higher and Ordinary Levels, namely, the original breakdown of grades obtained by examiners marking a representative sample of scripts based on the original marking scheme which was given to examiners at the marking conference
Conducted in accordance with section 22(2) of the FOI Act by Elizabeth Dolan, Senior Investigator, who is authorised by the Information Commissioner to conduct this review
07 June 2016
In a request dated 5 August 2015, the applicant sought access to records relating to the 2015 Leaving Certificate Mathematics Papers 1 and 2 at Higher and Ordinary levels, namely, the original breakdown of grades obtained by examiners marking a representative sample of scripts based on the original marking scheme which was given to examiners at the marking conference. In a decision dated 21 August 2015, the SEC refused the request under section 41(1) of the FOI Act and alternatively under section 30(1)(a) of the FOI Act, but noted that the breakdown of grades for the results of the 2015 Leaving Certificate at Higher and Ordinary Levels was available on its website and that the marking schemes were due to published on the website shortly. The SEC also noted that a more detailed analysis of candidate answering in the examinations would be available in the Chief Examiner's report on Leaving Certificate Mathematics that would be published on the website early in 2016.
On 18 September 2015, the applicant sought an internal review of the SEC's decision. On internal review, the SEC affirmed its original decision. In an application dated 22 October 2015, the applicant sought a review by this Office of the SEC's decision.
I have now completed my review in accordance with section 22(2) of the FOI Act. In carrying out my review, I have had regard to the submissions made by the SEC and the applicant and to the relevant provisions of the FOI Act. I have also examined the records forwarded to my Office for the purposes of my review. I have decided to conclude this review by way of a formal, binding decision.
This review is concerned solely with the question of whether the SEC was justified in refusing access to the records sought relating to the 2015 Leaving Certificate Mathematics Papers 1 and 2 at Higher and Ordinary levels under the FOI Act. It is not within my remit to consider how the SEC carries out its functions generally.
Section 22(12)(b) of the FOI Act provides that a decision to refuse to grant access to a record "shall be presumed not to have been justified unless the head concerned shows to the satisfaction of the Commissioner that the decision was justified." However, I note that, as a general rule, the actual or perceived reasons for a request must be disregarded in deciding whether to grant or refuse an access request under the FOI Act (section 13(4) of the FOI Act refers).
The SEC's position
In its decision, the SEC relied primarily on section 41(1) of the FOI Act as a basis for refusing the applicant's request. Section 41(1) is a mandatory exemption provision that applies where the disclosure of a record is prohibited by an enactment not specified in the Third Schedule to the FOI Act or where the non-disclosure of the record is authorised by any such enactment in certain circumstances and the case is one in which the head would, pursuant to the enactment, refuse to disclose the record. In support of its claim for refusal, the SEC referred to the Education Act 1998 in conjunction with the State Examination Commission (Establishment Order 2003), S.I. No. 373/2003, which gives the Commission all such powers as are necessary for or are incidental to the performance of its functions. The SEC's functions include making arrangements for the marking of work presented for examination and also issuing the results of examinations. According to the SEC, it has determined within the exercise of its powers that the aspects of examiners' work which relate to the training of individual examiners, the quality assurance of their work, and the standard-setting process for the examination as a whole need to remain confidential in the interest of maintaining the integrity and fairness of the examinations and public confidence in them.
In its submissions to this Office, however, the SEC has shifted the focus of its case to explaining the examinations system and why confidentiality is regarded as necessary for the effective conduct of the marking process; in doing so, the SEC has strengthened its claim for exemption under section 30(1)(a) of the FOI Act. In addition, it has invoked two new grounds for refusal, namely, section 29 of the FOI Act, which relates to the deliberative processes of an FOI body, and section 30(1)(b) of the FOI Act, which relates to the performance of an FOI body relating to management.
Section 30(1)(a)
Section 30(1)(a) provides that an FOI body may refuse access to a record if it considers that access could reasonably be expected to prejudice the effectiveness of tests, examinations, investigations, inquiries or audits conducted by or on behalf of an FOI body or the procedures or methods employed for the conduct thereof. In arriving at a decision to claim a section 30 exemption, a decision maker must, firstly, identify the potential harm to the functions covered by the exemption that might arise from disclosure and, having identified that harm, consider the reasonableness of any expectation that the harm will occur. To establish the reasonableness of the decision to my satisfaction, it is essential that the decision maker explain how and why he or she believes release of these particular records will give rise to the harm envisaged. Section 30(1) is subject to a public interest test under section 30(2).
In this case, the SEC acknowledges that there are inevitable fluctuations in the difficulty of newly set examination questions from one year to another, particularly as efforts are made to address the "predictability problem" through innovation and thereby to encourage the development of higher-order thinking skills. Accordingly, each examination cycle necessarily involves a standard-setting or calibration process in order to ensure fairness, accuracy, and consistency. The SEC explains that typically the standard-setting process for a curriculum-based examination such as the Leaving Certificate or the A-levels in the UK is based on an assumption termed the "similar cohort adage", i.e. that the level of achievement of a large cohort of students will not change radically from one year to the next in the absence of some significant systemic intervention. For further information on this assumption, the SEC refers to the following source: Newton, P.E. (2005) Examination Standards and the Limits of Linking. Assessment in Education: principles, policy & practice, 12(2), 105-123, cited in Predictability in the Irish Leaving Certificate Baird J. et al, Oxford University Centre for Educational Assessment and Queens University Belfast (2015) https://www.examinations.ie/about/Predictability-Overall-Report.pdf.
In Ireland, where the Leaving Certificate grade boundaries are fixed by regulation, the standard-setting process is done in relation to the scoring system through iterative or repeated trialling of the marking scheme. The SEC states:
"The standard-setting process involves the application of the expert professional judgment of senior examiners in the assessment of the response of the candidate cohort to the questions set each year. The potential for variations in the candidate cohort from year to year is recognised, and so minor variations in the grades achieved do not give rise to change to the marking scheme, though that is not to take away from the essential reliance on the similar cohort adage . . . ."
In addition, individual examiners receive training during the standard-setting process and their work is monitored with the aim of avoiding "inappropriate divergences from one examiner to another or from the intended standard".
The SEC also describes the fluid and rapid nature of the process:
"Given the iterative nature of the standard-setting process, all assessments of an individual exam script produced by a candidate are tentative and fluid until the standard-setting process is complete, at which point the chief examiner confirms that he or she is satisfied and issues an instruction, authorising individual examiners to complete the marking. The standard-setting system, then, involves an extensive deliberative process, which works towards a final marking scheme. The provisional or tentative script assessments and grade distributions are important communications in that deliberative process.
Critically, the standard-setting process must happen quickly. To allow for an orderly transition of students from second to third level each autumn, the marking of the June-time Leaving Certificate examinations must be completed within a 26 day period from the date of the main conference . . . . The senior team must work rapidly, and their practice involves intensive discussion of multiple options, the trialling of options on scripts in their possession, and multiple exchanges with examiners for speedy wider assessment of such options."
Thus, according to the SEC, the standard-setting process requires flexibility and resilience in order to be effective.
The SEC argues that release of the requested records could reasonably be expected to prejudice the effectiveness of the Leaving Certificate examinations, to prejudice the effectiveness of the standard-setting procedures employed in the conduct of those examination, as well as to have a significant adverse effect on the performance by the SEC of its functions relating to management. The management functions is described as "the evaluation and enhancement of quality in the conduct of the Leaving Certificate examinations, and the devising of new methods for the conducting of those examinations through the innovation of new forms of question designed to encourage the development of higher-order thinking skills". It argues that any meaningful comparison between the tentative grade distribution based on a draft marking scheme and the actual grade distribution in the relevant examination requires a full understanding of the relevant standard-setting process, disclosure of which would be seriously prejudicial because it would undermine the authority of the actual marking scheme and so of the examination generally. It says that "finality is an important factor in the authority of the marking scheme and of the examination", whereas disclosure would be likely to encourage "spurious challenges to marks awarded" and promote "spurious controversy about those awards".
The SEC also argues: "The release of the tentative grade distribution at any given stage of the standard-setting process points the way to the release of tentative grade distributions at each stage in that process, creating an apparent multiplicity of grade-distributions for a given examination, which in the opinion of the SEC CEO could reasonably be expected to tend towards minimising the number of revisions to the marking scheme, and so to the prejudicing of the effectiveness of the standard-setting process employed in the conduct of the Leaving Certificate mathematics examination in particular and the Leaving Certificate generally." Moreover, as "the tentative grade distributions at the sample stage are of no inherent information value in themselves", the SEC contends that their release would not fulfil the public interest in the achievement of greater openness and accountability or in strengthening accountability.
In contrast, according to the SEC, the prompt annual publication in full of the test instrument "allows members of the public to form their own view as to the standard of questions and the way in which marks were awarded and to undertake comparative analysis from year to year". The SEC acknowledges that it does not facilitate testing of the similar cohort assumption, but says that neither would the requested records: "it is widely accepted in assessment literature that examinations such as the Leaving Certificate are not capable of tracking gradual changes in cohort achievement over time". The SEC also notes that the public interest is further served by the publication of periodic reports from the Chief Examiners, "which provide detailed analysis of candidate achievement in the subject concerned, and include performance data and observations gathered from the individual examiners' reports as to candidate achievement in relation to particular knowledge and skills, and the examiners' observations as to improvement or deterioration over time in various aspects of the candidate work".
The applicant's position
The applicant suggests that confidentiality is merely a "smokescreen" which masks the flaws in "unsuitable papers" produced by the SEC which cause "unnecessary distress to students and parents". As an example, he points to the criticisms of the Ordinary Level Maths paper for 2015 and the adjustments to the marking scheme that were made to address the concerns expressed. He also argues that the adjustments made to the final marking scheme to ensure largely consistent results from year to year render it impossible to make judgments about the general standard of maths in Ireland. In particular, he contends that disclosure of the original breakdown of grades is required to shed light on whether the new Project Maths initiative is the "great success" that it is claimed to be. He describes the adjustment process as a form of "manipulation" that "undermines the original aims of the new course to improve standards in maths". In his view, greater transparency would enhance the public's faith in the examinations system, particularly in relation to concerns about the standard of maths education in Ireland.
Conclusions
In light of the SEC's detailed submissions identifying and describing potential harm to its functions, I accept that disclosure of the requested records could reasonably be expected to prejudice the effectiveness of the Leaving Certificate examinations and the standard-setting process employed for their conduct. The original grade distribution is derived from the tentative marking of exam scripts by examiners who have yet to complete their training. Moreover, it is based on the original draft marking scheme which has yet to be adjusted to take into account the fluctuations in the difficulty of the exam questions arising from the perceived need to avoid predictability in the setting of the questions. Thus, the original grade distribution is not a reliable indicator of student performance and it is reasonable to expect that its disclosure would undermine the finality and authority of the marks actually awarded following completion of the standard-setting process.
Moreover, I accept that the standard-setting process requires confidentiality in order to be effective. As described by the SEC, there is a need for flexibility and resilience in circumstances where, as described above, marking must be completed within a few weeks. The SEC is the body charged by the Minister with the function of making arrangements for the marking of work presented for examination. The arrangements that the SEC has made involve a confidential calibration process. I note that it is accepted in other contexts, such as performance management, that calibration requires confidentiality in order to be effective. In any event, based on the submissions of the SEC, I accept that the confidentiality of the process facilitates the flexibility and resilience required for the effective and timely marking of the Leaving Certificate examinations. It follows that disclosure could reasonably be expected to prejudice that process. I find, therefore, that subject to consideration of the public interest test at section 30(2), the section 30(1)(a) exemption applies.
The Public Interest
The purpose of the standard-setting or calibration process is to ensure fairness, accuracy, and consistency. The need arises from the efforts to be innovative in the setting of exam questions in order to avoid predictability and to promote higher-order thinking skills. Moreover, during the process, the work of individual examiners is monitored to ensure that the appropriate standard is being adhered to, i.e. to ensure that individual examiners are not marking exam questions too harshly or too leniently. As the SEC concedes, the process is not an exact or mechanical one. However, the options for standard-setting in Ireland are constrained given that the Leaving Certificate grade boundaries are fixed by regulation e.g. an "A1" is always 90%-100%. I do not believe that it would serve the public interest to undermine the process established by the SEC for the effective marking of examinations. Moreover, it seems to me that the value to be gained from the disclosure of the original grade distribution is minimal in the circumstances given both the need for further training of the examiners at that point and the absence of adjustments to take into account the fluctuations in the difficulty of the exam questions as compared to previous years. The SEC has explained that the benchmark used for making the adjustments is the assumption known as the "similar cohort adage". The assumption may not be infallible and again I note that the process is not exact or mechanical. However, I do not believe that it can seriously be disputed that there is a need for some mechanism for making adjustments to account for the differences in standards between papers (and individual examiners). On the contrary, according to the applicant's own arguments, certain papers may be regarded as "unsuitable", which itself points to the need for some form of calibration or standard-setting for fairness and consistency. If a paper is truly "unsuitable", then unadjusted marks which do not take this into account would not reveal any meaningful or reliable information about the standard of Maths education in Ireland.
On the other hand, I agree with the SEC that the public interest in openness, transparency, and accountability is served to a large extent by the material published each year, including the full test instrument, namely the questions set and the final marking scheme together with the final grade distribution. The publication of this material allows the public to see the level of difficulty involved and the manner in which the papers were ultimately marked, including where and how marks were allocated and how this compares to previous years. Moreover, while the reliance on the similar cohort adage would generally seem to prevent any "radical" changes in the results from year to year, the material does allow for comparative analyses to be undertaken as reflected by the commentary in the media and by organisations such as the Irish Business and Employers Confederation (IBEC) that follows its publication. I am satisfied that section 30(1)(a) of the FOI Act applies and that, on balance, the public interest would not be better served by release of these records.
Although the SEC's claim for exemption under section 30(1)(b) is closely related to its section 30(1)(a) claim, I do not consider it necessary in the circumstances to address this or the additional claims for exemption made under sections 29 and 41(1) of the FOI Act.
Having carried out a review under section 34(2) of the FOI Act, I hereby affirm the decision of the SEC in this case.
Section 24 of the FOI Act sets out detailed provisions for an appeal to the High Court by a party to a review, or any other person affected by the decision. In summary, such an appeal, normally on a point of law, must be initiated not later than four weeks after notice of the decision was given to the person bringing the appeal.
Elizabeth Dolan
Senior Investigator