Cinneadh
Mr X and The State Examinations Commission
Ó Oifig an Choimisinéara Faisnéise
Cásuimhir: 160107
Foilsithe
Teanga: Níl leagan Gaeilge den mhír seo ar fáil.
Ó Oifig an Choimisinéara Faisnéise
Cásuimhir: 160107
Foilsithe
Teanga: Níl leagan Gaeilge den mhír seo ar fáil.
Whether the SEC was justified in refusing access to records relating to the 2015 Leaving Certificate Ordinary Mathematics examination (Papers 1 and 2), namely, the original, draft marking schemes prior to the finalisation of the marking schemes and all other documentation informing adjustments to the schemes by the Chief Examiner
Conducted in accordance with section 22(2) of the FOI Act by Elizabeth Dolan, Senior Investigator, who is authorised by the Information Commissioner to conduct this review
07 June 2016
In a request dated 19 August 2015, the applicant sought access to records relating to the 2015 Leaving Certificate Ordinary Mathematics examination (Papers 1 and 2), namely, the original, draft marking schemes prior to finalisation of the marking schemes and all other documentation informing adjustments to the schemes by the Chief Examiner. In a decision dated 14 September 2015, the SEC refused the request under section 41(1) of the FOI Act and alternatively under section 30(1)(a) of the FOI Act, but noted that the final version of the marking scheme and a manual regarding the production of marking schemes were available on its website. The SEC also noted that a more detailed analysis of candidate answering in the examinations would be available in the Chief Examiner's report on Leaving Certificate Mathematics that would be published on the website early in 2016.
On 17 September 2015, the applicant sought an internal review of the SEC's decision. On internal review, the SEC affirmed its original decision. In an application received on 3 March 2016, the applicant sought review by this Office of the SEC's decision.
I have now completed my review in accordance with section 22(2) of the FOI Act. In carrying out my review, I have had regard to the submissions made by the SEC and the applicant and to the relevant provisions of the FOI Act. I have also examined the records forwarded to my Office for the purposes of my review. I have decided to conclude this review by way of a formal, binding decision.
This review is concerned solely with the question of whether the SEC was justified in refusing access to the records sought relating to the 2015 Leaving Certificate Ordinary Mathematics Papers 1 and 2. It is not within my remit to consider how the SEC carries out its functions generally.
Section 22(12)(b) of the FOI Act provides that a decision to refuse to grant access to a record "shall be presumed not to have been justified unless the head concerned shows to the satisfaction of the Commissioner that the decision was justified."
In its decision, the SEC relied primarily on section 41(1) of the FOI Act as a basis for refusing the applicant's request. Section 41(1) is a mandatory exemption provision that applies where the disclosure of a record is prohibited by an enactment not specified in the Third Schedule to the FOI Act or where the non-disclosure of the record is authorised by any such enactment in certain circumstances and the case is one in which the head would, pursuant to the enactment, refuse to disclose the record. In support of its claim for refusal, the SEC referred to the Education Act 1998 in conjunction with the State Examination Commission (Establishment Order 2003), S.I. No. 373/2003, which gives the Commission all such powers as are necessary for or are incidental to the performance of its functions. The SEC's functions include ensuring the preparation of examinations papers and other examination materials, making arrangements for the marking of work presented for examination, and issuing the results of examinations. According to the SEC, it has determined within the exercise of its powers that the aspects of examiners' work which relate to finalising the marking scheme, training of individual examiners, the quality assurance of their work, and the standard-setting process for the examination as a whole need to remain confidential in the interest of maintaining the integrity and fairness of the examinations and public confidence in them.
In its submissions to this Office, however, the SEC has shifted the focus of its case to explaining the examinations system and why confidentiality is regarded as necessary for the effective conduct of the marking process; in doing so, the SEC has strengthened its claim for exemption under section 30(1)(a) of the FOI Act. In addition, it has invoked two new grounds for refusal, namely, section 29 of the FOI Act, which relates to the deliberative processes of an FOI body, and section 30(1)(b) of the FOI Act, which relates to the performance of an FOI body relating to management.
Section 30(1)(a)
Section 30(1)(a) provides that an FOI body may refuse access to a record if it considers that access could reasonably be expected to prejudice the effectiveness of tests, examinations, investigations, inquiries or audits conducted by or on behalf of an FOI body or the procedures or methods employed for the conduct thereof. In arriving at a decision to claim a section 30 exemption, a decision maker must, firstly, identify the potential harm to the functions covered by the exemption that might arise from disclosure and, having identified that harm, consider the reasonableness of any expectation that the harm will occur. To establish the reasonableness of the decision to my satisfaction, it is essential that the decision maker explain how and why he or she believes release of these particular records will give rise to the harm envisaged. Section 30(1) is subject to a public interest test under section 30(2).
In this case, the SEC acknowledges that there are inevitable fluctuations in the difficulty of newly set examination questions from one year to another, particularly as efforts are made to address the "predictability problem" through innovation and thereby to encourage the development of higher-order thinking skills. Accordingly, each examination cycle necessarily involves a standard-setting or calibration process in order to ensure fairness, accuracy, and consistency. The SEC explains that typically the standard-setting process for a curriculum-based examination such as the Leaving Certificate or the A-levels in the UK is based on an assumption termed the "similar cohort adage", i.e. that the level of achievement of a large cohort of students will not change radically from one year to the next in the absence of some significant systemic intervention. For further information on this assumption, the SEC refers to the following source: Newton, P.E. (2005) Examination Standards and the Limits of Linking. Assessment in Education: principles, policy & practice, 12(2), 105-123, cited in Predictability in the Irish Leaving Certificate Baird J. et al, Oxford University Centre for Educational Assessment and Queens University Belfast (2015) https://www.examinations.ie/about/Predictability-Overall-Report.pdf.
In Ireland, where the Leaving Certificate grade boundaries are fixed by regulation, the standard-setting process is done in relation to the scoring system through iterative or repeated trialling of the marking scheme. The SEC states:
"The standard-setting process involves the application of the expert professional judgment of senior examiners in the assessment of the response of the candidate cohort to the questions set each year. The potential for variations in the candidate cohort from year to year is recognised, and so minor variations in the grades achieved do not give rise to change to the marking scheme, though that is not to take away from the essential reliance on the similar cohort adage . . . ."
In addition, individual examiners receive training during the standard-setting process and their work is monitored with the aim of avoiding "inappropriate divergences from one examiner to another or from the intended standard".
The SEC also describes the fluid and rapid nature of the process:
"Given the iterative nature of the standard-setting process, all assessments of an individual exam script produced by a candidate are tentative and fluid until the standard-setting process is complete, at which point the chief examiner confirms that he or she is satisfied and issues an instruction, authorising individual examiners to complete the marking. The standard-setting system, then, involves an extensive deliberative process, which works towards a final marking scheme. The provisional or tentative script assessments and grade distributions are important communications in that deliberative process.
Critically, the standard-setting process must happen quickly. To allow for an orderly transition of students from second to third level each autumn, the marking of the June-time Leaving Certificate examinations must be completed within a 26 day period from the date of the main conference . . . . The senior team must work rapidly, and their practice involves intensive discussion of multiple options, the trialling of options on scripts in their possession, and multiple exchanges with examiners for speedy wider assessment of such options."
Thus, according to the SEC, the standard-setting process requires flexibility and resilience in order to be effective.
The SEC argues that release of the requested records could reasonably be expected to prejudice the effectiveness of the Leaving Certificate examinations, to prejudice the effectiveness of the standard-setting procedures employed in the conduct of those examination, as well as to have a significant adverse effect on the performance by the SEC of its functions relating to management. The management functions is described as "the evaluation and enhancement of quality in the conduct of the Leaving Certificate examinations, and the devising of new methods for the conducting of those examinations through the innovation of new forms of question designed to encourage the development of higher-order thinking skills". It argues that the release of draft marking schemes and documentation informing the changes to those drafts would be seriously prejudicial because it would undermine the authority of the actual marking scheme and so of the examination generally. It says that "finality and certainty are very important factors in the authority of the marking scheme and of the examination", whereas release of the requested information "would present candidates with spurious alternative marking schemes by reference to which their individual scripts might be measured". According to the SEC, "[s]uch spurious alternative marking schemes would be the focus of intense interest, and would likely be used by challengers invalidly to propose counterfactual exam results, encouraging spurious challenges to marks awarded, and in the opinion of the SEC CEO thereby prejudicing the effectiveness of the Leaving Certificate". The SEC adds:
"[T]he release of draft marking schemes could reasonably be expected to tend towards minimising the number of revisions to the marking scheme, and so to the prejudicing of the flexibility and effectiveness of the standard-setting process. The tendency towards a less flexible standard-setting process will militate against innovation in the setting of questions and, in the opinion of the SEC CEO, can reasonably be expected to have a significant adverse effect on the performance by the SEC of its functions relating to management in the evaluation and enhancement of quality in the conduct of the Leaving Certificate examinations, and the devising of new methods for the conducting of those examinations."
In relation to the public interest, the SEC contends that the prompt annual publication in full of the test instrument "allows members of the public to form their own view as to the standard of questions and the way in which marks were awarded and to undertake comparative analysis from year to year". The SEC acknowledges that it does not facilitate testing of the similar cohort assumption, but says that neither would the requested records: "it is widely accepted in assessment literature that examinations such as the Leaving Certificate are not capable of tracking gradual changes in cohort achievement over time". The SEC also notes that the public interest is further served by the publication of periodic reports from the Chief Examiners, "which provide detailed analysis of candidate achievement in the subject concerned, and include performance data and observations gathered from the individual examiners' reports as to candidate achievement in relation to particular knowledge and skills, and the examiners' observations as to improvement or deterioration over time in various aspects of the candidate work".
The applicant's position
The applicant argues that, as the examination in question has already taken place, granting his request would not interfere in the marking process for students. He disputes that secrecy is required to uphold the integrity of the process, arguing that the reverse applies: "The best way of maintaining confidence is to have maximum transparency." He also argues that disclosure is required to determine if standards are improving in a particular subject in one year over the next. He says:
"Educational stakeholders and employers read the figures in this way, and comment on them with that assumption in mind. Given this commentary feeds into policy decisions, there is a public interest in knowing exactly how the distribution of grades is arrived at, and to what extent marking schedules are altered by the SEC in the assessment process to deliver an even distribution of grades."
In response to the SEC's arguments regarding the need for finality and certainty, the applicant suggests that such concerns could be ameliorated by releasing the draft schemes following a passage of time, for instance, several months after the examinations have been completed. He also disputes that examinations such as the Leaving Certificate are not capable of tracking gradual changes in cohort achievement over time. In support of his position, the applicant refers to commentary by the Irish Business and Employers Confederation (IBEC) and in the media suggesting that changes in grades are indeed viewed as changes in standards.
Conclusions
In light of the SEC's detailed submissions identifying and describing potential harms to its functions, I accept that disclosure of the requested records could reasonably be expected to prejudice the effectiveness of the Leaving Certificate examinations and the standard-setting process employed for their conduct. The original draft marking schemes require adjustments to take into account the fluctuations in the difficulty of the exam questions arising from the perceived need to avoid predictability in the setting of the questions. However, as the SEC argues, the release of the draft marking schemes and the documentation informing the adjustments would present an alternative framework for measuring individual scripts, which in turn could reasonably be expected to undermine the finality and authority of the actual marking scheme and of the examination results.
Moreover, I accept that the standard-setting process requires confidentiality in order to be effective. As described by the SEC, there is a need for flexibility and resilience in circumstances where marking must be completed within a few weeks. The SEC is the body charged by the Minister with making arrangements for the marking of work presented for examination. The arrangements that the SEC has made involve a confidential calibration process. I note that it is accepted in other contexts, such as performance management, that calibration requires confidentiality in order to be effective. In any event, based on the submissions of the SEC, I accept that the confidentiality of the process facilitates the flexibility and resilience required for the effective and timely marking of the Leaving Certificate examinations. It follows that disclosure could reasonably be expected to prejudice that process. I find, therefore, that subject to consideration of the public interest test at section 30(2), the section 30(1)(a) exemption applies.
The Public Interest
The purpose of the standard-setting or calibration process is to ensure fairness, accuracy, and consistency. The need arises from the efforts to be innovative in the setting of exam questions in order to avoid predictability and promote higher-order thinking skills. Moreover, during the process, the work of individual examiners is monitored to ensure that the appropriate standard is being adhered to, i.e. to ensure that individual examiners are not marking exam questions too harshly or too leniently. As the SEC concedes, the process is not an exact or mechanical one. However, the options for standard-setting in Ireland are constrained given that the Leaving Certificate grade boundaries are fixed by regulation e.g. an "A1" is always 90% - 100%. I do not believe that it would serve the public interest to undermine the process established by the SEC for the effective marking of examinations. Moreover, it seems to me that the value to be gained from the disclosure of the original draft marking schemes is minimal in the circumstances given the need for adjustments to take into account the fluctuations in the difficulty of the exam questions as compared to previous years. The SEC has explained that the benchmark used for making the adjustments is the assumption known as the "similar cohort adage". The assumption may not be infallible and again I note that the process is not exact or mechanical, but I do not believe that it can seriously be disputed that there is a need for some mechanism for making adjustments to account for the differences in standards between papers.
On the other hand, I agree with the SEC that the public interest in openness, transparency, and accountability is served to a large extent by the material published each year, including the full test instrument, namely the questions set and the final marking scheme together with the final grade distribution. The publication of this material allows the public to see the level of difficulty involved and the manner in which the papers were ultimately marked, including where and how marks were allocated and how this compares to previous years. Moreover, while the reliance on the similar cohort adage would generally seem to prevent any "radical" changes in the results from year to year, the material does allow for comparative analyses to be undertaken as reflected by the commentary in the media and by organisations such as IBEC that follows its publication. I am satisfied that section 30(1)(a) of the FOI Act applies. Although the SEC's claim for exemption under section 30(1)(b) is closely related to its section 30(1)(a) claim, I do not consider it necessary in the circumstances to address this or the additional claims for exemption made under sections 29 and 41(1) of the FOI Act.
Having carried out a review under section 34(2) of the FOI Act, I hereby affirm the decision of the SEC in this case.
Section 24 of the FOI Act sets out detailed provisions for an appeal to the High Court by a party to a review, or any other person affected by the decision. In summary, such an appeal, normally on a point of law, must be initiated not later than four weeks after notice of the decision was given to the person bringing the appeal.
Elizabeth Dolan
Senior Investigator