Examiner newsletter jan 2015 (e) by International Baccalaureate

Examiner newsletter January 2015

Note from the editor Our research shows that some examiners open their newsletters but do not read any articles. If you’re tempted to close the newsletter – please read on! There are many interesting articles which have been written specifically for IB examiners. Read about the recent meeting of Chief Examiners in Cardiff, interviews with examiners and IB staff, an explanation of ‘QIGing’ (this will be coming your way too eventually), November 2014 session statistics, the human cost of examiners who do not complete their marking and updates on the MYP eAssessments, Visual Arts uploads and the Japanese project. Many thanks to the many examiners who responded to the survey set out after the May 2014 session. The feedback is tremendously helpful to us in our planning. The free text comments are still being analyzed to pull out the rich details, but the table below show that most examiners find their IB examining experience rewarding, professionally valuable and want to continue.

Chief Examiners’ conference October 2014 A successful and stimulating 2014 Chief Examiners’ conference was held in the new IB Assessment Centre in Cardiff. Thirty-one Chief Examiners (including the Chief Assessor in theory of knowledge) attended as well as four examiners responsible for group 1 and 2 languages. Here are some of the conference highlights and key discussion topics.

Examiner review of the May 2014 session Group 1 There was a high rate of examiner drop out (up to 50% in some languages), which led to concerns about the stability of the examiner teams, with such high rates of turnover. It is still unclear why so many examiners dropped out before they completed their marking and we’ll be contacting some of those who dropped out to try and understand their reasons. Group 2 There was general agreement that the experience of eMarking was more positive than last year. Group 2 was the first group for which all internally assessed work (IA) was uploaded and accessed through IBIS. This reportedly worked really well. Receiving audio files via upload rather than waiting for their delivery via DHL, was one reason given as an improved efficiency. Some reported poor experiences when using SharePoint for authoring question papers. The new process “2 to 4 standardization”, which was devised to help involve examiners from small examining teams in the standardization process, received positive feedback. Markschemes in group 2 are being revised with the aim of making expectations clearer and examining more consistent. It is hoped that this will help to improve consistency and provide greater guidance across all languages. Group 3 and core There was discussion on extended essays. A diploma candidate is able to choose an extended essay subject that is not related to one of their other DP subjects. This can result in a poor quality essay, owing to a lack of understanding of the topic area. Changes are being made to the extended essay to help address this. An effort has been made in group 3 subjects to improve the standardization process to help ensure consistency of marking from examiners and this appears to be effective. Group 4 It was felt that the introduction of QIGing in physics increased pressure during standardization, where a large time commitment is required. There was a discussion on how new authors could be tested in advance of paper editing meetings. Suggestions included asking them to write questions in advance.

Group 5 Maths IA went well as part of the new curriculum. It was felt that this was largely due to good team leader training. ` Group 6 The visual arts upload was much improved from last year. There was a delay in getting marking completed on time due to some examiners dropping out and others who were slightly late. One of the causes was felt to be examiners delaying the beginning of their marking; examiners are encouraged to start marking as soon as possible.

Conference session: What makes a good examiner? The following skills and abilities were identified in good examiners: •

accurate and complete their marking on time

•

feel valued and confident in assessing skills, not only subject knowledge

•

aware that they can ask “silly questions” of team leaders

•

resilient if they fail seeds/standardization and are willing to persevere

•

refer to practice scripts to understand why marks are allocated and ask questions if they are not clear

•

willing to persuade IB teachers about the benefits of being an examiner

•

appreciate the professional development relating to being an examiner

The following skills and abilities were identified in good Team Leaders: •

good markers as well as mentors

•

have good coaching and mentoring skills

•

good communicators

•

available for their team and respond promptly

•

able to agree with and explain the Principal Examiner’s standard

•

Ideally, all team leaders should be included in standardization, but in large teams this will be hard to manage.

Interview with Rowena Denner: Interim Head of Assessment Design and Innovation (ADI) How long have you worked with the IB? I joined in May 2009, so about five and a half years now. I joined as a Business Analyst working on our eMarking/eCoursework projects, and three months ago took over the role of Interim Head of ADI, which is all about bringing in change to assessment. What did you do before the IB? I worked as an operational manager for British Gas and moved into a business change role. I liked the aspect of change and improving things, which can be quite exciting, so I moved into business analysis at other blue chip companies and local government before joining the IB. What is the biggest challenge of your role? Having enough time to do it all! We are constantly looking to improve things, and we consult widely. Refining those ideas, planning them and scheduling them with our internal development teams and our external development partners takes a lot of time and effort. We can’t always do as much as we’d like and my team and I are always very busy, both in introducing change, but also supporting everyone once the change has been implemented. What aspect of the transformation from paper-based marking to eMarking most excites you? The constant change—we’re always eager to find better ways of managing our work, and it’s always exciting working with my colleagues, coordinators and examiners to improve our assessment processes. It keeps the brain cells active as we’re trying to think of new and innovative ways that we can improve the services we offer. What are the most important qualities that everyone in your teams must have? Not being afraid of change, and the ability to think outside of the box. What do you most enjoy doing outside office hours? I enjoy cooking and find that it relaxes me. My husband tells me that he’s a good eater, so we make a good pair!

Question Item Grouping (QIGing) For the first time, in May 2014, IB assessment split up scripts for the marking of some components. The process, called Question Item Grouping (QIGing), was implemented in order to yield the following benefits: 

Efficiency for examiners—an examiner can concentrate on qualifying for and marking a particular question or group of questions. An examiner can more readily retain the markscheme for a single question or group of questions in their working memory than the markscheme for a whole examination paper.



Increased marking reliability—previous studies of marking by QIG have shown that it increases marking consistency and reliability.



Better retention of examiners—some examiners are unable to pass qualification for a whole script but can pass for individual questions, and through QIGing their work can be retained. In whole-script marking these examiners are not likely to have passed qualification and would

not have been able to mark as they had planned. This would have lost their marking effort to the IB for the session. Did the IB’s first QIGing experience deliver the benefits listed above? Were examiners happier? Was there better quality marking, and more timely completion of marking? What was successful? QIGing was pioneered in English literature paper 1, Spanish literature paper 1, business and management paper 2, geography paper 2 and physics paper 2. The IBIS development was of very high quality, worked well and worked first time. What was less successful? The reports from the RM Assessor did not provide all the information that Principal Examiners and IB staff required, or at least not in an easily accessible form; the assignment of candidate answers to QIGs was affected by errors and a great deal of additional work was required from senior examiners and IB staff during the standardization meetings and subsequently during monitoring and management of marking. So was it worth the additional work? In five out of six components, marking quality, indicated by the rate of component level mark changes and subject grade changes after enquiry upon results, was found to be better than in May 2013. However, the findings about examiner retention were inconclusive. In business and management paper 2, marking quality was slightly reduced, but retention of examiners was significantly improved. We have learned a great deal from our first adventures with QIGing that will inform our gradual extension of its use in other components from May 2015. Feedback from senior examiners and IB staff has fed into the commissioning of further software development and action to improve our processes related to QIGing. There is much to do, but our first experience has encouraged our belief that QIGing holds promise of significant benefit for the IB community.

Interview with Richard Harvey: Vice Chair of the IB Examining Board How many years have you been an examiner with the IB? I started examining for the IB the year that film was first examined as a pilot subject in 2003. Before this I was part of the team that developed the course programme. Before examining for the IB I had examined film, theatre, media and English literature for a variety of examining boards, mostly in the UK. I taught these subjects at secondary and tertiary level for more years than I care to remember: and this after a rather inauspicious career in film, TV and theatre. I have had the opportunity in recent years to return, for brief periods, to TV and film. What attracted you to film? My passion for film began literally at my mother’s knee as she managed three independent cinemas in Kent in the UK through the 1950s and 1960s. I could run the projection room by the age of 14 (although not unsupervised) and could splice a damaged reel before the audience got too restless. My early years were almost a British version of Cinema Paradiso. Early on in my teaching I was able to include units of visual literacy into the curriculum for 11 to 18 year olds and piloted film and media programmes of study for post-16 students. It was a privilege to be able to launch film as part of DP group 6. © International Baccalaureate Organization 2015 International Baccalaureate® | Baccalauréat International® | Bachillerato Internacional®

What are your top three tips for examiners? 1. Get organized at least three weeks before you think you need to. 2. Plan your marking schedule—spread the load and don’t try to do too much all at once. 3. Keep to deadlines—the impact if you don’t may be much greater than you think. Where is your favourite place? It’s difficult to choose just one place. Our garden is a place of refuge and pleasure (and the grass area large enough to warrant the luxury of a ride-on mower). We have views across the Blackmore Vale, Thomas Hardy country, in a village where Hardy set Tess of the d’Urbervilles. Other than that I would choose the tiny island Komandoo in the Maldives where I proposed to my wife. As for cities it has to be either San Francisco or New York: both so vibrant and exciting in their own individual ways. What do you like most about being an examiner for the IB? Certainly the sense of being part of an international community. I meet my colleagues, both examiners and staff members, face to face all too infrequently, but always try to keep in touch. I feel very fortunate to be able to enjoy their friendship. Of course, our students are a significant part of this community and it is a great pleasure to share their knowledge and understanding, and rewarding to recognize that I always have something fresh and new to learn from them.

November 2014: some statistics Number of:

November 2013

November 2014

Increase (%)

May session schools entering candidates

580

585

0.86

November session schools entering candidates

209

221

5.74

May session school candidates

1,817

1,903

4.73

November session school candidates

8,592

9,025

5.04

Raw marks required

182,437

193,777

6.22

Subject Grades to be issued

56,675

60,073

6.00

Examiners allocated to marking/moderation

1,139

1,165

2.28

Candidates’ examination scripts eMarked

90,437

96,081

6.24

Theory of knowledge essays eMarked

6,418

6,887

7.31

Who pays the price when examiners do not complete their marking? The price of examiners who do not complete their marking commitments is paid for by their colleagues. Other examiners are asked to step up to examine the scripts left unmarked by examiners who drop out before their marking is completed. This is particularly acute in small teams, where the Principal Examiner and one or two trusted colleagues can end up having to examine literally hundreds of © International Baccalaureate Organization 2015 International Baccalaureate® | Baccalauréat International® | Bachillerato Internacional®

scripts. It’s either this or face the impossible position of letting down our students and providing no grades on results day. We often find that during the session examiner workloads increase and prior planning needs to be reviewed because a large number of examiners inform us at very short notice that they are not available or simply complete no marking; the examiners “drop out” of the session. A recent study, focusing on larger subjects within the Diploma Programme, found that over 10% of examiners across the programme drop out during the marking period. This effect was more noticeable in group 1: language and literature and group 3: individuals and societies. We recognize that there will be instances where examiners who fully intend to examine find that they are unable to do so. However, with over one in ten examiners dropping out of the session we wanted to highlight the effect this has and the ways in which this can be avoided. When an examiner drops out of the session, the work that During the May 2014 session, 24 examiner was expected to complete needs to be shared examiners marked over 1,000 among other examiners. In some cases this can be scripts to complete all marking completed through small increases across many examiners. on time. This is not always possible though. Too frequently the additional work is picked up by a small number of individuals. These examiners take this work on to ensure that candidate grades can be awarded on time. Many of these examiners also have full-time day jobs and their commitment to ensuring the timely and accurate assessment of candidate is a credit to the IB’s values. Speaking with examiners who marked 200 or 300% of the standard marking target, a sense of the potential for personal disruption becomes clear: “Actually, I had not planned to mark so much, but it affected me that the deadline was so close and that we still had a lot of marking to do […]. I think that it is a problem that people sign up and don't even start.” “I had quite a few examiners who either didn't start marking or who were stopped for poor quality marking—this meant others such as myself taking on more papers. I wonder if people are not making it a high enough priority.” “The number of dropouts did affect [the] team. Especially since the number of dropouts was large this time.” These late changes to the examining teams create issues for marking deadlines, meeting schedules and arrangements relating to the statistical analysis of data. Overall it also causes a significant increase in work for IB colleagues, and one which adds further complexity to a multifaceted, timesensitive process. If your availability changes, please inform us at the earliest opportunity so that we can plan accordingly. Allocation of work for the May session is usually completed for all components by February. At this time examiners are asked to acknowledge their allocations, and this represents the last time at which we can make changes to examination session plans without disruption.

Indicating your availability for an examination session should be viewed as a firm commitment to undertake work for the IB.

To plan ahead for future sessions, the IB will be implementing practices focusing on examiner availability across sessions: 

If an examiner is not available for three consecutive years then they will not be offered a renewal of contract.



If an examiner drops out of marking twice in two years they will not be offered a renewal of contract.

In both cases the individual can re-apply to become an examiner but the circumstances around their previous term will need to be investigated. If examiners do not anticipate being available for multiple sessions, we would invite them to contact us via examiner_maintenance@ibo.org. We can then put the examiner on hold until they contact us to say that they are available once more. If you wish to provide feedback regarding this article, please send this to assessmentnews@ibo.org.

Changes to the font style of question papers for May 2015 onwards The IB is committed to ensuring that its examination papers are accessible and clear to all of its candidates. Working with the Assessment Access and Inclusion Manager, we have reviewed the style of Diploma Programme examination papers. Several changes have been made to make our papers easier to read and will be in place from the May 2015 examination session onwards. The main changes are explained in this article, however, sample documents are in the library on IBIS that demonstrate the new look of the examination papers and can be accessed here. These are style changes and not changes to the structure or assessment criteria of the papers. The samples illustrate the new style and are not examples of the types of questions or the number of marks. The most significant and obvious change to the examination papers is in the font used for text. Previously, the text on the covers of examination papers has been in and the text inside the examination papers in Times New Roman font. From May 2015 all text, apart from where a different font is normally used (for example, in the formatting of texts in language B paper 1), will be in Arial font. This is far more accessible for candidates. However, in order to accommodate the needs of different subjects, there are two major exceptions to this:  

examination papers in group 5 (mathematics) will retain Times New Roman for mathematical notation, variables and numbers; all other text will be in Arial examination papers in group 4 (sciences) will retain Times New Roman for Greek letters and f, l, i, x and y characters; all other text will be in Arial.

In addition to this, the front covers have been slightly rearranged so that the name of the paper and instructions to candidates appear in the clearest possible format. On advice from design and access specialists, other style changes include the left alignment of text, rather than full justification as previously, and the discontinuation of the use of italic text (used previously for instructions to candidates). New versions of the current chemistry and physics data booklets will be published at the end of January to reflect these style changes.

News in brief Visual arts upload Improvements in the IT system and guidance meant that the upload of visual arts coursework went much better in May 2014 than in May or November 2013, but it is still a cumbersome and unfriendly process for many schools. IB Assessment is working with a developer of market-leading educational

software to create a user interface for IB World Schools that will make the upload process a much more elegant experience for the May 2015 session and beyond. Middle Year’s Programme (MYP) update The Middle Years Programme (MYP) assessment will be introduced for first assessment in May 2016. From then on, in order to achieve an MYP Certificate, students will sit at least five subjects examined on-screen (the student responses will then be uploaded to RM Assessor), as well as complete two ePortfolio subjects and the personal project. These will be assessed internally by teachers and be electronically uploaded and moderated using the new dynamic sampling approach. These new MYP eAssessments represent a significant investment in assessment technologies by the IB to establish a completely electronic assessment system. The International Baccalaureate Electronic Assessment Development System (IBeADS) has been developed with a German technology company. The system allows for staff in our newly dedicated team in MYP Examination Preparations to efficiently combine rich media and imaginative questions to produce an immersive eAssessment environment. A major challenge has been developing the content of the examinations. The MYP has previously been assessed through a moderation-only process. Developing the right kind of questions to ask, that reflect the values of the MYP, has been a voyage of discovery for the new team of subject managers in MYP assessment. The exams are developed using the MYP Unit Planner approach, with the content of the exams being informed by high-level topic lists, MYP marking criteria, chosen key and related concepts and global contexts. The rich, immersive environment of IBeADS is well suited to the concept-rich MYP. The newly established MYP subject management team which joins the existing subject operations team has made excellent progress. Examinations for a pilot session to be held in May 2015 as well as a range of other on-screen examinations being offered as specimen papers are being worked on for release to schools in September. Production of the 2016 examinations is also well underway after recruiting authors. For ePortfolio subjects, an arts trial was held from which the team learned a great deal and have since launched trials in language acquisition, design, and physical and health education. The broader challenge of including the MYP in assessment processes that were originally developed for the DP, will continue to involve the input of a broad range of staff from across the IB offices. In the coming months the team hope to appoint our first Chief Examiners for the MYP. These will be key leadership positions for teachers with examining experience and a chance to make a significant contribution to the IB. Anyone interested in applying should visit the recently re-launched www.ibo.org for information. Japanese project In November 2016 the IB will be offering assessments with Japanese as a response language. The subjects being assessed in Japanese are listed below.      

Economics HL/SL History HL/SL route 2 (probably Asia/Oceania HL) Biology HL/SL Physics HL/SL Chemistry HL/SL Mathematics HL

 

Mathematics SL Theory of knowledge

In preparation for these assessments we wish to identify examiners able to assess candidates in Japanese immediately. If you are able to examine these subjects in Japanese please contact examiner_maintenance@ibo.org. Examiners’ fees to be raised by 2.3% in 2015 We are pleased to announce that the IB’s Board of Governors has approved a 2.3% fee increase for IB examiners in 2015. The 2015 fees schedule, available shortly on IBIS, will indicate the new marking fees in each component. Examiner newsletter feedback The July edition of the Examiner newsletter was read by 8,597 examiners in English, French and Spanish. If you have any feedback on the newsletter or would like to suggest an article topic, please e-mail assessmentnews@ibo.org.