Mercer Review: Annex 5 – IPReg Middlesex Review of FD4, 6 March 2017

FD4/P6 Exam Review

Abstract

The main focus of the current research project was to investigate issues prompted by the low and variable pass rate for trainee Patent Attorneys taking the FD4/P6 examination paper. The specific aims of the research were to review: the adequacy of the preparation of candidates for the FD4 examination; the appropriateness of the current assessment methodology; the extent to which the current teaching, learning and assessment arrangements are in line with leading pedagogical practice in the area, and the appropriateness of the technical content of the assessment, given the different technical backgrounds of the candidates.

The research project adopted a mixed methods approach, employing both qualitative and quantitative research tools to gauge the breadth and depth of candidate preparedness in training to qualify as a Registered Patent Attorney and to examine the effects of the design of the examination on how candidates prepare. Initially, an online candidates survey was conducted among the 2014 and 2015 cohorts. The next phase of the research involved conducting in-depth telephone interviews with key stakeholders of the FD4/P6 exam, in this case, examiners, tutors and mentors, and employers. The semi-structured interviews with key stakeholders of the FD4/P6 exam provided an opportunity to validate (or not) the concerns and issues arising in the candidates survey.

The findings revealed that there were mixed views regarding the perceived ‘fairness’ of the FD4/P6 exam in terms of the technical content. Candidates in the survey and the majority of employers felt that the FD4/P6 exam tended to favour trainees from a mechanical engineering background and disadvantage those from a chemistry, life sciences or electronics background. However, the majority of examiners, tutors and mentors indicated that the technical subject matter of the FD4/P6 exam paper was not technically demanding, since it was based on everyday mechanical devices. Nor did the outcome of the exam support this perceived technical bias. In terms of tutor support, one concern identified by some of the candidates in the survey was that tutors who had passed the FD4/P6 exam a long time ago would be employing out-of-date techniques in their training. This view was not shared by many tutors who indicated that they kept abreast of the requirements of or changes to the FD4/P6 exam through their trainees or, more importantly, by doing the paper themselves.

There were mixed views on the important issue concerning whether the FD4/P6 exam appropriately tests knowledge and skills in infringement and validity. The majority of employers thought that the exam was well aligned with real life practice. Some, however, noted that the ‘artificiality’ of the exam precluded the testing of commercial reality and client care skills. A notable finding was the level of uncertainty expressed by some examiners when asked about how effectively the learning outcomes were met by the assessment process, commenting that the FD4/P6 exam was not an academic exercise but rather a test of practical skills. In addition, some of the examiners and employers were concerned about the ‘strict marking schedule’ for the FD4 exam, such that marks appeared to be given away for trivial points which did not reflect real life patent practice nor test the overall competence of candidates in advising their client. There was consensus amongst all stakeholders about calls for greater transparency of the FD4/P6 exam paper in terms of how the examiners write the questions and what they are looking for in the allocation of marks, in order to bridge the gap in understanding between examiners’ expectations and what is required of candidates to pass the FD4 exam.

Practising past papers was the main pedagogic approach by examiners, employers, tutors and mentors in preparing candidates for the FD4/P6 exam. Other effective techniques put forward by key stakeholders included: attempt all parts of the paper; be consistent in the analysis; and think about the paper as a whole.

Overall, candidates attributed failure of the exam to inadequacy in the transparency of the exam requirements and lack of training/support, rather than their own lack of preparation. Candidates who passed thought that they were greatly assisted by training, guidance and feedback opportunities, employing strategies for passing and exam preparedness.

Practical suggestions to tackle the issue of the low and variable pass rate of the FD4 exam have been generated from the findings in this report and include: ‘training the trainers’; the opportunity for trainees to shadow a real life validity and infringement opinion; reviewing the appropriateness of the exam format in terms of implementing computer-based examinations and enhancing the focus of the FD4 exam to one that is developmental and has a practice requirement, such as doing coursework, undertaking a viva or a case study; reviewing the FD4 exam mark scheme for upcoming exams; and aligning the learning outcomes to the assessment criteria.

Background to the project

The Intellectual Property Regulation Board (IPReg) had commissioned research by Middlesex University into a review of a specific part of the examinations necessary to qualify as a patent attorney, based on the Quality Assurance Agency’s (QAA) report (2015)[1] about the apparently low and variable pass rates of the FD4/P6 exam. The FD4/P6 exam paper tests the knowledge and skills expected of a registered patent attorney in advising on UK patent infringement and validity issues.

Using PEB data for all candidates, the FD4/P6 exam pass rates have fluctuated over the years, but it still has the lowest average pass rate of all the Final Diploma papers in the last five years, reaching as low as 36.45% in 2012, rising to 41.62% in 2014 but then dropping to 38.81% in 2015.

Following the QAA recommendation to investigate carefully the reasons why the FD4/P6 exam paper should have had such a poor candidate success rate over an extended period, the Patent Examination Board (PEB) in collaboration with IPReg, agreed on IPReg’s proposal to fund independent research to investigate this issue. As a result, IPReg established a Steering Group to commission and manage this external research project.

IPReg commissioned the expertise and experience of a research team based in the Institute for Work Based Learning (IWBL) and the School of Law at Middlesex University, to undertake independent research in reviewing the preparedness of trainee Patent Attorneys and PEB’s assessment methodology for the FD4/P6 examination. The IWBL has a strong track record in practice-based research, pedagogical approaches for work-based learners and professional learning programme content and assessment.

Research aims

The focus of the research is to investigate issues prompted by the low and variable pass rate for trainee Patent Attorneys taking the FD4/P6 examination paper.
Based on the original Research Brief, the main aims of the research are to review:

Adequacy of the preparation of candidates for the FD4 examination
Appropriateness of the current assessment methodology
Extent to which the current teaching, learning and assessment arrangements are in line with leading pedagogical practice in the area
Appropriateness of the technical content of the assessment, given the different technical backgrounds of the candidates.

Research methods

The research project adopted a mixed methods approach, employing both qualitative and quantitative research tools (detailed below). Adopting a mixed methods approach was seen as important to gauge the breadth and depth of candidate preparedness in training to qualify as a Registered Patent Attorney and to examine the effects of the examination on how candidates prepare.
In addressing the key aims of this project, the following research tools were employed:

1. An online candidates survey – the initial phase of the project employed a short, online questionnaire, using specialist online survey software, Qualtrics, which is a technique that the research team has used on other projects, and where results can be quickly produced. The questionnaire was designed in collaboration with the Steering Group. It focused on the key research aims and comprised questions on the following topics:

Preparation for the FD4/P6 examination
Practice opportunities in the workplace
Opportunities of support and take up to assist candidates in their training and development for the FD4/P6 exam, including extra support offered to re-sitters
What candidates believe they need to do to be successful in the exam
Perceived reasons why candidates felt they failed the exam
Perceived ‘fairness’ of the FD4/P6 paper
Specialist knowledge advantage/ disadvantage

The survey was administered to 2014 and 2015 candidates of the FD4/P6 exam via the Chartered Institute of Patent Attorneys (CIPA) between 6 and 24 September 2016. The survey was disseminated to 340 candidates and 176 respondents completed the questionnaire, giving a response rate of almost 52%. In research, this is considered a good response rate for a survey and, presumably, reflects the importance of reviewing the FD4/P6 examination experiences among trainee patent attorneys.

2. Semi-structured interviews – the next phase of the research involved conducting in-depth telephone interviews with key stakeholders of the FD4/P6 exam, in this case, examiners, tutors and mentors, and employers. Seven examiners participated in the research and included the Chief Examiner of the Finals, The Principal Examiner of the FD4 exam and five marking examiners. In conducting interviews with examiners, it was important to distinguish between setters of the paper and the marking scheme and those who mark according to the mark scheme given to them. In this way, we could ask the former group particular questions on the design of the FD4/P6 examination such as, how the exam paper is constructed, and how the construction of the exam and the mark scheme is linked to the learning outcomes.

In terms of training or coaching the FD4/P6 candidates, the interview participants were either tutors or mentors or both. Eight tutors and mentors had participated in the research with diverse experiences of tutoring and/or mentoring FD4/P6 candidates.

The Head of Education at CIPA and a former member of the CIPA Informals Committee facilitated access to examiners and tutors/mentors respectively during October 2016, disseminating at the same time a one page information sheet outlining the research project and the topics for interview. Initially, there was a low response from examiners in their ability to participate in an interview, mainly due to the timing of the research interviews leading up to the FD4 exam in October 2016. As a result, the research team was advised by IPReg to conduct the interviews with examiners at a later date in order to increase their engagement levels. Several interviews with examiners were conducted in late January 2017.

The employer interviews were conducted late November and early December 2016. IPReg facilitated access to the employers by circulating an information sheet to their main points of contact of IPReg registered firms and to the IP Federation requesting that it be disseminated to their members. In total, eight employer interviews were conducted, seven were law firms and one was an industry firm with an in-house patent department. Two of the employers were large firms (i.e. 250 or more employees), three of the employers were medium size firms (i.e. 50-249 employees) and three of the employers were small firms (less than 50 employees). There was an initial concern by the research team about the ‘imbalance’ of interviews with private practice and industry firms. However, this situation was more representative of IPReg’s regulated community (IPReg, personal communication).

The majority of interviews lasted between 35 and 45 minutes, using a semi-structured interview guide, and permission was sought beforehand to tape record the interviews. For the examiners, tutors/mentors and employers, similar topics were covered, as outlined below:

Design and aims of the FD4/P6 exam paper/knowledge and skills assessed (Examiners)
Support and training opportunities
Candidates’ concerns of the FD4/P6 exam and expectations of tutor/mentor
Candidate performance in the exam and reasons for low pass rates
Time pressure of exam
Technical content of FD4/P6 exam and advantage/disadvantage
Candidate preparation for the FD4 exam and techniques for passing
The interviews with examiners, tutors and mentors, and employers provided an opportunity to validate (or not) the concerns and issues arising in the candidates survey.
The research project and research tools had received ethical approval from Middlesex University’s Ethics Committee, ensuring the anonymity and confidentiality of the research participants.

Analysis

Data from the survey were analysed using statistical software (e.g. SPSS), to explore the data and identify patterns. Qualitative data techniques were used to analyse the semi-structured interviews, identifying themes to interpret the data, quoting freely from the interviews to illustrate the results. In presenting the findings, the research team have triangulated the different data sources, synthesising both quantitative and qualitative findings, so as to offer an integrated account of issues that impact pass/fail rates, thereby meeting the aims of the research.

Primarily, the analysis has been developed in collaboration with the Steering Group who seek to learn more about the reasons for the low pass rates of the FD4/P6 exam and identify effective strategies to address this issue. Following a presentation of the emerging survey findings to the Steering Group on 10 October 2016, it was recommended that the research team drill down further into the data to identify whether there are any significant relationships between the type of employer (e.g. private practice, industry) that candidates worked for and candidate preparedness, as well as the year that candidates passed the FD4/P6 exam and opportunities of support and take up – since the 2015 candidates were exposed to greater transparency and further support of the FD4 examination, receiving a new style syllabus with the learning outcomes clearly aligned, a mark scheme with the marks clearly allocated, an Examiner’s Report that was clearly set out and sample pass scripts showing the percentage mark awarded (CIPA communication).

It should be pointed out here, that a large proportion of candidates in this survey were employed in private practice, either at the time of passing the FD4/P6 exam or, if they had not passed the exam, when they last sat the exam (demonstrated in the findings). As a result, it was not possible to undertake statistically meaningful analysis of the data that identified differences in candidate preparedness and support between those employed in private practice and those employed in industry. However, there are instances in the analysis where mention is given to observed differences between candidates employed in private practice and industry, taking into account the relatively small sample size of respondents employed in industry.

Similarly, a large proportion of candidates in the survey had passed the exam in 2015 or, if they had not passed the exam, almost all of the candidates had last sat the exam paper in 2015 (demonstrated in the findings). As a result, it was not possible to undertake statistically meaningful analysis of the data that identified differences between candidates’ opportunities and take up of support in 2014 and 2015. However, there are instances in the analysis where observed differences between 2014 and 2015 candidates who had passed the FD4/P6 exam are indicated, taking into account the relatively small sample size of the 2014 candidates.

Suggestions to tackle the issue of low pass rates of the FD4 exam

The following ideas and suggestions have been generated from the findings in this report, in this case, from the candidates in the survey and the examiners, employers and tutors and mentors who participated in the semi-structured interviews. These ideas and suggestions will need to be explored further with members of the Steering Group and other interested parties.

Preparing mentors and trainers

The research shows that there is a mismatch between the preparation carried out by tutors and mentors and what is expected of the candidates in the FD4/P6 examination. The majority of tutors and mentors have not undertaken any formal training or systematic briefing, although, they welcome the opportunity to have a clearer idea of the requirements of the exam. Only one of the tutors/mentors had attended a ‘tutoring the tutors’ programme more than five years ago organised by JEB, which was found to be very helpful since,

“[it provided] clear information about how the paper is going to be marked, which is obviously relevant to how you do the tutorial… It had guidance on how to set up the answer and how to manage your time and it had examples of typical bad reasons for failing”.

Training the trainers would be particularly beneficial for tutors who had sat the FD4/P6 exam a long time ago and, therefore, they may employ an out-of-date technique in approaching their training as indicated by examiners and candidates in this research. This form of preparation is also important because trainers act as the bridge between candidates and the examination, with the potential to inhibit or facilitate understanding of the FD4/P6 exam as identified in the present research. It is suggested that PEB should provide such courses to ‘refresh’ tutors’ and mentors’ ideas and approaches to the exam. Moreover, provision of these ‘training the trainer’ activities has been considered important in other professions, such as, the Association of Chartered Certified Accountants (ACCA),[2] which enables the tutors to develop their skills in training their students for professional exams.

In addition, the employers in this study would also welcome information from the examiners about how they can support and prepare their trainees for the FD4 exam as well as inform them about what approach they would need to take to help them meet the appropriate standards.

Shadowing a real life infringement and validity opinion

The findings highlighted the lack of validity and infringement experiences identified by the candidates themselves, which is not surprising given the scarcity of such cases in private practice, and the lack of delegation of such high value work to trainees. As a result, one of the mentors in the research, who was previously a tutor for the FD4/P6 exam candidates, suggested that trainees be given the opportunity to shadow a colleague working on an infringement and validity opinion, evaluating this as a useful training aid: “so I have trainees, on occasion, I have asked them to shadow me when I’m doing infringement and validity… [it’s] really really helpful”. Shadowing a real-life validity and infringement opinion could be part of the employers’ checklist of experiences for each of their trainees in the 2-3 years leading up to the FD4 exam. For those without access to such an opportunity a training resource which gives an exemplar of a real case including interviews with experienced practitioners, illustrations of the preparation required, steps undertaken, timelines, etc. would be useful.

The appropriateness of handwriting for exams

A common issue raised with regard to many examinations is whether it is still necessary for them to be completed in long-hand when almost everyone in the modern workplace and at home uses a keyboard to write text. In the current research, in tackling the issue of time pressure in the exam, which is exacerbated by candidates having to write their answers by hand, the findings suggest that candidates be allowed to use a word processor in the exam since they can type their responses faster. This is the more favourable tool identified by the candidates since they feel that it would enhance their performance in the exam, and a more appropriate exam format for many of the examiners in so far as it improves the marking experience, due to the poor hand writing skills of many candidates taking the exam.

While there has been little research on the use of computer-based examinations in professional contexts, it has been explored in some studies in higher education.[3] It can be concluded that there is very little difference between hand writing and word processing on the ability of students to complete responses or on the nature of the responses themselves. Some students express a preference for one mode or another, but this is often dependent on their prior experience with them. As word-processing has become more ubiquitous, even more so for office workers than for students, it can be concluded that there is no substantive reason why word processing not be permitted for examinations. The only issues of concern are practical ones. If word processing were used with an unseen examination, then it would need to be conducted in a computer laboratory or with supplied laptops with internet access disabled. A brief practice session prior to the unveiling of the paper should be permitted to enable familiarity with the machine to be gained. Allowing candidates to bring their own machine would not be appropriate in an unseen examination; it would not be practicable to disable internet access under such circumstances.

Is a conventional exam sufficient to judge the desired learning outcomes?

Another key finding of the research was a suggestion for a more practical test of students’ abilities, skills and understanding in validity and infringement, in other words, including a developmental or ‘professional level’ activity as part of the overall exam. Suggestions given by one of the employers included coursework or a viva (and to this could be added a case study) whereby candidates would explain to an examiner how they would handle a real life infringement opinion, which would also demonstrate their ‘client care’ skills set. This practice requirement has been built into other professional qualifications.

Reviewing the FD4 exam marking design scheme for upcoming exams

As the research identified, candidates in the survey felt strongly that there needed to be more transparency of the marking scheme, so that they could focus their time accordingly. However, the examiners differed in their views about providing the allocation of marks on the exam paper being taken, since to do so would either impact candidates’ ability to view and analyse the whole exam paper (concern of an examiner) and, at the same time, lead them to the answers (view of some of the examiners) or, conversely, facilitate candidates’ ability to plan and write their answers (view of some of the examiners). These findings do suggest that further discussion is needed in reviewing whether (or not) to provide a mark scheme for the upcoming exam and how this will impact candidates’ ability to respond to the paper (either favourably or unfavourably).

Another concern raised by some examiners was the rigidity of the marking schedule such that it had prevented them from awarding marks that they would like to give as these were outside of the marking schedule. The examiners have suggested a ‘fundamental review’ of the marking scheme to ascertain whether it is an academic mark scheme which tries to find points or whether it seeks to be a practice paper and test the competence of candidates in giving good advice.

There was also some ambiguity amongst the examiners in terms of holistic marking versus section by section marking. For instance, some of the examiners felt strongly that doing well in one section cannot compensate for doing badly in another section of the paper since the exam is marked holistically in assessing a range of skills. One of the examiners expressed the opposite view in that candidates who give poor advice in one section can compensate in another section of the paper, although this was seen as a negative aspect of the paper. Still another marking examiner held mixed views, identifying that in some cases it is possible to compensate for performing poorly in one section while in other cases this is not possible. This would suggest that examiners revisit the holistic versus section-by-section marking of the FD4 exam paper to ensure transparency and clarity.

Aligning learning outcomes to the assessment criteria

The findings revealed a level of uncertainty among most of the examiners about how effectively the learning outcomes are met by the assessment process. Moreover, the way the exam is marked does not allow any inferences to be made about whether trainee patent attorneys have met the learning outcomes.[4] Given the improved changes to the 2015 FD4/P6 exam syllabus, in terms of the provision of learning outcomes and examination guidance notes, it is important that examiners (and tutors/mentors) understand and actively engage with the ‘language’ of learning outcomes and their understanding of how these align with the assessment process – rather than view these as ‘educational jargon’ – so as to meet PEB’s original objective to improve transparency for candidates, making it clearer what knowledge and skills are being tested in the exam so that candidates can be better prepared as well as providing greater transparency on how marks are being awarded.

The ways in which marking was discussed by a range of participants, including examiners, led the research team to a consideration of how the exam operated to enable judgements to be made about the successful meeting of requirements. It was heartening that a key step had been undertaken through expressing requirements in terms of learning outcomes. In a standards-based framework, which is what has been adopted, it is necessary for there to be transparency about the relationship between the elements of the exam and learning outcomes. Any form of assessment must ensure that those who pass can be shown to have met the threshold standards with respect to each learning outcome. The research team found it difficult to discern that this was occurring. The meeting of threshold requirements means that doing well in one part of a test that addresses different outcomes cannot compensate for lack of sufficient attainment with respect to another. The collapsing of marks into a single ‘pass mark’ does not permit assurance that this has occurred. In a standards-based framework, an overall pass mark takes on lesser significance than reporting that each outcome has been attained. Our experience in other professional contexts suggests that such an approach may require adjustments to statements of learning outcomes, standards and criteria as well as to reporting processes. The disaggregation of performance by outcome provides a subsidiary benefit in that candidates and their advisers are better able to identify what they need to address when they fail to meet the requisite standard.

Notes and links

(QAA, 2015) External assurance of the patent examination board’s policies, procedures and processes, Professor Steve Bristow.
See ACCA new tutor excellence programme – www.accaglobal.com/ie/en/discover/news/2016/august/tutor-excellence-programme.html
See e.g. Melody Charman (2014). Linguistic analysis of extended examination answers: Differences between on-screen and paper-based, high- and low-scoring answers, British Journal of Educational Technology, 45, 5, 834–843. DOI:10.1111/bjet.12100
Sigal Eden and Yoram Eshet-Alkalai (2013). The effect of format on performance: Editing text in print versus digital formats, British Journal of Educational Technology, 44, 5, 846–856. DOI:10.1111/j.1467-8535.2012.01332.x
Nora Mogey, John Cowan, Jessie Paterson and Mike Purcell (2012). Students’ choices between typing and handwriting in examinations, Active Learning in Higher Education, 13(2) 117–128. DOI:10.1177/1469787412441297
Nora Mogey & James Hartley (2013). To write or to type? The effects of handwriting and word- processing on the written style of examination essays, Innovations in Education and Teaching International, 50:1, 85-93, DOI: 10.1080/14703297.2012.748334
Nora Mogey and Andrew Fluck (2015). Factors influencing student preference when comparing handwriting and typing for essay style examinations, British Journal of Educational Technology, 46, 4, 793–802. DOI:10.1111/bjet.12171
Taking the QAA definition of learning outcomes: ‘What a learner is expected to know, understand and/or be able to demonstrate after completing a process of learning’. (www.qaa.ac.uk/)

Middlesex University Research Team

Professor Carol Costley (Project Lead);
Professor David Boud (Assessment Expert);
Dr Nico Pizzolato (IWBL Senior Researcher);
Susan Scott-Hunt (School of Law Senior Researcher);
Dr Lisa Clarke (Research Fellow).

Journal