Tackling Differential Attainment in
Summative Assessments in Postgraduate Medical Education
ROUNDTABLE REPORT 17.09.2020
The United Kingdom National Health Service (NHS) is the 5th largest employer in the world with one of the most diverse workforces in the public sector. Nearly 40% of its employees are of BAME (Black, Asian, and Minority Ethnic) background, have migrated from outside the UK, or have disabilities or other protected characteristics. This rich diversity of professional and cultural experience of its workforce would ideally serve as a national resource for the NHS, but instead, the recognised phenomenon of differential outcomes or attainment has become a hurdle for many, based on gender, ethnicity, disability, or other protected characteristics. The experience is often far worse for employees who have qualified overseas.
Differential Attainment or Outcomes or award (DA) is a euphemistic phrase that describes the historical and persistent differences in award levels based on factors other than academic/ professional capability or effort. The impact of this endemic discrimination of affected individuals/ groups has a profound impact on careers and wellbeing on a personal (micro) level on them, productivity, team-working or patient safety impact at organisational (meso) level and at a much larger societal/ socio-economic level (macro) level for the country.
This roundtable sponsored jointly by the British Association of Physicians of Indian Origin (BAPIO) and the Royal College of Physicians (RCP) is the first of a series of events organised by the BAPIO Institute for Health Research (BIHR) exploring DA across the journey of a medical professional from entry to medical school all the way to retirement.
This roundtable is focussed on DA as observed in summative assessments in high stakes professional examinations, which are essential for entry, progression, or accreditation. The findings and recommendations of this series will be published in the rainbow paper “Bridging the Gap” celebrating diversity in the NHS. This report should be read in conjunction with the scoping paper which presents the evidence base for the discussion and recommendations. (Dave et al., 2020)Key Findings
- In the UK doctors from BAME groups, and International Medical Graduates (IMG) i.e. doctors whose primary medical qualification (PMQ) is from a medical school outside of the UK have, consistently, poorer demonstrated outcomes in assessments when compared to white doctors and UK medical school graduates.
- The quantum of this difference varies across examinations and cohorts but usually equates to about 10-15% gap between UK BAME and UK White doctors and about 30-50% gap between IMGs and UK graduates. Clearly, these are group differences with many individual outliers that defy the norm.
- The impact of DA is manifold and pernicious.
- It is a moral imperative that assessment systems and practices are equitable. Implicit bias and discrimination lead to a non-compassionate organisational perception amongst its staff fundamental principles of ‘Our People Plan’ (NHS England » We Are the NHS: People Plan for 2020/2021 – Action for Us All, n.d.) and the ‘NHS Constitution’.(The NHS Constitution for England, n.d.)
- Assessments are important in maintaining clinical standards and patient safety. However, inequitable assessment systems lead to loss of psychological safety and reduced freedom to speak up with consequences for patient safety.
- The financial cost of inequitable assessments for the NHS is significant. It is estimated that in 2019, it costs £88,000 for an additional year of training for general practitioners. Moreover, it limits the number of new trainees that can be trained, creating further pressures on workforce capacity.
- There is a significant wellbeing cost of inequitable assessments in terms of psychological and physical weathering, moral injury, and demoralisation. Demoralisation in the workforce is associated with production costs and poorer patient outcomes. Individual stories of personal and professional tragedy abound, in many cases associated with significant morbidity and sadly in some with untimely mortality.
- DA in summative examinations offer a lens to the differential experiences and outcomes that occur throughout the educational journey and as such systemic issues should not be lost sight of, whilst focussing on high-stakes examinations. Workplace-based assessments or supervisor assessments may not be free from bias or from grade-inflation and therefore may not offer a silver bullet to replace summative examinations. Improving educational supervision, educational, and clinical supervisors’ confidence and skill in addressing the differential learning needs of trainees including IMGs and BAME trainees is important.
- However, the disproportionate impact of COVID19 on BAME patients and the BAME workforce and the Black Lives Matter movement have shone a spotlight on the real impact of structural inequalities. There are a necessity and an opportunity to be bold and courageous in our response, in contrast to the traditional ways of dealing with complex issues that are often ponderous even if incremental and this applies specifically to the inequalities in outcomes associated with summative examinations.
- Summative examinations offer a single point of assessment to determine competence and progression. Making a judgment about competence to practice is a complex process. The reliance on this single point to determine progression ignores the richness of assessment data available from the educational journey. Integrating and formally linking information available from multiple assessment points conducted by a range of assessors over a period of time, allows for multi-dimensional assessment and can potentially neutralise the bias that may reside in a single point assessment.
- Discussions about DA are not new. DA has been highlighted for over two decades but previous efforts at addressing DA have failed to bear fruit despite high profile legal challenges (BAPIO) and high-level meetings with Colleges. This coalition comes at a time when many summative examinations and assessments have had to be reformed or reshaped completely in the wake of COVID19. This offers a unique opportunity to transform the assessment landscape for postgraduate summative assessments in the UK.
- Developing an equitable assessment process that does not discriminate between candidates is not only fair but is also vital in retaining public confidence in the assessment system. Patients and the public need to have the confidence that doctors who are excellent clinical practitioners are able to progress and not failed by the system of assessment.
- Build on the consensus achieved with partners including Health Education England (HEE), General Medical Council (GMC), NHS England and Improvement (NHS E/I), and the Royal Colleges at the roundtable and declare a firm, public commitment, and accountability for addressing the inequalities demonstrated through DA and strive to create a fair, responsive and transparent system.
- Identify and clarify the roles of individual stakeholders in ensuring a coordinated 4-nation response in addressing DA. (NHSE/I)
- To retain the current system of summative assessments but demonstrate robust equality and diversity impact assessment of the entire process from question writing, standards-setting, and training of examiners. (Academy of Medical Royal Colleges)
- To reform the provision of targeted support to candidates known to be disadvantaged by DA and supervisors in increasing awareness, cultural competency, and resources to provide mentorship, educational guidance, and career oversight. (Health Education England)
- Set up a Task and Finish Group (T&FG) from members of this roundtable, learners able to offer a ‘lived experience’ and subject experts to create an options appraisal, identifying alternatives to rethink the current assessment systems that facilitate a transition to a fairer, comprehensive, multi-modality and valid judgment about the candidate that will determine progression/ accreditation. (HEE & GMC)
ISSAR, Prerana, Chief People’s Officer, NHS England & Improvement
POWIS, Stephen, National Medical Director for England, NHS England
GODDARD, Andrew, President, Royal College of Physicians
MEHTA, Ramesh, President, British Association of Physicians of Indian Origin
- BASI, Jagtar Singh, Lay Member
- CANNON, Jane, General Medical Council, UK
- CHAKRAVORTY, Indranil, Chair, Bapio Institute for Health Research
- DAVE, Subodh, Vice-Chair, Bapio Institute for Health Research
- O’DONOGHUE, Donal, Registrar, Royal College of Physicians,
- ELDER, Andrew, Ex-Medical Director, MRCP UK
- ESMAIL, Aneez, Professor of Primary Care, University of Manchester
- EVANS, Navina, CEO, Health Education England
- FREETH, Della, Royal College of Physicians
- Macleod, Sheona, Acting MD, Health Education England
- MELVILLE, Colin, General Medical Council, UK
- MENON, Geeta, Postgraduate Dean, Health Education England (South London)
- PATEL, Mumtaz, Vice President, RCP Global
- PETTY-SAPHON, Katie, Medical Schools Council, UK
- SINGHAL, Parag, Hon Secretary, British Association of Physicians of Indian Origin
- WOOLF, Katherine, Research Department of Medical Education, University College Hospital, London
Overview of Evidence
- In their introduction to the roundtable, the sponsors explained the background to differential attainment (DA) in postgraduate exams, as a significant issue for thousands of doctors. A story of a talented, ambitious, hardworking 42-year-old doctor who failed the Clinical Skills Assessment (CSA) exit exam for the Royal College of General Practitioners (RCGP) and therefore was no longer able to be registered or work as a General Practitioner. This disappointment proved to be too much, as the doctor sadly ended their own life in tragic circumstances. This anecdote was described as an extreme but not an isolated example of the stress and mental health impact of assessments for doctors. To medical professionals, success is often paramount, and failure is often an utterly unacceptable outcome. Often interpreted as being unable to cope or that one is not considered good enough.
- Yet DA in summative assessments is an established fact. It is widely accepted that DA is a complex issue, and endemic. Over its 24 years of existence, the British Association of Physicians of Indian Origin (BAPIO) has tried various methods (including legal challenges) to find a solution.
- The turmoil that has been caused to every established system by the advent of COVID-19 pandemic in 2020, has also presented an opportunity for radical rethinking. BAPIO has embarked on a major project on ‘Bridging the Gap’, tackling DA in the healthcare professions. This project will bring together the triad of grassroots professionals with real-life ‘lived’ experience of DA, international experts in education, and key stakeholder organisations responsible for designing, delivering, and quality assuring the education and training of the medical workforce.
- It is anticipated that this comprehensive exploration of DA in the medical profession over its entire journey will provide meaningful and effective solutions. There was an acknowledgment of the support received for this project from the establishment including the General Medical Council (GMC), Health Education England (HEE), and the Royal Colleges. This collaborative project will publish its findings and recommendations in 2021, as a ‘rainbow paper’, celebrating the ideals of equality and diversity.
- The primary aim of the roundtable was to find consensus on the next steps and tangible solutions to the issues. The Sponsors reiterated that the evidence for the existence of DA in medical professions, the impact individuals and their families, as well as the healthcare system, was already established, hence the focus will be on where in the system, the efforts are best placed over the next few months.
- In their opening, the Chairs described how important they felt this issue of DA was to every stakeholder in the profession. They described their passion for the impact on clinical colleagues with differential outcomes and agreed that the data showed very clearly that there is an inclusion issue, there’s an equality issue or equity issue.
- Participants discussed an overview of the extent of DA and the evidence proving its existence. Differential attainment (DA) is the variation observed across groups when split by a number of protected characteristics, including age, gender, ethnicity, and race. This is independent of individual ability or effort but the causes are complex and often include the intersectionality of a number of historic, social, cultural, economic, and environmental factors. In the spirit of fairness and equality in any society and obligations of the public sector equality duty, there is a need to eliminate unlawful discrimination and aspire to equality of opportunity for all.
- DA in medical professions has been recognised for over four decades, and the system recognises that International Medical graduates (IMGs i.e. those with a primary medical qualification from countries outside of the European Economic Area) and UK Black and Minority Ethnic (BAME) doctors consistently underperform in summative assessments compared to UK white graduates. Overall, there is data showing that there's a 10 to 15% gap in pass rates of UK BAME candidates and for an IMG, this is much larger.
- Headline figures from data published by the GMC, on differential pass rates for summative examinations conducted by different colleges were discussed. In the case of the examination conducted by the Royal College of Psychiatrists, the pass rate for a UK white candidate was 88% but for an IMG this could be as low as 31%, and for a BAME candidate in a UK training program, this figure may be even lower.
- DA refers to group differences on average, and the participants recognised that there was overlap in the distribution of scores between different groups, and there are many high achieving BAME, or IMG doctors as well as, many low achieving white UK graduates. Therefore, it would be important not to make assumptions about individuals based on group differences. Although these group differences do reflect inequalities that need addressing.
- DA when described, should be clarified as being differences in the average attainment of two or more groups. The discussion in this roundtable is about DA as applicable to IMGs versus UK graduates or differences between a black Asian and minority ethnic group versus a white group, among UK graduates. There is also DA based on ethnicity among IMGs.
- This distinction is important when exploring causal factors and seeking solutions. Causal factors are often connected to each other in quite complex ways. These could be related to living, working in a different country to the one that one has qualified in, those that relate to being of a different ethnic group to the white majority. Sometimes candidates can be affected by more than one factor. It is critical to remember the inter-relatedness and inter-connectedness of causal factors.
- It is also important to recognise the diversity within BAME or IMG categories and being careful not to amalgamate diverse groups into one.
- However, the phenomenon of DA has a significant impact on the National Health Service (NHS) which is one of the largest employers of migrant workers in the world. Almost 40% of the NHS workforce were from BAME communities including IMGs, for whom the DA effect appeared to be double.
- The participants discussed that ‘differential attainment’ wasn't the preferred term to use anymore and this would be better described as ‘differential award’ which demonstrated that it's actually the system that's creating the difference rather than the attainment by the individual.
DA Impact on Wellbeing
- The participants discussed that the term ‘BAME’ was unpopular as it was considered to be just too ‘all-encompassing’ and did not recognise the significant diversity and different characteristics within the group. It was agreed that it was better to describe the groups by ethnicity rather than consider an artificial category such as BAME. Some of the UK census categories such as mixed white and black Caribbean, for example, could be quite a small group. Under the census data, there are clearly defined categories.
- However, under the Equality Act, it is a legal duty to the group, people together to find out where there are inequality and BAME may still be a useful category differentiate between white and black. It is also our responsibility to look at the intersectionality of BAME.
Causes of DA- a systematic issue
- Research shows that racial gaps, unemployment, and overall socioeconomic indicators haven't moved for the last 20 years. Expressions of explicit prejudice have come down, but implicit bias and discrimination haven’t, and the stereotypes or implicit bias has not changed much over time. It was a moral imperative that if there was a level playing field, and an assessment process or system and practices that led to equity within the NHS, then this will ultimately lead to, equality of health outcomes for the population.
- However the weathering effects of any kind of discrimination, racism, or sexism, were the same, and those weathering effects had an adverse impact on the physical and psychological health of medical professionals. And this manifests as problems for the workforce, whether it's in terms of sickness absence, physical and moral injury, and a non-compassionate culture. That can only result in worse patient outcomes and demoralized staff.
- And that, this was completely contrary to what was agreed in the NHS People Plan. Therefore, the impact on leadership and culture was quite significant. If the medical professionals felt the system was unfair, this would have a direct impact on their performance and outcomes for the organisation.
- The participants discussed that there were key moments where employees were engaging with the values of an institution. And that, the assessments and examinations were key moments of interaction with an institution, so if the workforce felt that there was institutional inequality, discrimination, or that their contribution was not valued, this would have a direct consequence on their engagement, motivation, and leadership.
- A workforce which felt that the institution did not treat or assess them fairly would also not have the psychological safety to raise concerns which may impact on clinical standards and patient safety.
- Participants were urged to ‘be bold and brave’ due to the heightened awareness in society as a result of the cross-section of the Black Lives Matter movement and the disproportionate impact of COVID on sections of the population.
What is the economic and workforce impact of DA?
- All stakeholders accepted that there was an attainment gap and an ethnic attainment gap. That multi-level, multifactorial analysis suggested that it was, ethnicity, that was driving the attainment gap in many examinations. However, it was unclear is if it was simply the structure, or the implicit bias within an examination setting, that was causing the differential.
- Participants believed that it was an entire system issue, and very complex at both, systemic level in terms of the processes around recruitment, selection, and other factors such as relationships with educational supervisors, the quality, regularity, and accuracy of feedback that's given to doctors in training, and possible issues related to an exam experience such as a perception of implicit bias in an exam.
- Although the examinations provided a lens on the problem of DA, the participants felt that the exams were not, the problem in themselves. DA occurred across all specialties, across multiple professions and was not unique to the UK.
- As assessments were at the end of the educational journey, it suggested that there was something amiss about the journey, the environment of education, the environment of learning, the culture in which we are working or training. If one seeks to fix the endpoint one may not actually be dealing with the root cause.
- Participants discussed that if a candidate fails the exit exam for General Practice (GP) and needs to spend another year in training, it costs the UK at least GBP 88,000. Due to a planned expansion in GP numbers, this will probably limit the ability of the system to train more GPs. If people have an extension it is important to recognize that a trainee lower down can't be recruited to that program. So there is a significant cost to train someone from entry into a program for longer.
- However what matters much more is the cost to the individual going into training knowing all of this, knowing that they're more likely to fail at the end of three years, they fail and feel a failure and after a year, they get their exams and move on.
- There are examples of successful early intervention. By identifying people early and by being honest in admitting that there is an attainment gap, one can allow people to go through the training pathway, with their head held as high as anyone else, and know that they're being supported.
Impact of Assessments on Wellbeing
- The COVID pandemic has provided an opportunity to look differently at recruitment and assessment systems. As this year, systems were modified significantly, the Academy of Medical Royal Colleges is working with the GMC to analyse the impact of these COVID related changes and especially the effect this may have on the bias.
- As an industry, it is estimated that on average GBP 200,000 is saved to the public purse by importing a doctor from abroad. The reality is that at the moment there often very little investment, in a thorough induction, when a doctor joins the workforce. Albeit, there are exceptions.
- Examinations are a number one cause of stress among candidates as these are an important determinant of future progression
- It is known that more IMGs and more BAME doctors end up in non-training grades or SAS (Specialty and Associate Specialist), not by choice. This often leads to a lack of professional fulfillment and consequent disengagement, often adversely affecting patient outcomes, equally poor performance, and progression, which can lead to a lack of wellbeing and lead to poor mental health.
- The participants heard examples of quotes from trainees from a study commissioned by the GMC on differential attainment. And from a white trainee saying, ‘failing massively affected me almost to the point of depression at the time, I was really committed to the specialty, I couldn't understand why I loved my job, I got really good feedback, people said I was good at it. And then it wasn't materializing in the ways that it was meant to, I failed the primary three times. It doesn't matter how many times people tell you oh it's okay, you'll get there, it's fine. You feel so stupid.’
- The impact of summative assessments on wellbeing can affect candidates of all denominations. However, the support that Black Asian minority ethnic and IMG trainees get is often different and experience additional risk, that's especially around low expectations, fear or experience of discrimination or bias, lack of support, poor learning environment, and lack of learning opportunities.
- An example from a white European IMG, a radiology trainee saying, ‘you know, I didn't pass one exam by point four marks, the other one by two marks and I was very upset. Most of the consultants have been supportive, but my educational supervisor wasn't she said, “Oh, are you sure you want to do radiology.”
- The idea of low expectations has been described many times by IMGs being concerned knowing that failure rates are lower, creates an unfair disadvantage before even attempting the exam. A Black UK graduate said, ‘with black ethnic candidates, there's an expectation that they won't do well. It has a psychological effect, I'm expecting a lower mark because I know it's a stupid way of thinking, but actually, it got to the point where I was thinking; What is it, am I, I wasn't sure if it was my knowledge anymore. I wasn't sure if it was my confidence. I wasn't sure if it was my skin colour’ And if someone's expectation from a candidate is low, subconsciously performance will be low.
Importance of early intervention -the HEE iTAP tool
- There is a lack of oversight and connectivity between different training programs hence opportunities are lost for providing a holistic oversight or mentorship to doctors in training by their supervisors. This situation of supervision is even worse for locally employed doctors.
- So, in some ways trying to identify those in formal and informal training programs who need additional support in order to get through the exam process, needs to be a lot more interconnected. For disadvantaged trainees, recurrent failure and lack of support may become a steady debt that accrues over time and becomes unsurmountable.
- Participants discussed that there was yet no convincing argument that if one were to get rid of the summative exams, that would solve the problem because if one were to use workplace-based assessments (WBA) for example, there was as much risk of bias and variability in different qualitative assessments.
- What is needed, is to have reliable valid quality and, investment in the infrastructure to support doctors, and that's what one should focus on.
Purpose of Assessments - Retain vs Reform
- Providing timely and supportive feedback is important. Health Education England (HEE) has been developing a tool for early identification of trainees at high risk of failing’ using their training assessment portfolio. The (Scarborough) (i-TAP) tool helps to highlight the trainees that need targeted interventions at an early stage, not just for the trainee but also for the education faculty, and to the system.
- Firstly, this tool identifies those in need of enhancing the induction (eg. IMGs) at the beginning, who need additional support getting into the system.
- Secondly undertaking a neurodiversity screening, and then developing personalized learning plans for each of these trainees. This is then facilitated at the level of the educational supervisor, as well by training program directors, and then implement interventions at an early stage.
- Finally, the system needs to be developing the skills of cultural diversity among supervisors who are training these trainees in a culturally different environment.
- It is fundamental to decide what the purpose of the assessment should be. It is to make sure that at the end, when a medical professional go through that process, the public can be sure that that person is going to be competent.
- Participants recognised the link between clinical parts of MRCP and MRCGP exam (scores) and the risk of being referred to or being sanctioned by the GMC, regardless of ethnicity. Yet, if the current exams were assessed against the purpose (of producing a safe and competent doctor), it was clear that the current system was not achieving what it was set up to do. Thus there was a need to reform or rethink the process to achieve a better outcome.
- The problem (of DA) is much wider than just the summative exam, and that there is a need for a mechanism of assessment in order to ensure safety and assurance to the public. However, the system needs to recognize that in virtually all adult education assessment systems, there are a variety of assessment modalities, which include summative assessments, but which also include monitoring and assessing an individual in real-life situations, (like workplace-based assessments).
- However, part of the problem is that assessment methods are sometimes used as a shortcut. And when one moves towards ‘supervisor based’ assessments there is likely to be a lot more people meeting the bar, due to a more comprehensive assessment, than just one final summative assessment.
Moving from Assessment to Judgement in Progression
- There was caution about any reform or rethink that resulted in completely losing the clinical skills exams, as this would certainly cause a deterioration in vital clinical diagnostic and decision-making, to the potential detriment of patient outcomes.
- Equally, if the system were totally dependent on WBAs or Supervisor-led assessments, there was also a potential for bias. Candidates were much more worried about that in fact than they were about bias in exams. Some candidates were indeed concerned about the fact that we might take away the opportunity to have an external assessment.
- Participants discussed the advantages of moving away from talking about assessments to talking about judgment. What the system needs are competent doctors and competence is a process that develops; one can be competent to do something with a supervisor sat in the coffee room, or to perform independently.
- Assessments are often used to make a single point judgment about whether someone is competent and that is not probably viable, however much one may be reassured by a perception of high reliability and therefore, suitability.
- Participants heard that GMC was exploring with colleges, the role of summative assessments in making career progression decisions. Such that progression judgments are not hinged on assessment alone.
- Participants were in agreement that summative assessments (exams) needed to be retained, as the current system relied on this for professional progression. They recognised the potential danger of sweeping away a system that was working (for the majority) before a fairer and more comprehensive system was established. It is likely that the whole system will need to go through cycles of evolution and relatively quickly.
What is the role and responsibility of the academic leadership to tackle DA?
- The participants agreed to start thinking of potential solutions that would remove the barriers, although these may not achieve absolute equity, but can narrow the attainment gaps. This can be achieved by developing exploring the issues in granular detail and find tangible reforms through a task and finish group.
- Participants recognised the role that academic leaders in tackling DA. It was recognised that not all colleges were aware of, or proactive in addressing the recognised DA in their assessment systems.
- It was felt that unfortunately in some cases the burden of raising awareness had fallen on to IMGs and BAME doctors and that needed to change. It was important that all stakeholders needed to recognise and take responsibility for addressing DA and be accountable.
- Participants felt that finding solutions and reforming the current system was an evolutionary process but there was a risk that if the problem was perceived to be too complex, then one may be tempted to retreat into inaction. .
The conclusion of this section is obviously retaining exams, but that the kind of outcomes that we are seeing from the entire assessment process is not the outcomes that we want, and therefore, the system as it is with these outcomes cannot be redeemed.Assessment of learning vs assessment for learning
Integrating multiple components of training and assessment
- It is important to recognise that often assessment is of what learning has been achieved. Then if people haven't quite achieved the learning or haven't had the opportunities to achieve that learning, then the assessment is actually not assessing the individual, but assessing the system that got them to that place. Whereas, if one thinks about moving onto a system where one assesses for learning; so one finds out what has been achieved, and what has not, therefore how one should move forward. Then, we might be thinking of a culturally different focus which also incorporates the education and training system and not solely the candidate.
HEE’s commitment to fairness and value for money
- Participants recognised that there was no formal linkage between the different parts of the system, such as recruitment, training, assessment (in real-life settings), summative examinations, and when the assessment of progression (in Annual Review of Competency Progression – ARCPs). There is no robust system of information sharing between each component of an individual’s journey. And there are situations frequently, where the individuals result in a summative assessment may be incongruous with their overall performance in other assessment interactions.
- Therefore there is a need to think about a system that integrates all the processes including formative and summative assessments in real-life workplace scenarios, conducted not by a single assessor but a whole range of assessors.
- Data Source 1 – Multi-assessor, Multi-dimension Training Portfolio; There exists for most trainees a comprehensive portfolio of multiple assessments, for example, supervised learning events done by 10 different individuals or multiple individual feedback (Multisource feedback, Multiple consultant reports), and there is a potential that this can balance /compensate for any theoretical biases that might exist in that.
- Data Source 2 – Externally assessed multi-dimensional summative assessment; Considering examinations, currently, one looks at the exam as a pass or fail, and we set an artificial standard/benchmark, whereas the exams often assess in multiple dimensions. Every exam has multiple domains, (like communication, physical examination, and decision-making skills that one is assessing), yet when assessing an individual, the system ignores all the multi-domain data and accepts a dichotomous, arbitrary threshold.
- Triangulation and Progression Decision - And finally, the role of an independent ARCP panel, which assesses all the evidence, and then makes a progression decision.
- Although this system of integrating the different sources of information on competency at a multi-dimensional level does not directly address the DA phenomenon, there is a real chance that the assessment system which receives information from multiple real-life encounters with a range of (appropriately trained and supported) supervisors, an appropriate multi-dimensional summative assessment and a training system which recognises and supports, the differential needs of individuals, is much more likely to offer fairness and equality.
- A review of ARCPs for Health Education England found that ARCP outcomes are often hugely variable by specialty and by deanery and, but also it appeared that the process, as well, was very variable. The participants agreed there was a significant reduction in the inconsistencies in ARCP decisions following the reform and standardisation.
- ARCPs are not designed to be assessment but a process for making a judgment based on the information presented; so information from exams, information from supervisors, but the ARCP outcome is just that ‘have you learned enough to progress?’
On the role of Clinical Supervisors
- The participants recognised the established partnership between HEE, NHS E/I, GMC, and the colleges and the consensus for support around tackling DA.
- There was a recognition of the medical workforce challenges and the responsibility of the system to reform such that medical training is delivering value for public money as well as safe and competent doctors.
- There was an acknowledgment that structural racism existed in some organisational processes of recruitment, assessment, and in the examination process. Some of it overt, some covert, and that one would be complicit if one were to accept this.
- There was an agreement to establish a fair, transparent, and robust pathway for an individual that is coming into training, through stages of progression, assessment, Continuous Professional Development (CPD), and right the way to retirement. To set up processes to be accountable and outcomes that are measurable. There was a resolve to take decisive action at any point where this was found to be failing. But to do this in a way that the final outcome is free of bias and prejudice.
- The participants discussed ways to support clinical and educational supervisors, as there was often observed that there were inconsistencies in marking informal assessments compared to the rigour and robustness accompanying an external summative assessment.
- There was agreement that focus has to be much more on how a supervisor does, have the time, but also have the tools and the confidence to help people with their learning. If the system feels happier that supervisors are able to make that judgment about capability, then this may be more reliable than a single point of assessment i.e. the exam.
- It was recognised that colleges can have real input by educating supervisors. By helping them to understand how to give feedback to people and how to make sure that trainees are getting the feedback they need, throughout the year. Hence, additional support can be initiated before any unexpected debacles at the end of the training.
Summary – a way forward
- There was agreement on the need for improved educational supervision, not just for those in training rotations, but for all doctors in the system. The participants agreed that there was a real challenge in clinicians in (clinical and educational) supervisory roles having the time in their job plans to supervise. The system needs to support the supervisors, as the knock-on effect of poor supervision is manifest in poor clinical outcomes, litigation, delayed career progression, and poor retention with significant economic consequences.
- The participants agreed that there was a risk of stigmatisation if some people are singled out for additional support but if every doctor has a ‘personalised training plan’ then it is likely that the culture will recognise that individuals have different training needs, rather than as a sign of weakness or incompetence.
- Previously the need to extend training was seen as a failure, yet the COVID pandemic has changed that perception. And one of the things that COVID has done is allowed people to accept extensions because it's ‘no-fault’, but the culture shift should make it acceptable that people have individualised training needs.
The participants agreed
What can the college(s) do?
- that assessment needs to change but that needs to be done in a structured way
- that the system needs to identify the trainees early and give them support to get through the system,
- that there needs to be an investment in developing a better support system for supervisors in training, time, and skills.
- Continue to raise the issue of differential awards.
- Continue to provide a comprehensive equality and diversity training and unconscious bias training within the exam systems, for the clinical and the written exams.
- Develop better support for people going through the assessments
- Develop educational support tools for education and clinical supervisors working closely with HEE and GMC to do that because it needs to apply across all 4 nations. This may be best delivered through the academy (AoMRC) while being careful it doesn’t lose momentum or get embroiled in bureaucracy.
- The roundtable concluded with a vote of thanks from the Sponsors and a proposal that a focussed Task & Finish group is set up to explore the solutions as a deep dive and present concrete steps at the forthcoming workshop.
- The Chairs offered their personal commitment to move this issue towards a solution. There was agreement on setting up a smaller task and finish group, exclusively focussed on summative assessments, which takes some membership from this group but broadens it out to include trainees and international experts. And then takes the outcome of that into a follow-up workshop in a few months’ time, timeline to be determined.
- Dave, S., Chakravorty, I., Menon, G., Sidhu, K., Bamrah, J., & Mehta, R. (2020). Differential Attainment in Summative Assessments within Postgraduate Medical Education & Training: 2020 Thematic Series on Tackling Differential Attainment in Healthcare Professions. Sushruta Journal of Health Policy & Opinion, 13(3). https://doi.org/10.38192/13.3.15
- Scarborough, N. Differential Attainment case study: How Health Education East Midlands (HEE EM) are using early performance indicators to provide tailored support for GP trainees. https://www.gmc-uk.org/-/media/documents/itap-case-study_pdf-75325035.pdf