Section 7: Assessment criteria – E&I

Introduction

This section should be read alongside Section 6: E&I guidance in the Guidance, which sets out the definitions of engagement and impact for REF 2029, and the requirements for submitting impact case studies (ICS). The sub-panels will assess Engagement and Impact (E&I) in accordance with this framework.  

The panels wish to encourage the submission of a broad range of impact types across all disciplines, including those arising from new or emerging forms of research and engagement and those underpinned by diverse forms of research activity and output. The main and sub-panels have determined that no single model will be considered intrinsically preferable. Each ICS will be assessed on its own merits. The panels recognise that excellent impact can be achieved through diverse approaches, and responsible research and/or engagement can be key contributors to achieving impact. The panels welcome ICS that demonstrate how impact has been achieved through responsible and inclusive practices appropriate to the discipline. Where responsible practices shaped how impact was achieved, sustained or extended, accounts of the practice will strengthen the case made for reach and significance. Further definition of responsible practices and engagement can be found in sub-section 6., Section 8: Assessment criteria – SPRE and sub-section 12., Annex C: Additional guidance on ICS development both in the Panel criteria and sub-section 4.1., Section 6: E&I guidance in the Guidance.

In drawing up their assessment criteria, the main panels strongly advise institutions that the provided examples in Annex C: Additional guidance on ICS development in the Panel criteria are illustrative only. They should not be read as exhaustive or prescriptive, and the panels anticipate that excellent impact will be submitted which does not relate to any of the examples provided in the guidance. The examples are offered to assist institutions, not to constrain them.

The panels acknowledge that there are multiple, diverse pathways through which research has impact, with variations reflecting:

• non-linearity: the relationship between research and impact can be indirect, non-linear and arise through planned or unanticipated engagement pathways, or be serendipitous. However, the relationship should be clearly evidenced
• timescales: impact may arise after the research as an outcome or be demonstrated during the research process itself (for example, through participatory and engaged research or co-production)
• duration: impacts may be temporary or long-lasting. The assessment will focus on the reach and significance of the change or benefit, not solely on its duration
• beneficiaries: impacts may change or benefit diverse groups, ranging from individuals and local communities to entire nations or international organisations. Engagement may be an important feature of many impacts as the mechanism by which the impact claimed has been achieved

Engagement and impact assessment criteria

The sub-panels will assess the ‘reach and significance’ of impacts on the economy, society and/or culture that were underpinned by research aligning to the UoA descriptor conducted in the HEI, according to the generic criteria and level definitions. This section provides a descriptive account of how the sub-panels will interpret and apply the generic criteria for assessing impact:

• reach will be understood as the extent and/or diversity of the beneficiaries of the impact, as relevant to the nature of the impact. Reach will be assessed in terms of the extent to which the potential constituencies, number or groups of beneficiaries have been reached; it will not be assessed in purely geographic terms, nor in terms of absolute numbers of beneficiaries. The criteria will be applied wherever the impact occurred, regardless of location whether in the UK and/or abroad
• significance will be understood as the degree to which the impact has enabled, enriched, influenced, informed and/or changed the performance, policies, practices, products, services, understanding, awareness or wellbeing of the beneficiaries

The sub-panels will make an overall judgement about the reach and significance of impacts, rather than assessing each criterion separately. While ICS need to demonstrate both reach and significance, the balance between them may vary at all quality levels. The sub-panels will exercise their judgement on reach and significance without privileging or disadvantaging either reach or significance.

The advisory panels will work with the sub-panels to ensure consistent recognition of appropriate engagement practices and approaches to impact generation across all UoAs.

HEIs may submit ICS describing impacts at any stage of development or maturity. However, the assessment will be solely on the impact achieved during the assessment period, regardless of its stage of maturity. No account will be taken of anticipated or future potential impact, nor of impact that occurred outside the assessment period.

The role of engagement

For the purposes of the REF, engagement is defined as an interaction between the HEI and relevant individuals, groups, communities, organisations, the public, commercial partners or policy makers that is purposeful, responsible and context appropriate (see sub-section 4.1., Section 6: E&I guidance in the Guidance).

Engagement is not assessed. The reach and significance of the impact is assessed. Where an account of engagement helps sub-panels understand how that impact came about by explaining who was reached and why they were the relevant constituency, why a particular approach suited that context or how the research came to be taken up, it will be taken into account as part of the impact claim. It will not be scored separately. Examples of ways to discuss the methods and domains of engagement can be found in sub-section 12., Annex C: Additional guidance on ICS development in the Panel criteria. The widest range of approaches to engagement are welcome, as appropriate.

There is no expectation that discussion of engagement will be included in all ICS and where provided it will not be scored separately from the impact.

Each case study will be assessed on a fair and equal basis regardless of whether or not an engagement narrative is included.

Accounts of engagement activity that are not connected to the impact achieved will not strengthen an ICS.

Range of engagement and impacts

The sub-panels welcome ICS that describe any type(s) of impact which fulfil(s) the definition of impact for the REF (see sub-section 4., Section 6: E&I guidance in the Guidance). They acknowledge that impact may take many forms and occur in a wide range of spheres. All sub-panels welcome ICS which describe impacts that have provided effect, change or benefit to one or more areas of the economy, society, culture, public policy and services, health, production, environment, international development, public understanding or quality of life. The sub-panels will also welcome impacts that describe effects, changes or benefits resulting from research that leads to a decision not to undertake a particular course of action. Sub-panels recognise the value of co-produced impact and expect institutions to ensure ICS of this nature clearly acknowledge the work and contribution of partner organisations. ICS of this nature must also clearly define the scope of the institution’s research aligning to the UoA to which the ICS is submitted.

Impact can be manifested in many spheres with many entities including, but not limited to, individuals, organisations, communities, industry, regions and other entities. Impacts can include those on products, processes, behaviours, policies, practices and understanding and avoidance of harm or the waste of resources in the widest sense. Impact of any type may be local, regional, national or international, in any part of the world.

Research may underpin impact which produced changes or provided benefits in more than one area. An ICS may therefore describe more than one type of impact arising from such bodies of work; for example, a new drug can generate both health and economic impact, a new energy technology can generate both environmental and economic impact and a new exhibition or performance can generate cultural, economic and social changes or benefits.

Indicative examples of types of impact, types of engagement, and evidence of impact are provided in Annex C: Additional guidance on ICS development in the Panel criteria as a guide to the range of potential impacts that may be eligible as ICS. The list is not exhaustive or exclusive and does not rank examples in any way.

In making use of this to assist with the preparation of submissions, HEIs should note that:

• the list of types and examples of impacts is not intended to be exhaustive, and some examples are relevant to more than one type of impact. Sub-panels wish to encourage HEIs to submit ICS describing any impacts that meet the generic definition of impact
• HEIs are not required to align ICS with the types of impact defined in the list or to categorise their engagement. While HEIs are required to provide evidence of the impact(s) claimed, no particular form of evidence is inherently privileged over any other. The resource is to provide ideas and indicate the breadth of engagement and impact welcome in ICS
• submitting units should describe their impact using relevant evidence of the change or benefit
• submitting units should describe their engagement approach where it will enhance their claims for reach and significance (see paragraph 2.0.1.)

Impacts on or through teaching within and beyond the submitting institution may be submitted, and examples are included in Annex C: Additional guidance on ICS development in the Panel criteria. Where impact on teaching within the submitting unit’s own institution is included in an ICS, sub-panels will give consideration to the following:

• the reach of the impact, that is, the extent or diversity of the communities affected by the change to teaching practice
• the significance of the impact, that is, the extent to which teaching practice was enriched, influenced or changed at the organisation(s) involved and/or the extent to which individuals experiencing changed teaching practice were enriched, influenced or changed

Sub-panels expect that impact on teaching within the submitting unit’s own institution may most convincingly form a component of a wider ICS that also includes impacts beyond the institution.

The panels acknowledge that there may be impacts arising from research which take forms such as holding public or private bodies to account, or subjects proposed changes in society, public policy or business practices to public scrutiny. Such holding to account or public scrutiny may have had the effect of a proposed change not taking place; there may be circumstances in which this of itself is claimed as an impact. There may also be examples of research findings having been communicated to, but not necessarily acted upon, by the intended audience, but which nevertheless can be evidenced to have made a change or benefit to critical public debate around policy, social or business issues. The panels also recognise that research findings may generate critique or dissent, which leads to impact(s). For example, research may find that a government’s approach to a particular social, health, food/biosecurity or economic issue is not delivering its objectives, which leads to the approach being questioned or modified.

Continued impact ICS

As set out in sub-section 7.1., Section 6: E&I guidance in the Guidance, ICS which represent a continuation of ICS submitted in 2014 and/or 2021 will be eligible for submission in REF 2029.

The sub-panels will assess each ICS on merit and do not wish to receive information on whether ICS are continued ICS. There will be no continuing ICS flag for REF 2029.

The sub-panels encourage submitting units to submit their strongest ICS irrespective of whether they are new examples or represent continuing impact from those submitted in REF 2014 and/or REF 2021.

Evidence of impact

Each case study must provide a clear and coherent narrative that includes an account of who or what audiences, constituencies, groups, organisations, places, publics and/or sectors have benefited, been influenced, acted upon or co-created with. The sub-panels will use their expert judgement regarding the integrity, coherence and clarity of the narrative of each case study but will expect the key claims made in the narrative to be supported by evidence and indicators.

In assessing ICS, sub-panels will consider both the evidence linking research and bodies of work (see paragraph 8.6.3.,Section 6: E&I guidance in the Guidance) within the submitting unit to the impact(s) claimed, and the evidence of the reach and significance of the impact.

The link between the underpinning research and the impact can be indirect or non-linear, and examples of methods of engagement that may help explain this link are listed in Annex C: Additional guidance on ICS development in the Panel criteria.

Responsible practice is often part of the explanation for why impact occurred, rather than a separate matter of process. As such, brief descriptions of responsible practice may, if relevant, support the evidence of impact.

Within their narrative account in the case study, submitting units should provide the indicators and evidence most appropriate to support the impact(s) claimed.

Submitting units should focus on providing evidence of the impacts achieved to demonstrate both the reach and significance of the impact(s) claimed, including methods of engagement where these add to the claims for reach and/or significance. As for REF 2021, evidence of dissemination on its own is not evidence of impact. Conversely, a rationale for a particular mode of engagement may contextualise the reach of an impact by showing adaptation to the relevant audiences or co-creators of the impact.

Submitting units should ensure that, so far as possible, any evidence cited is independently verifiable. Details of verifiable sources for key evidence and indicators should be provided in the sources section of the ICS template (see sub-section 9.2., Section 6: E&I guidance in the Guidance). For evidence available as a web link, the web link should be included in the list of sources. Files and digital artefacts should be uploaded to the Submission system as a zip file where possible. Where the evidence cannot be provided as a digital file or link, it should be retained by the institution for provision in the event of audit and not submitted routinely.

The main panels recognise that some evidence in ICS may be of a confidential or sensitive nature. The arrangements for submitting and assessing ICS that contain confidential elements are outlined in sub-section 11.1., Section 6: E&I guidance in the Guidance. The arrangements for submitting and assessing ICS that contain material with a national security classification are set out in paragraph 3.3.3., Section 2: Submissions in the Guidance.

The panels anticipate that ICS will refer to a wide range of types of evidence, including qualitative, quantitative and tangible or material evidence, as appropriate. Individual ICS may draw on a variety of forms of evidence and indicators (as set out in Annex C: Additional guidance on ICS development in the Panel criteria).

The panels encourage submitting units to use evidence most appropriate to the impact claimed. A diversity of evidence is welcome, and no type of evidence is inherently preferred over another. Cited evidence should provide a convincing and verifiable link between the underpinning research or bodies of work and the impact claimed, as well as convincing and verifiable evidence of the reach and significance of the impact.

Testimonials should draw on statements of fact and relate specifically to the impact(s) claimed. There may be occasions where opinion-based testimonials are appropriate (for example, where the impact is on public understanding of an issue). Where such testimonials are cited as evidence in ICS, sub-panels will consider the extent to which the testimonial evidences the reach or significance of the impact. Sub-panels recognise the varying degrees to which evidence and indicator information may be available to HEIs. Where testimony is cited, it should be made clear whether the source is a participant in the process of impact delivery (and the degree to which this is the case) or is a beneficiary or reporter.

Where corroborating evidence is reviewed as a consequence of audit, it will be used solely to verify the claims made about the impact. Additional information included in such evidence will not be used to supplement or strengthen the ICS narrative.

The examples in Annex C: Additional guidance on ICS development in the Panel criteria provide a guide to potential types of evidence or indicators that may be most relevant to each of the broad areas of impact. However, institutions should note that:

• this is not intended to be exhaustive
• many indicators are relevant to more than one type of impact
• sub-panels will consider any relevant, verifiable evidence

Underpinning research

Sub-panels welcome the full diversity of research methods and output types for underpinning research as described in sub-section 2., Section 1: Overview in the Guidance.

Sub-panels need to be assured that the impact claimed is based on research, and panels will consider if the underpinning research meets the REF definition of research.

The quality of the underpinning research will not be taken into consideration as part of the assessment of the reach and significance of the claimed impact. However, an ICS must have at least one output that meets the REF 2029 definition of research and without this it will be classified as ‘ineligible’. Where one of the references cited does not meet the definition of research, it will be disregarded and the ICS assessed on the remaining references.

The underpinning research must align with the UoA descriptor for the UoA the ICS is submitted to, or be part of an aggregate submission.

Submitting units are required to identify the research that underpins the impact described in the ICS (see paragraph 8.1.1., Section 6: E&I guidance in the Guidance). They should provide references for up to six key pieces of research produced by the HEI and aligning to the UoA descriptor published in the period 1 January 2008 to 31 December 2028. This underpinning research must have made a distinct and material contribution to the impact described in the ICS. Sub-panels will expect to see clear narrative evidence of this in the ICS.

Where engagement forms part of the underpinning research, the underpinning research section should be used to describe where researchers worked with users, partners, practitioners and communities to shape the research questions, refine research design or test emerging findings against the setting that the research was intended to affect.

The panels recognise that several groups, institutions or organisations may have made distinct research contributions to a given impact and strongly advise submitting units to ensure that the contribution of the institution’s research aligning to the UoA is specified clearly and that the contributions of other units and institutions are acknowledged.

Underpinning research referenced in an ICS may, but does not need to, be included in a submission as an output for CKU. Where it is also submitted for CKU, the assessment of the ICS and the assessment of the output for CKU will remain independent processes.

There will be many cases where a researcher has moved to a different institution during the period in which a body of research underpinning an ICS was produced. Where this is the case, the submitting institution should make clear that the research undertaken during the period the researcher spent at that institution made a distinct and material contribution to the impact claimed.

Preparing impact case studies (ICS)

The sub-panels recommend that institutions refer to the following list of characteristics when preparing ICs:

• all the material required to make a judgement should be included in the ICS template – no further reading should be required. URLs should only be included for the purpose of verifying or corroborating claims made in the submission. Panels will not follow URLs to access additional evidence or information to supplement the submission
• there should be a clear definition of the impact beneficiaries and/or co-creators, and what has changed as a result of the research
• the narrative should be coherent, clearly explaining the relationship between the researchers, the underpinning research, the impact, the nature of the changes or benefits arising (noting that narratives differ according to the areas of impact claimed) and, if pertinent to the claims of reach and significance, the methods and/or domains of engagement (see Annex C: Additional guidance on ICS development in the Panel criteria)
• indicators used should be relevant, contextualised and precise in support of the ICS, and the evidence should be verifiable, focused and concise
• there should be a brief explanation of what is original or distinctive about the research insights that contributed to the impact
• specific and appropriate sources of corroborating information, independent of the submitting HEI, should be supplied. Extracts from corroborating statements may be included within the ICS, where appropriate
• where the research was carried out in collaboration with other HEIs, or was part of a wider body of research, this should be acknowledged and the specific contribution to the impact of the submitting unit’s research clearly described. In such cases, units (whether within or across HEIs) may provide common descriptions of the impact arising, where they so wish
• where relevant and considered to enhance the reach or significance of the impact, the engagement strategy (and activities that enabled impact) and any relevant indicators of responsible practices (which enabled, sustained or enhanced the engagement or the impact) may be included