Section 6: Assessment criteria – Contribution to Knowledge and Understanding (CKU)

CKU criteria and level definitions

In REF 2021, each main panel provided their own interpretation of the CKU criteria. For REF 2029, the main panels have worked together to provide a harmonised interpretation to reduce burden for HEIs in assessing the full range of broad research practices and output types including those that cross UoA boundaries and interdisciplinary research (IDR). In doing so, the main panels are keen to emphasise that this harmonisation is not a change to the generic criteria or an alteration in expectations of the quality required at each starred level, but an exercise in agreeing mutual language that works across all panels.

Sub-panels assess each CKU submission against the generic starred level definitions, as set out in Annex A: Assessment criteria and level definitions in the Panel criteria. The supplementary criteria below apply across all four main panels and are intended to assist sub-panels and reviewers in applying those generic definitions consistently and equitably across the full range of disciplines.

These criteria complement, rather than replace, the generic quality level definitions. Sub-panels will apply them holistically as a set of indicative characteristics, recognising that not all characteristics need be present for a contribution to be assessed at a given level. They will be applied with appropriate sensitivity to disciplinary and/or interdisciplinary context and to the full, diverse range of output types submitted to REF 2029, including but not limited to the types of output listed in Annex B: Output glossary in the Guidance. An underpinning principle of REF 2029 is that all types and formats of research output will be assessed on a fair and equal basis. Sub-panels will not regard any particular form of output as of intrinsically greater or lesser quality than another, and all subject matter and research practices will be assessed on a fair and equal basis, without bias.

The terms ‘world-leading’, ‘internationally’ and ‘nationally’ are quality benchmarks. They relate to the actual, likely or deserved influence of the work and do not imply any requirement for international exposure in terms of publication venue, geographic scope of the research topic or language of dissemination. Work published in languages other than English, Welsh or BSL will be assessed equitably.

Level definitions

In assessing contribution to knowledge and understanding, sub-panels will look for evidence of originality, significance and rigour, and apply the generic definitions of the starred quality levels. Sub-panels recognise that such evidence may take different forms across Units of Assessment, output types and formats. The following supplementary characteristics are intended to guide assessment at each level.

Four star: world-leading

In assessing work as being four star, sub-panels will expect to see evidence of, or potential for, some of the following characteristics:

Originality

• outstanding contribution to knowledge through developing novel concepts, paradigms, insights, methods, practices or resources
• exceptional expansion of the range and depth of knowledge and its application

Significance

• agenda-setting research with influence on thinking, practice or paradigms within its field and potentially beyond it, an essential point of reference
• leading influence, where appropriate, on policy or practice, including management, processes, production, performance or audience engagement

Rigour

• exemplary articulation and rationale, convincing supporting evidence (as appropriate) and comprehensive engagement with its subject
• exemplary research design, methods and techniques of investigation, practice, performance and/or analysis

Three star: internationally excellent

In assessing work as being three star, sub-panels will expect to see evidence of, or potential for, some of the following characteristics:

Originality

• important contribution to knowledge through developing novel concepts, paradigms, insights, methods, practices or resources
• substantial expansion of the range and depth of knowledge and its application

Significance

• research with substantial influence on thinking, practice or paradigms within its field, an important point of reference 
• substantial influence, where appropriate, on policy or practice, including management, processes, production, performance or audience engagement

Rigour

• strong articulation, rationale and supporting evidence (as appropriate) and thorough engagement with its subject
• robust research design, methods and techniques of investigation, practice, performance and/or analysis

Two star: recognised internationally

In assessing work as being two star, sub-panels will expect to see evidence of, or potential for, some of the following characteristics:

Originality

• valuable incremental contribution to knowledge through extending existing concepts, paradigms, insights, methods, practices or resources
• demonstrable expansion of the range and depth of knowledge and its application

Significance

• research with demonstrable influence on thinking, practice or paradigms within its field
• demonstrable influence, where appropriate, on policy or practice, including management, processes, production, performance or audience engagement

Rigour

• good articulation, rationale, supporting evidence (as appropriate) and completeness of engagement with its subject
• thorough and professional research design, methods and techniques of investigation, practice, performance and/or analysis

One star: recognised nationally

In assessing work as being one star, sub-panels will expect to see evidence of, or potential for, some of the following characteristics:

Originality

• minor contribution through the application of existing concepts, paradigms, insights, methods, practices or resources
• minor expansion of the range and depth of knowledge and its application

Significance

• research with limited influence on thinking, practice or paradigms within its field
• minor influence, where appropriate, on policy or practice, including management, processes, production, performance or audience engagement

Rigour

• limited articulation, rationale, supporting evidence (as appropriate) and completeness of engagement with its subject
• sound research design, methods and techniques of investigation, practice, performance and/or analysis

Unclassified

A contribution will be graded unclassified if it falls below the quality threshold for one star.

Ineligible

A contribution will be graded as ineligible if it does not meet the published definition of research for the purposes of REF 2029, or otherwise does not meet eligibility requirements for submission.

Output types

The main panels welcome all forms of research output that fulfil the eligibility criteria for the REF (set out in sub-section 5., Section 4: CKU guidance in the Guidance). All forms of output, irrespective of language, will be considered equitably, with no distinction being made between the type of research or form of output submitted. The sub-panels will neither advantage nor disadvantage any type of research or form of output. The main panels encourage submitting institutions to refer to the glossary of output types for information on the categories under which outputs may be submitted for assessment (see Annex B: Output glossary in the Guidance).

No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of outputs. No output will be privileged or disadvantaged on the basis of the publisher, where it is published or the medium of its publication.

Reviews, textbooks and edited works (including editions of texts) and translations may be included if they embody research as defined in sub-section 5., Section 4: CKU guidance in the Guidance. Editorships of journals and other activities associated with the dissemination of research findings should not be submitted as an output.

Each submitted output needs to have a single classification selected from the list of eligible output types (see Annex B: Output glossary in the Guidance). The purpose of the classification is to assist in the management of the collection and distribution of outputs, the allocation of outputs to expert reviewers and a post-submission analysis of types of outputs submitted. The sub-panel will assess the research content of the material submitted regardless of the classification.

Main Panel D supplementary criteria – output types

1. The classification of ‘Translation’ has been retained from REF 2021 for the submission of works of translation of literary or scholarly texts or other cultural documents that constitute original, significant and rigorous research. Translations that meet the definition of research will often exhibit a deep insight into the source material, while drawing on and reflecting specialist knowledge of its historical, political, social and cultural contexts, and will also rely on a detailed engagement with style in both the source and target languages. Research may, as a result, be reflected in the critical apparatus associated with a translated text but will also be inherent in the translation process itself. Such outputs will often contribute to the development and maintenance of intellectual infrastructure of subjects and disciplines. They may demonstrate research practice that is critical and/or creative, and may also serve as substantial interventions in intellectual and cultural life in their own right.
2. For any output type, for indicative guidance on what material to include in the submission, please refer to the table of output types in Annex B: Output glossary in the Guidance and Annex D: Summary of additional information about outputs in the Panel criteria.

Outputs with significant material in common

As stated in sub-section 5.7., Section 4: CKU guidance in the Guidance concerning significant material in common, where two or more research outputs within a submission include significant material in common, the sub-panels will assess each output taking account of the common material only once. Where a sub-panel judges that these output(s) do not contain sufficiently distinct material and should be treated as a single output, an ineligible score will be given to the ‘missing’ output.

Where a submitted output includes significant material in common with an output submitted to REF 2021, submissions should explain how far the earlier work was revised to incorporate new material (maximum 100 words).

Co-authored outputs

For co-authored outputs:

• the number of other authors will be required (as outlined in paragraph 10.7.1., Section 4: CKU guidance in the Guidance)
• an output (whether single or co-authored) may only be included once in a given UoA submission, by any given HEI
• for co-authored outputs where there are multiple substantive links to the institution, the output can be submitted in as many different UoAs as the number of substantive links to the institution, provided that the output meets the descriptor for those UoAs and all other conditions stipulated in the CKU guidance are met
• where the co-authors have substantive links to different institutions, the output can be submitted by each HEI

For outputs to be eligible for submission, the author upon whom the identified substantive link is based must have made a significant research contribution to the output. The main panels set out below their requirements for information demonstrating a significant research contribution.

Where information is requested and a sub-panel judges that the author upon whom the substantive link is based has not made a significant research contribution to a co-authored output, the sub-panel will grade that occurrence of the output as ineligible.

Information may also be requested through audit to verify that the author upon whom the substantive link is based made a significant research contribution to the output. Where this cannot be verified the output will be graded as ineligible.

Once a sub-panel has determined that the author upon whom the substantive link is based has made a significant research contribution, it will assess the quality of the output as a whole, taking no further regard of each individual co-author’s contribution.

Outputs may only be submitted to UoAs where they meet the UoA descriptor, except where the conditions for aggregating disciplines into a single submitting unit are met (as outlined in sub-section 1., Section 5: Assessment criteria – Submissions in the Panel criteria). Submission to multiple UoAs is only appropriate for multi-authored outputs that align to UoAs where the UoA descriptors overlap, or where the output is interdisciplinary.

The sub-panel executive will be able to check in the Submission system whether outputs have been submitted to multiple UoAs if necessary to support consideration of interdisciplinarity, joint assessment and audit requests.

Main Panel A supplementary criteria – co-authored outputs

1. Additional information on the substantial research contribution of the author upon whom the substantive link is based is required in the following instances:
   1. the output has 50 or more authors and the identified substantive link is via an author who is not clearly identified as either lead or corresponding author
   2. the author upon whom the identified substantive link is based is part of a consortium or group in the authorship listing
2. In these instances, institutions are required to affirm the substantial contribution to the research by the staff member upon whom the substantive link is based. This should be done by entering the staff member’s name (and that of the consortium or group if relevant) and up to 100 words in which the author contribution is articulated, including at least one element from each of the points below:
   1. the author upon whom the substantive link is based made a substantial contribution either to the conception and design of the study; or to the organisation of the conduct of the study; or to carrying out the study; or to analysis and interpretation of study data
      
      AND
      
   2. the author upon whom the substantive link is based helped draft the output; or critiqued the output for important intellectual content

Main Panel B supplementary criteria – co-authored outputs

1. Additional information on the substantial research contribution of the author upon whom the substantive link is based is required in the following instances:
   1. the output has 50 or more authors and the identified substantive link is via an author who is not clearly identified as the corresponding author
   2. the author upon whom the identified substantive link is based is part of a consortium or group in the authorship listing
2. In these instances, institutions are required to affirm the substantial contribution to the research by the staff member upon whom the substantive link is based. This should be done by entering the staff member’s name (and that of the consortium or group if relevant) and up to 100 words in which the author contribution is articulated.
3. In cases where a statement on substantial research contribution is not required, the sub-panels may seek to verify a contribution via audit in accordance with paragraph 4.0.4.

Main Panels C and D supplementary criteria – co-authored outputs

1. The sub-panels in Main Panel C do not require the submission of information about the contribution to co-authored outputs by the author upon whom the substantive link is based and, if received, will take no account of such statements. The sub-panels may seek to verify a contribution via audit in accordance with paragraph 4.0.4.

Requests to double-weight outputs

The main and sub-panels recognise that there will be cases where the scale of academic investment in the research activity and/or the scope of the research output is considerable. The main and sub-panels want to recognise and double-weight such outputs in the assessment so that they will count as two outputs both in a submission and in the calculation of the CKU sub-profile. The main panels have set out below their expectations in relation to receiving requests for double-weighting.

Institutions’ requests for double-weighting must be accompanied by a statement of up to 100 words explaining how the output meets the criteria for double-weighting. The statement should not simply describe the nature and form of the output as this is unlikely to provide sub-panels with the information required to judge whether an output should be double-weighted.

As set out in sub-section 18.2., Section 4: CKU in the Guidance, a reserve output may be submitted where a request for double-weighting is made. The reserve output may be based on any eligible substantive link at the HEI, but HEIs should consider the expectation, in the event the reserve output is assessed, that there will generally be no more than five outputs associated with a single substantive link within a submission at UoA level. Submitting units will provide a list of ‘reserve’ outputs in priority order and, if a reserve output is required due to the double-weighting request being declined, the sub-panel will assess the reserve item at the top of the reserve list.

Sub-panels will double-weight an output only if a request is made by the submitting institution and the case is accepted by the sub-panel. Sub-panels will not double-weight any output for which a request has not been made by the institution.

Sub-panels will assess the claim for double-weighting separately from assessing the quality of the output. When assessing claims for double-weighting, the sub-panel will not privilege or disadvantage any particular form of research or type of output.

Double-weighted outputs will be assessed and scored against the criteria of originality, significance and rigour in the same way as all other outputs.

Sub-panels will assess the claim for double-weighting separately from assessing any extended process or long-form output exception requests. Granting of an extended process/long-form exception does not confer a double-weighted status. Further details on extended process/long-form outputs are available in sub-section 6.

Main Panels A and B supplementary criteria – requests to double-weight outputs

1. The sub-panels encourage submission of outputs which derive from substantial academic endeavour for consideration as double-weighted outputs. Such endeavour might be understood in terms of (but is not limited to) the ambition of the research.
2. Considering the patterns of publication across Main Panels A and B’s areas of activity, the sub-panels expect that such requests will occur only exceptionally. In particular, the sub-panels anticipate that outputs published as journal articles and conference papers will not normally embody work of this nature, and they therefore do not normally expect to receive requests for double-weighting these types of outputs.

Main Panels C and D supplementary criteria – requests to double-weight outputs

1. The sub-panels strongly encourage submission of outputs arising from research of extended scale and scope for consideration as double-weighted outputs. The submission of a statement to evidence the claim for double-weighting is required and should briefly outline the reasons for the request, addressing the characteristics below. The sub-panels in Main Panels C and D have identified the following characteristics which might apply (individually or in combination) to the research effort associated with a double-weighted output:
   1. the production of a longer-form output (for example, book, long-duration creative work or multi-component output) demonstrating sustained research effort
   2. the generation of an extended or complex piece of research
   3. the collection and analysis of a large body of material
   4. the use of primary sources which were extended, complex or difficult to access
   5. the presentation of a critical insight or argument which was dependent upon the completion of a lengthy period of data collection or investigation of materials
   6. the undertaking of a complex, extended and/or multi-layered process of creative investigation (individual or collective)
   7. the investigation of a given theme in considerable depth, from different perspectives and/or in relation to different contexts
   8. significant fieldwork which may be of prolonged duration, complex in nature and/or with difficult to access participants 
   9. research that required the creation and analysis of a very large original dataset, database or repository 
2. It is recognised that in some instances the characteristics listed above may apply to short-form outputs such as journal articles, shorter monographs, journal special issues, edited collections or critical editions, book chapters and short-duration creative work and justify the double-weighting of such items.
3. It is expected that most research monographs, scholarly/critical editions, book-length creative works (for example, novels, poetry collections), book-length translations and longer-form outputs warrant double-weighting, although requests will not automatically be accepted.

Extended process and/or long-form exception

The funding bodies are committed to encouraging diversity in research, with the assessment framework recognising all types of research and forms of output. In support of this commitment, and to mitigate any unintended consequences arising from the assessment framework whereby institutions are discouraged from investing in research methods or output formats that typically take longer to produce, the funding bodies consider that it is appropriate that there is an exception in place in REF 2029 to the requirement for an eligible employment relationship to have occurred either during or before the point at which an output has been made first publicly available. Further details on this exception are available in sub-section 6.4., Section 4: CKU guidance in the Guidance.

Extended process and/or long-form outputs exception requests

For REF 2029, submitting units may submit extended process or long-form outputs which were first made publicly available up to five years before the author (or equivalent) began an eligible employment relationship with the HEI (see sub-section 6.10., Section 4: CKU guidance in the Guidance).

In order to submit outputs of this type, a 100-word statement must be included which outlines why the output meets the criteria for being extended process and/or long-form. The criteria for these are detailed in sub-section 6.2 and sub-section 6.3.. Sub-panels will make informed judgements as to whether an output satisfies the requirements for either of these exceptions, using the 100-word statement as supporting information. Assessment of this statement will focus on fit to the exception(s) only and will not include any consideration of the quality of the output. All outputs deemed in scope of either exception will then be assessed against originality, significance and rigour as per the standard assessment processes for all outputs. 

If an output is deemed by sub-panels not to meet the criteria for extended process or long-form, this output will be deemed ineligible.

It should be noted that where an output is subject to a request for double-weighting and a request to use the extended process and/or long process exception, two separate statements must be supplied; one each for the request for double-weighting and the request for the extended process and/or long-form exception.

Where an output is also subject to a request for double-weighting, a positive assessment as to whether the output is in scope of either the extended process or long-form exceptions does not inform the double-weighting request. Any decision to accept a double-weighting request will be treated as a separate assessment against the criteria for double-weighting outlined in sub-section 5.

It is possible that a request for an extended process or long-form exception may be accepted but that a double-weighting request for the same output is denied.

Extended process outputs

Extended process outputs are the product(s) of a substantial, discrete research endeavour which, due to the nature of the approach or methodology, involved sustained activity, usually over a period of five years or longer. They may be in any format or of any length. 

It is expected that these outputs will have involved one or more of the following methodological approaches or circumstances: 

• the generation of an extended or complex piece of research
• the presentation of a critical insight or argument which was dependent upon the completion of a lengthy period of data collection or investigation of a large body of materials
• significant fieldwork which may be of prolonged duration, complex in nature and/or involving multiple sites or participant groups
• a complex, specialised and/or extensive original dataset, database, repository or numerical model
• research that required prolonged investigation into specialised and/or difficult to access materials, including archival research 
• a significant period to establish governance and/or infrastructure and/or collaborative relationships (including co-design with academic and non-academic stakeholders) 
• the undertaking of a complex, extended and/or multi-layered process of creative investigation (individual or collective) 
• the investigation of a given theme in considerable depth, from different perspectives and/or in relation to different contexts

Not all sub-panels will consider the use of these exceptions as standard. However, all panels will consider each output on a case-by-case basis with reference to the 100-word statement submitted.

Extended process – Main Panels A and B supplementary criteria

1. Sub-panels within Main Panels A and B will not generally accept outputs under this exception unless a clear and compelling justification is provided demonstrating the exceptional nature of the research process underpinning the work. This is because they consider the exception for extended process/long-form outputs less directly applicable to the disciplines within their remit as many of the outputs in these disciplines may reasonably meet the definition of an extended process output, making this routine rather than rare.
2. Only sub-panels 7 and 10 will routinely consider exception requests.

Extended process – Main Panels C and D supplementary criteria 

1. All sub-panels will routinely consider exception requests.

Long-form outputs

Long-form outputs are defined as outputs of extended scale or scope.

The main panels expect that this exception will normally include the following output types. Where a specific output type is not listed here, HEIs may consider whether the output meets the definition of extended process above. 

Book length creative works: includes but is not limited to:

• novels 
• work for the stage 
• audio drama 
• creative non-fiction (memoir, biography or travel writing) 
• collections of verse 

Catalogues: detailed inventories of research materials and special collections.

Corpora: large structured and organized collections of spoken, written or signed texts, objects or material culture stored as digital databases.

Long-form creative outputs: including but not limited to:

• designs of products, artefacts and/or buildings/infrastructure 
• exhibitions 
• extended scale compositions 
• extended scale performances 
• series of art works or design collections
• extended translations
• extended scale films and documentaries

Dictionaries or thesauri: printed or electronic reference tools that arrange their content (subjects, words, topics etc.) according to a particular design (for example, alphabetically, thematically and so on). May be single or multiple volumes.

Large scale reports, systematic review and/or meta-analysis: large scale reports that are either: 

• policy-focused analyses that synthesise and translate cutting edge research for a practitioner audience
• a major report on a key issue, policy, or context, where the submitting author is an author of the output as a whole
• NIHR reports 
• an extended synthesis of relevant studies or different sources of data 

Major databases and repositories: electronic collections of information (data, texts, sounds, images and so on) on particular subjects or domains.

Monographs: includes:

• single and multi-authored research monographs of extended scale and scope
• scholarly/critical editions of extended scale and scope

Multi-component outputs: including:

• portfolio of creative or critical work, including collections of writing or practice-based outputs on a theme or topic, whether curated by a publisher, media or digital platform
• creative/practice-based portfolios arising from a single research process, including editions, design of artefacts, collections, series, games 
• multi-component portfolio with different elements (for example, practice-output plus journal article/s) 

Software: substantial new software built to process massive datasets, run complex simulations and/or support complex IDR.

Translations: a book length output which may include but is not limited to:

• translations of novels, collections of poetry, theatre texts or other literary works
• translations of primary or secondary sources from across a broad range of disciplines or fields

While all sub-panels welcome the submission of long-form outputs where they meet the wider eligibility criteria, only some sub-panels will routinely consider the long-form exception. However, all panels will consider each output on a case-by-case basis, with reference to the 100-word statement submitted.

Long-form – Main Panels A and B supplementary criteria

1. No sub-panel will routinely consider exception requests.

Long-form – Main Panel C supplementary criteria

1. No sub-panel will routinely consider exception requests unless the criteria for extended process are met.

Long-form – Main Panel D supplementary criteria

1. All sub-panels will routinely consider exception requests.

Requests to use the five-year pre-publication time period for extended process and/or long-form outputs

The 100-word statement for extended process and/or long-form outputs outlined in sub-section 6. should also be provided for extended process and/or long-form outputs submitted which were first made publicly available between two to five years after the substantive link ended. For further guidance, see paragraph 6.4.3., Section 4: CKU guidance in the Guidance.

The same main panel expectations on the use of this process apply to requests for exceptions where the substantive link occurred before the output was first made publicly available, as outlined in sub-section 6.

If an output is deemed by sub-panels not to meet the criteria for extended process or long-form this output will be deemed ineligible.

Additional information for outputs

The requirements for additional supporting information for each main panel are set out below. A summary of additional information for outputs required by the main panels is set out in Annex D: Summary of additional information about outputs in the Panel criteria.

HEIs are requested to ensure that additional supporting information for outputs is succinct, verifiable, and, for Main Panel B, externally referenced where appropriate (see Main Panel B supplementary criteria applying to Sub-panel 12 (Engineering)). No other information should be included, and sub-panels will take no account of any such information if submitted.

Where additional information is accepted, the information provided must not include citation data or journal impact factors. Any panels that make use of citation data will be provided with the data by the REF team. Sub-panels will take no account of any citation data provided directly by the HEI. No sub-panel will use journal impact factors or any hierarchy of journals in their assessment of outputs.

For research outputs in languages other than English (see sub-section 16.3., Section 4: CKU guidance in the Guidance) submitted to all sub-panels, a short abstract (up to 300 words) in English should be provided to describe the content and nature of the work. This abstract will be used only for the purpose of allocation to appropriate assessors. See Main Panel D supplementary criteria below for an exception applying to Sub-panel 26 (Modern Languages and Linguistics).

For research outputs in Welsh, a short abstract (up to 300 words) in English should be provided to support allocation to an assessor with the appropriate subject and Welsh-language expertise. The abstract will be used solely for this administrative purpose and will not form part of the assessment. The output itself will be assessed directly in Welsh, without reliance on an English translation, and will be treated no less favourably than an output submitted in English.

For research outputs in BSL, a short abstract (up to 300 words) in English should be provided to support allocation to an assessor with the appropriate subject and BSL-language expertise. The abstract will be used solely for this administrative purpose and will not form part of the assessment. The output itself will be assessed directly in BSL, without reliance on an English translation, and will be treated no less favourably than an output submitted in English.

Evidence of research dimensions and research contribution (300-word supporting statement)

All submissions should provide sufficient information to allow a sub-panel to understand the alignment of the output with the REF 2029 definition of research (see sub-section 5.3., Section 4: CKU guidance in the Guidance) and the research contributions on the basis of which a substantive link has been established between the output and the submitting institution (see sub-section 6., Section 4: CKU guidance in the Guidance).

REF 2029 is committed to the equitable recognition of the full diversity of output formats and research practices. Often, the nature of the research and the specific contributions to it will be evident within the materials submitted, requiring no further information. However, where this may not be evident to the assessors from the submitted material, the submitting institution is encouraged to supplement outputs with a statement of up to 300 words. The statement is important in informing the sub-panels’ decisions about the eligibility of outputs but will not itself be assessed.

The statement should clearly articulate the research dimensions of the output and should only be used to supplement, rather than repeat, the information already provided within the output itself (irrespective of output type). For the avoidance of doubt, the statement should not be used to provide evidence of the originality, significance or rigour of the output concerned.

The statement should focus upon how the output aligns with the REF 2029 definition of research, and may include but is not restricted to information about:

• research aims: the objectives, questions, hypotheses and/or issues being explored; the research problem, challenge or critique addressed
• research approach: the process of discovery or inquiry; the methods and/or methodologies deployed; the creative and/or intellectual context or literature review upon which the work draws; the research practices involved in the production of the output
• research insights: the findings, results, discoveries or creative outcomes of the research and how the assessor can navigate the output in order to most effectively understand it
• dissemination: when, how and where the insights or discoveries were ‘effectively shared’ by making the output publicly available for the first time. The output must have been publicly shared for the first time between 1 January 2021 and 31 December 2028
• research contribution: if not explicit in the output itself, the specific research contributions associated with the substantive link to the institution, on the basis of which the output is submitted

Main Panel A supplementary criteria – additional information for outputs

1. Information about the research process and/or content: for any submitted output where the nature of the research and/or the specific contributions are not evident to the assessors from within the materials submitted, the submitting institution is encouraged to supplement outputs with an up to 300-word statement as outlined above.
2. Factual information about significance: the sub-panels do not wish to receive additional information about the significance of outputs and, if received, will take no account of any statement of this kind.
3. Multi-component outputs: for multi-component outputs, the sub-panels would anticipate that these would generally consist of a combination of more than one output type. Specifically, they would not expect to see collections of multiple journal articles submitted, except in the circumstances where together they evidence a single process of investigation leading to new insights, effectively shared.

Main Panel B supplementary criteria – additional information for outputs

1. Information about the research process and/or content (all sub-panels): for any submitted output where the nature of the research and/or the specific contributions are not evident to the assessors from within the materials submitted, the submitting institution is encouraged to supplement outputs with an up to 300-word statement as outlined above.
2. Factual information about significance (sub-panel 12 only): Sub-panel 12 (Engineering) encompasses both fundamental and applied research, where the significance of applied or industrial research may not be fully apparent from the output itself. A short factual statement can evidence such significance, ensuring fair assessment alongside fundamental research or applied research where sufficient evidence can already be found in the output.
3. Where appropriate, a short factual statement (maximum 100 words) could include, for example, additional evidence about how an output has gained recognition or advanced practice through technical innovation, industrial application, influence on standards, adoption by practitioners or recognition by the wider research community.
4. It should not comprise a synopsis of the output, a volunteered opinion as to the quality of the output, or citation data. It is expected that, in most cases, sufficient information will be provided in significantly fewer words than the 100-word limit.
5. The statement will inform the evaluation of the significance of the research reported in the output. It will not lead directly to an increase in the score for the output, but provide additional context for evaluation. The absence of a statement will not be regarded negatively or interpreted as a lack of significance, nor will the inclusion of prohibited information.
6. HEIs are requested to ensure that any evidence provided is succinct, verifiable and externally referenced, where appropriate, with the name and contact details of a senior industrialist to allow verification of claims.
7. Multi-component outputs (all sub-panels): for multi-component outputs, the sub-panels would anticipate that these would generally consist of a combination of more than one output type. Specifically, they would not expect to see collections of multiple journal articles submitted, except in the circumstances where together they evidence a single process of investigation leading to new insights, effectively shared.

Main Panel C supplementary criteria – additional information for outputs

1. Information about the research process and/or content: often, the nature of the research and the specific contributions to it will be evident within the materials submitted, requiring no further information. However, for any submitted output where the nature of the research and/or the specific contributions may not be evident to the assessors from within the materials submitted, the submitting institution is encouraged to supplement outputs with an up to 300-word statement as outlined above.
2. Factual information about significance: the sub-panels do not wish to receive additional information about the significance of outputs and, if received, will take no account of any statement of this kind.
3. Multi-component outputs: for multi-component outputs, the sub-panels would anticipate that these would generally consist of a combination of more than one output type. Specifically, they would not expect to see collections of multiple journal articles submitted, except in the circumstances where together they evidence a single process of investigation leading to new insights, effectively shared.

Main Panel D supplementary criteria – additional information for outputs

1. Option to submit supporting contextual information: for outputs where the role of the researcher upon whom the identified substantive link is based or the research process is not evident in the submitted output to the assessor, submitting units have the option to submit supporting contextual information to enable the panel to access the research dimensions of the work and to assess its originality, significance and rigour. This contextual information may include moving image, sonic, visual or other digital media or written text, as appropriate, and can be submitted to support both single-item outputs and multi-component outputs.
2. HEIs are encouraged to also submit a 300-word statement as outlined above and to provide information within this about what is the output and what is contextual information.
3. The sub-panels will ignore any additional material that includes evaluative commentary on the perceived quality of a research output or comments on its originality, significance or rigour.
4. Translations: where a translation is submitted it requires the following:
   1. a 300-word supporting statement as outlined above, to ensure assessors have a reasonable understanding of the research that has led to the translation output
   2. a copy of the source text which can be achieved via provision of the physical book or digital link to the source text
5. Multi-component outputs: it is not unusual for a multi-component output submitted to the sub-panels in Main Panel D to encompass a number of different output types, such as:
   1. a ‘design’ output which includes a journal article and a patent application
   2. an ‘artefact’ or prototype that has been the subject of an exhibition
   3. a dataset or database which includes critical insight or analysis, with an accompanying paper/journal article
   4. a ‘composition’ that has also been a performance or recording; or an exhibition where the research may be curatorial (or involve or support co-curation) and/or evident in the development of the interpretative strategy, exhibition text/narrative or catalogue
6. Sub-panels in Main Panel D would not expect to see collections of multiple journal articles submitted, except in circumstances where together they evidence a single process of investigation leading to new insights, effectively shared. Submitting institutions should select the multi-component output type, and the panel will judge the research content of the material submitted regardless of the range of output types included.
7. Statement on rationale for grouping short items: substantial dictionary or encyclopaedia entries and groups of short items including, for example, collections of creative writing, series editorials or related critical works (where such work embodies research as defined for the purposes of the REF in sub-section 5.3., Section 4: CKU guidance in the Guidance), may be submitted as either a single output or a multi-component output as appropriate, along with an explanation of the rationale for grouping such items as supporting contextual information of up to 300 words.
8. HEIs are encouraged to also submit a 300-word statement as outlined above and to provide information within this about what is the output and what is contextual information.
9. Information for outputs in languages other than English: for research outputs in languages other than English (see sub-section 16.3., Section 4: CKU guidance in the Guidance) a short abstract (of up to 300 words) in English should be provided to describe the content and nature of the work. This abstract does not form part of the assessment of the submitted output, but enables the panel to allocate the output to an assessor with the appropriate language and subject expertise. This requirement is waived for outputs submitted in UoA 26 (Modern Languages and Linguistics) if the output is produced in any of the languages within the remit of that UoA. 

Citation data

Panels will continue to rely on expert review as the primary means of assessing outputs in order to reach rounded judgements about the full range of assessment criteria (‘originality, significance and rigour’). They will also recognise the significance of outputs beyond academia wherever appropriate and will assess all outputs on an equal basis, regardless of whether or not citation data is available for them.

The use of citation data will follow principles of responsible research assessment, considering best practice frameworks such as the San Francisco Declaration on Research Assessment (DORA) and the Coalition for Advancing Research Assessment (CoARA).

The main panels set out below which of the sub-panels will use citation data. Where sub-panels use citation data, the following criteria apply:

• where available and appropriate, citation data will be considered as an indicator of the academic significance of the research output. This will only be one element to inform peer-review judgements about the quality of the output, and will not be used as a primary tool in the assessment
• panel scoring will only ever be influenced upwards by citation data, usually in marginal circumstances where relevant. Citation data will never disadvantage an output
• the absence of citation data for an output will not be taken to mean an absence of academic significance
• the sub-panels recognise that the citation count is sometimes, but not always, a reliable indicator. They are also aware that such data may not always be available, and the level of citations can vary across disciplines and across UoAs. Sub-panels will be mindful that citation data may be an unreliable indicator for some forms of output (for example, relating to applied research), and the limitations of such data for outputs in languages other than English and for recent outputs. Sub-panels will take due regard of the potential equalities implications of using citation data
• sub-panels will use citation data only where provided by the REF team and will not refer to any additional sources of bibliometric analysis, including, in particular, journal impact factors and other journal rankings

Main Panel A supplementary criteria – citation data

1. All sub-panels in Main Panel A will use citation data, where appropriate and available, as a potential indicator of academic significance to inform the assessment of output quality.

Main Panel B supplementary criteria – citation data

1. Sub-panels 7, 8, 9, and 11 will receive appropriately normalised citation data as a potential indicator of academic significance to inform the assessment of output quality.
2. Sub-panels 10 and 12 will neither receive nor make use of citation data.

Main Panel C supplementary criteria – citation data

1. Sub-panel 16 (Economics and Econometrics) will receive citation data, where available, and will make use of the data supplied by the REF team where it is considered appropriate as an additional piece of supplementary evidence to support the initial assessment of outputs, not as a determining factor. Sub-panel 16 will take account of the well-known limitations of citations, including equality, diversity and inclusion issues.
2. The remaining sub-panels within Main Panel C will neither receive nor make use of citation data.

Main Panel D supplementary criteria – citation data

1. The sub-panels in Main Panel D will neither receive nor make use of citation data.