Section 10: Panel working methods

The REF panel interaction 

The REF Panel Executive comprises the chairs of all main and advisory panels and will also include the IL SPRE Panel chair once appointed.

The Panel Executive, along with the panel advisors, will meet regularly with the REF Director throughout the assessment phase to discuss progress, identify issues for early action and inform the work of all the assessment panels. This will include planning and reporting on calibration exercises, including cross-main panel and IL SPRE Panel calibration, and reviewing emerging and final outcomes across the four main panels and the IL SPRE Panel. Specific actions will be identified to support consistency across those sub-panels in different main panels that have a significant overlap (for example, sharing of some of the material used in calibration exercises, and identifying opportunities for appointing assessors to work across those sub-panels). Specific discussion will also take place to share calibration insights between the IL SPRE Panel and the main panels in regard to SPRE.

The Panel Executive’s responsibilities include:

• calibration of panel criteria and working methods
• recommendations on required panel training during the submission phase
• review of further panel recruitment programmes
• calibration of assessment procedures 
• signing off the IL scores and combining these with the UL sub-profiles to confirm the overall quality profile to be awarded to each submission
• end of exercise report recommendations

Institutional-Level Strategy, People and Research Environment (IL SPRE) Panel working methods

The IL SPRE Panel will assess the information provided in the ILS, informed by the quantitative data and standard analyses and the context section. The IL SPRE Panel will build up a graduated sub-profile by assessing the range of components in each submission, using the starred levels defined in Annex A: Assessment criteria and level definitions in the Panel criteria.

The assessment criteria for IL SPRE have been developed by the main and sub-panels for REF 2029, ahead of the appointment of the IL SPRE Panel. These assessment criteria have received expert input from the SPRE experts on each main panel who will go on to be co-opted as full members of the IL SPRE Panel. The IL SPRE Panel will be responsible for assessing IL SPRE submissions across all submitting HEIs, applying the published criteria and working methods, and recommending outcomes to the Panel Executive.

The full working methods of the IL SPRE Panel will be developed and published following the appointment of the full panel. Institutions can expect that these working methods will align with the working methods of other assessment panels outlined in this Panel criteria.

Main panel working methods

Each main panel has worked with its sub-panels to review and define common assessment criteria. Main panels will work with their sub-panels throughout the assessment process to ensure that the published procedures are followed and that the overall assessment standards are applied consistently. Each main panel will also be responsible for deciding on the unit level sub-profiles to be awarded to each submission in each of the UoAs in its remit, after recommendations have been made by the sub-panels.

Each main panel will work with its sub-panels as follows.

Main panel meetings: the main panels will meet regularly throughout the planning and assessment phases to ensure close working and communication between sub-panels, to identify issues for early action, seek advice on handling specific cases, resolve emerging differences, share developing good practice and provide assurance on the procedures being followed. Sub-panel chairs will report to the main panel meetings on general progress and on the implementation of working methods, particularly on issues where cross-panel consistency is significant, including:

• interdisciplinary research (IDR) outputs
• joint assessment and cross-referrals
• the range of output types
• double-weighted outputs
• extended process and long-form exceptions
• impact case studies (ICS)
• UL SPRE statements (ULS)

Main panel member attendance at sub-panel meetings: the chair and members of the main panel will attend some sub-panels meetings, to provide assurance that practices are consistent across the group of sub-panels:

• the members of the main panel advising on IDR and the RDAP member associated with each main panel will be engaged in calibration processes relating to IDR outputs, supporting the interdisciplinary advisors on the sub-panels, and advising on the consistency of assessment standards for IDR
• the international members of the main panel will, in particular, be engaged in sub-panel calibration processes and in the formation of sub-profiles, to ensure consistency with international standards
• main panel user members will, in particular, be engaged in briefing and calibration among sub-panel user members and assessors, providing support and focus for them, and advising on consistency of method and efficient use of expertise and knowledge in assessing ICS
• the main panel SPRE experts will, in particular, be engaged in sub-panel calibration processes, providing support and focus for them, and advising on consistency of method in assessing SPRE
• the main panel chair and main panel members will attend a sample of sub-panel meetings as agreed with the main panel, especially at an early stage in the assessment process

Advice and support to panels: a group of panel advisors and panel secretaries has been appointed to support the work of each main panel and its sub-panels. The secretariat will be briefed and trained in providing advice and guidance to their group of panels on the assessment procedures. Each member of the panel secretariat will work with several sub-panels within a main panel, providing consistent support and advice across them and providing feedback to the main panel chairs as appropriate.

Cross-panel appointments: individual academic members and assessors, and individual user members or assessors, may be appointed to work with more than one sub-panel, particularly where there are substantial overlaps between UoA descriptors, to contribute to consistency in the assessment of work on the boundaries. In considering the selection and appointment of further members and assessors, the main panel will identify where such boundaries could benefit from joint appointments.

Calibration exercises: each main panel and its sub-panels will undertake calibration exercises at an early stage in the assessment to develop a common understanding of the assessment standards and the application of the quality levels. International and user members of the main panel will participate in these exercises to assist in benchmarking judgements. The main panel chair and members of the main panel will attend a selection of the sub-panel meetings that deal with calibration exercises, and main panels will receive and discuss reports from sub-panel chairs on these exercises.

Reviewing emerging assessment outcomes: the main panels will review the emerging assessments at UoA level from their sub-panels during the course of the assessment phase, to support the consistent application of assessment standards. To facilitate this review, the group of sub-panels within each main panel will adopt a common process for the formation of each of the three sub-profiles and a common sequence in which each sub-profile will be formed. In considering the emerging assessment outcomes from sub-panels, the main panels will seek advice from the international members about the application of internationally referenced standards, and from the user members about the assessment of impact.

Deciding on the outcomes: when considering the sub-profiles recommended by its sub-panels, each main panel will confirm that the published assessment procedures and criteria have been applied by the sub-panels, and that the sub-panels have consistently applied the overall standards of assessment. The main panels recognise that there may be a range of overall profiles across their respective UoAs reflecting the relative strength of the disciplines in the UK. Each main panel will require that any substantial differences in the overall profiles for each UoA are investigated and understood before approving the sub-profiles recommended by its sub-panels. Where the recommendations of a given sub-panel for the overall results for that UoA are at substantial variance from the other sub-panels, the sub-panel chair will need to justify this to the main panel with reference to external evidence where available.

In addition to the main panels’ approaches to ensuring consistency within each group of sub-panels, to support appropriate consistency across the four main panels, generic assessment criteria and working methods across all main and sub-panels have been developed, as set out in the Panel criteria. These include standard weightings for each of the elements of the assessment (CKU, E&I, and SPRE), generic criteria for assessing each element, and consistent working methods and procedures.

Sub-panel working methods

Overview

Each sub-panel will be responsible for assessing submissions in its UoA, applying the published criteria and working methods, and recommending the outcomes to the main panel. This section sets out how the sub-panels will undertake their work at each stage of the assessment process.

Appointing the full assessment phase sub-panel

In early 2028, the sub-panels will examine institutions’ submission intentions and identify their requirements for further members and assessors to ensure the sub-panel has the breadth and depth of expertise required to carry out the assessment. These members and assessors will be appointed prior to the start of the assessment to ensure that sub-panels have access to appropriate expertise to reach robust and valid judgements with regard to the material anticipated in submissions.

Additional members and specialist assessors (for outputs and ICS) may also be appointed, including individuals from outside academia. Panels are recruited to cover the full range of topics in each subject area, and to balance experienced and new members. The procedures for appointing members and assessors are described in sub-section 3.1., Section 3: Overview in the Panel criteria.

Sub-panels will consider the breadth of work in actual submissions early in the assessment phase in 2029 in order to confirm that the sub-panel collectively has the breadth and depth of expertise to assess the work submitted. Where necessary, sub-panels may recommend the appointment of further additional assessors, or request that specific parts of submissions should be cross-referred to or jointly assessed with another sub-panel as indicated, where appropriate, in Annex B – UoA descriptors and taxonomies in the Panel criteria.

Each sub-panel will include research user members and impact assessors, with appropriate expertise to contribute fully to the assessment of the engagement and impact element of submissions, alongside academic members of the sub-panels. The research user members and impact assessors will be appropriately briefed (for example, with respect to the details of the REF process and key issues, including equality and diversity, and research diversity) alongside the sub-panel members and output assessors.

Allocating work

The sub-panel chair, consulting with the deputy chair, interdisciplinary advisor(s) and sub-panel members as appropriate, will allocate work to members and assessors with appropriate expertise, taking account of any conflicts of interest and the sub-panel’s approach to identifying outputs with significant material in common (see sub-section 4., Section 6: Assessment criteria – CKU in the Panel criteria). This allocation may be at the level of individual or groups of outputs, individual or groups of ICS, and whole ULS. Sub-panels may choose to use automated output allocation based on the research specialisms tagged for each submitted output to inform the allocation of outputs to panel members and assessors.

Each member and assessor on a sub-panel will be allocated a significant proportion of material to assess, so that each member and assessor makes a significant contribution to the sub-panel’s overall recommendations.

The sub-panel executive, in consultation with interdisciplinary advisors and the wider sub-panel where appropriate, may determine that the sub-panel and its appointed assessors do not have the required expertise to assess outputs that otherwise meet the UoA descriptor. In these circumstances sub-panels can consider the processes of joint assessment with other sub-panels or cross-referral to other sub-panels for advice.

In taking decisions on whether to cross-refer or jointly assess outputs, sub-panels will consider information regarding any other UoA to which the output has been submitted, where an output is tagged with the ‘Other for the purposes of aggregation’ taxon at broad research area level, and any requests from either the HEI or the submitting unit for cross-referral or joint assessment of the output. Where a sub-panel cross-refers an output for advice, the procedures in sub-section 4.10. will be followed. Where a sub-panel jointly assesses an output with another sub-panel, the procedures in sub-section 4.9. will be followed.

Each ICS will be allocated to at least one academic member and one user member or assessor, wherever practicable. User assessors will be allocated ICS and may be allocated relevant parts of the ULS. User members may – in addition to ICS – be allocated whole ULS and/or outputs in particular areas where they are willing and have appropriate expertise to assess them.

Where a sub-panel refers outputs to external specialist language advisors the procedures in sub-section 16., Section 4: CKU guidance in the Guidance will be followed

Calibration of assessment standards

Sub-panels will undertake early calibration exercises with respect to CKU, E&I and SPRE, to ensure sub-panel members and assessors develop a common understanding of the quality levels. The calibration exercises will be based on samples of a range of outputs (whether submitted to the REF or sourced from elsewhere by panel members), and on samples of submitted ICS and ULS.

In addition to sub-panel members, the assessors who will subsequently be involved in assessing either CKU or E&I, will take part in the relevant calibration exercises.

After these initial calibration exercises, the sub-panels will continue to discuss the application of the quality levels and will keep under review the scoring patterns of members and assessors, to ensure consistency in the sub-panel’s standards of assessment. Sub-panels may choose to use automated processes to support the tracking and understanding of scoring patterns.

Assessing submissions

Sub-panels will assess all components of unit-level submissions: CKU, E&I and SPRE. This reflects an underpinning principle that sub-panels will assess each submission in the round. They will not make collective judgements about the contributions of individual researchers. Sub-panels will make collective judgements about the range of submitted information in order to develop the sub-profiles and recommend these to the main panel for each unit being assessed.

All the outputs listed in submissions will (unless prevented by reasons beyond a sub-panel’s control) be examined by panel members and/or assessors. They will be examined with a level of detail sufficient to contribute to the formation of a robust sub-profile for all the outputs in that submission. Panels will only use citation counts as part of their assessment where they have declared they will do so (see sub-section 9, Section 6: Assessment criteria – CKU in the Panel criteria) and with no detriment. Expert review of the outputs will remain the primary means of assessing them.

Sub-panels will examine all the submitted ICS, and all the information submitted in the ULS together with the standard quantitative data and analyses.

Sub-panels will meet during the course of the assessment phase to discuss their assessment of each element of submissions. Assessors will attend those meetings at which the relevant element of submissions is being discussed, so that they contribute fully and on an equal basis with members to the development of the relevant sub-profile.

During the course of the assessment, the sub-panels will be asked to draw attention to any data they would like the REF team to verify through an audit. This data will be investigated by the REF team (in addition to the REF team auditing a proportion of submitted information from each institution).

Assessing outputs that may align to more than one UoA

Outputs must align to the UoA to which they are submitted according to the UoA descriptor, unless the criteria for aggregate submissions are met as outlined in sub-section 1., Section 5: Assessment criteria – Submissions in the Panel criteria. However, some outputs may align with the descriptors for more than one UoA and may have been submitted to more than one UoA. Where an output has been submitted to more than one UoA, this information will be available to each sub-panel executive to which the output has been submitted. Outputs meeting the descriptor for more than one UoA may or may not be interdisciplinary.

Institutions may indicate that additional assessment input would be appropriate for outputs meeting the descriptors for more than one UoA by requesting either cross-referral or joint assessment. When requesting cross-referral or joint assessment for particular outputs, HEIs will be able to indicate which other UoA the output may benefit from assessment input from and will be able to tag the output with a broad research area taxon from that second UoA. Sub-panels, working with their IDR advisors where relevant, will consider these requests as part of their consideration of the most appropriate means of assessing the output. The decision to cross-refer or jointly assess any output remains the sub-panel’s to which the output was submitted.

Assessing outputs and ICS submitted as part of an aggregate submission

Where a submission is made that meets the criteria for aggregate submissions outlined in sub-section 1., Section 5: Assessment criteria – Submissions in the Panel criteria, outputs may be tagged as ‘Other for the purposes of aggregation’ at broad research area level, allowing the submitting HEI to indicate the UoA and relevant broad research areas to which the output better aligns. This information will be used to support sub-panels, working with their IDR advisors where relevant, to consider the most appropriate means of assessing the output.

Indicating that an output is ‘Other for the purposes of aggregation’ will not automatically trigger cross-referral or joint assessment of the output but cross-referral or joint assessment may be amongst the assessment routes followed. Where a submission is made that meets the criteria for aggregate submissions outlined in Section 4: Panel descriptors in the Panel criteria, aggregated ICS may be exceptionally considered for joint assessment or cross-referral by the sub-panel.

Assessing interdisciplinary work

Sub-panels recognise that interdisciplinary work may cross UoA boundaries or sit within one UoA. Sub-panels will recognise and assess appropriately all IDR, regardless of whether the interdisciplinarity overlaps multiple UoAs.

For REF 2029, HEIs will not be asked to flag interdisciplinary outputs in the Submission system. Sub-panels will use their judgement to decide whether outputs are interdisciplinary solely for the purposes of monitoring, administration and post-assessment analysis.

Where interdisciplinary work crosses UoA boundaries, HEIs will be able to request cross-referral or joint assessment of particular outputs where the output may benefit from assessment input from more than one sub-panel, as outlined in paragraph 4.6.2.

All interdisciplinary outputs will be assessed on a fair and equal basis and will neither be privileged nor disadvantaged in the assessment on the basis of output form or type of research. All sub-panels will have members experienced in IDR and other modes of research, and the processes for joint assessment and cross-referral will be available for interdisciplinary outputs as for all other outputs.

Sub-panel executives will appoint at least two interdisciplinary advisors per sub-panel. Their responsibilities are:

• to offer guidance to the sub-panels in their assessment of interdisciplinary outputs to enable their robust and valid assessment
• to advise, as necessary, on the allocation of outputs and the calibration and moderation of scoring
• to liaise with interdisciplinary experts on the main panels on issues relating to interdisciplinarity
• to contribute to the main and sub-panel reports at the conclusion of the exercise
• to work collaboratively with their counterparts on other sub-panels (across all four main panels) to review outcomes from the initial calibration of interdisciplinary outputs and provide advice on appropriate expertise to review interdisciplinary outputs
• to advise on the appropriateness of joint assessment and cross-referral for specific IDR outputs
• to discuss and share best practice

Collectively, the IDR advisors on the sub-panels and the IDR experts on the main panels will constitute the IDR forum. It is expected that the IDR forum will meet at key points during the assessment phase to share good practice for reviewing joint working arrangements and identify wider expertise requirements. RDAP will provide advice and support to the forum on these aspects during the calibration and assessment phases.   

The sub-panels’ approach to assessing outputs identified as interdisciplinary will be reviewed across the main panels, to ensure an appropriate consistency of approach. Analysis of the emerging and final scores for the group of outputs identified as interdisciplinary by sub-panels will be conducted by RDAP during and upon completion of the assessment process. 

RDAP will have a role in overseeing the assessment of interdisciplinary outputs to ensure that agreed principles and processes for assessment are applied and that there is consistency in approach across sub-panels. RDAP will not advise on the assessment of individual outputs, but will advise on process, and provide advice and support for cross-panel calibration. The main panel interdisciplinary experts will work with RDAP during the assessment phase to ensure the appropriate application of the principles for the assessment of IDR.

Joint assessment

The sub-panels’ preferred approach is to assess work within the sub-panel to which it was submitted and, informed by the survey of submission intentions, to appoint further members and assessors where required to enable this. Where the sub-panel considers that it has some of the relevant expertise, but not sufficient expertise to assess specific parts of submissions, it may undertake a joint assessment with other relevant sub-panels (including both within and across main panels).

The submitting institution may request that specific outputs (but not ICS or entire submissions) should be considered for joint assessment with another sub-panel.

Where the sub-panel executive considers there is sufficient expertise within the sub-panel to reach a robust judgement, the work will be assessed within the sub-panel. The sub-panels expect that this will normally be the case, except in some cases of aggregation. The sub-panel can also consider cross-referral as an option for assessment.

The decision and process to undertake joint assessment will be supported by RDAP and the main panel interdisciplinary experts to ensure consistency of approach in joint assessment.

Where joint assessment is undertaken, the work will be allocated to panellists both on the original and on the other sub-panel(s) involved. The allocated panellists (from all sub-panels involved) will work together as required, to provide a recommendation to the original sub-panel. Where outputs are jointly assessed, advice will be sought and given on the basis of any specific assessment criteria and procedures (including, for example, use or not of citation data) for the UoA in which the work was originally submitted.

Joint assessment with an appropriate sub-panel will be undertaken in a timely manner, and advice provided within a reasonable window.

Cross-referral

The sub-panels’ preferred approach is to assess work within the sub-panel to which it was submitted and, informed by the survey of submission intentions, to appoint further members and assessors where required to enable this. Where the sub-panel considers that it does not have the required expertise to assess a submission in full, it may cross-refer outputs or ICS to other relevant sub-panels (including both within and across main panels).

The submitting institution may request that specific outputs (but not ICS or entire submissions) should be considered for cross-referral to another sub-panel.

Where the sub-panel executive considers there is sufficient expertise within the sub-panel to reach a robust judgement, the work will be assessed within the sub-panel. The sub-panels expect that this will normally be the case, except in some cases of aggregation. The sub-panel can also consider joint assessment as an option for assessment.

The sub-panels’ approach to cross-referral will be discussed within the main panels to ensure consistency of approach. RDAP will also provide support as will the main panel interdisciplinary experts to ensure consistency of approach to interdisciplinary assessment.

The original sub-panel will specify the scope of advice that it is seeking. This will be limited to advice relating to the quality of the work. It may not include advice on other matters such as significant research contribution to outputs or double-weighting of outputs.

Where outputs are cross-referred, advice will be sought and given on the basis of any specific assessment criteria and procedures (including, for example, use or not of citation data) for the UoA in which the work was originally submitted; cross-referred outputs will be assessed on the same basis as work which is not cross-referred.

The original sub-panel will retain responsibility for recommending the sub-profiles for all work that was submitted in its UoA. 

Cross-referral to an appropriate sub-panel will be undertaken in a timely manner, and advice provided within a reasonable window.

Developing and recommending sub-profiles

Sub-panels will develop a sub-profile for each of the three elements – contribution to knowledge and understanding, engagement and impact, and unit-level strategy, people and research environment – of each submission.

Contribution to Knowledge and Understanding sub-profile: each output listed in a submission will be assessed against the quality levels: 4 star, 3 star, 2 star, 1 star or unclassified. Outputs deemed ineligible will not be assessed against the quality levels and will be recorded as ‘ineligible’. The CKU sub-profile will be formed by calculating the percentage of outputs listed in a submission that are assigned at each quality level, with each output contributing an equal proportion to the sub-profile. The following exceptions and rules apply:

• any submitted output that is found to be ineligible will be entered into the CKU sub-profile as ‘ineligible’
• any outputs that are missing from a submission (that is, where fewer outputs have been submitted than the number required and where no unit reduction has been agreed) will be graded as ‘ineligible’
• where a request to double-weight an output has been accepted by the sub-panel, the quality level assigned to the output will be entered twice into the CKU sub-profile. Where a request to double-weight an output is not accepted by the sub-panel a reserve output will be assessed. If no reserve output has been submitted, the output will contribute to the sub-profile as a single output and one instance of ‘ineligible’ will be entered into the CKU sub-profile
• where the sub-panel determines that the staff member upon whom the substantive link is based did not make a substantial contribution to the output, the output will be entered into the CKU sub-profile as ‘ineligible’
• where a sub-panel judges that two outputs within a submission do not contain sufficiently distinct material and should be treated as a single output, the missing output will be entered into the CKU sub-profile as ‘ineligible’

Engagement and Impact sub-profile: each ICS included in a submission will be assessed according to the definitions of the starred levels in Annex A: Assessment criteria and level definitions in the Panel criteria. Any ICS that are missing from a submission (that is, where fewer ICS have been submitted than the number required) will be graded as ineligible. Where the impact or the underpinning research does not meet the eligibility requirements set out in the guidance, the ICS will be deemed ineligible. The E&I sub-profile will be formed by calculating the percentage of ICS listed in a submission that are assigned at each quality level, with each ICS contributing an equal proportion to the sub-profile.

Strategy, People and Research Environment sub-profile: sub-panels will assess the information provided in the ULS, informed by the quantitative data and standard analyses, the ILS and the context section. Sub-panels will build up a graduated sub-profile by assessing the range of components in each submission, using the starred levels defined in Annex A: Quality profiles in the Guidance. The sub-panels are exploring the most appropriate way to build the graduated sub-profile for SPRE ahead of assessment to allow the recognition of quality across the different elements of the statement. Ahead of assessment, all sub-panels and the IL SPRE Panel will agree a single approach to building the graduated sub-profile.

The assessment of the ILS and ULS will be undertaken independently and respectively by the IL SPRE Panel and sub-panels and both the institution- and unit-level sub-profiles will be published separately in addition to a combined sub-profile.

In recommending the sub-profiles for each submission to its main panel each sub-panel will:

• reach a collective decision, within the framework of the exercise and in accordance with the Panel criteria. Each sub-panel will debate the reasoning behind the sub-profiles in sufficient detail to reach collective conclusions and will make recommendations to the main panel on the basis of its collective judgement. Each sub-panel will seek to achieve a consensus on all the sub-profiles to be recommended to its main panel. If a consensus cannot be achieved after reasonable effort, decisions will be taken by majority vote, with the chair holding a casting vote
• confirm to the main panel that each submission has been assessed against the published criteria for that UoA (including in cases where parts of submissions have been cross-referred to other sub-panels for advice or have used the joint assessment process) and according to the published procedures
• confirm that each submission has been examined in sufficient detail to form robust judgements, and that appropriate expertise has been deployed in assessing submissions

Advisory panel working methods

The assessment panels are supported by two advisory panels who provide expert guidance in their respective areas. These are: 

• People and Diversity Advisory Panel (PDAP) 
• Research Diversity Advisory Panel (RDAP) 

The work of the two advisory panels will overlap in places, as matters concerning the diversity of research and the diversity of people are strongly interlinked. The two panels will work together where there is benefit in doing so and both have observers on the main panels and will also have observers on the IL SPRE Panel.

People and Diversity Advisory Panel (PDAP) 

PDAP has met regularly throughout the criteria setting phase and will continue to meet through the assessment phase. Their focus is on developing strategies to support recognition of a diversity of roles, careers and career stages within the REF 2029 development, delivery, and assessment; and advancing equality, diversity, and inclusion (EDI) throughout REF 2029.

PDAP advises the REF team and Steering Group on embedding consideration of people and diversity on an ongoing basis throughout the exercise. Where emerging issues relating to people and diversity are identified, PDAP will work proactively with the REF team to consider and act upon them. PDAP also, via the Chair of PDAP, contributes to the Panel Executive.

PDAP will focus on:

• diversity of roles and careers – Recognition and inclusion of the full rangeof professions and careers and career stages that are part of UK research system, in the development and delivery of the REF and the assessment process 
• equality, diversity and inclusion – Due regard and approach to advancing equality of opportunity in relation to protected characteristics and the diversity of roles, careers and career stages, in the development and delivery of the REF, and in the assessment process

PDAP works alongside the panels and the REF team as follows:

• advising on the development and delivery of inclusive REF 2029 assessment criteria
• supporting the development and delivery of Codes of Practice
• advising on the recruitment processes for the REF 2029 panel roles
• working with panels and the REF team on the development and delivery of panel member training programmes
• advising on calibration exercises throughout the criteria and assessment phases
• providing expert advice to panels during the assessment phase, including, in particular, the people element of SPRE
• advising on equality-related unit reduction requests
• contributing to the evaluation of the overall effectiveness of the REF’s equality and diversity aspects and identifying lessons for the future

At the end of the REF 2029 exercise, they will publish a report on their assessment of people and diversity considerations.

Research Diversity Advisory Panel (RDAP) 

RDAP has met regularly throughout the criteria setting phase and will continue to meet through the assessment phase. Their focus is to develop strategies to support the equitable recognition of diverse forms of research, within REF 2029 development, delivery and assessment. They will focus on:

• diverse research practices (for example, IDR, practice research, meta-analysis, replication studies)
• diverse output formats (for example, practice research outputs, audio and video content, scholarly editions, software, datasets, and artefacts)
• diverse engagement and impact types and formats and corresponding responsible practices (for example, interdisciplinary impact, civic and regional impact)

RDAP advises the REF team and Steering Group on embedding consideration of research diversity on an ongoing basis throughout the exercise. Where emerging issues relating to research diversity are identified, RDAP will work proactively with the REF team to consider and act upon them. RDAP also, via the Chair of RDAP, contributes to the Panel Executive.

RDAP works alongside the panels and the REF team as follows:

• advising on the development and delivery of inclusive REF 2029 assessment criteria
• advising on appropriate mechanisms for assessing diverse research, engagement and impact
• advising on the recruitment processes for the REF 2029 panel roles
• working with panels and the REF team on the development and delivery of panel member training programmes
• providing expert advice to panels during the assessment phase, especially on matters related to the assessment of diverse outputs, impacts and interdisciplinary work
• working with the IDR forum and support analyses of IDR-identified outputs

At the end of the REF 2029 exercise, they will publish a report on their assessment of research diversity considerations.  

Recording panel decisions

The panel secretariat will minute details of the procedures followed by panels, and these will be published after the conclusion of the exercise. Panels will not make or record collective judgements about individuals’ contributions to submissions. The panel secretariat will record the panels’ collective judgements about the sub-profiles and overall quality profiles in respect of each submission.

Panel use of AI

We recognise that many panel members use AI and automation tools routinely in their academic work. To ensure transparency in the assessment process, the panels have adopted the following protocols regarding their own use of technology:

• confidentiality: panel members will not upload the content of a submitted item (redacted or otherwise) to an AI or automation tool, in line with their duty of confidentiality
• human judgement: panel members will not use AI tools to generate judgements on quality, recognising that reliable quality assessment requires the exercise of human judgement. This includes preparation of translations for outputs in languages other than English, which will not be prepared using AI
• data processing: panels may make use of AI/automation to process scoring data, with a view to improving the robustness of assessment – for example, by correcting for systematic differences in scoring behaviour of panel members. Any such use will be calibrated by the REF team, transparently communicated to the sector and will never be a substitute for panel judgement
• administrative efficiency: the REF team may explore the use of AI/automation tools for strictly administrative processes, such as the allocation of outputs to reviewers based on expertise matching. Any such use will be transparently communicated to the sector and will never be a substitute for the relevant chair’s final oversight on process