Guidelines for Selection Board Members—Application Stage

Background

The Canada First Research Excellence Fund (CFREF) is a tri-agency initiative of the Social Sciences and Humanities Research Council (SSHRC), the Natural Sciences and Engineering Research Council (NSERC) and the Canadian Institutes of Health Research (CIHR). It is administered by the Tri-Agency Institutional Programs Secretariat, which is housed within SSHRC. The competition was announced in November 2021 to allocate up to $1.4 billion to support large-scale research programs focusing on a strategic thematic area related to the institution’s research excellence strengths and aligned with the Government of Canada’s science, technology and innovation (ST&I) priorities for the CFREF and Canada Excellence Research Chairs programs (CERC).

CFREF aims to boost the strengths of Canadian postsecondary institutions so that they can achieve global success in research areas that create long-term social and economic advantages for Canada.

CFREF invests approximately $200 million per year through a highly competitive peer review process to support selected Canadian postsecondary institutions in turning their key strengths into world-leading capabilities.

CFREF objectives are to support the full range of research—from fundamental to applied—and to give institutions the ability to:

  • pursue the best in the world for talent and partnership opportunities to enable breakthrough discoveries;
  • seize emerging opportunities and strategically advance their greatest strengths on the global stage; and
  • implement large-scale, transformational and forward-thinking institutional strategies.

Incremental and small-step evolution of research objectives will not meet the goals of this program.

There is no maximum or minimum limit for the grant value, and the number of awards depends on the amounts requested in successful applications. Each institution can submit only one letter of intent (LOI) and application as the lead institution. There is no limit to the number of applications on which an institution can be listed as a partner.

The 2022 competition has a budget of approximately $1.4 billion over seven years. Program staff will conduct an administrative review of the budget items. A list of allowable costs can be found here: Eligible and ineligible expenses.

Equity, diversity and inclusion excellence

The three federal research funding agencies are committed to excellence in research and research training, and therefore to achieving equity, diversity and inclusion (EDI) in the Canadian research enterprise. EDI is essential to creating the excellent, innovative, impactful research needed to seize opportunities and respond to global challenges. To be successful, applications must demonstrate and implement the highest EDI standards.

Overview of the application evaluation process

Awards are made following a rigorous, competitive, international peer review process. Only institutions that are successful at the LOI stage are invited to submit full proposals at the application stage. 

Applications must demonstrate that the institution has the capacity for global leadership, is capable of being among the top 10% of centres of academic excellence globally and can sustain world leadership in the proposed area(s). The program’s objective is not to develop capacity where there is emerging potential. The proposed initiatives must integrate partner involvement and strong knowledge mobilization throughout the application. By doing so, institutions will further demonstrate the sustainability of their initiatives.

All applications will be held to the same standards of world-leading research excellence.

A three-stage review process will be applied at the full application stage: (1) written external expert reviews, (2) multidisciplinary review panel evaluation, and (3) selection board adjudication.

  1. External reviewers – International external reviewers will be recruited to evaluate the strengths and weaknesses of the application’s Scientific strategy (Part B of the application), in relation to Criterion 1: Scientific merit and demonstrated capacity to lead on an international scale; and Criterion 2: Strategic relevance to Canada. The Implementation plans (Part A of the application) and the LOI will also be provided as context. Reviewers are screened to ensure they do not a have a conflict of interest in reviewing the applications. External review evaluations will be provided to the review panel.
  2. Review panel – A multidisciplinary review panel will evaluate in detail the relative strengths and weaknesses of submitted applications, based on the selection criteria (excluding those covered in the LOI stage). The review panel provides individual ratings, consensus ratings and rankings to the selection board ultimately to competitively award the funds available to proposals demonstrating the greatest potential to achieve world-leading research outcomes in areas that promise long-term economic advantages for Canada.
  3. Selection board –The selection board provides recommendations to the Tri-Agency Institutional Programs Secretariat Steering Committee (see below) on the most strategic way to award the available budget for the competition among the proposals demonstrating the greatest potential to achieve globally leading research outcomes in ST&I priorities for the CFREF and CERC programs. The selection board will benefit from the reports from all previous review stages of the competition in making its final recommendations to the Steering Committee.

The Tri-Agency Institutional Programs Secretariat Steering Committee is composed of the presidents of CIHR, NSERC, SSHRC and the Canada Foundation for Innovation (as an observer), as well as the deputy ministers of Innovation, Science and Economic Development Canada and Health Canada. The committee reviews the adjudication process to ensure that it is rigorous, objective, transparent and consistent with the program’s objectives. The Steering Committee will ratify the funding recommendations of the selection board.

Equity, diversity and inclusion and early career researchers in research teams

Equity, diversity and inclusion (EDI) are embedded as a foundational principle in CFREF’s objectives, expected outcomes and application and reporting requirements. To achieve its excellence-based objectives and outcomes, the program requires the participation and contributions of students, trainees, personnel and researchers from underrepresented or marginalized groups. These groups include, but are not limited to, racialized minorities, Indigenous Peoples, persons with disabilities and women and individuals from LGBTQ2+ communities.

Supporting early career researchers is a tri-agency priority, as it enhances Canada’s position as a world leader in building talent and strengthening the research ecosystem. CFREF initiatives are expected to implement measures that specifically support early career researchers.

EDI in the selection process

CFREF-funded initiatives are expected to demonstrate exceptional leadership in contributing to transforming their research discipline and Canada’s research ecosystem to help it become equitable, diverse and inclusive. Institutions are required to take active and rigorous measures to prevent the systemic barriers that may result in individuals from underrepresented groups receiving unequal access to, or being excluded from participating in, the opportunities provided by the grant. This includes the composition of the initiative’s governance committees, evaluation processes used to provide funding within the initiative, and recruitment and selection processes.

All funded CFREF initiatives will be required to develop an initiative-specific EDI action plan within the 12 months following the grant start date. This EDI action plan will be assessed by the Tri-Agency Institutional Programs Secretariat, either administratively or by expert reviewers. The quality of the strategy that will be used to develop this comprehensive and innovative EDI action plan will be assessed in Criterion 3: Quality of implementation plans. 

The quality of the research proposal in considering and implementing EDI (i.e., Gender-Based Analysis Plus (GBA+) / Sex- and Gender-Based Analysis Plus (SGBA+) in the research questions, design, methodology, data collection, analysis and interpretation, and dissemination of results, as relevant, will be assessed in Criterion 1: Scientific merit and demonstrated capacity to lead on an international scale.

The quality of the proposed research in its inclusion of Indigenous research that is co-created and co-led by and with First Nations, Inuit and Métis peoples—as investigators, students, trainees, partners and collaborators—and in its recognition of Indigenous ways of knowing, and how it will extend research knowledge in the field that is significant for Indigenous Peoples and communities, will be assessed in Criterion 1: Scientific merit and demonstrated capacity to lead on an international scale.

All reviewers are required to consistently guard against the possibility of unconscious bias influencing the decision-making process, whether this bias is based on a school of thought; fundamental versus applied research; certain subdisciplines; areas of research or approaches (including emerging ones); size or reputation of an institution; or the age, personal factors, sex or gender of the applicants. All reviewers are cautioned against judging an application based on these factors. Before selection board members are able to view any applications, they must first complete the required Unconscious Bias Training Module. Board members are encouraged to read the following documents that were shared with applicants to help them with addressing EDI considerations: Best Practices in Equity, Diversity and Inclusion in Research, EDI considerations at each stage of the research process, Women and Gender Equality Canada’s information on GBA+, and the SGBA+ section of the CIHR website.

For applications involving Indigenous research, selection board members should familiarize themselves with SSHRC’s Indigenous Research Statement of Principles and Guidelines for the Merit Review of Indigenous Research (specifically, the section describing key concepts for the merit review of Indigenous research).

The multidisciplinary review panel and the selection board will include EDI and Indigenous champions, who will help support the program’s EDI priorities in the selection process.

Accessing the review material

You must complete your review using the Convergence Portal. The portal is supported only on the latest versions of Google Chrome, Microsoft Edge, Apple Safari and Mozilla Firefox. The portal may appear to function in other browsers, but technical problems can occur, such as information being improperly captured in the system without the user being aware. Use of an unsupported browser is strongly discouraged.

As a selection board member, you will receive an email with detailed instructions asking you to log in (or create an account) in the Convergence Portal to complete your profile and accept the reviewer terms and conditions. After accepting the terms and conditions, you will have access to the reviewer dashboard on the Convergence Portal. From your dashboard, you must first declare any conflict of interest (COI), if applicable. The Ability to review tab will provide you with a summary of the application(s) to help you determine whether you have a COI.

Once the applications are assigned to external reviewers, you will have access to the Committee Assignments tab. You will have access to a PDF copy of each applicant’s submission, both their input into modules in the Convergence Portal and supporting documents they prepared according to Application instructions.

To assess the application and the supporting documents (listed below), you will also have access to the full LOI submission, the LOI review panel documents and external expert reviews, for reference. Selection board members are encouraged to first read the LOI and review panel feedback provided to understand the context in which the full application was developed.

The PDF document is organized as follows:

  • Application details
  • Invite partner institutions
  • Partner institutions
  • Partners
  • Collaborators
  • Core personnel
  • Socio-economic objectives
  • Science, technology and innovation research priority areas
  • Fields of research
  • Alignment with federal research funding agencies
  • Keywords
  • Summary of the implementation plans
  • Summary of the scientific strategy
  • Proposed budget
  • Additional suggested reviewers and reviewer exclusions

Supporting documents

In the Convergence Portal, you will have access to the following supporting documents, which are provided as a single PDF document:

  • Changes from the LOI (2 pages in English, 2.4 pages in French)
  • Implementation plans – Part A (15 pages in English, 18 pages in French)
  • Performance management plan – Part A
  • Scientific strategy – Part B (30 pages in English, 36 pages in French)
  • References (no page limit)
  • Budget table
  • Budget justification (20 pages in English, 24 pages in French)
  • Letters of support from partner institutions (2 pages per letter in English, 2.4 pages in French)
  • Letters of support from partners (2 pages per letter in English, 2.4 pages in French)
  • Letters of support from collaborators (2 pages per letter in English, 2.4 pages in French)
  • Core personnel biosketches (20 pages total in English, 24 pages total in French)
  • Panel members’ comments and ratings of the LOI) 

The document can be downloaded and saved on your computer or device, but please ensure it is kept confidential and deleted when the review process is completed.

Applications submitted in French

Applications submitted in French are assigned to selection board members who have a reading comprehension of the French language. The page limit is higher for French applications, to ensure an equitable amount of space for applications written in either official language. Evidence demonstrates that documents written in French require approximately 20% more space than similar documents in English.

Selection board member assignments

The selection board is responsible for conducting a strategic review. Specifically, it conducts an in-depth review of the strategic relevance under Criterion 2, and of the implementation plan under Criterion 3. For its in-depth review, the selection board is informed by the input from the individual expert reviewers and the multidisciplinary review panel and confirms final ratings for each proposal for these two criteria. The multidisciplinary review panel ratings provided for Criterion 2 (strategic relevance to Canada) and Criterion 3 (quality of the implementation plan) are considered preliminary, and the selection board may revise these ratings, as appropriate.

The expert reviews and the multidisciplinary review panel ratings provided for Criterion 1 (Scientific merit and demonstrated capacity to lead on an international scale) are considered final and cannot be revised by the selection board. Selection board members are not required to evaluate Criteria 1.

The selection board may also provide comments and suggestions on ways to optimize the implementation plan to align with program objectives. For example, the selection board may recommend improvements to the Performance Measurement Plan in a proposal. The Secretariat would then communicate the recommendations for improvement to the applicant. These suggestions would highlight key risks or key progress milestones to monitor over time as critical success factors for the funded initiative. 

The following describes the assessment roles played by selection board members:

Readers A and B: Indicate in the score sheet provided whether you agree with the review panel’s rating and assessment for Criteria 2 and 3. If you do not agree, please enter your revised rating(s) and some brief comments to support your assessment of each criterion. Please also consider:

  • any suggestions you have to optimize the implementation plan to align with program objectives;
  • any areas for improvement to the Performance Measurement Plan to monitor progress; and
  • potential questions to ask the applicants if they are invited for an interview.

Reader C: Indicate in the score sheet provided whether you agree with the review panel’s rating and assessment for Criteria 2 and 3. If you do not agree, please enter your revised rating(s). You do not need to provide any written comments. However, we encourage you to make notes (for yourself) and to be prepared to discuss the application at the meeting.

While the first priority of a selection board member is to review the applications they are assigned (as Reviewer or Reader), members should also familiarize themselves with the other applications not assigned to them in order to participate fully in the discussions and consider any questions they would like to ask applicants in the final selection round.

Based on the multidisciplinary review panel ratings and results, the full applications will be sorted into three tiers for further consideration by the selection board (see section on ratings, below):

  1. Top tier – This tier includes proposals rated Fully satisfies and exceeds (FSE) across all three criteria that are highly recommended by the review panel.
  2. Mid-tier – This tier includes proposals with ratings of Fully satisfies and Fully satisfies and exceeds (FS/FSE) for the three criteria that meet the minimum threshold and standard for consideration.
  3. Lower tier – This tier includes proposals with ratings of Mostly satisfies or Does not satisfy (MS/DNS) for one or more of the criteria that do not meet the minimum threshold and standard for consideration and are not recommended by the review panel.

Focus of selection

The focus of selection will be whether an institution meets the threshold and standard for consideration under the program. That is, does the proposal appropriately respond to the program’s intended goals and ambitious aims, based on the key elements of the selection criteria? Specifically, a proposal must meet a minimum rating of “Fully Satisfies” on all three criteria, as rated by the multidisciplinary review panel and selection board, to be considered for funding. Proposals will ultimately be differentiated based on their potential to achieve globally leading research outcomes in areas of long-term economic advantages for Canada.

Submitting your review (ratings and comments)

Each application will be assigned to three or more selection board members, taking into account conflicts of interest, language and workload balance.

After reading the guidelines, the application, the external reviewer reports, and the panel reviews, follow the instructions provided in the Convergence Portal to complete and submit your ratings and comments.

In the comment boxes in the Convergence Portal, please provide objective feedback describing both the strengths and weaknesses of the application for each element of the criteria. Your written comments should be clear and concise, using objective and non-inflammatory language. Because your comments will be provided as feedback to the applicant, you should ensure that you provide sufficient detail. To ensure the confidentiality of the review process, please avoid providing information that identifies you.

When you are ready to submit your assessment in the Convergence Portal, review your ratings and comments to ensure they are complete. Click the “Submit Assessment” button to complete the assessment of each assigned Application. The status of your submission will change to “Assessed.” Once all assessments are completed, select “CFREF-2022-1” from the Competition dropdown list, then select “Application” from the Stage dropdown list. Finally, click the “Submit All Assessments” button. The status of your submission will change to “Submitted.” No changes can be made after that point.

We ask that your reviews be completed in the Convergence Portal by the date specified in your email invitation.

Selection board meetings

A first meeting will be moderated by the selection board chair in December 2022 to discuss individual application scores, with a focus on any divergent reviews (significant discrepancies or disagreements). The meeting provides an additional forum to discuss the applications, share perspectives and consider various opinions. All selection board members will be invited to participate in the discussion of these applications.

Prior to the meeting, members will be invited to participate in a preparatory briefing session that will include additional training on the assessment of equity, diversity and inclusion.

The meeting date and agenda will be provided in advance. The selection board members may make special requests regarding the agenda to help with its assessment.

Ratings will be considered divergent if ratings provided by the assigned board members are misaligned―for example, a mix of Fully Satisfies (FS) or Fully Satisfies and Exceeds (FSE), together with Mostly Satisfies (MS) or Does Not Satisfy (DNS).

To resolve divergent ratings, assigned selection board members will not be required to modify their individual ratings, but the selection board will be asked to reach a consensus rating for Criteria 2 and 3.

Based on the consensus ratings obtained during this first meeting, the selection board will rank the applications and establish a threshold above which institutions will be asked to present their proposed CFREF initiative to the board and answer any questions the selection board may have. The selection board may submit a list of questions for the institutions to address during their meeting with the board. Applicants will receive the consensus comments and ratings of the selection board.

A second virtual meeting, also moderated by the selection board chair, will be held in January 2023 to meet with representatives of the invited institutions. After meeting with all invited institutions, the board will finalize its ratings and comments for Criteria 2 and 3 for those applications and submit its funding recommendations for the Steering Committee’s approval.

The selection board chair may be asked to present a report to, or address questions posed by, the Steering Committee.

Evaluation criteria and rating scale

The Definition of Ratings is provided for the three selection criteria and includes the rating scale you will use to assess them. As a member of the selection board, you will use these ratings to finalize the score for Criteria 2 and 3 for each application assigned to you.

The Application instructions provided to applicants are available for your information.

Applications must meet a minimum rating of “Fully Satisfies” for all three criteria. Applications will ultimately be differentiated during the LOI and the full application stage based on their potential to achieve world-leading research outcomes in areas of long-term economic advantages for Canada.

Rating scale

After the selection board reviews are completed, each rating will be converted to a numerical score to create a ranking of the applications for the selection board’s consideration:

  • Fully satisfies and exceeds (FSE)
  • Fully satisfies (FS)
  • Mostly satisfies (MS)
  • Does not satisfy (DNS)

The final ranking of the applications will be based on the final consensus ratings.

Adjudicated elements

While all of the information in the application will be considered, the following tables describe how three key supporting documents of the overall application correspond to the two criteria adjudicated by the selection board.

Implementation plans (Part A of the application)

Vision and commitment – This section describes the institution’s vision and willingness to commit resources and to support the initiative. Criterion 3: Quality of implementation plans
Equity, diversity and inclusion strategy – This section describes the strategy that will be used to develop a comprehensive EDI action plan.
Implementation plan – This section outlines plans for allocating the grant funding, governance and a high-level work plan.
Risk and mitigation plan – This section outlines the major risks and mitigation measures.
Sustainability plan – This section outlines the plan to optimize the use of leveraged and supporting funds to ensure the longer-term sustainability of the initiative.

Performance management plan (Part A of the application)

This is an element of Part A but is submitted by the applicant as a separate supporting document.

Performance management plan – This section outlines the performance measurement plan for ongoing reporting, monitoring and evaluation Criterion 3: Quality of implementation plans

Scientific strategy (Part B of the application)

Only the following specific elements of the scientific strategy are assessed by the selection board:

Potential for the proposed research area to create long-term economic advantages – This section describes prospective benefits to Canada from the proposed research strategy. Criterion 2: Strategic relevance to Canada
Ability of the proposed initiative to leverage additional resources and promote knowledge mobilization through partnerships – This section provides evidence of the relevance of the proposed research to research users by demonstrating the initiative’s potential to leverage contributions with the private and/or public sector, international research institutions, as well as academic and philanthropic organizations, both in Canada and abroad.
Potential for the research results to foster innovation – This section outlines the required knowledge mobilization strategy including plans for innovation, commercialization and contribution to public policy, as appropriate.

A research program proposal (or scientific strategy) will have one common strategic thematic area. The proposal can include multiple sub-themes within the thematic area. The research program proposal must clearly demonstrate how these sub-themes are linked and how they can be combined to provide a coherent strategic focus under an integrated institutional leadership and governance structure. In its proposal, the institution must make the case for the choice of theme and sub-themes and must describe how advancing these would serve the institution’s overall strategic focus for global excellence. Institutions are advised that submitting a scientific strategy with multiple sub-themes, some of which do not meet the standards of excellence expected for the CFREF, would weaken the overall competitiveness of the proposal. Such an approach could raise doubts about the institution’s ability to self-assess its global position and to establish objectives aligned with its demonstrated strengths (as assessed in Criterion 1). It could also reveal a weakness in the institution’s ability to make the difficult choices that will undoubtedly be required to successfully implement an initiative of CFREF’s scale and scope.

Budget requests

Both Parts A and B of the application will include a high-level budget request and justification. Reviewers must provide comments on the budget requests. Please consider the following:

  • Does the amount requested seem reasonable, appropriate, and justified, particularly for infrastructure upgrades and equipment purchases?
  • Are there any items that could be cut from the budget or item amounts that could be reduced?
  • Does the initiative/activity appear feasible given the amount requested?
  • Has the EDI strategy/action plan been sufficiently budgeted for?
  • Do you have any other concerns or items to flag for our attention?

The budget request also enables reviewers to see the planned distribution of the proposed investment across budget categories and to assess whether this aligns well with their own appreciation of where an institution needs to ramp up its capacity and its activities.

Handling documents used in peer review

Review documents contain personal information as well as information that, if disclosed in an unauthorized manner, could reasonably be expected to cause serious injury (such as prejudicial treatment or loss of reputation or competitive advantage) to an individual, organization or government. Therefore, these documents are subject to regulation under the Privacy Act, the Access to Information Act and the Policy on Government Security. Protocols must be followed to ensure that information contained in applications, internal and external reviews, and panel discussions remains strictly confidential. Improper or unauthorized collection, use, disclosure, retention and/or disposal of this information can result in a privacy breach. Please refer to the Guide on Handling Documents Used in Peer Review for further details.

Legal and ethical information

Responsible conduct of research

Canada’s federal research funding agencies—CIHR, NSERC and SSHRC—are committed to fostering and maintaining an environment that supports and promotes the responsible conduct of research. The Tri-Agency Framework: Responsible Conduct of Research sets out the responsibilities and corresponding policies for researchers, institutions and the agencies that, together, help support and promote a positive research environment.

Confidentiality

Members appointed to the selection board must read and agree to the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers describing expectations and requirements.

Privacy Act

Personal information refers to any information about an identifiable individual. Based on the Privacy Act, personal information provided by applicants must be used only for assessing applications, making funding decisions, and related uses describing applicants at the time that their personal information is collected. Members are reminded that the use or disclosure of this information for any other purpose is illegal. It is important for panel members to adhere strictly to the guidelines set out in the Conflict of Interest and Confidentiality Agreement for Review Committee Members, External Reviewers, and Observers.

Canadian Human Rights Act

The activities of CIHR, NSERC and SSHRC are subject to the Canadian Human Rights Act. See Purpose of the Act. The Act is intended to ensure that all individuals have an equal opportunity to make for themselves the lives that they are able and wish to have. They should also have their needs accommodated, consistent with their duties and obligations as members of society, without being hindered in or prevented from doing so by discriminatory practices based on race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital status, family status, genetic characteristics, disability or conviction for an offence for which a pardon has been granted or in respect of which a record suspension has been ordered.

Official Languages Act

All members of the selection board and program staff must be aware of their obligations and rights under the Official Languages Act.