Peer Review Detailed Assessments
During the recent scheme rounds of the National Competitive Grants Program (in 2017), the ARC's Peer Review team received feedback regarding the quality of assessments from our Detailed Assessors. The quality of assessments is a key factor in ensuring the good reputation of the Peer Review process of the ARC's National Competitive Grants Program, and so in an effort to improve quality and consistency, the ARC have written to Deputy Vice-Chancellors and Directors of Research Officers with the information below on how assessors should allocate scores and write assessment text that assists applicants and the ARC College.
This information may be relevant to research staff and any other relevant stakeholders in the university who may conduct assessments for the ARC. However, it may also benefit applicants for ARC funding when preparing proposals or a rejoinder. These points should be kept in mind when next conducting an, or preparing an application for, assessment under the ARC National Competitive Grants Program Schemes.
The ARC received notification of inappropriate assessments through applicant rejoinders, research office correspondence and from members of the College of Experts. The following types of assessments were identified as inappropriate in recent scheme rounds:
- minimal text provided in the assessment (e.g. a one sentence statement)
- assessment text does not match the scores provided in the assessment (e.g. low scoring, but positive, encouraging text and vice versa)
- comments copied across multiple assessments or from previous assessments/scheme rounds
- assessment text indicates the assessor has not actually read the PDF
- scores are detailed within the text
- assessment text includes information that indicates an undeclared conflict of interest
- inappropriate comments, e.g. malicious or denigrating remarks. (Note: hard, constructive criticism is permitted)
The following two concepts illustrate two key points of good assessment:
- Assessments show consistency between the text and score
While assessment text is released to applicants for rejoinder, assessment scores are not - they are only visible to the ARC College of Experts. Consequently, it is important that text and score agree, as this allows the applicant to address weaknesses in the proposal in their rejoinder and provides clearer information for the ARC College of Experts to better understand the scores the assessors have assigned against the selection criteria. Positive text with a weak score can give the applicant a false sense that the reviewer supports their proposal and also does not provide an opportunity for applicants to learn how to improve proposals in the future.
- Assessments give clear details on the strengths and weaknesses of the proposal
This assists the applicant and gives the ARC College of Experts a clear understanding of the assessor's arguments and how these underpin the scores assigned to proposals. This is immensely valuable as College members rely on the expertise of assessors to bring to light the merits and weaknesses of proposals.
Please don't hesitate to contact email@example.com if you have any queries in relation to ARC Assessment.