At the November 20-21, 1996, meeting of the Peer Review Oversight Group (PROG), a major topic was the Rating of Grant Applications (RGA). At the previous meeting (July, 1996), PROG members had discussed the recommendations in the RGA report, and considered information which had been obtained from the scientific community through a variety of channels. At that time, the PROG decided to table several of the ten recommendations in the RGA report and to focus primarily on whether to use explicit criteria to structure the review, if so what these criteria should be, whether to score/rate those individual criteria, and whether to retain reviewer assignment of a global score or derive an overall score from criteria subscores. Pilot experiments were recommended, and the preliminary results of pilots by both the Division of Research Grants and the National Institute of Allergy and Infectious Diseases were presented to the members at the November meeting.

Based on the information available, the following decision was made after the November meeting: There will be no changes in the basic numerical scoring system at this time. There was enthusiasm for having reviewers continue to assign a global score to each application and that practice will continue. There was generally low enthusiasm for the idea of assigning scores, whether numeric or alphabetic, to the individual criteria or deriving an overall score using such subscores, and those practices will not be adopted, but will be discussed further at the Feburary PROG meeting. The issue of the rating scale itself has been tabled for future discussion; for the present time, the 1.0-5.0 rating scale, with 1 as the best possible score, will be retained.

In addition, there was much discussion of the information on the use of criteria, and indications are leading in the general direction of using criteria to structure the written critique and the review discussion. A final decision on this issue will be made in January or February 1997 based on additional information currently being obtained from program staff. The criteria to be used would include at least Impact, Feasibility, and Investigator/Environment; whether Innovation/Creativity would be best dealt with as a separate criterion is under continuing consideration. There was general agreement on the importance of fostering innovation and creativity in research, and considerable discussion on how best to ensure that such applications were submitted, receive appropriate review and are supported by the Institutes and Centers. Regardless of whether a separate review criterion is adopted, the consideration of Innovation/ Creativity in review and in research in general will continue to be of the utmost importance for the PROG and for the NIH.

The criteria under consideration are stated below, and will be accompanied by some guidelines or suggestions/examples of what they might encompass:

IMPACT: The extent to which, if successfully carried out, the project will make an original and important contribution.

FEASIBILITY: The extent to which the conceptual framework, design, methods, and analyses are adequately developed, well integrated, and appropriate to the aims of the project.

INVESTIGATOR/ ENVIRONMENT: The extent to which the investigators, available resources, institutional commitment, and any other unique features will contribute to the success of the proposed research.

INNOVATION/CREATIVITY (working definition): The extent to which the project employs novel concepts, approaches or methodology.

It was emphasized at the meeting that it is important for the scientific community to understand that the use of these criteria does not represent a change in the PHS 398 instructions, nor does it add criteria to the areas that reviewers are instructed to use in considering the scientific merit of applications during the review process. Rather, this effort represents a reformatting or reorganization of the presentation of the information, in the hope that it will result in more informative and perhaps more balanced critiques. This should be an advantage to those who rely on this information, both program staff in the Institutes and Centers and research investigators.

A working group was formed to gather information on the views of NIH program staff regarding the use of a specific criteria for creativity will be chaired by Dr. Claude Lenfant, Director of the National Heart Lung and Blood Institute and will include representatives from the biological, clinical and behavioral sciences.

Note: The above summarizes only the issue on the Rating of Grant Applications discussed by the PROG; for more detail on these deliberations and a summary of the other issues discussed, see the Summary Minutes of the PROG for November, 1996, which will be posted on the World Wide Web at http://www.nih.gov on the Grants page under Peer Review Issues. At the next PROG meeting (Feb. 13-14, 1997) members will discuss the issue of how to evoke creative, innovative projects, the possible use of a separate criterion for creativity, and the wording of any further explanation of the criteria to be used, along with other agenda items.