Minutes of the Meeting of the Peer Review Oversight Group (PROG)

November 3-4, 1997

Conference Rooms 9100-9104, Rockledge Center 2, Rockville, MD

New Investigators

Dr. Marvin Cassman, Director of the National Institute of General Medical Sciences, and cochair (with Dr. Ellie Ehrenfeld) of the NIH Working Group on New Investigators, presented the findings of that group. Dr. Harold Varmus, NIH Director, commissioned this group one year ago to address how we might best support new investigators. The group examined various mechanisms which support new investigators; new investigators were defined as those who had never before served as principal investigator on NIH research project grants. In this presentation to the PROG, Dr. Cassman focused on the utility of the R29 mechanism, presenting data from 1980-1995 which indicates that new investigators do not fare any worse than other applicants competing for new awards. New investigators applying for the R29 have a higher success rate when applying for their initial award, but when competing for their next grant, those holding R29s fare less well than investigators who initially received R01 awards. The working group recommended that the R29 mechanism be eliminated, but that the most popular feature, the five year time period, be retained as an option for the new R01 applicant. Because there is no dollar limit, awards to new investigators will cost more, but estimates vary and no exact figure can be given. The goal is to maintain a steady state of entry between new investigators and loss rate (those not receiving continued awards); this replacement rate is about 9%. (For updated information, see http://grants.nih.gov/grants/news.htm.)

The PROG members praised the report, and agreed that integrating the R29 into the R01 mechanism and guiding all new investigators to apply for the R01 is a good idea. They shared anecdotes indicating that the R29 is in fact stigmatized in some institutions as a lesser accomplishment than receipt of an R01 award, and that for some clinical researchers the dollar limits on R29s have been too restrictive. The group was strong in the support of efforts to encourage new investigators to enter research, stressing that reviewers should be encouraged to recognize that new investigators may have valuable ideas but be less adept at grant writing per se and that review by explicit criteria may be helpful. The members felt it would be interesting to track the success of new investigators over the next several years. It was explained that there is some variation across Institutes and Centers (ICs) on paylines in general and in the type of special attention given to new investigators (in any given funding cycle some ICs may have a more generous payline for new investigators), but that across the NIH the replacement rate of 9% mentioned by Dr. Cassman has held steady for the past few years. The group agreed that the report provides useful data and information and should be shared publicly; they also expressed unanimous support for the recommendations in the report..

Update on the Center for Scientific Review (CSR)

Dr. Ellie Ehrenfeld, Director of the Center for Scientific Review, announced that the organization's name has been changed to more accurately reflect the role of this component of the NIH, i.e. scientific peer review rather than grant-making. She outlined several high priority issues that the CSR has been addressing, including implementing a new organizational structure with three scientific review divisions, Molecular and Cellular Mechanisms, Physiological Systems, and Clinical and Population-Based Studies, each containing approximately six Initial Review Groups (IRGs). As new scientific review groups are formed through the integration of review for the National Institutes of Mental Health, Drug Abuse and Alcohol Abuse and Alcoholism, they are becoming part of this new organization. For example, the 21 new review groups formed as part of the integration of the review of neuroscience applications resulted in three new IRGs, one in each of the three CSR divisions.

Dr Ehrenfeld is also commissioning a Blue Ribbon Panel to make recommendations about the organization of CSR on a macro level. Through the CSR Advisory Committee, she is also establishing oversight groups for some of the IRGs, and has already created an evaluation office within CSR to assess changes being made in the review process. Special efforts are being made to deal with groups who perceive that they are at a disadvantage in the review process; for example, a consultant will soon report on the review of clinical applications, and an examination of the review of behavioral science applications is planned. Reinvention activities have focused on the receipt and referral process. Training of Scientific Review Administrators, reviewers and scientific review group chairs is a priority for CSR; there have been several changes, including the impending use of the new explicitly stated review criteria, which require special communication with and orientation of reviewers. Dr. Ehrenfeld indicated that there is a variety of styles and approaches to the orientation and training of reviewers, and that she is establishing a training committee to address areas where there is a need for greater uniformity, and to develop formal instructional materials where necessary. A meeting of scientific review group chairs is also planned, as part of this training effort.

The PROG members were enthusiastic about the changes at CSR, and made suggestions that might be considered in the training effort. For example, a training video of a review group meeting, a mock review prior to real meetings for standardization purposes, enhanced communication and feedback between SRAs and program staff, a minimum set of standards for all SRAs and review group chairs were suggested. It was also suggested that this be done not only for CSR but for all review across the NIH. Dr. Ehrenfeld agreed, pointing out that promoting such trans-NIH review efforts is a major purpose of PROG. She noted that reviewer training has already begun with dissemination of the new criteria statements.

Review of Clinical Research

Dr. David Kupfer, Chair of the PROG Working Group on the Review of Clinical Research, reported that, in light of other current activities on this topic, the group felt it best to share their recommendations to date and stand by for ad hoc service as necessary. Dr. Michael Simmons, a member of the group and consultant to CSR, is finalizing a report with specific recommendations regarding the review of clinical research.

The group initiated the development of a process for profiling the adequacy of a scientific review group to review clinical research, with the hope that such a process could also serve as a model for other areas of science, since the issue of how to profile review group expertise is a generic one, not unique to the review of clinical applications. Using expertise codes, degrees, institutional and departmental affiliations, and NIH funding histories for reviewers, the group attempted to profile 14 scientific review groups (SRGs). Of 305 reviewers, the members felt there was sufficient information to make a designation for 235 (about 77%). Review groups were designated as "high", "mid" and "low" for clinical applications, based on the proxy measure, proportion of clinical awards which had been reviewed in that SRG. For the five SRGs designated as "high," the proportion of "clinically expert" reviewers ranged from 62-100%. Of the five "mid" SRGs, the range was 13-90%, and for the "low" SRGs 0-28%. Since the group was developing the process as they tried it out, these data represent only indicators that the process might prove a viable starting point for profiling SRG expertise. However, it does not address the issue of whether there is an optimal proportion of reviewers or whether the number of clinical reviewers should be within a certain range of the proportion of clinical applications to be reviewed.

In developing this process, the working group identified information requirements for a useful and meaningful profiling process. The group specifically noted a need to examine how and by whom reviewer expertise codes are assigned and verified, and what kind of background information is provided on each reviewer, including additional or prior degrees which may indicate clinical skills (e.g. DDS, MPH.) It would be useful to determine whether there actually is a critical mass of applications required to ensure a fair and thorough review for clinical applications (and other types of applications with specific technical or other criteria.) Several variables will affect this: the review committee chair, the scientific review administrator, and the proportions of clinical applications and clinical reviewers. It is likely that threshold ranges may be more appropriate than a set number or percent and should probably be established both for applications and for reviewers. The profiling procedure will require further development and refinement, when complete, credible, accurate data are easily retrievable through the electronic databases currently under development as part of the peer review module of the Electronic Review Administration. Reviewer orientation and SRA training will also be important components of the review of clinical research grant applications. The working group recommends the development of best practices guidelines, parameters for optimizing review of clinical applications, and training modules for staff and reviewers.

Behavioral Science and AIDS Review Integration

Dr. Virginia Cain, Office of Behavioral and Social Sciences Research, is coordinating the integration/ reorganization of the review of behavioral and social sciences research grant applications. She reported that the guiding principles established for the neuroscience activity are being used, as is the essential process developed by that group. The behavioral sciences integration of review activity has as its goals to assure that CSR review groups in behavioral and social science reflect the current state of the science, to create a structure that can adapt to future developments in science, and to ensure high quality peer review that identifies the most meritorious applications for each institute to consider for funding. A total of eleven Institutes and the CSR are participating in this effort, with others staying informed for possible future participation. The timeline is geared toward presentation of a final plan by November, 1998, for implementation for review of applications to be funded in 2000.

Dr. Gil Meier, of the Center for Scientific Review, addressed the group about the integration of review for AIDS-related research grants applications. The goals for this integration effort are parallel to those presented for the behavioral and social science. The group has identified four major research categories: basic HIV/AIDS research; epidemiology and clinical HIV/AIDS research; behavioral and social science HIV/AIDS research; and vaccine research for HIV/AIDS. The committee is now seeking comments from current review group members, and is exploring various means for soliciting input from the broader scientific community, including presenting the descriptions on the CSR homepage, presenting them at scientific and NIH advisory council meetings, and publishing them in newsletters. The timeframe for implementation is for review of applications submitted in May, 1998, for potential funding in the fall of 1998.

Review of Neuroscience Integration: Plan for New Scientific Review Groups

Dr. Elliot Postow, Center for Scientific Review, presented an overview of the Neurosciences Working Group's plan for 21new neuroscience review groups, which have been grouped into three new IRGs, one in each of the three scientific divisions of CSR. The plan will be posted on the CSR homepage in December, with rosters of reviewers and SRAs for the new scientific review groups. The PROG members were enthusiastic about the plan and are eager to see it implemented. The effort has not only resulted in a plan for the review of neuroscience research grant applications but also has developed a process that is serving as a model for the other integration efforts, allowing them to progress more quickly to implementation.

Creativity

The PROG met in closed session to discuss summary statements of both funded and unfunded applications provided by Institute program staff as examples of creative or innovative research ideas. As reported in the open session, the members noted that there were some summary statements that were succinct and clear, but many focused on feasibility, methodological details and other minor criticisms. Often there was not enough attention given to the importance of the project. The members suggested considering a pilot that would focus reviewer attention on the review criteria more broadly by limiting critique length to one or two pages.

Visits to Scientific Review Groups

The members attended scientific review group meetings that included both regular review groups and Special Emphasis Panels, and covered a wide range of types of applications and approaches; mechanisms reviewed included applications for research project grant, fellowship, and Small Business Innovation Research/ Small Business Technology Transfer. The goal of the visits was to give the members an opportunity to broaden and update their familiarity with current review processes, as well as enabling them to consider issues such as where greater uniformity in review procedures might be advisable or where variability is an asset within the review. The members agreed that uniformity in orientation of review groups is highly desirable, and that the roles and relationship between the Scientific Review Administrator and the review group chair are among the most important of review procedures. The members reinforced to Dr. Ehrenfeld the importance of the training she intends to develop and of making this a trans-NIH effort, encompassing all review at the NIH. This experience was considered sufficiently valuable that the members would like to visit additional review meetings. The next PROG meeting will be scheduled during the June round of review meetings, to afford them the opportunity to attend additional review meetings, including not only those conducted through the Center for Scientific Review but also those conducted through the review branches in the Institutes and Centers.

The NIH Review Rebuttal Process.

At the last meeting, the PROG members endorsed the idea of eliminating the second level appeal process and moving towards greater uniformity of the rebuttal process across the NIH. As a followup on this issue, Dr. Yvonne Maddox of the National Institute of Child Health and Human Development and Dr. Ken Warren of the National Institute on Alcohol Abuse and Alcoholism presented summaries of the procedures those Institutes use in handling rebuttal letters from investigators. They explained that investigators may rebut the following issues in the peer review of a research grant application: irregularities in the review process, factual errors, perceived bias or conflict of interest, or lack of appropriate scientific expertise on review panels. In both Institutes (and across the NIH) an applicant who has serious concerns with the review process has the option of discussing these issues with the program official and of formally addressing these issues in writing. A formal rebuttal is either resolved by the program and review staff or taken to the Institute's national advisory council or board for adjudication. Frequently, even when the rebuttal process is likely to result in a recommendation for re-review, the program official and applicant agree will that it would be better to revise the application. While there are some variations in the way that different Institutes handle rebuttals, the basic procedures are the same. These procedures are currently under revision, and the streamlined procedures will likely result in even greater uniformity across the NIH. The PROG members recommended renaming the process as Appeals rather than Rebuttals to make it more understandable, and stressed the importance of presenting the new procedures clearly to the research community.

Agenda Items for Next Meeting

At the next meeting, the PROG will visit additional scientific review group meetings, and will continue their discussion of the need for uniformity in procedures. During the interim period before the June meeting, the members will receive periodic updates on efforts currently underway, e.g. the integration of review in the neurosciences, AIDS, and behavioral and social sciences, and on the review of clinical research. These updates will be posted on the world wide web at see http://grants.nih.gov/grants/oer.htm.) in the PROG section under Peer Review. The members will be notified by e-mail when updates are posted. An agenda will be developed prior to the June meeting, through electronic communication with the members.

Summary and Conclusions

During the two day meeting, several topics were addressed. Dr. Marvin Cassman presented highlights of the report of the NIH Working Group on New Investigators, with specific attention to the R29 (First Independent Research Support and Transition award), including the working group's recommendation that the R29 be absorbed into the R01, and that all new investigators should be guided to apply for R01 grants. The PROG members endorsed this recommendation.

Dr. Ellie Ehrenfeld, Director of the Center for Scientific Review (CSR), presented an update on the Center's reorganization, including the changing of the name from Division of Research Grants to CSR. She enumerated activities taking place at the Center, including the integration of review of applications in the neurosciences, behavioral and social sciences, and AIDS and AIDS-related research, and establishment of a Blue Ribbon panel to make recommendations on the macro level organization of the Center. In preparation for implementation of the new explicitly stated review criteria, CSR staff are disseminating information to reviewers. A training committee is being formed at CSR to address issues of staff training and reviewer orientation. The PROG members suggested that this be a trans-NIH activity, and Dr. Ehrenfeld agreed to work with the Institutes and Centers on this effort.

Several updates and verbal reports were presented. Dr. David Kupfer reported on the activities of the PROG Working Group on the Review of Clinical Research. The group developed a framework for profiling the clinical expertise of a review committee, which could also be applied to other research areas, and concluded that any such process will need to be further developed at such time as complete, credible and accurate data are easily retrievable through electronic databases, such as those currently under development. Updates were given on review integration. Drs. Virginia Cain and Gil Meier presented overviews of the processes being followed and progress to date on the integration of review for the behavioral and social sciences and AIDS and AIDS-related research, respectively. Dr. Elliot Postow presented the final plan for 21 new neuroscience scientific review groups, which will comprise three new Initial Review Groups distributed across the three scientific divisions at CSR. These review groups will be reviewing applications beginning with those received in February, 1998; the scientific review group guidelines and partial rosters of reviewers will be posted on the CSR homepage in December, 1997.

The NIH Rebuttal and Appeal Process, which is currently under revision, was discussed by Drs. Yvonne Maddox of the National Institute of Child Health and Human Development and Ken Warren of the National Institute on Alcohol Abuse and Alcoholism. While there are some variations in the way that different Institutes handle rebuttals, the basic procedures are the same. These procedures are currently under revision. The PROG members recommended renaming the process Appeals and presenting it in clear simple terms to the research community. (See NIH GUIDE, Vol. 26, No. 38 - November 21, 1997, at http://grants.nih.gov/grants/guide/1997/ 97.11.21/index.html

During a closed session, the PROG members discussed summary statement of both funded and unfunded innovative, novel or creative research grant applications. In open session, they summarized their discussions. The members voiced the opinion that there was generally an overemphasis on feasibility, and expressed the hope that use of the new explicit criteria might address this problem in part. It was also suggested that a pilot be conducted to determine whether limiting the length of reviewers' critiques might illicit a tighter focus on the overall importance of the projects.

The members visited scientific review group meetings, to update and broaden their knowledge of review procedures. They observed the review of various mechanisms in regular scientific review groups and Special Emphasis Panels. The most consistent observation was that the roles and preparation of and interaction between the Scientific Review Administrators and review group chairs is one of the most important elements in the conduct of scientific review meetings. The members expressed the desire to attend additional review meetings during the next PROG meeting.

The next meeting will be held in June 1998, and will be scheduled in conjunction with Institute and CSR review meetings, so that members can again visit scientific review meetings. In the interim, updates on various issues will be posted on the peer review/ PROG section of the NIH Grants page of the world wide web (http://grants.nih.gov/grants/oer.htm) with e-mail notification to members each time an update is posted. The agenda for the June meeting will be developed through correspondence with the members prior to the next meeting.