June 17-18, 1998

Conference Rooms 7-8, Building 31, NIH, Bethesda, MD

Dr. Baldwin welcomed the group, and updated them on two important activities at the NIH which relate to peer review. First, she reported that the Bioengineering Consortium (BECON) held a very well-attended symposium last February. BECON has raised the issue of peer review of applications involving bioengineering, and is working with the Center for Scientific Review to address this; the PROG should be hearing a report of this work at the next meeting. Second, Dr. Baldwin noted that this is the last review cycle for R29 applications; hereafter new investigators will be submitting R01 applications designated as being from "New Investigators" and there will no longer be an R29 grant mechanism. She added that while there are fewer R29 applications this cycle, the number of applications from new investigators overall is higher, which confirms that the word has been received that the NIH will continue to support new investigators through the R01. Reviewers are being given a list of new investigators prior to the meetings and the application form itself will soon have a new box on the face page which investigators will use to indicate "new" status. Dr. Ehrenfeld commented that Scientific Review Administrators and scientific review group chairs have been specifically instructed to remind reviewers that these applications should not be expected to have as much preliminary data or as extensive track records as those of experienced investigators.

Update on the Center for Scientific Review

Dr. Ehrenfeld reminded the group of the six priority areas she had previously announced for the Center for Scientific Review (CSR): study section organization, reviewer quality/study section composition, perception that segments of the community are ill served in review, speed and consistency of the review process, responsiveness of CSR to the NIH Institutes and Centers, and enhancement of Scientific Review Administrators’ (SRA) function. She updated the group on activities in each of these areas.

Study section organization is being dealt with through the integration activities for the review of neuroscience, AIDS/HIV, and behavioral and social sciences research applications, and more generally through a special panel of extramural scientists, the Panel on Scientific Boundaries for Review. In addition, CSR is forming advisory groups for each of the Initial Review Groups (IRGs, or clusters of scientifically related scientific review groups.) CSR is addressing reviewer quality by piloting efforts toward greater flexibility for terms of reviewer service and new reviewer nomination procedures.

To address communities concerned about possibly being disadvantaged in review, CSR has had an individual from an extramural university on a visiting appointment to develop recommendations for the review of clinical research (which are currently being implemented) and will soon announce a similar appointment for the behavioral and social sciences. In addition, the integration of review activities are addressing the adequacy of the behavioral and social sciences study section composition. Special attention is currently being given to the review of applications involving bioengineering; a working group formed this summer should have recommendations by this fall; this effort will address both design-driven research that addresses device development and broader bioengineering research. Commenting on the diversity of the field of bioengineering, Dr. Ehrenfeld said that defining the issues will be a first step, and that this effort will be done in conjunction with the NIH Bioengineering Consortium.

The speed and consistency of review are being addressed trans-NIH through reinvention of the receipt and referral processes. In response to a question about referral, Dr. Ehrenfeld commented that, since applicants may elect to self-refer to IRGs, CSR is working to ensure that there is sufficient information available to enable applicants to make an informed choice. Referral to the study section level is made by the IRG chief. Addressing the final two issues, Dr. Ehrenfeld stated that internal best practices are being developed regarding communication between CSR and the Institutes and Centers, and an internal CSR committee is developing best practices and training for SRAs. In addition a planning meeting of scientific review group chairs will address training for chairs and reviewers and recruitment procedures.

Dr. Ehrenfeld also brought up for discussion the issue of review of the applications from study section members, and that it may be a disincentive to service on a review group if one’s grant application will be submitted during the term of service as a reviewer. She indicated that this is an area where she welcomes suggestions. The discussion did not address review of these applications, but rather focused on the need for flexibility in terms of service: members felt that review service over four years with three meetings a year is quite demanding, given that those eligible for such duty also are busy conducting research, writing grant applications, and performing other professional responsibilities.

Consumers on Advisory Panels

Dr. Baldwin explained that the NIH has used lay members or consumers for years on advisory panels; all national advisory boards and councils are made up in part of lay members. In addition, various Institutes and Centers include consumers on panels to advise on specific issues. Most recently, the National Cancer Institute (NCI) established the Directors Consumer Liaison Group (DCLG), to increase the representation of consumer-advocates in program and policy development, to increase communication with the scientific community at the NCI, and to help NCI widen the pool of qualified consumer-advocates who can be called upon to serve on NCI advisory committees and other groups. In addition, Dr. Baldwin noted, recently consumers also have served on some peer review panels, such as the Department of Defense’s breast cancer research initiative, as well as in two of the NIH Institutes. Such an innovation in peer review should be considered by the PROG.

Dr. Hortencia Hornbeak presented information on the experiences of the National Institute of Allergy and Infectious Diseases (NIAID). She defined a consumer advocate as someone who is knowledgeable of a disease, condition or subject from a patient perspective, is interested in public policy, and is active in disease-related community organizations. The role of a consumer on a peer review panel is to contribute an informed patient perspective to the evaluation of research proposals. It is important that those selected for such activities not only have the requisite knowledge and an affiliation with a community-based organization related to a particular disease or condition, but also that they offer no apparent biases or conflicts of interest. They must demonstrate objectivity, good communication skills, and understand the NIH peer review system.

Dr. Hornbeak indicated that the NIAID has used more than 60 consumer advocates, representing less than one percent of the more than 7,000 peer reviewers used during the past 10 years by the Institute. NIAID has used consumer advocate reviewers on panels reviewing applications for clinical trials, epidemiologic studies, education and demonstration projects, centers, and one basic science review of projects involving gene therapy. NIAID staff carefully orient consumer reviewers, providing not only the same materials that other reviewers receive, but also an explanation of the consumer’s role and it’s value to the committee; that role is clearly delineated as to what aspects of the proposal the consumer reviewer is being asked to address. These areas usually are issues such as feasibility of plans for recruitment, retention and followup of research participants; feasibility of protocols with specific populations with regard to complexity and compliance; outreach; composition and role of Community Advisory Boards; human subjects protection and consent forms. Consumer reviewers are given assignments, vote, prepare written critiques, and participate actively in discussions, generally brining to the table a somewhat different perspective from that of the scientist reviewer. The NIAID experience has been that consumer reviewers help to bridge the knowledge and communication gaps among the scientific community, policy makers and patients.

The experiences of the National Cancer Institute were shared by Dr. Marvin Kalt. He agreed with Dr. Hornbeak’s assessment of the value of consumers in peer review, adding that applicants appear to focus applications more clearly once they are aware of the presence of consumers on review panels, and that the presence of consumers reaffirms the validity of the review process to an important constituency, patients and advocates. The enhanced communication which results aids consumers’ understanding of science and scientists, and improves scientists’ understanding of patient concerns. NCI has noted favorable responses by both researchers and consumers. Dr. Kalt noted that orientation for this activity must begin with staff, and that the nomination process for consumer reviewers is important. He then discussed the various possible sources of nominations (including local and national advocacy groups, other granting agencies, awardees, and experienced reviewers.) A formal recruitment process is needed, and the NCI now has a specific Office of Liaison Activities, established in 1996, which offers "one-stop shopping" for organizations within the NCI; the office assists those seeking to use consumers in various capacities, including peer review. The DCLG has developed a stringent selection process, and is developing a pool of potential consumer reviewers from which a Scientific Review Administrator could select consumer reviewers as needed.

Dr. Kalt also pointed out some potential challenges of the use of consumers in peer review: the extensive workloads often levied on reviewers may be a significant burden on someone in frail health; the time commitment required may be difficult for a non-academician who must take annual leave from work; the reimbursement process for travel expenses can present financial difficulties; and the applicability of consumer input may be less apparent in the review of basic research. Dr. Kalt cautioned that substantial differences in programs, review criteria, and procedures make comparability across agencies or even NIH Institutes and Centers difficult, and that differences in focus and scope in some review settings may limit the use of consumers. The NCI currently uses consumers in the peer review of applications submitted in response to some Requests for Applications and Requests for Proposals, and specialized research centers, and is considering their use in reviews of cancer center and clinical cooperative group applications.

In the subsequent discussion, PROG members agreed that there are obvious values to including consumers in the peer review process in certain cases, and that this is more apparent for disease-specific and clinically oriented research, reviewed within the NIH Institutes, than for much of the research reviewed within the Center for Scientific Review. They affirmed the importance of careful recruitment and selection process, agreed on the apparent value of bringing the consumer/ patient perspective to the review process, and applauded the enhancement of communication between the scientific and consumer communities that results. The PROG will undertake an examination of the general process of using consumers in peer review, considering the experiences of other agencies and organizations. Since this practice at NIH is relatively new and is limited to just two Institutes, Dr. Baldwin suggested that a working group of PROG might develop principles that might guide and assist other Institutes and Centers as they consider the use of consumers in peer review.

Scientific Review Group Visits

PROG members spent the morning visiting institute or CSR scientific peer review group meetings. They were invited to comment on the review process, based on both their morning’s observations and their own previous experiences. Overall, members felt that the use of the new review criteria seems to be progressing well. Some groups are explicit in using the criteria as guiding "headers" for their discussion of applications, while others are less formal in structuring their comments but do mention the criteria within their discussions. PROG members felt that posting the criteria in the meeting room stated seemed to serve as a good reminder for reviewers; greater uniformity of such practices might be helpful. There was some discussion about whether or not more refined definitions of the criteria would be helpful; while there were some mixed opinions, most agreed that there is a danger of becoming overly restrictive with definitions that become very refined. It was suggested that sample summary statements might be useful in guiding those reviewers who are still inconsistent in writing their critiques to specifically address the criteria; there are such samples available and some SRAs are already sharing them. The use of the criteria will be assessed when they have been in use for at least a year, but reviewers seem to be gaining familiarity with them and becoming comfortable using them.

Another observation was that some chairs and/or SRAs are more formal than others in their structuring and management of review meetings. It was suggested that there are certain areas such as orientation of members and training for new reviewers that should be addressed with a degree of uniformity. At the next PROG meeting, there will be a CSR presentation on SRA, chair and reviewer training and orientation.

Among the other topics discussed was streamlining of applications. It is presumed that review groups will discuss and score approximately half of the applications reviewed in a given cycle, although this proportion may fluctuate from one cycle to the next. Dr. Baldwin noted that streamlining is to be applied to projects for which further discussion will not enhance the assessment of their scientific merit, and members indicated that it was probably unwise to try to tie it too tightly to a numeric score. They also commented that the current process of bringing back for full discussion any application on which the streamlining decision is not unanimous is a good safeguard, and that it is important to know why an application was not scored; therefore the provision of a summary statement with the written critiques remains important.

Integration of Review

Issues for the Next PROG Agenda

At the next meeting, PROG has requested to hear from CSR on their initiative for training and orientation of Scientific Review Administrators and review committee members and chairs. They will also hear a report on the review of bioengineering research applications. A working group on consumers in peer review will be formed and principles will be developed. There was also a request to discuss the usefulness of the new criteria, with specific attention to innovation. It was agreed that the next meeting will be in late January – early February, 1999.