|
|
|
|
|
 |
|
|
|
|
|
Dr. Baldwin welcomed the members and gave a general overview of the agenda. Each Peer Review Oversight Group (PROG) member was given a "Talking About NIH" packet; Dr. Baldwin explained that these packets contain information to assist members of the scientific community who serve on various advisory committees and councils, and might be asked to talk generically about the NIH.
Dr. Baldwin noted that we are within the comment period for revisions to the PHS 398 grant application form, but emphasized that any changes should decrease rather than increase respondent burden. An example would be decreasing the number of pages or eliminating some part of the form. There was brief discussion of the fact that what represents a change in burden from the administrative point of view of the Office of Management and Budget (OMB) does not always coincide with what would represent an increase or decrease in effort for applicants, such as a checklist to replace a page of text. Any increases would need to be offset with equal or greater decreases in applicant burden, as defined by OMB.
Dr. Baldwin presented an update of the information provided in the SBIR report (available on the NIH/Office of Extramural Research homepage at http://grants.nih.gov/grants/funding/sbir.htm). Dr. Ehrenfeld commented that at some point the Division of Research Grants would appreciate having PROG discuss and analyze the review of SBIR applications.
Go back to the Table of ContentsTo begin the discussion, Dr. Baldwin pointed out that the timing for a decision is important as we are now at a crucial point in the "life cycle" for review and funding for the next fiscal year; therefore, there is a need for closure on any advice the committee would like her to carry to Dr. Varmus, who will make the decisions regarding implementation of any RGA recommendations. The discussion focused on the use of "creativity" as an independent criterion, not the value that NIH places on creative research. NIH does value creativity in research and the question before the group was to recommend the best ways to elicit and identify such applications.
Dr. Claude Lenfant shared the report of the "creativity criterion" working group which he chaired, whose charge was to obtain the opinions of program staff on this topic. Dr. Lenfant indicated that the findings of this group uniformly indicated that, while all recognize the importance of funding innovative and creative applications and agree that these issues should be addressed in reviewers' critiques, creativity and innovation permeate all three areas addressed by the other three criteria sets, and should not therefore be set apart in a separate criterion. Dr. Leshner noted that he had had separate discussions with extramural scientists who offered the same views. There was again discussion of the public perception of whether creativity and innovation are valued by the NIH and by scientific peer reviewers, and it was noted that creativity and innovation should be made explicit within the other three criteria and be incorporated into reviewers' education. Dr. Lenfant favored persuasion rather than edicts.
Dr. Baldwin stated that NIH is strongly in favor of eliciting, identifying and funding creative and novel research projects, but that in her view this requires a complex intervention. She presented a six step plan for addressing creativity and innovation, not all of which she noted would fall within the purview of PROG. These six steps address ways to change the perception that NIH does not want or reward creative research applications, by (1) more clearly communicating interest in such research through the science press, world wide web homepage activities, and speeches of our senior officials; (2) communicating to review groups that we expect them to identify creative/innovative applications or parts of applications; (3) assuring the applicant community that reviewers themselves represent creative thinkers who will recognize these qualities in the applications they review; (4) ensuring that the language used to guide reviewers in the preparation of critiques indicates the importance NIH places on the identification of these applications; (5) encouraging the use of selective pay strategies to ensure that creative or novel ideas are well represented in the Institutes' research portfolios; (6) publicizing ways that we have succeeded in soliciting and encouraging creative and novel research. The fourth of these six points is the one that clearly falls within the domain addressed by the PROG.
Generally the group endorsed this approach. Dr. McGowan commented that NIH is funding some very creative work, and agreed that an educational campaign as outlined by Dr. Baldwin would be a good way to increase the visibility of existing efforts and increase them, and endorsed using three criteria, with creativity and innovation incorporated, as a good step in this direction. Dr. Yamamoto vigorously endorsed a fourth separate criterion of creativity; he indicated that creativity was not subsumed under other criteria but was independent. Those recommending four criteria voiced the view that this would send a signal to the research community that they should send in creative applications and to the reviewer community that they needed to be alert to such applications. Dr. Lenfant made the point that we must balance among various kinds of research, and that some very valuable projects may not be ground breaking with regard to novelty but are crucial to the advancement of science. At this point Dr. Ehrenfeld indicated that she viewed Impact as the most important of all the criteria sets discussed, stating that it should be listed first and clearly elevated in importance. Dr. Marrett agreed, stating that Impact is so important that it needs to be highlighted for both applicants and reviewers. The group generally agreed. Dr. McCombs argued for changing the label "Feasibility" to "Approach"; the group agreed that this should be considered.
Dr. Baldwin summarized reasons to stay with the three explicit criteria sets being discussed, with creativity/innovation incorporated within them: the strong preference of both NIH staff and some elements of the scientific community for this approach, the results of the pilot studies, the belief that creativity and innovation are dispersed throughout those three areas, the view that the problem is a complex one requiring a broader intervention (as outlined above), and the fact that program staff have indicated that a fourth separate criterion for creativity would not be helpful. Although there were dissenters and abstainers, a plurality voted to recommend the adoption of three criteria. It was agreed that this information along with the members' alternate proposals would be presented to Dr. Varmus.
Go back to the Table of ContentsThe Report of the NIH Director's Panel on Clinical Research was discussed, with specific attention to the fourth recommendation: "Ensure fair and effective reviews of extramural grant applications for support of clinical research; panels must (a) include experienced clinical investigators and (b) review an appreciable number of such applications." There was some discussion of the definition of clinical research put forth in the report; this definition was accepted and praised. Dr. Kreek indicated that she felt the definition for clinical research was very good and extremely useful. Dr. Norman Anderson pointed out, however, that the definition as currently written would include all behavioral studies as clinical research. As there is a large community of behavioral and social scientists who do not consider themselves clinical scientists, but who study fundamental behavioral processes in healthy individuals (e.g. learning, memory, language, social loss), he suggested that point 1.b. of the definition should specify the types of behavioral studies that should (and should not) fall under clinical research. Several PROG members commented on the need to bring experienced clinical researchers into the process of review of clinical applications. There was concern over the impact of the time requirements and some discussion of the use of flexible mechanisms to ensure that the broadest group of expertise could be brought to bear. It was agreed that implementation of the review-related fourth recommendation of the Director's Panel on Clinical Research is a suitable topic for the PROG. A working group will be formed, chaired by Dr. David Kupfer with other PROG members and members of the extramural clinical research community participating. Questions to be addressed will include how to quantify what would constitute appropriate reviewer qualifications and study section/scientific review group composition for the review of clinical research applications, and what flexible options for reviewer service might be employed in order to accommodate the special needs of clinical researchers.
Go back to the Table of ContentsMr. Geoff Grant, Director of the Office of Policy for Extramural Research Administration, and his staff presented an update of the activities occurring at the NIH in Electronic Research Administration (ERA), including the reengineering of the NIH public and internal working databases for tracking grant application referral, review, and award. Electronic demonstrations of modules in active service, such as Edison for invention reporting, and under development, such as the NIH "Commons", were presented. This information is presented in the "NIH Reinvention Activities: Status Report" available on the world wide web.
Go back to the Table of ContentsDr. Anthony Demsey, Acting Deputy Director of the Division of Research Grants, presented some preliminary information about study sections to examine possible diurnal trends in the scoring metric during review meetings. Data were collected from eight study sections whose orders of review were organized alphabetically or numerically by IC; no distinctive differences in scoring patterns were found for time blocks nearer the beginning or end of the meetings. Based on these preliminary data, it appears that timing does not play a role in scoring patterns. The group agreed that there was no indication that this issue needed to be pursued further.
Go back to the Table of ContentsAll applications reviewed in DRG were examined in terms of scoring patterns. When examined by Initial Review Group (IRG) as well as overall, it appears that there is an increased probability of scores averaging whole numbers (e.g. 150 or 160 rather than 147,151, or 159.) These data could serve as the background for examining some scoring issues, such as what happens if reviewers are limited to five or seven choices rather than having a scale in tenths of a point. How are they using the choices they already have? What effect on scoring might a truncated scale have? It was suggested that the group might request information on whether the discussion at the meeting alters what reviewers initially posited as scores, on how many reviewers actually read the applications, what parts of the applications appear most relevant to the reviewers in assigning scores. It was agreed that these are provocative questions that the group may want to discuss at future meetings.
Go back to the Table of ContentsDr. Steve Hyman, Director of the National Institute of Mental Health, gave a brief update on the integration of the review of neuroscience applications. He emphasized the cooperation and collegiality with which this activity is being implemented across Institutes both at the Directors' and staff levels. He indicated that staff realize that this reorganization will allow for the best assessment of science for funding, but that the referral guidelines will be crucially important. He pointed out that one topic that will need to be addressed as review moves to the DRG is whether several mechanisms are reviewed within single review groups, which is currently the practice at NIMH and the National Institute for Drug Abuse. The issue of where program projects will be reviewed needs to be addressed thoughtfully. The review of clinical neuroscience applications will also need to be addressed. Dr. Hyman acknowledged that the transition period will be one which will require careful monitoring, but concluded that the results of this major undertaking should result in benefits to all, including the research investigators, since "in the end, funding is all about high standards." There was some discussion of the intention to reorganize the review of behavioral science applications next.
Go back to the Table of ContentsDr. Ellie Ehrenfeld, the new Director of the Division of Research Grants, spoke of the new directions she sees for DRG. She put these issues into six broad categories: (1) issues around the reorganization of study sections, such as entitlements, mapping of science to study sections, and keeping up with emerging areas of science; (2) reviewer issues, such as their qualifications, terms of service, and roles; (3) investigators' concerns in specific scientific areas, such as clinical research, behavioral research, or emerging areas; (4) review process issues, such as speed, efficiency, and consistency across study sections; (5) communication issues between DRG and the ICs, and (6) staff issues, such as Scientific Review Administrator training and recruitment. All of these issues have both long and short term aspects, and DRG has already begun to address some, of these, one being the reorganization of study sections, being addressed through the Integration of Neuroscience working group. PROG members acknowledged the importance of these issues, and Dr. Baldwin commented on the opportunity for partnership between DRG and PROG, since many of these issues are relevant to review across all of NIH. She asked that PROG members think about which of these issues they might want to discuss in the future. Dr. Ehrenfeld stated that she welcomed PROG members' thoughts on all of these issues.
Go back to the Table of ContentsDr. Baldwin reminded members that part of the charge of the PROG is to address location of review (DRG vs. IC) and how much this should be influenced by the mechanisms involved and how much by the science. One aspect of this question is whether multiple mechanisms can or should be reviewed within a single study section. Dr. William Grace, an SRA from NIDA, discussed how NIDA reviews multiple mechanisms within single study sections. The review is based on the assumptions that each application should have the best scientific expertise applied to it, and that reviewers can learn new material and be flexible in applying what they learn. He pointed out that often the structure of the application itself will cue a reviewer, such as a fellowship application which contains a training plan and information on mentors. Meetings are also structured to facilitate the process, by grouping like mechanisms for review. Instructional information is provided with the applications, and the SRA orients reviewers for each different mechanism at the meeting. Dr. Grace presented information indicating that score distributions for the various mechanisms were similar, and added that some reviewers held the opinion that combining mechanisms in one review meeting enhanced the review. Typically a study section might review as many as six different mechanisms (R01, R03, R29, K02, F31, and F32.) When asked how he knew if the process works well, Dr. Grace replied that he could only answer subjectively, but it is his impression that reviewers give equal attention to each mechanism.
Dr. Jay Pettegrew, Department of Psychiatry and Neurology, University of Pittsburgh Medical Center, and chair of an NIMH study section, also presented views on review of multiple mechanisms within a single study section. He views the study section as being able to be inclusive, integrative, and not elitist, and finds having the study section broadly defined and covering a broad spectrum of applications very useful. He added that having laboratory scientists and clinicians together in one room is beneficial, as it enhances understanding and communication among them, which is good for science. He sees the review of various mechanisms not only possible but desirable. He also commented that having a mix of senior and more junior scientists together on a panel allows opportunities for mentoring by the more experienced reviewers. He added that the SRA sets the tone for the study section, and that the chair and SRA must work well together.
Dr. Marcia Litwak, an SRA from DRG, also manages a study section which reviews multiple mechanisms. Dr. Litwak described the process by which her study section, which was reviewing only R01 and R29 applications, gradually added mechanisms, one mechanism at a time, over a period of years. She pointed out the importance of preparing the reviewers in advance, educating the chair who then worked closely with her, and orienting the group as they reviewed. She presented similar scoring distributions for the various mechanisms, and stated that while there is a learning process in moving from reviewing only research project grants (R01, R29) to reviewing multiple mechanisms, the transition can be made relatively easily.
There was some discussion among the PROG members regarding pros and cons of mixing the review of various mechanisms, such as its contribution to breadth and flexibility within a group, whether study sections can change from a strict research project grant mindset, and whether the value of reviewing multiple mechanisms within one group is worth the effort of initiating behavior change in those who do not currently use this practice. Dr. Schachman pointed out the value of attaining a critical mass of applications in a review group. For example, if there are multiple review groups with very similar science, it might be preferable for all the related fellowships to be assigned to one review group each round. It was agreed that the solution need not be an all-or-nothing one, and that some scientific review groups might use either strategy. Further discussion will be around what the relative value of the practice is, and for whom (program staff, applicants, reviewers); whether the practice be incorporated, and in what situations; and what the limits might be.
Go back to the Table of ContentsThe group agreed that the main issue that the PROG will address next is the implementation of the Director's Panel on Clinical Research recommendation on the review of clinical applications. They agreed that the working group should proceed as soon as they could reasonably do so and not be necessarily constrained by meeting dates of the PROG.
Another major issue is the integration/reorganization of the review of neuroscience applications; by May the Neuroscience Integration will have held its major meeting of extramural scientists to obtain ideas for the design of new study sections and will be reporting that information to the PROG.
The issue of review of multiple mechanisms in study sections will be addressed. The members have heard information from scientific review groups which currently use this practice, and were charged to think about the parameters of a continuum of use for this practice. Other issues that they might address at future meetings also include reviewer and SRA education, evaluation, and mentoring, and those aspects of Dr. Baldwin's six step plan on creativity that relate to trans-NIH peer review.
Addition to Previous Minutes: There was one correction/addition to the minutes from the November, 1996, meeting: In the discussion of clinical research at that meeting, Dr. Norman Anderson commented that not all behavioral research is clinical research, and that it is important to distinguish between clinical and non-clinical behavioral research.
Go back to the Table of ContentsSeveral topics were discussed at this meeting. Dr. Baldwin noted that we are within the comment period for revisions to the PHS 398 grant application form, but emphasized the importance of respondent burden with any changes. The group did not opt to take up this topic for further action. Dr. Baldwin presented a brief summary of the SBIR meeting, noting that the report is available on the WWW at htpp://grants.nih.gov/grants/funding/sbir.htm, and Dr. Ehrenfeld noted that she would welcome future PROG input on the review of SBIR applications.
A discussion of the Rating of Grant Applications focused on whether or not to adopt creativity as a separate criterion or incorporate it throughout the three already recommended for adoption. Dr. Lenfant shared the report of his subcommittee on program staff views; while the importance of funding creative work is acknowledged, the group indicated that since creativity and innovation permeate all of the other three areas (Impact, Feasibility, and Investigator/Environment), the recommendation was not to have a separate fourth criterion. Dr. Baldwin presented a six step plan for addressing the issue of creativity: these included communicating clearly to the research community that NIH values creativity, encouraging review groups to identify creative applications, ensuring that review committees include reviewers who will recognize and value creative applications, including language in the review guidelines to indicate the importance of creativity, ensuring that creative or novel projects are included in the Institutes' research portfolios, and publicizing success in soliciting and funding creative research. The group generally endorsed this approach. There was additional discussion on the value of a separate fourth criterion, and a vote was taken on whether to recommend three or four criteria. Although some members dissented and some abstained, there was a plurality that voted to recommend three criteria: Significance, Feasibility (renamed Approach), and Investigator/Environment. This information along with alternate proposals will be presented to Dr. Varmus.
Other activities included a brief discussion of the review of clinical research, with specific attention to the fourth recommendation in the report of the NIH Director's Panel on Clinical Research, which addresses the review of clinical applications. It was agreed that a working group should be formed to address possible ways to implement that recommendation, and Dr. David Kupfer agreed to chair the working group. Dr. Norman Anderson suggested that a bullet be added to the definition of clinical research in the Panel on Clinical Research report, to specify the types of behavioral studies that should and should not be categorized as clinical (e.g. fundamental behavioral processes such as learning, memory, language, and social loss in healthy individuals).
An informational update on NIH's Electronic Reinvention Administration was presented by Mr. Geoff Grant and his staff. Dr. Steve Hyman, Director of the National Institute of Mental Health (NIMH), gave an update on the Integration of Neuroscience Review, indicating that this initiative is proceeding well, and indicating that the transition period for implementation will require careful monitoring but that the results of this reorganization process should ultimately be of benefit to both investigators and NIH staff. Dr. Ellie Ehrenfeld, Director of the Division of Research Grants (DRG), presented her views on new directions for the DRG, including major issues around study section organization, makeup and recruitment; reviewer and Scientific Review Administrator (SRA) selection, education, and roles; and other important review issues. She expressed enthusiasm for having PROG members share their thoughts with her on many of these issues.
Data were presented on some potential issues which PROG might consider in the future. Dr. Demsey presented some preliminary data on constancy of the scoring method across the meeting time of study sections. As no distinctive differences in scoring patterns were noted for time blocks nearer the beginning or end of the meetings, the group agreed that there appeared to be no compelling reason to pursue this issue further. Dr. Baldwin presented data on scoring patterns across various Initial Review Groups in the DRG which indicated that there appears to be an accelerated probability in some IRGs of reviewers assigning whole number scores; she suggested that these data could serve as the background for examining some scoring issues, such as number of points needed on the rating scale and the effect a truncated scale might have. Experiences reviewing multiple mechanisms within single study sections were shared by Dr. William Grace, SRA at the National Institute on Drug Abuse, Dr. Jay Pettegrew, chair of an NIMH scientific review group, and Dr. Marcia Litwak, SRA at DRG. All were positive about having the review include various mechanisms, and Dr. Pettegrew offered the opinion that it enhances the review. All agreed that the SRA and chair play crucial roles and must be in synchrony for this to succeed.
Agenda items for the next meeting will include review of clinical applications, results of the Neuroscience Integration meeting of extramural scientists taking place in March, and further discussion of the review of multiple mechanisms within single scientific review groups. In addition, at the next or some future meeting the group may discuss reviewer and SRA education, evaluation and mentoring, and peer-review related aspects of Dr. Baldwin's six step plan on creativity.
I hereby certify that, to the best of my knowledge, the foregoing minutes are accurate and complete.
Peggy McCardle, Ph.D.
MPH, Executive Secretary Peer Review Oversight Group
Wendy Baldwin, Ph.D.
Deputy Director for Extramural Research
Go back to the Table of Contents
| |||
| |||
| |||
| |||
| |||
|
| [OER Home | NIH Home | DHHS | FirstGov] |
| [Accessibility | Privacy Notice | Disclaimer | Contact Us] |
| [Site Search | Site Map | Document Index | Help Downloading Files] |
| Web Posting: 3/28/2002 |
| Webmaster |