Minutes of the Meeting of the Peer Review Oversight Group (PROG)
January 5-6, 1999
Conference Room F, Natcher Building, NIH, Bethesda, MD

Welcome and Introduction

Dr. Baldwin welcomed the group. She noted that a major topic of discussion for this meeting would be the important initiative to consider the use of consumer representatives in the peer review of some NIH grant applications., indicating the goal of finalizing the guidance document that the working group has drafted on this topic by the end of the current meeting. She noted that Dr. Varmus has had an initial meeting to obtain advice on the establishment of the Director's Council of Public Representatives (COPR), for which nominations are now being received. The first official meeting of the COPR will occur in the spring (http://www.nih.gov/news/information/summary.html.)

Addressing the recent increase to the NIH budget, Dr. Baldwin indicated her feeling that the NIH will have no trouble absorbing the increase and spending it wisely. One major effect of the budget increase is a "healing" - for example, the NIH has been able to increase the training stipend by 25%, and is trying to step away from making budget reductions to actually making awards without cuts. But the budget increase will raise certain challenges for peer review. One of these challenges is that new mechanisms are being developed; the training of scientific review administrators and their orientation of reviewers to these new mechanisms is crucial to their success. In addition, there are some new issues in review, such as how research will be done in the post-genomic era, where quantities of data as well as types of data to be dealt with will be of a new magnitude. As Dr. Varmus noted in his address to the Association for the Advancement of Science on February 13, 1998, (www.nih.gov/welcome/director/021398.htm) review groups will be addressing types of applications and skills sets we have not seen before. There are two areas that have implications for peer review and that the PROG has been discussing and will continue to discuss: clinical research, for which there are now new mechanisms for which applications are being received, and bioengineering, for which two program announcements are now published in The NIH Guide to Grants and Contracts. These are examples of opportunities for the NIH portfolio to support new research.

Dr. Baldwin also noted that another new initiative, the modular grants approach to grant application budgets, offers a chance to step away from the detail in applications and move toward the assistance role in making research awards. It does not change the fiscal responsibility of institutions in managing their awards; it does allow the NIH, especially in the review process, to focus more on the science and less on details of budget. Information on the modular grants approach can be found on the Word Wide Web at grants.nih.gov/grants/funding/modular/modular.htm).

Another change with an impact on peer review that Dr. Baldwin discussed was the recent confirmation from the Small Business Administration that the guidelines for time and amount of Small Business Innovation Research grants are recommendations, not rules, and therefore have some flexibility. While this greater flexibility is unlikely to change the majority of applications, it will allow those projects requiring greater time or funds to request what they need to complete the project, with appropriate justification.

Review of Bioengineering Research Grant Applications

Dr. Baldwin explained to the group the function of the NIH Bioengineering Consortium (BECON), which she has chaired for the past two years; the group's mission is to provide a central focus at the NIH for bioengineering and related research. Major accomplishments to date have included a national symposium that helped provide direction for bioengineering at the NIH (http://www.nibib.nih.gov/becon/report_19980228.pdf) (PDF) and two program announcements to encourage research and research collaboration in the area (http://grants.nih.gov/grants/guide/pa-files/PAR-99-009.html). One issue that BECON seeks to address is the review of grant applications in bioengineering and bioengineering-related research. This issue is currently being addressed by a panel of extramural scientists convened by the Center for Scientific Review (CSR).

Ms. Linda Engel of CSR described the panel and gave an overview of their tentative recommendations. She presented the panel roster (which can be found on the CSR Web pages at www.csr.nih.gov/committees/roster.asp?CID=101236) and the charge of the group, to make recommendations that will enable peer review to keep pace with the changing ways that scientific research is accomplished, with more and broader collaborations, greater interdisciplinarity, and a focus on the creation, adaptation and increasing use of advanced technology. Ms. Engel noted that this group complements but will not overlap with BECON, and that their recommendations will be provided not only to Dr. Ehrenfeld for consideration at the CSR but also to the CSR Panel on Boundaries for Scientific Review (www.drg.nih.gov/events/broster.htm). The panel's recommendations included provision of suitable venues for the review of bioengineering and related research grant applications within both a biological and methodological context, which might include augmenting expertise on existing review panels or establishing Special Emphasis Panels for the applications; development of a program - review partnership among NIH staff with regard to these applications; and inclusion of the extramural scientific community's input in shaping the review process for these applications. Ms. Engel indicated that the group would be refining their recommendations at their March meeting. There was some discussion of the great breadth of science that is covered by the term "bioengineering," which includes many diverse groups. It was noted that while greater representation in review panels of members of these scientific areas has been recommended, there has been no assessment of the extent to which such groups are currently represented.

Consumer Representation in Peer Review

Dr. Baldwin noted that the discussion of this topic began at the June 1998 PROG meeting, when the group heard the experiences of two Institutes that had included consumer representatives on selected peer review panels. Prior to that meeting, Dr. Baldwin had also met with Colonel Irene Rich, then director of the Department of Defense (DOD) Congressionally Directed Medical Research Programs, about their experiences. As a result of that meeting, Dr. Baldwin invited her to address the PROG about the DOD's experiences in building a comprehensive program for inclusion of consumers in their peer review process. Dr. Baldwin stated that the PROG's consideration of this topic and development of guidelines for the inclusion of consumers in peer review is in no way a dictum to Institutes and Centers to include consumers. Rather, since additional ICs are considering this step at present, this is an effort to assist them in the significant investment that this will involve, offering some guiding principles.

Colonel Irene Rich is now Director Emeritus of the DOD Congressionally Directed Medical Research Programs (CDMRP), and Lieutenant Colonel Yvonne Andejeski is Director, Consumer Work Group, CDMRP; both were instrumental in the development of the programs' current inclusion of consumer representatives on peer review panels. Drs. Rich and Andejeski indicated that they had modeled their peer review program on that of the NIH, with a two-tiered review, based on recommendations from a 1993 Institute of Medicine panel. That panel had also recommended that they seek consumer involvement in their funding decision-making process. While they included in their initial planning the idea of including consumer representation in both levels of review, they phased in consumer participation in the second year of their research program. They noted that both the careful planning that went on during that time and the evaluations performed at each stage of their phasing in of consumers in peer review were crucially important in the success of this initiative. They established a consumer working group whose charge was to define goals and objectives, define roles, develop recruitment and selection strategies, establish timelines, and design orientation strategies and an ongoing evaluation process. Their overall goal was to provide on opportunity for consumers and scientists to work together on peer review panels, and they did this by ensuring a fair and impartial process, broad representation of consumers, clear role definitions, education and orientation for all participants about those roles, and ongoing evaluation of participation.

The CDMRR consumer role is to serve as a reader on the research proposals being reviewed, to prepare written comments on the value and importance of these proposals based on evaluation questions provided, to participate in the review discussion and scoring of proposals, and to represent the experiences and interests of a large cadre of consumers rather than speaking only from their own experience. Two consumers are included in each panel, with one being experienced in review and serving as a mentor for the other, a new consumer reviewer; each is assigned approximately 15 proposals. They are asked to focus their attention on the relevance of the research, and a lay abstract of the research is provided (written by the applicant as part of the application) to facilitate their reviews.

Drs. Rich and Andejeski provided details of the recruitment, selection, and orientation of consumer reviewers, orientation of the panels on which consumers serve, and some thoughts on their evaluation of the process to date. They concluded that consumer involvement in peer review is valuable in that it adds a perspective which scientists might not otherwise have considered, thereby ensuring that human dimensions of disease are incorporated into scientific considerations. Over the four years that they have had their program of consumer-reviewer inclusion in place, they have more clearly defined roles and expectations, refined orientation materials to include instructions on how to read scientific proposals, and developed information resources, including a direct point of contact for consumer reviewers throughout the process, an education packet with a glossary of scientific terms, and a mentoring system of experienced advocates. They offered to share their compiled materials with Institutes and Centers considering the use of consumers in the peer review of research grant applications.

In response to PROG members questions, Drs. Rich and Andejeski observed that generally the consumer representatives were well-received by the scientists on the review panels and that their scoring of applications did not differ substantially from that of other reviewers. Their 1995-96 evaluation of the program led them to discontinue the lengthy telephone interviews that had previously been used to screen consumer reviewer nominees as part of their selection process, but they still make telephone contact for orientation purposes. They script the review administrators' remarks regarding the confidentiality of the application information and the discussion within the review meeting to ensure standard orientation for all reviewers in all groups, and include in their orientation materials some vignettes of what is not acceptable behavior. Consumer reviewers are encouraged to bring any scientific questions to the review administrators.

During the discussion, Drs. Rich and Andejeski reinforced that while the core principles of their approach are generalizable to the review of applications focused on various topics and diseases, the definition of consumer must be tailored for each program. They emphasized that consumers do not replace scientists on the panels but augment the expertise on the committees. In response to questions about the mentoring program, Dr. Andejeski indicated that, by design, half of the consumer reviewers in the program are new for each meeting and half are experienced mentors. She said that the use of mentors appreciably reduced the level of effort required by staff in orienting new consumer reviewers. She added that many consumer reviewers wished to participate more than once or to mentor, and that about 40 percent get involved in other research-related activities, such as service on Institutional Review Boards, after serving on a peer review panel.

A discussion of the draft document developed by the PROG working group on consumers in peer review followed. Dr. Baldwin again stressed that the purpose of this document was to provide internal best practice guidelines for those ICs that chose to include consumers in the peer review process for specific research initiatives, but that it does not represent a mandate for such inclusion.

Best Practices in Orientation and Training of Reviewers, Chairs and Scientific Review Administrators

During the past two PROG meeetings, members visited scientific review group (SRG) meetings. The outcome of those visits was an increased awareness of the variability that exists among SRGs, and an affirmation of the importance of orientation and training for reviewers, chairs and Scientific Review Administrators (SRAs) to ensure the quality of the scientific peer review process. Dr. Baldwin noted that the IC review branches face different issues and challenges from CSR, but that there are practices that are common to all of peer review at the NIH which could be shared. Therefore, the experiences of the CSR and the review branches of three ICs were shared with members.

Dr. Ehrenfeld, Director of CSR, drew an analogy with the Center's flexible Initial Review Group organizational structure (which clusters SRGs by scientific content areas) and university academic departments, where researchers function independently but the department has authority and responsibility to develop curricula. CSR is establishing IRG oversight and advisory groups from the external scientific community; these groups will advise CSR about scientific content and review group operation. The IRG concept is being used to meet certain needs for discussion and communication among SRAs, for opportunities to convert best practices to standard operating procedures, and for monitoring and oversight. Resources must be allocated to allow for professional performance and development.

In the area of SRA training, Dr. Ehrenfeld shared notes from a full-day course for new SRAs set up by the CSR SRA training committee. This course covered many of the tasks for which SRAs are responsible and which are required for a competent review: referral of applications within and between IRGs and individual assignments to reviewers, conflicts of interest, reviewer recruitment, administrative review of applications, pre- and post-meeting communication, and review meeting operation. The committee also plans additional minicourses, such as the review of Small Business Innovation Research grant applications, the roles of program and grants management staff, and streamlining. They are also considering the development of uniform packets of information for reviewers, to provide some uniformity and to promote the development of best practices models.

Related activities include the establishment of a CSR Advisory Committee's working group to develop best practices guidelines for review group chairs, the development of a structured feedback process for IRG chiefs to use in mentoring SRAs, and some novel work practices for SRAs. For example, SRAs may now work at home one day per week during summary statement preparation, and they may now apply for a "sabbatical" in which they may serve in some other capacity within the NIH for one review cycle every three or four years. Dr. Ehrenfeld pointed out that all of these efforts are in the initial stages.

In response to questions, Dr. Ehrenfeld said that there are about 22 IRGs, with from 5-8 review groups in each; some are small but the structures are being reorganized. SRA recruiting procedures and criteria have been changed recently, but recruitment of SRG chairs has not been changed. She added that chair recruitment will probably undergo changes as standard practices are defined. Members affirmed the importance of thorough, uniform orientation for review committee chairs, and there was some discussion of the qualities desirable in SRG chairs, terms of service for reviewers and chairs, and whether it might be advantageous to orient chairs in groups or to have them meet occasionally.

Dr. Teri Levitin and Dr. William C. Grace of NIDA presented an overview of their training for reviewers. Because not all scenarios can be predicted, NIDA uses a systematic, continuous training approach that emphasizes the basic principles and values underlying peer review. The review staff provides all reviewers with uniform materials and guidelines, including templates to ensure that all areas are addressed in writing critiques. The review staff is seeking to make materials more uniform, more available, and available more quickly; they intend to meet all three goals by use of a web site, on which they also hope to convey their excitement and enthusiasm about peer review and its importance. The web site will be organized using a management philosophy where SRAs are managers who give the big picture and major resources, with details available on the web. Materials for review are temporally organized into pre-, during and post- review meeting segments, with the underlying philosophy of why certain demands are made of reviewers. The web site is intended to supplement current training by SRAs.

PROG members were enthusiastic not only about the use of technology which would enhance efficiency and promote uniform procedures, but also about the idea of maintaining interest and enthusiasm about peer review. They applauded the emphasis on the value of peer review service.

Practices at the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK)

Dr. Ann Hagan of NIDDK presented information about that IC's training of SRAs, which she emphasized differs from CSR's in that it is more individualized, based on the smaller number of individuals entering the branch at any given time. Training is viewed as a long-term phased process over perhaps six months, during which the new SRA is mentored by a more senior, experienced SRA. Informal luncheons are held periodically to facilitate the sharing of experiences and ideas among SRAs, and the new SRA takes advantage of trans-NIH training, such as the Fundamentals of Extramural Research, a general orientation course offered for newly hired NIH staff. Interaction with not only review staff but program and grants management staff within the Institute is stressed. Sample documents and templates for standard documents are provided on the local area network to facilitate knowledge of policy, procedures, and document preparation.

Dr. Hortencia Hornbeak described the training and orientation of SRAs, reviewers and review committee chairs with a specific focus on Special Emphasis Panels (SEPs) at NIAID. While the Institute has three standing review committees, they also conduct a large number of (approximately 50) SEPs annually. Because these panels are constituted anew for each review, reviewer and chair orientation require special attention.

Review committee chairs are selected for their broad-based scientific knowledge, objectivity and fair-mindedness, and skills in group dynamics. Chairs for chartered committees are selected in a manner similar to selection in other ICs. Each fall, NIAID staff updates the committee on new policies and procedures, and program staff members highlight new initiatives or changes in their respective programs. Orientation for chairs of SEPs begins at the time of invitation and is tailored to the experience of the chair and the type of applications to be reviewed. The SRA usually meets with the chair the evening prior to each meeting to discuss any additional issues or concerns. The decision to have a premeeting orientation was based on input form reviewers who participated in prior SEP reviews for complex initiatives.

In addition to the basic training that all NIH SRAs receive, those at the NIAID also receive specialized training by senior Institute staff, with specific attention to the guiding principles that govern policies and procedures. One unique resource for NIAID SRAs is a system of centralized electronic files which can be used to produce tailored reviewer manuals for each review; thus uniform materials can be individualized for the needs of any particular reviewer or group of reviewers and for any specific type of review meeting. For all SRAs there are bimonthly policy meetings, at which not only policies and procedures are discussed, but also the rationale and an implementation plan for each policy, to ensure uniformity among SRAs. In addition, brown bag luncheons are organized in which SRAs share experiences. There are even awards for the worst or most trying experiences. Planning forms are used to ensure coordination of activities, uniformity of approach, and completeness.

The NIAID has implemented electronically assisted reviews, using secure websites. Internet-assisted review requires not only specialized training for SRAs but also computer-literate reviewers. Dr. Hornbeak reports that while both SRAs and reviewers are still in transition with this process, reviewers are requiring less assistance and there are fewer who do not have computer access. Most reviewers have reportedly had a positive response to these internet-assisted reviews. All innovative pilots and procedures are continuously monitored and evaluated, and NIAID has held occasional focus groups to obtain feedback on increasing efficiency.

Integration Update: Review of Behavioral and Social Sciences Applications

Dr. Virginia Cain handed out descriptions of the newly formed behavioral and social sciences review panels that resulted from the integration of review of the former alcohol, drug abuse and mental health institutes. The review group descriptions were posted on the web in September and have been refined based on community input from more than 200 organizations and individuals. They were then "tested" on a group of application abstracts, and the revised descriptions were posted on the CSR website in November, 1998. Implementation is just beginning, and the goal is to have these review groups in place for review of applications submitted for the February 1, 1999, receipt date.

FOIA Access to Grant Data

Dr. Baldwin informed members of a recent congressional action with implications for biomedical research. In the Omnibus 1999 Appropriations bill, Congress directed the Office of Management and Budget (OMB) to amend OMB Circular A-110 (Grants and Agreements with Institutions of Higher Education, Hospitals, and Other Non-Profit Organizations; (<http://www.whitehouse.gov/WH/EOR/ORM/html/circulars >)) to extend the Freedom of Information Act (FOIA) to "require Federal awarding agencies to ensure that all data produced under an award will be made available to the public under the [FOIA]. …If the agency obtaining the data does so solely at the request of a private party, the agency may authorize a reasonable user fee equaling the incremental cost of obtaining the data" (Statutory Language from HR 4328, Title III: PL 105-277 10/21/98, which can be accessed at the Library of Congress Website at <http://thomas.loc.gov>). She indicated that while the NIH endorses strategies that support the sharing of research data, the present language does not address several important concepts that must be clarified to ensure that data sharing could be undertaken while still supporting the integrity of data, the confidentiality of participants, the investments of the researchers and the continuance of important research.

Since this legislation extends FOIA, an existing statute, it is useful to recall that that act provides for exemptions, for example for proprietary data or data whose release would be an invasion of privacy. However, FOIA applies to records that are in the possession of the Federal government, whereas the present language applies to data in the possession of the grantees. Dr. Baldwin outlined the issues that will need to be addressed. The law provides no guidance about concepts such as the definition of data, the timing of release of data, and protections necessary for clinical studies. Also, while the legislation states that there can be compensation for the expense of responding, there is no explanation of how this would be done and FOIA does not provide examples that might serve as models. She encouraged members to watch for the Notice of Proposed Rule Making, and to take advantage of the 60-day comment period to make their concerns known. (See Implications of Notice of Proposed Rule Making, OMB Circular A-110 at http://grants.nih.gov/grants/news.htm, which contains a link to the NPRM.)

Report on CSR Panel on Scientific Boundaries

Dr. Keith Yamamoto, who serves as CSR liaison to the PROG, gave an update on the activities of the Panel on Boundaries for Scientific Review. He explained that the panel is examining de novo the scientific organization of scientific review groups (SRGs or study sections) and Initial Review Groups (IRGs). Their goals are to define organizing principles for IRGs and SRGs that can provide fair and rigorous review for current biomedical research, accommodate changes in science, and recognize and capture newly emerging fields. Some examples of organizing principles include possibly organizing IRGs by an organ/disease system and a trans-system orientation, where applications in the same scientific area might be assigned to either orientation. This would provide more than one review venue for each application; this is an explicit effort to have sufficient overlap to allow projects to move from one orientation to another, plus providing an alternative for the review of SRG members' applications. One principle would be to re-examine the assignment of each competing continuation application rather than automatically reassigning it to the SRG that reviewed the original application. In addition, they will define operating principles for SRGs that could establish a "culture" that promotes the highest quality of review and to help recapture the joy of serving on a review group. Some examples of areas in which they will develop operating principles are roles and responsibilities of SRAs, reviewers and review group chairs, expectations of applicants, content and length of critiques, and involvement of senior scientists in review. They hope to complete their work by June 1999. Dr. Yamamoto predicted that there will be substantial changes recommended in the way IRGs are organized.

Members applauded the broad, cross-cutting approach and the vertical-horizontal matrix of organ/disease vs. trans-system organization. They discussed the parallel between internal changes at the NIH such as those which may be proposed by this panel and changes in the culture of extramural scientists, and how that might affect the preparation of the next generation of scientists. It was agreed that mentoring requires significant attention in the scientific community, and that work needs to be done in both the NIH and the extramural arenas. In response to questions from members, Dr. Ehrenfeld emphasized the preliminary nature of the information about the panel's activities and indicated that CSR will solicit extensive input from the scientific community in a manner similar to that used for the integration activities.

Use of the Innovation Criterion

Dr. Baldwin gave the result of a preliminary snapshot of how the new review criteria are being used in review groups. She cautioned that it is too early to really assess how well the use of the new explicit statements of criteria are working, since these went into effect in May 1998 and thus have been in use less than a year. However, a brief examination of how well the behavior change has taken effect indicated that reviewers are organizing their critiques using these new criteria headings.

Sixty-nine summary statements were randomly pulled (from more than 100, after excluding special mechanisms.) These 69 summary statements came from 64 different review groups and 18 different ICs. There were 132 R01s and twelve each of R03s, R21s and R29s. Roughly half were from each of the two councils rounds that have been completed since this change took effect - i.e., 86 from May '98 and 82 from October '98. The summary statements contained a total of 168 critiques that were examined for inclusion of explicit statements addressing the criteria, set off by specific labels for those criteria. Only 11 critiques (7%) did not use the labels at all, so that general compliance was approximately 93%. An examination of individual criteria showed that 97% of reviewers addressed significance and investigator, 96% addressed approach, 92% addressed environment and 91% addressed innovation. Thus, reviewers are generally doing what they have been asked to do. While this was not a systematic assessment of the success of the use of the new criteria, it does indicate that reviewers are complying with the mechanical task.

The staff members who examined the summary statements reported that references to innovation often occur in other sections of the review. However, it does seem easier to read the summary statements when they are formatted and organized by criteria.

If a reviewer is generally negative about an application, that is usually indicated in various places, under many of the criteria and in the overall evaluation, which many reviewers add as a separate section. Similarly, if they find an application creative, unique, and innovative, they will address that in more than one place. If the approach is particularly novel, they will say that in both the approach section and the innovation section - but the critiques did not seem redundant as a result. Dr. Cassman agreed that the summary statements are easier for staff to read, and that having reviewers address the specific categories seems to have value.

Issues for Future Meetings

Several issues were identified for discussion at the next meeting: possible decrease in page limits and overall length of applications, as an effort to both reduce reviewer work load and assist applicants in focusing on the overall purpose of the project rather than methodologic detail; the various approaches to shortening time from receipt to award -- models, costs, and barriers; guidelines for the selection process for review committee chairs, plus role, orientation and training; data sharing, NIH policy and legislative requirements; and possibly an update on data sharing in the human genome project.

Note: If you need help accessing any files above that are labeled as PDF, RTF, MS Word, Excel, PowerPoint or RealPlayer, see Instructions For Downloading Documents and Electronic Forms.