July 10-11, 2000

Cloister (Bldg. 60), The Chapel, NIH, Bethesda, MD

Welcome and Introduction

Dr. Wendy Baldwin welcomed members of PROG and its subcommittee, the Regulatory Burden Working Group, to the first joint meeting of the two groups. Dr. Baldwin explained the interrelationship between the two groups and their complementary functions. Dr. Baldwin also welcomed the visitors in attendance, including the members of the science press.

Since the members of the two groups were meeting together for the first time, Dr. Baldwin asked the members to introduce themselves. Dr. Baldwin then acknowledged that the meeting's agenda was a full one, and indicated that it was designed to provide a maximum number of items that were of interest to both groups.

The NIMH Experience with Lay Reviewers

Dr. Jean Noronha described the recent experience that the NIMH Review Branch has had with the use of lay (public) members in the peer review process. Their experience has been primarily in the review of intervention research. Initially, NIMH has recruited a cohort of 15 such individuals. To date, recruitment has been by telephone, without in-person interviews. Members are recruited on an ad hoc basis, with no term of appointment. Three public participants have been used on each review panel and a public participant has been assigned to each application. The critiques of lay members are sent to applicants as part of the summary statement. In future recruitments, the most proficient lay members will be used to mentor new public participants.

Training in the Center for Scientific Review (CSR)

Dr. Camilla Day presented the status of training for Scientific Review Administrators (SRAs) in CSR. She noted that there has been extensive hiring recently, with more than two dozen SRAs recruited per year over the past two years. An internal training committee has been formed to maintain uniformity, and their functions are complementary to the training that Integrated Review Group (IRG) Chiefs provide to their new recruitments. During the discussion, concern was raised by group members about the quality of the Resume and Summary of Discussion of summary statements. It was suggested that additional/reinforced training should be provided to SRAs, and Dr. Day agreed that this might be a good topic for a "refresher" course. Another member noted that while the Resume was important, the best information he has received has come routinely from institute program staff who have attended the study section meeting. Dr. Day noted that IRG Chiefs attend study section meetings routinely to monitor their SRAs' performance, but that they cannot attend every SRA's meeting each round. Another observation was that the study section chairperson is often underutilized; it was suggested that they should be less active in the review of applications, and more active in the management of the process. Also discussed was the process for making assignments of applications to the individual reviewers. Dr. Day concluded her remarks by mentioning that continuing education is provided to SRAs by voluntary case study workshops.

Preliminary Data in Grant Applications

Dr. Anthony Demsey presented the findings of a small working group that was assembled to look at the issue of how preliminary data may be viewed as being an essential part of the application, both by applicants and by reviewers. The working group found that nearly all applications included preliminary data, and nearly all summary statements commented on preliminary data, generally as an indication of the likelihood of success for the proposed project. The working group recommended that the instructions to applicants should be more explicit in emphasizing the importance of preliminary data for investigator-initiated R01 applications, and that the NIH should perhaps make greater use of the R21 mechanism to encourage "risk-taking" applications with little preliminary data.

Group members acknowledged that RFAs have been a useful way to encourage applicants and reviewers to concentrate less on preliminary data. It was suggested that it would perhaps be interesting to focus specifically on the experience of applications from new investigators, and on how various types of research (e.g., clinical) may differ in the presentation of and expectation for preliminary data. One member noted that certain non-Federal grants programs focus on short applications, with little preliminary data, and with short turn-around to funding. Dr. Baldwin indicated that perhaps the information to applicants in the PHS 398 application form should be recast, to emphasize the inclusion of the applicant's unpublished data. One member noted that study sections tended to have an ethos - some "require" preliminary data, others pay little attention to it. Additional discussion related to how many reviews mentioned the presence of no preliminary data as a negative, and to whether some reviewers always expect preliminary data.

The consensus was that the current system is sound, but may require a little fine-tuning. Suggestions included changing criteria headings, shortening the application, and structuring the 398 form to correspond with how science is currently presented and conducted. It was reiterated that the sections in the reviewers' critiques on experimental design and methodology should be shortened, and that reviewers should be asked to provide more details on the critique of the investigator's line of thinking and hypotheses.

Update on Modular Grant Applications

Dr. Ronald Geller described the NIH experience with modular applications after one year of implementation. He indicated that overall, there has been no trend in escalation of requested dollars, and study sections report spending less time on review of budgets. Dr. Geller indicated that several clarifications will be announced in the NIH Guide for Grants and Contracts shortly, for implementation effective September 1. Applicants will be asked to list and describe the roles and responsibilities of all personnel, and applications that are not submitted in modular form will not be reviewed unless revised to the modular format. It was noted that raising the limit for use of the modular format from $250,000 per year to $500,000 would encompass approximately 95% of all research grant applications, and that consequently, most study sections would see only modular applications.

Conflict of Interest

Mr. Gary Thompson described his experience with recent "proactive compliance" or "not-for-cause" site visits to three universities, and plans for 7 more site visits over the next two months. The purpose of the site visits is to emphasize the importance of the issue and to discuss and develop best practices to compliance issues in general. Additional discussion focused on the complementarity of the Bayh-Dole Act and conflict of interest regulations, and the ethics of disclosing to patients the financial interests of clinicians involved in clinical studies and trials. The overarching theme is "objectivity in research" and several recent announcements in the Guide have been directed toward this issue and the need for appropriate training at research institutions. Dr. Baldwin noted that the issue of how to handle institutional conflicts is a new and serious one.

Structured Review Experiment

Dr. Demsey described an experiment conducted in the Division of Research Grants in the late 1980s on the use of a structured review format for reviewers' comments, subsequently leading to a more structured and focused discussion of applications at the study section meeting. He noted that the experiment as conducted was not favorably received by reviewers, SRAs or program staff. However, it was also noted that over the past decade, many of the general directions of the structured format have been incorporated into the current review process. The suggestion was made that the 398 form should be modified to instruct applicants to prepare applications specifically to address the individual review criteria. Another suggestion was to combine the structured review "checklist" format with short paragraphs on each of the criteria. It was acknowledged that the critique of the "scientific approach" is often the longest, and that it should instead be kept to no more than half a page. It was again emphasized that the Resume and Summary of Discussion is a critical part of the summary statement. It was suggested that useful information for a subsequent discussion of this issue would be other Federal agencies' experiences with forms, preliminary data, length of applications and reviews, and what their guidelines specifically elicit from applicants and reviewers. Other ideas that had been mentioned previously but were reiterated at the meeting included the possibility for applicants to provide abstracts understandable to the public, and the suggestion that page lengths could vary, depending on the mechanism and/or kind of science proposed.

Institutional Costs of Compliance

Dr. Baldwin initiated the discussion by noting that the idea for "uncapping" the 26% allowance for administration in Facilities and Administrative (F&A) costs has gained increased attention recently. However, she also noted that approval for removing the cap seems unlikely at this time. Other options were discussed, including carving out the costs of compliance activities from overall administrative costs. Less attractive was the possibility of charging compliance costs as direct costs on grants. It was noted that the current policy on the 26% cap is restricted to universities, and not applied to small businesses or not-for-profit institutions. Dr. Baldwin noted that recently the focus has been on Institutional Review Boards (IRBs), which might seem appropriate as attributable to direct costs, but that the issue is really compliance overall. It was suggested that it would be useful to have an estimate of what institutions incur in administrative costs for compliance activities.

Hazardous Waste

One of the considerations within the NIH's efforts to reduce unnecessary regulatory burdens on research institutions has been that of hazardous waste disposal. Addressing the members on this issue was Ms. Elizabeth Cotsworth, Director of the Environmental Protection Administration (EPA) Office of Solid Waste. Ms. Cotsworth acknowledged the difficulties that the Resource Conservation and Recovery Act causes the academic research community. She noted that the issue is primarily one of scale - the treatment of 55-gallon drums of waste at a chemical factory versus milliliter quantities on a laboratory bench. Ms. Cotsworth indicated that she shares the goal of making the regulations better fit the arena of academic waste generation, and at this point, the EPA is trying to determine what path is best for them to pursue in this effort. Their Project XL, in conjunction with several New England institutions, is an effort to develop some strategies at the research institution level. These universities are prepared to operate under Project XL with the upcoming academic year, and will be expected to provide data for EPA monitoring as well as for supporting future changes. This project is now one year into the four-year study. Ms. Cotsworth also mentioned the activities of the Howard Hughes Medical Institute (HHMI) in attempting to develop best management practices through pilots with universities in the ten U.S. regions. A new coordinator of these activities and interactions has been named at EPA - Ms. Gail Hansen.

In response to a question of whether EPA can defer to state/local requirements, Ms. Cotsworth indicated that while state/local requirements must be at least as stringent, EPA must still be involved. In response to a question of how the community can make its concerns better known, she noted that to some degree the community already has, as evidenced by recent VA/HUD Appropriations Committee directives to EPA. However, she noted that there are always competing demands for reform - some to be more lenient, some to be more strident - and that these need to be balanced. She also indicated that the EPA would likely want to rely to some degree on the work of HHMI and the NIH regulatory burden report. Finally, in response to a question about the seeming reluctance of EPA to endorse best practices, Ms. Cotsworth indicated that so long as regulations still exist, there would always be an inherent conservatism on the part of the environmental inspectors.

The Regulatory Burden Report - Progress and Next Steps

Dr. Demsey gave a brief synopsis of the major changes that have been implemented as a result of the recommendations from the regulatory burden report and the activities of the advisory working group. The full description of changes can be found at http://grants.nih.gov/grants/policy/regulatoryburden/update_june_2000.htm.

Dr. Nelson Garnett discussed the latest issues and upcoming challenges in the area of animal welfare, one of the major being the likelihood that USDA regulations will be extended to rats, mice and birds, possibly creating a nine-fold increase in tracking and reporting burden. Other issues brought up by the members included that of regulations on outdated drugs, and the number of reviews of animal facilities and procedures that are currently required.

Office of Human Research Protections

Dr. Baldwin introduced Dr. Greg Koski, director designee of the Office of Human Research Protections (OHRP), and outline for him the groups previous discussions on compliance. After a brief introduction, Dr. Koski opened the floor for questions. One member expressed concern regarding IRB review of full grant applications, characterizing the effort as counterproductive, increasing burden on the IRBs, and introducing a lack of clarity as to what has actually been "approved" related to the application's five years of work. Another member suggested the importance of communicating the study section's review of human subjects protocols to the IRB, if IRB consideration were awaiting the results of peer review. Both the group members and Dr. Koski agreed that this opportunity for dialog was useful.

Wrap-up and Future Issues

Dr. Baldwin expressed her satisfaction with this first joint meeting of the two groups and asked if such joint meetings in the future would be welcome. The consensus was that overlapping meetings would be useful - i.e., having the meeting start with items of interest primarily to one group, transitioning to items of interest to both, and then concluding with items of interest primarily to the other group. In this way, members of the two groups could overlap for part of the meeting, but need not be present for the entire two days if their schedules won't permit. Dr. Baldwin indicated that we would try to accomplish such a schedule for a meeting in January 2001.

Dr. Baldwin listed the following potential issues for future consideration by the group: (1) continuation of the discussion of preliminary data, with additional information from other agencies and organizations; (2) the length of applications; (3) A-110 and data sharing; (4) costs of compliance; (5) good-faith efforts on the part of Federal agencies in identifying potential burdens on the research community before regulations are imposed; (6) institutional conflict of interest; (7) hazardous waste; (8) just-in-time for IACUC approval; (9) ORI issues related to education requirements; (10) transport of biological materials; (11) the issue of multiple surgeries on animals - USDA Policy 14; (12) USDA Policy 12; and (13) the frequency of animal facility/procedures reviews. Dr. Baldwin suggested that perhaps the last four items could be addressed in preparation for the next meeting by a subcommittee of Drs. Greger, Rich, Theran and Vandeberg.

Dr. Baldwin thanked the members of the two groups for their time and efforts, and adjourned the meeting.

Executive Summary

The July 2000 meeting of PROG was held jointly with its recently established subcommittee, the Regulatory Burden Working Group. The meeting began with a welcome and charge to the group by Dr. Wendy Baldwin, then the members introduced themselves.

The first presentation was by Dr. Jean Noronha who described the recent experience that the NIMH Review Branch has had with the use of lay members on peer review panels. To date, lay members have been used in the review of intervention research, and the experience has been favorable. Dr. Camilla Day then presented the status of training for Scientific Review Administrators in the Center for Scientific Review. Much discussion surrounded the issue of the quality of the Resume and Summary of Discussion in summary statements. Following this discussion, Dr. Anthony Demsey presented the findings and recommendations of a working group that looked at the inclusion and review of preliminary data in grant applications. It was agreed to reconsider this topic at a subsequent meeting after additional information has been collected.

Dr. Ronald Geller provided an update on the NIH experience with modular applications. Contrary to some initial concerns, there has been no trend in increasing escalation of requested dollars. There is now some consideration being given to raising the requested dollar limit for the use of the modular format. Following this agenda item was a presentation by Mr. Gary Thompson on recent experiences with proactive compliance site visits to research institutions. Dr. Baldwin noted that an important issue for the future will be that of institutional conflicts related to clinical studies. The first day ended with a presentation by Dr. Demsey on the results of an experiment in structuring reviewers' written critiques and subsequent discussion at the review meeting. This experiment was conducted in the late 1980s and although it was not favorably received at the time, many of the features of the experiment have now been incorporated into the NIH peer review process.

The second day began with a discussion of the costs that institutions bear in complying with Federal regulations. This issue is under serious discussion at the Department level, and the consensus of the group was that those costs must in some fashion be considered separate from other administrative costs in the F&A costs, and thus be fully reimbursable. The next agenda item, on hazardous waste, was introduced by Dr. Demsey, who then presented Ms. Elizabeth Cotsworth of the EPA Office of Solid Waste who presented an overview of the issues from her perspective. It was acknowledged by all that EPA requirements must be adapted to apply appropriately and sensibly to the academic research environment.

Dr. Demsey then gave an update on progress in addressing recommendations in the NIH regulatory burden report. Dr. Nelson Garnett spoke specifically to progress in the area of animal welfare, and described upcoming challenges in this area. Finally, Dr. Greg Koski, director designee of the Office of Human Research Protections, addressed the group and engaged in a fruitful dialogue with the members about current issues and future directions of his office. The meeting concluded with a wrap-up by Dr. Baldwin. She outlined a number of issues that might make interesting topics for future meetings, thanked the group for their service to the NIH, and adjourned the meeting.

We hereby certify that, to the best of our knowledge, the foregoing minutes are accurate and complete.

Anthony Demsey, Ph.D.
Executive Secretary, Peer Review Oversight Group

Wendy Baldwin, Ph.D.
Deputy Director for Extramural Research