Inclusion of Public Representatives /
Participants in Scientific Peer Review

The NIH includes public members as participants in the National Advisory Boards and Councils of each Institute and Center (IC). Recently, the potential value of having members of the public participate in the peer review of research grant applications, particularly for studies that directly involve human subjects, has been recognized. This is especially true of clinical studies seeking to develop new treatments, to investigate health behaviors, and to discover etiologies of diseases and disorders. As a result, there has been an increased desire for nonscientist members of the public to participate in the process where their perspectives can enhance that process.

While it is often neither necessary nor appropriate to include consumer representatives in peer review, the use of consumer representatives may be extremely helpful in the review of certain areas of research. The use of consumers in scientific peer review should be considered when their expertise is deemed appropriate and necessary to the context of the review. This determination will be made individually by each IC, with specific rationale based on the scientific programs being reviewed. The nature of involvement of consumer reviewers should relate directly to the applications' relevance to a disease, health status, or public health problem.

The Peer Review Oversight Group (PROG), which provides advice to the Director of NIH regarding peer review policies and issues, has undertaken the development of general guidelines for the inclusion of public representatives as participants in the NIH peer review process. The following are the recommendations of the PROG on this topic.

The role of consumer reviewers is to offer a different perspective to the review process, such as (1) the feasibility of the research (e.g. willingness of patients to participate in a clinical trial, or the probability that patients would remain involved in a study); (2) whether a particular regimen is realistic; and (3) the relevance and content of the documents seeking informed consent. Consumer reviewers are expected to use their expertise as it relates to the research under review; there is no expectation that they should develop expertise in specific scientific areas, such as research methods or design.

Definition of Consumer Representative/Reviewer: No single definition is likely to fit all situations, given the variety of types of research, different award mechanisms used by ICs, and diseases and conditions being studied. Therefore, general guidance should be that each IC should have in place a clear definition of "consumer reviewer" for the research being reviewed. Definitions may differ among ICs but also may vary for specific research activities such as Requests for Applications (RFAs) on specific diseases or conditions. For example, an IC conducting disease-related research could require that consumer reviewers be survivors of that specific disease. However, for a disease with unfavorable survival rates or one with cognitive sequelae, this might not be feasible, and the definition could include those with intimate experience of the disease or disorder, such as family members or caregivers. In cases in which multiple diseases or disorders are to be reviewed in a single panel, the review criteria for consumer reviewer selection might be more general, to include experience related to one or more of several diseases or disorders.

Source of Nominations: The nomination process should be an open one, essentially the same as that used for nomination of scientists for NIH scientific peer review panels and other committees. That is, self-nomination as well as nominations from interested parties and from consumer groups may be accepted. Regardless of the source of the nomination, nominees should address the specific selection criteria and indicate their ability to represent broadly the interests of the group of public consumers they would represent, which need not be limited to a single disease. Nominees should also identify consumer groups to which they belong, as well as references who could be contacted to discuss their qualifications to participate as a reviewer.

Nominations should typically be solicited and accepted on an annual basis. Applications for consideration should be updated on an annual basis since there may be changes in the status of consumers and in their interest and ability to serve as reviewers. Some consideration should be given to recruitment of lay persons who have had experience serving on Institutional Review Boards or other similar involvement related to human subjects research, although recruitment should not be limited to these groups.

Criteria for selection should include the ability to communicate effectively; knowledge of or experience with a particular disease/condition; the ability to represent a broad constituency - not just one's own agenda or the agenda of a particular advocacy group; and the ability to contribute to an effective group process (able to be cooperative, constructive, flexible). Nominations must address these criteria and include names of potential references; letters of reference/recommendation need not accompany the application. Each IC should have a specific process for review and selection of applicants, which could range from a staff group (within a particular IC or across ICs for collaborative RFAs) to the use of external consultants. The selection process should be tailored to the needs of the IC and should be made clear to potential consumer reviewers.

As is the usual practice in scientific peer review, orientation materials should be sent to each consumer reviewer along with assignment information. On-site orientation should take place prior to the meeting; this could take place as a large group activity if several meetings are being held simultaneously and there are many consumer representatives serving as reviewers, or in a small group. Both advance materials and the on-site in-person orientation should include information about the NIH scientific peer review system (dual-level review, importance of confidentiality, avoidance of conflict of interest, etc.) and specific procedures of peer review, as well as the roles of the consumer reviewer, scientific reviewers, scientific review administrator and review panel chair. In addition, the review panel orientation at the beginning of the review meeting should include an introduction of the consumer reviewers (as part of the standard introductions of reviewers) and an explanation of their role on the panel.

It should be made clear to the consumer reviewers that the importance of their role in the review is based on their experience and knowledge in areas other than scientific methods and analysis. It should be emphasized that there is no expectation that they develop expertise in scientific methods or research design; their role is to offer a perspective that most scientists do not bring to the review.

When possible, consumer representatives who have already served on peer review panels should be assigned as mentors to new consumer reviewers; the mentor and new consumer reviewer should be assigned to the same review panel, and should be put in touch with each other in advance of the meeting. Training of potential mentors should be provided.

Specialized training/orientation for scientific review administrators and review committee chairs should be developed. Such preparation will be crucial to the success of the use of consumer representatives as reviewers and to their acceptance within the review groups. For example, guidelines and case studies could address (1) appropriate management of speaking opportunities for all members, emphasizing that consumer reviewers should be recognized to speak when assigned and given the opportunity to comment on all applications, similar to scientific reviewers; (2) a clear articulation for the review panel of the role of the consumer on the panel; and (3) appropriate monitoring and intervention when the peer review process is not proceeding appropriately.

Review Assignments: Consumer reviewers are to be used when their specific expertise, knowledge or experience will augment the consideration of merit; they are not to replace or reduce the number of scientific reviewers, but are added, to augment the review. Scientific review administrators should be mindful of the types of applications to which assignment of consumer reviewers is appropriate, and to the load distribution of review assignments. Consumer reviewers should be requested to focus on specific areas in which they have experience and expertise, which might include informed consent, recruitment/retention strategies, relevance of the research, and feasibility of the approach. This process will be facilitated if applicants provide a lay abstract as part of the application. Consumer reviewers will be expected to participate in scoring and discussion just as scientific reviewers do; i.e., they will be full participants and voting members of the committee. As is expected of any reviewer, if consumer reviewers intend to score outside the scoring range recommended by other reviewers, there must be open discussion of their reasons for this divergence. The number of consumer reviewers assigned to a panel should be flexible, but there should be a minimum of two.

Terms of Service: Consumer reviewers should be invited initially to attend peer review meetings as temporary members, just as scientific reviewers typically are. They may subsequently be invited to become regular members of the review group for terms of one year. Each one-year term would be a segment of "availability" to participate in review meetings, with actual service at each meeting based on the need for their experience/ expertise given the applications to be reviewed. While a consumer reviewer could be renewed for a second consecutive term of service availability, attention should be given to factors such as fatigue, health status, and available time (since review activities are unlikely to be seen as a professional responsibility, as is generally the case for scientific reviewers.)

Reimbursement: Consumer reviewers should be compensated in the same manner and at the same rates as scientific reviewers (honorarium plus travel and per diem). However, since consumer reviewers may not have the same ability to pay for their own travel in advance and await reimbursement, cash advances may be necessary. ICs should attempt to minimize financial issues as barriers to a consumer representative's ability to serve on a peer review panel.

Assessment: Ongoing assessment should be performed. Feedback from reviewers should be obtained at the end of each meeting. This can be done in a brief open discussion, with program staff permitted to be present. In addition, all reviewers should be given the opportunity to provide written comments if they wish to do so. Suggestions that are easily implemented can be dealt with administratively within individual review groups, but a centralized effort to share feedback within and across ICs, perhaps through the Review Policy Committee and the Extramural Program Management Committee, should be developed in order that all groups may benefit from the comments/ suggestions.

The use of consumer representatives in peer review should be an annual agenda topic for the Peer Review Oversight Group, which will monitor progress and make necessary and appropriate recommendations to the Director, NIH.