The 1993 Government Performance and Results Act (GPRA)
emerged from a bipartisan effort to improve accountability, productivity,
and effectiveness of Federal programs through strategic planning,
goal setting, and performance assessment. The annual assessments
will serve the dual purpose of guiding subsequent planning decisions
and of communicating program outcomes and impacts to the public.
This document, developed under the auspices of the Committee
on Fundamental Science of the National Science and Technology
Council (NSTC), serves to establish a broad framework for GPRA
implementation in assessment of fundamental science programs.
GPRA anticipates the need for flexibility in designing
the planning and assessment methodology appropriate to the great
variety of Federal programs. Clearly, all programs are intended
to contribute ultimately to over-arching national goals, such
as national security, quality of life, and economic prosperity.
Nevertheless, the manner in which specific programs do so is
dramatically different. For example, procurement of military
hardware contributes rather directly to the national security
goal, whereas an undergirding activity, such as fundamental scientific
research, contributes broadly to national goals over a very long
time period. Assessment techniques are in relatively early stages
of development in all areas and are only in their infancy for
areas such as fundamental science. GPRA anticipates the need
for time and experimentation in developing assessment techniques
by building in a phase-in period of several years for implementation.
The central issue in assessing fundamental science
lies in defining the goal against which progress is measured.
The Administration's science policy statement, Science in
the National Interest, establishes that goal as leadership
across the frontiers of scientific knowledge. This is
the critical measure for assuring that American scientists are
expanding the knowledge base at the leading edge.
We stress that leadership evaluation does not entail
simplistic numerical ranking of national programs. Our national
interest in leadership rests in having our research and educational
programs perform at the cutting edge--sometimes in competition,
but often in explicit collaboration, with scientists from other
nations.
This goal is the principal guideline for government
stewardship of science in the national interest. It is an enabling
or intermediate objective with respect to the over-arching goals
of improved health and environment, national security, economic
prosperity, and quality of life. Historical experience, particularly
over the last fifty years, has shown that a world-leading American
science program has been a necessary contributor to the national
interest. This will almost certainly be even more crucial in
the knowledge-based society of the twenty-first century.
However, we also know that science drives progress towards the
over-arching national goals over a long time period and only as
part of a larger enterprise requiring a complex interplay of science
and technological innovation with fiscal, regulatory, intellectual
property rights, and trade policies. Consequently, the enabling
goal of maintaining broad scientific leadership is that which
guides the management and assessment of today's science investments.
It provides the principal yardstick for GPRA assessment strategies
for fundamental science programs.
This goal naturally determines other broad features
of the assessment strategy. Retrospective evaluation, over several
decades, will need to be updated periodically to examine the link
between fundamental science and the over-arching national goals.
Both specific examples and broader economic analyses will be
needed.
Also in support of the over-arching national goals
are four additional enabling goals defined by the Administration
in Science in the National Interest. These are to:
Balanced assessment of the various dimensions of
program performance in an agency will require multiple sources
and types of evidence. In addition to retrospective merit review,
retrospective performance reports might draw on quantitative indicators,
qualitative indicators, descriptive indicators or narrative text,
examples of outstanding accomplishments and of more typical levels
of achievement, information about context, and findings from special
studies and analyses.
Because pre-existing measures of research results
were developed primarily for other purposes, they have not yet
been adapted for use in reporting at the agency level. Pre-existing
measures capture only a subset of the spectrum of research outputs
and outcomes. They do not map neatly or cleanly onto GPRA concepts.
Consequently, these measures (e.g., publication counts, citation
counts, and rate-of-return and related economic measures summarized
in Section III below) can serve only as a starting point for agency
thinking about how to design the most effective assessment strategies.
The insufficiency of quantitative measures per se is one reason
why other sources of evidence, such as merit review of past performance,
narrative discussion, and descriptions of outstanding accomplishments
and more typical levels of achievement should be included in annual
performance reports. New initiatives or programs in fundamental
science will continue to be evaluated by merit or peer review
within the framework of leadership at the scientific frontier;
i.e., agency program managers should be guided by the quality
of the research proposed relative to world standards as determined
by expert reviewers.
Effective assessment methods will be agency-
and program-specific. A set of principles for assessment
of fundamental science in individual agencies is summarized in
the box on the following page:
In addition to the principal enabling goal of leadership
across the frontiers of scientific knowledge, subsidiary goals
may include contributions to the education and preparation of
outstanding scientists and engineers, contributions to public
literacy in science and technology, or stimulation of advanced
technologies. The appropriateness of these or other subsidiary
goals will vary widely for specific agency programs.
As stressed above, this document provides only the
framework for assessing fundamental science in individual agencies.
Program-specific activities are needed to flesh out the
skeleton consistent with the goals and principles presented here.
As anticipated in GPRA, special studies and some period of experimentation
will be required. The pilot projects now underway will yield
new assessment methods. These developments, within the unifying
framework outlined here, will enable aggregation of program contributions
into a well managed Federal portfolio of fundamental science programs.
For evaluating current programs in individual
agencies, merit review based on peer evaluation will continue
to be the primary vehicle for assessing the excellence and conduct
of science at the cutting edge. Methods for international comparison
will need to be developed for NSTC assessment of the leadership
status of United States science overall. Assessment of US standing
internationally will be addressed in a future NSTC effort. This
paper focuses on assessment in individual agencies.
