I. INTRODUCTION

This is NSF’s second GPRA performance plan. It is based on NSF’s GPRA strategic plan, submitted to Congress in September, 1997, and the FY 1999 performance plan. The mission, outcome goals, and critical factors for success from the strategic plan are outlined below.

Mission

NSF’s continuing mission is set out in the preamble to the National Science Foundation Act of 1950 (Public Law 810507):

To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense; and for other purposes.

The Act authorizes and directs NSF to initiate and support:

• basic scientific research and research fundamental to the engineering process,
• programs to strengthen scientific and engineering research potential,
• science and engineering education programs at all levels and in all the various fields of science and engineering,
• an information base for science and engineering appropriate for development of national and international policy.

NSF carries out its mission primarily by making merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S.

Outcome Goals

The outcomes of NSF investments, the results stemming from the grants and cooperative agreements we make, provide the evidence for NSF’s success as an investment agent. NSF staff pursue the following outcome goals, the general goals of the strategic plan, as they develop the NSF award portfolio:

1. Discoveries at and across the frontier of science and engineering;
Connections between discoveries and their use in service to society;
2. A diverse, globally-oriented workforce of scientists and engineers;
3. Improved achievement in mathematics and science skills needed by all Americans; and
4. Relevant, timely information on the national and international science and engineering enterprise.

Critical Factors for Success: Goals for Management

Excellence in managing the agency’s processes is an NSF goal on a par with our mission-oriented outcome goals. In the GPRA strategic plan, NSF articulated four critical factors in managing for excellence that provide the framework for annual performance goals.

• Operating a viable, credible, efficient merit review system;
• Exemplary use of and broad access to new and emerging technologies;
• A diverse, capable, motivated staff that operates with integrity; and
• Implementation of mandated performance assessment and management reforms in line with agency needs.

Means and Strategies

NSF’s primary business is to make merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S. By providing these resources, NSF contributes to the health and vitality of the U.S. research and education system, which enables and enhances the nation’s capacity for sustained growth and prosperity. The individuals and organizations in which NSF invests conduct the work that ultimately determines the outcomes of the investment process that NSF manages.

NSF uses merit review with external peer evaluation to select about 10,000 new awards each year from about 30,000 competitive proposals submitted by the science and engineering community for its consideration. Work continues through another 10,000 awards made in competitions of previous years. NSF's role in the fabric of federal funding of science and engineering is defined by the fundamental nature of the problems our grantees propose and explore, the innovative nature of the research and education we support, and our integrative approach to research and education.

NSF’s GPRA strategic plan outlines key investment strategies and an action plan for achievement of each of the outcome goals. There are common themes running through these investment strategies, and this performance plan reflects the importance of emphasizing activities that influence achievement of multiple objectives. Common strategies include: (1) broad support for activities across science and engineering research and education using competitive merit review with peer evaluation to identify the most promising ideas from the strongest researchers and educators; (2) integrating research and education to strengthen both; (3) extending NSF’s reach to underserved communities, including enhancing the diversity of the human resource base for science and engineering; (4) emphasis on emerging opportunities, particularly those that drive science and engineering forward at disciplinary interfaces while adding to the knowledge base in areas of national interest; (5) building partnerships with other agencies and other sectors; and (6) assuring that both NSF and the research and education communities reap optimal benefit from the revolution in information, communications, and computing technologies. In addition, NSF is committed to using committees and panels of external experts to assess its effectiveness and directions on a regular basis.

NSF’s management uses information on past performance (where available) and applies the strategies for enhancing outcomes to allocate available resources. NSF staff, advised by the merit review process, select the individual projects to be supported, managing toward the optimal mix of outcomes, given the available resources. At the NSF level, response to these common strategic elements has led to the identification of broad thematic areas for investment. Three thematic areas are emphasized in the FY 2000 Request: Information Technologies, Biocomplexity in the Environment, and Educating for the Future. These areas for enhanced investment stimulate coordination and synergy outside the usual budget lines, that is, across NSF organizations and with other agencies.

Deployment of Resources

Proposals and awards are managed through eight programmatic organizations, seven directorates and the Office of Polar Programs.²

• The Biological Sciences (BIO) directorate supports research ranging from the study of the structure and dynamics of biological molecules, such as proteins and nucleic acids, through cells, organs, and organisms, to studies of populations and ecosystems. (FY 2000 Request: $408.62 million)
• The Computer and Information Science and Engineering (CISE) directorate supports research on the theory and foundations of computing, system software and computer system design, and human-computer interactions, as well as prototyping, testing and development of cutting-edge computing and communications systems. (FY 2000 Request: $422.53 million)
• The Engineering (ENG) directorate supports research in bioengineering and environmental systems; chemical and transport systems; civil and mechanical systems; electrical and communications systems; design, manufacture, and industrial innovation; and engineering education activities. (FY 2000 Request: $378.53 million)
• The Geosciences (GEO) directorate supports research in the atmospheric, earth, and ocean sciences. (FY 2000 Request: $485.48 million)
• The Mathematical and Physical Sciences directorate supports research in mathematics, astronomy, physics, chemistry, and materials science. (FY 2000 Request: $753.97 million)
• The Social, Behavioral and Economic Sciences (SBE) directorate supports research on human characteristics and behavior of individuals and groups, including research on economic systems; coordinates and supports the Foundation’s international activities; and provides informational tools for tracking the human and institutional resources that make up the nation’s science and engineering infrastructure. (FY 2000 Request: $143.01 million)
• The Office of Polar Programs (OPP) supports multidisciplinary research in Arctic and Antarctic regions. (FY 2000 Request: $250.63 million)
• The Education and Human Resources directorate (EHR) supports education and training activities at every level, informal education, research on teaching and learning, human resource development, and development of research capabilities in every region of the country. (FY 2000 Request: $711.00 million)
• In FY 1999, NSF established an additional budget activity entitled Integrative Activities to manage funds for selected Foundation-wide activities such as the Opportunity Fund and the Major Research Instrumentation (MRI) program. (FY 2000 Request: $161.23 million)

NSF directorates make investments in three functional categories of activity: research projects, research facilities, and education and training. Approximately 95% of NSF’s budget goes directly to these investments. A fourth function, administration and management, provides support for the immediate activities of the agency. Each of these functions supports progress toward multiple outcome goals and is distinct in how that support is structured.

The FY 2000 Request leads to the following distribution of NSF budget resources across the key functions, with a total Request of $3.954 billion.

Investments in Research Project Support fund the cutting edge research that yields new discoveries. These investments help to maintain the nation’s capacity to excel in science and engineering, particularly in academic institutions. The store of knowledge produced by NSF-funded research projects also provides a rich foundation for broad and useful applications of knowledge and the development of new technologies. Research projects contribute to the education and training of the next generation of scientists and engineers by giving them the opportunity to participate in discovery-oriented projects. NSF centers provide a particularly rich environment for broad interdisciplinary research and education at all levels. The FY 2000 Request for Research Project Support is $2,316 million.

NSF provides support for large, multi-user Research Facilities that meet the need for access to state-of-the-art research facilities that would otherwise be unavailable to academic scientists and engineers. Support includes funding for construction, upgrade, operations, and maintenance of facilities, and for staff and support personnel to assist scientists and engineers in conducting research at the facilities. Support for these unique national facilities is essential to advancing U.S. research and education capabilities, and is driven predominantly by research opportunities and priorities. Planning for facilities involves selecting from among many exciting ideas those where facilities will be initiated or upgraded. It also involves phasing down or terminating those components of facilities that no longer support cutting edge research. The FY 2000 Request for Research Facilities totals $744 million.

Investments in Education and Training help ensure an adequate, well-prepared workforce of scientists and engineers that can maintain leadership in science and technology, both now and in the future, and help all students to achieve the mathematics and science skills needed to thrive in an increasingly technological society. NSF’s programs produce scientists and engineers knowledgeable about the most recent scientific and technical advances. These highly educated people then reach every sector of society and actively disseminate and use that knowledge in the service of innumerable social goals. They also provide well-prepared teachers and instructional materials and technologies that influence mathematics and science education at all levels. The FY 2000 Request for Education and Training totals $704 million.

The FY 2000 Request for Administration and Management of $190 million provides support for salaries and benefits of persons employed at the NSF; general operating expenses, including key activities to advance the agency’s information systems technology and to enhance staff training; and audit and Inspector General activities.

NSF is exploring revision of its budget account structure to track more closely with this functional approach to use of funds. The following table describes a crosswalk for the directorates and Integrative Activities with respect to key program function and staffing levels.

⁴ Other staff include all central administration and management. Other budget items include Major Research Equipment ($85 million, Research Facilities); Salaries and Expenses ($149 million, Adminstration and Management); and Office of Inspector General ($5 million, Administration and Management).

⁵ Numbers may not add due to rounding.

Just as several investment strategies support progress toward more than one outcome goal, so do the directorates and key functions. The table below provides a picture of the interactions between outcome goals and key functions. The most heavily shaded areas for each outcome goal indicate the key functions that most directly support it; the more lightly shaded areas show the key functions that influence progress toward the outcome goal less directly. By comparing this table with the crosswalk above, it is possible to determine how directorates exert their influence on attaining the outcome goals.

Cross-Cutting Areas with Other Agencies

Many other agencies support or conduct research and education activities in science and engineering in support of their missions. Frequently they will define outcome and performance goals that are similar to those NSF has defined. However, an agency’s mission will have an impact on the nature of the outcome and performance goals, so, in general, they are not identical. The FY 2000 government-wide performance plan for research and development contains a number of common performance goals related to like sets of activities. These relate to use of merit review in the awarding of funds for research and to construction and operation of research facilities.

Appendix 4 in NSF’s GPRA strategic plan discusses cross-cutting areas with other agencies in programs and processes in support of research, education, and development of information systems. It describes the mechanisms NSF and other agencies use to cooperate in addressing the broad spectrum of activities needed to manage the federal science and technology programs while avoiding inappropriate overlap and duplication.

External Factors Bearing on Success

Appendix 1 of NSF’s GPRA strategic plan discusses external factors bearing on NSF’s ability to achieve the outcome goals. In general, these factors result from changes in the environment for the conduct of research and education activities in the federal sector, the private sector, and in academe. Similar factors have a bearing on success in meeting NSF’s performance goals. They stem largely from the fact that NSF does not conduct the research and education activities directly and, therefore, influences outcomes rather than controlling them. Where particular factors could have exceptional impact on individual performance goals, they are discussed in the appendices that accompany this performance plan.

Assessing Agency Progress toward Outcome Goals

The challenge of performance assessment for NSF is that both the substance and the timing of outcomes from research and education activities are largely unpredictable. Moreover, NSF staff do not conduct the research and education projects. They provide support for others to undertake these activities based on proposals for the work to be done, the best information available as to the likely outputs and outcomes. and their knowledge of NSF’s outcome goals and the strategies for achieving them. They influence rather than control the outputs and outcomes.

Annual outputs of NSF awards, while in a sense quantifiable and certainly important to the success of individual projects, frequently shed little light on and consistently understate progress toward outcomes. NSF staff and external experts assess the performance of individual projects in the merit review process when the principal investigator requests funds for future activities. Assessment of agency performance must integrate across the outputs of many projects and address the impact of those outputs on meeting agency outcome goals, including information on NSF’s contribution as compared to that of many other possible factors. Whether assessing an individual project for future funding or assessing performance of the agency, the collective quality of the outputs is generally more important that the quantities. Assessing quality requires expert judgment.

Thus, NSF has developed performance goals for results of NSF’s investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format. The descriptive standards characterize successful and minimally effective performance. The stream of data and information on the products of NSF’s awards provides the basis for assessing NSF’s performance through annual reporting processes and the expert judgment of independent external panels.

Much of this performance assessment is retrospective, addressing investments made at some point in the past. In order to tie this effectively to current issues in management of the programs, the assessments must also address the quality of the set of awards made in the fiscal year under consideration. The focus of this portfolio assessment is the likelihood that the package of awards will produce strong results in the future. Special emphases within the plans for the fiscal year merit special attention in the assessment process.

Another way to tie performance to current issues in management of the programs is to address NSF performance in implementing the strategies articulated in the GPRA strategic plan. NSF staff have control over budget allocations and the decision processes that determine the set of awards. NSF performance goals for investment processes, along with those for management of the agency, are generally quantitative. They refer to processes conducted during the fiscal year that are captured in NSF systems.

[Return to Table of Contents]

II. SUMMARY TABLE OF ANNUAL PERFORMANCE GOALS

NSF’s annual performance goals for FY 2000 fall into three categories:

• Results (rationale, description of measurement approach, means and strategies, and baseline information found in Appendix 1);
• Investment process (rationale, description of measurement approach, means and strategies, and baseline information found in Appendix 2); and
• Management (rationale, description of measurement approach, means and strategies, and baseline information found in Appendix 3).

⁸ Elements in italics are highlighted in the FY 2000 government-wide performance plan.

⁹ Additional information on these quantitative annual performance goals is found in Appendix 1.

¹¹ Performance goals comparable to those in italics are highlighted in the FY 2000 government-wide performance plan.

¹² This performance goal is stated in the alternative format provided for in GPRA legisltation. See Appendix 2 for the complete statement of the performance goal, including the minimally effective standard.

[Return to Table of Contents]

APPENDIX 1

PERFORMANCE GOALS FOR RESULTS

The main text of this performance plan discusses the challenges to performance assessment of results of NSF research and education programs. In response the these challenges, NSF has developed performance goals for results of NSF’s investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format. The descriptive standards characterize successful and minimally effective performance. For each of the research and education outcome goals, there is a single alternative format performance goal that covers several elements of performance that NSF regards as most important in the current environment. In some instances there are a few related performance goals stated in a more standard format. The stream of data and information on the products of NSF’s awards provides the basis for assessing NSF’s performance through annual reporting processes and the expert judgment of independent external panels.

Much of this performance assessment is retrospective, addressing investments made at some point in the past. In order to tie this effectively to current issues in management of the programs, the assessments must also address the quality of the set of awards made in the fiscal year under consideration. The focus of this portfolio assessment is the likelihood that the package of awards will produce strong results in the future. Budget emphases for FY 2000 merit special attention in the assessment process. These special emphases are noted in the table of the main text and there is added discussion in this appendix. In most instances they refer to assessment of the likelihood that the current portfolio of awards will achieve the objectives of the special emphasis. For a few, NSF has highlighted the activity for several years, and it is possible to include some assessment of results.

Use of Advisory Committees¹ in Addressing Performance Goals for
Outcomes in Research and Education

Use of the alternative format performance goals for outcomes in research and education allows for human judgment to consider both quantitative and qualitative information on performance and weigh it in a balanced assessment. The descriptive performance goals are designed to be used as tools in NSF’s management process through a combination of internal self-assessment and review by external panels managed through the directorate advisory committees. Members of these committees are highly credible experts in their fields. Because of the need for the judgments of potential users and international perspectives on quality, committee membership will evolve to provide enhanced independence of judgements.

In FY 1997 and early FY 1998, NSF has run a series of experiments with advisory committees and their subcommittees to determine how to adapt existing processes² to obtain effective assessments of outcomes. For FY 1999, all advisory committees will give judgments of program effectiveness, describing strengths and weaknesses using the target levels of performance. The credibility of these reports will rest on the provision of qualitative detail about program results, to demonstrate which of the levels of performance are actually achieved. Advisory committees will have full access to a variety of data sources.

² NSF has a long-standing proactice of reviewing all programs on a three-year cycle for their performance in administering the merit review process. The review is performed by a Committee of Visitors (COV), usually set up as a subcommittee of a directorate advisory committee. The COV members form an independent group of external experts. COV reports are routed through the Advisory Committees to the directorates and NSF's Director. Once approved by the advisory committee, they become public documents and are available upon request. NSF is experimenting with using COVs to assess results as well as processes.

Results from NSF awards appear over time. Thus, in assessing performance toward the outcome goals during a given year, NSF must look carefully at (1) the noteworthy achievements of the year based on NSF awards, (2) the ways in which projects collectively affect progress, and (3) expectations for future performance based on the current set of awards.

As a basis for NSF’s annual performance report, each directorate will develop an annual self-assessment that addresses the three elements described above, articulating achievements, collective progress, and expectations for the future in understandable terms. The first two of these elements will provide means of assessing the performance of past NSF investments; the last, a means of assessing the characteristics of the current investments, with a focus on the likelihood of strong performance in the future. Directorate advisory committees, panels composed of experts external to NSF, will take the self-assessment and other available sources of information³ into account in benchmarking the directorate’s performance against the performance standards. The advisory committees will be expected to address both strengths and weaknesses of the directorates, including ways in which the performance is exceptionally strong. NSF management, working with the advice and guidance of the National Science Board, will integrate the advisory committee recommendations into the GPRA performance report. All of the performance goals are addressed to aggregate activities, and both advisory committee and NSF-wide reports will assess progress toward them based on the performance of collections of awards within key program functions.  

Outcome Goal 1: Discoveries at and across the frontier of science and engineering

Performance Indicators: level of outputs, quality of outputs, importance of discoveries, introduction of new ideas, development of new tools and technologies, interplay of disciplinary and interdisciplinary research, balance of the portfolio.

FY 2000 Performance Goal: NSF’s performance toward this outcome goal is

• Successful when NSF awards lead to important discoveries; new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and high-potential links across these boundaries;
• Minimally effective when there is a steady stream of outputs of good scientific quality.

Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.⁴

Elements of exceptionally strong performance include outcomes where NSF awards lead to advances of major scientific or engineering importance. These might, for instance: create new paradigms, stimulate important new areas of inquiry, or otherwise excite national and international attention.

⁴ During FY 1998, 11 of the 14 COV reviews held included new processes to assess NSF's performance. While each addressed GPRA they did not use the same mechanism and a number of reports are still outstanding. However, of the five reports reviewed to-date, performance for this goal was deemed as exceptionally strong.

FY 2000 Resources Supporting Achievement of the Successful Level of Performance:

• Investments made through the Research Project Support and Research Facilities key functions are the principal contributors toward reaching this goal. Resources are allocated to these key functions to ensure they support attainment of the outcome goal.
• FY 2000 plans call for increases of $173 million in Research Project Support and $14 million in Research Facilities for respective totals of $2,316 and $744 million.
• Focused increases for the Information Technology Initiative, IT², ($110 million on a base of approximately $700 million) and Biocomplexity ($50 million on a base of approximately $600 million) provide the basis for describing areas of emphasis for consideration within the total portfolio of awards in the assessment process.

Capital Investment to Enhance Future Performance:

• NSF’s capital investment is funded through the Major Research Equipment account. The FY 2000 Request for MRE is $85 million.
• Continuing projects include South Pole Station Modernization, Phase I of the Millimeter Wave Array, and the Large Hadron Collider.
• New projects include upgrade of an additional LC-130 for polar air support, initiation of a Network for Earthquake Engineering Simulation, and Terascale Computing Systems.
• (Performance goals for these capital construction projects are found in Appendix 2.)

Areas of Emphasis in FY 2000:

NSF will ask all directorate advisory committees to examine the directorate’s entire FY 2000 portfolio of research project support activities, identify activities they would characterize as high risk, multidisciplinary, or innovative, and make an assessment of overall scientific quality and balance with respect to these specific characteristics. The focus of this review will be to ensure that NSF is positioned well to attain the discovery-oriented performance goal. Directorates will structure their self-assessments to facilitate this review.

All directorate advisory committees will also be asked to examine the extent and appropriateness of investment in new types of scientific databases and the tools to use them. This is a critical component of activity under Knowledge and Distributed Intelligence, one of NSF’s areas of emphasis in FY 1999 and related to the IT emphasis in FY 2000.

Life in Extreme Environments, begun as a focused investment theme in FY 1997, reflects an aspect of Biocomplexity in the Environment. Preliminary results coming from awards made then may be available for assessment in FY 2000. Advisory committees will also be able to examine the active portfolio of awards for their potential influence on progress in this exciting area. Relevant directorates are BIO, GEO, MPS, and OPP.

Biocomplexity represents a new focused investment opportunity in FY 2000. An assessment of the resulting portfolio of investments will be part of the FY 2000 performance report.

In addition, Nanoscience and Engineering is just beginning to develop as an area for coordinated investment. Advisory committees in ENG and MPS will be asked to examine the set of awards in this area for their potential influence on progress.

Comment:

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that focus on discovery.
• Factors leading to priority for allocation of resources to directorates include emerging opportunities for major advances.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 2000 plans and use of resources are described in the main text of the performance plan. Appendix 2 lays out performance goals for investment processes that support these strategies.
• The most important single factor in attaining this outcome goal is the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix. There are no specific factors relative to this performance goal that merit additional attention.

Cross-Cutting Areas with Other Agencies:

• Facilities, with their large capital construction base and continuing operating costs, are particularly important elements of interagency planning. NSF has both formal and informal agreements with several other agencies to ensure that needed infrastructure is available to U.S. researchers without inappropriate overlap or duplication. For example, NSF and the Department of Energy cooperate in support for U.S. involvement with the Large Hadron Collider at CERN. This is a formal agreement between the two agencies on behalf of the U.S. and with the international partnership building the collider. On a more informal level, generally NSF develops and supports ground-based astronomy facilities, while NASA does the same for space-based facilities.
• Many other agencies support basic research in academic institutions, usually with a mission orientation. Their activities contribute to developing discoveries at and across the frontier of science and engineering. The introduction to this appendix describes NSF’s general approach to working with other agencies to ensure complementary sets of activities. Certain interactions are particularly important for fundamental research: National Institutes of Health (biosciences, genomics, biomedical research, chemistry); Department of Energy (high energy and nuclear physics, materials, high end computing, genomics); Department of Defense (engineering, computer and information science and engineering, mathematics); Department of Commerce (ocean and atmospheric sciences, global climate change, meteorology, atomic and molecular physics); National Aeronautics and Space Administration (astronomical sciences, global climate change); Department of Agriculture (biosciences, plant genomics); Department of Education (research on education); Environmental Protection Agency (environmental research).

Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.⁵

Exceptionally strong performance is characterized by NSF staff and grantees actively reaching out to potential users, and NSF-supported work playing critical roles in important innovation or problem solving for society.

FY 2000 Resources Supporting Achievement of the Successful Level of Performance:

• Investments made through the Research Project Support and Research Facilities key functions are the principal contributors toward reaching this goal. Some elements of these key functions lend themselves to fairly immediate connections between discoveries and their use in service to society. Resources may be allocated to these areas to ensure they support attainment of the outcome goal. For other elements, the connections may be less apparent, and the identification of resources supporting those connections less clear.⁶
• Research on education is included in the Education and Training key function. Such activities result in discoveries with potential for almost immediate impact on practitioners, and impact of results of research on the practice of education will be assessed under this performance goal.

Capital Investment to Enhance Future Performance:

• In FY 2000, the requested funding for Terascale Computing Systems and for initiation of a Network for Earthquake Engineering Simulation would provide a means of linking fundamental discovery to immediate service to society.

FY 2000 Areas of Emphasis:

Within NSF’s FY 2000 guidance level, a number of multidisciplinary areas of research are identified as ripe for progress and of particular importance for their potential connections to use in service to society. Generally speaking, these fit within the Foundation’s broad themes of Information Technology (IT), Biocomplexity in the Environment (BE), and Educating for the Future (EFF). In implementing focused research activities in these areas, NSF works in partnership with other agencies. Relevant directorate advisory committees will be asked to pay particular attention to these areas in their assessments, examining results of past investments, where results are available, and the contents of the current portfolio for (1) appropriateness in meeting the goals of the interagency program; (2) quality of the NSF research and infrastructure activities; and (3) balance among related areas of activity within NSF.

⁶ This is an example of how NSF resources may serve multiple purposes. Achievement of this performance goal is dependent on a continuing stream of discoveries. The resources address discovery and connections simultaneously.

Providing balanced, terascale computing resources access to the academic science and engineering community is critical to educating the next generation of computer and computational scientists. As part of the Foundation’s IT theme and the Administration’s initiative, Information Technology for the 21^st Century (IT²), all parts of NSF will address relevant issues in these areas. Advisory committees will review progress in developing a strong portfolio of activity.

Global Change, fitting within the broad theme of BE, has been an important area of focused research investment at NSF for several years, in conjunction with NSF’s participation in the U.S. Global Change Research Program. Advisory committees in BIO, GEO, OPP, and SBE will be asked to address performance in global change, addressing both past results and the current portfolio.

Research on Learning and Education, an element of EFF, was given high priority in the report of the President’s Committee of Advisors in Science and Technology on the Use of Technology to Strengthen K-12 Education in the United States (March 1997). NSF, in partnership with the Department of Education, will build on past investments in this area with an initiative in FY 1999 that will continue in FY 2000. While all directorates will participate in this effort, the EHR advisory committee will be asked to provide an assessment of the new investments for NSF as a whole.

Plant Genome Research received a major funding increase in FY 1998. The BIO directorate has initiated a program of support for research and infrastructure development, consistent with the recommendations of the 1998 report of the National Science and Technology Council entitled National Plant Genome Initiative. In FY 2000, the BIO advisory committee will review progress in developing a strong portfolio in this area, including interactions with other agencies, other nations, and the private sector. They will pay particular attention to completing sequencing of the model plant Arabidopsis.

Urban Communities incorporate a mix of natural and built environments. They are frequently neglected in efforts to obtain baseline information in natural, pristine environments. Yet their impact on quality of life in the U.S. is evident. BIO, ENG, and SBE play the leading roles in addressing issues related to urban environments as part of LEE.

Science and Technology Centers: Integrative Partnerships have responsibilities in the area of knowledge transfer that go beyond those of individual investigators by design. The first new set of STCs in almost a decade will be awarded in FY 2000. Assessments will address both the effectiveness of existing centers in establishing connections and the potential for strong performance in the new class.

Comment:

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that focus on discovery and that create or have the potential for connections with use in service to society.
• Factors leading to priority for allocation of resources to directorates include potential for connections with use in service to society.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 2000 plans and use of resources are described in the main text of the performance plan. Appendix 2 lays out performance goals for investment processes that support these strategies.
• Potential for use in service to society is an element in the merit review criteria used in the decision process.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff. Attention to all relevant merit review criteria is a factor in the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards. Staff outreach efforts accelerate with activities with strong potential for connections with use in service to society; frequently including specialized program announcements.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix.
• Many investigators do not think about the possible connections their work might have with use in service to society. Many potential users are not aware that results from NSF awards could be useful to them. NSF can conduct outreach and awareness efforts, but, generally, cannot force the connections.

Cross-Cutting Areas with Other Agencies:

• Many other agencies support basic research in academic institutions, usually with a mission orientation. Their missions contribute to developing connections between discoveries and their use in service to society. The introduction to this appendix describes NSF’s general approach to working with other agencies to ensure complementary sets of activities. Certain interactions are particularly important for fundamental research linked to missions of other agencies: National Institutes of Health (biosciences, genomics, biomedical engineering, chemistry); Department of Energy (high end and nuclear physics, materials, high end computing, genomics); Department of Defense (engineering, computer and information science and engineering, mathematics); Department of Commerce (ocean and atmospheric sciences, global climate change, meteorology, atomic and molecular physics); National Aeronautics and Space Administration (astronomical sciences, global climate change, engineering); Department of Agriculture (biosciences, plant genomics); Department of Education (research on education); Environmental Protection Agency (environmental research).
• NSF is an active participant in many interagency planning activities coordinated through the National Science and Technology Council (NSTC) that link fundamental research to national priorities. The include: U.S. Global Change Research Program, High Performance Computing and Communications, Program for the Next Generation of Vehicles, Education Research, Integrative Science for Ecosystem Challenges, Children’s Research, Plant Genome Research, among others. In all these activities, NSF’s role is at the fundamental end of the research and development spectrum. These interagency planning efforts coordinate among agencies to reap optimal benefit from the federal investment.
• NSF has been designated the lead Federal agency for an initiative on Information for the 21^st Century (IT²) – a six agency initiative including the Departments of Energy and Defense, the National Aeronautics and Space Administration, the National Institutes of Health, and the National Oceanic and Atmospheric Administration. IT² grew from the efforts of several agencies and responds to recommendations made by the President’s Information Technology Advisory Committee (PITAC). NSF’s FY 2000 investment includes $110 million for research in software systems, scaleable information infrastructure, high-end computing, and socioeconomic and workforce impacts. An additional $36 million for development of terascale computing systems will enable U.S. researchers to gain access to leading edge computational systems.

Baseline: There is no formal baseline at present. Preliminary efforts to pilot the use of expert judgment in performance assessment indicate NSF is successful in meeting this goal.⁷

Exceptionally strong performance is characterized by recognition of scientists or engineers who received NSF support during their training; and when the production of degree recipients in science, mathematics, and engineering increases markedly for underrepresented groups.

FY 2000 Resources Supporting Achievement of the Successful Level of Performance:

• The Research Project Support, Research Facilities, and Education and Training key functions all contribute to attaining this outcome goal.⁸
• Within the Research Project Support the Research Experiences for Undergraduates (REU) and Faculty Early Career Development (CAREER) programs are designed to address this goal. In FY 2000, REU will increase by $1.5 million to $37 million, and CAREER will increase by $5.4 million to $98 million.
• Funds supporting research assistants (both undergraduate and graduate) and postdoctoral researchers have a less direct, but noticeable effect on this performance goal.
• Within Education and Training, funding for the Integrative Graduate Education and Research program increases by $1.7 million to $29 million for graduate and postdoctoral activities.
• NSF investments in support of workforce development are based on good information on demographics of the current workforce, the future needs of the public and private sectors for the workforce, and, where appropriate, on valid research on teaching and learning.

⁸ This is an example how NSF resources may serve multiple purposes. In general, resources provided through the Research Project Support and Research Facilities key functions are not directed at serving this goal, but they influence its achievement. Likewise, the higher education components of Education and Training serve this goal most directly, but the K-12 and informal education activities also have an impact in the quality of the students entering higher education.

FY 2000 Areas of Emphasis:

NSF’s activities under the theme of Educating for the Future (an increase of $25 million on a base of about $400 million) describe several areas of emphasis focused on developing the science and engineering workforce.

Providing opportunities for participation in integrative research and education experiences is one of NSF’s key investment strategies for addressing this outcome goal. In order to influence the development of integrated approaches, NSF has developed a number of programs intended to synergize the integration of research and education. The most prominent of these programs are the CAREER, REU, and IGERT programs. Each of these is aimed at using NSF’s close interaction with academic science and engineering to support on-going efforts within academe to draw research and education together. They are supported across the Foundation. All advisory committees will assess the activities of the directorates they advise in the CAREER and REU programs, examining results of past investments and the level and quality of current investments. IGERT is a newer program that was undertaken in response to a National Science Board recommendation that NSF experiment with alternative forms of graduate student support and training. It builds on earlier investments in the BIO and EHR directorates. All directorates will address the quality of the current set of awards. The BIO and EHR advisory committees will also assess the results of past related investments and provide guidance on NSF’s implementation of IGERT.

The three programs addressed above, along with NSF-supported centers, are key elements of NSF’s activities to increase the participation of underrepresented groups in integrative research and education experiences. Their program announcements ask proposers to address how the activity proposed will impact diversity in the science and engineering workforce, and participation by members of underrepresented groups is strong. All directorate advisory committees will be asked to include an assessment of the impact of past activities on participation of underrepresented groups and the extent to which the current portfolio can be expected to have an impact on increasing participation.

See Appendix 2 for additional information and performance goals relative to these two areas of emphasis.

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that influence development of the science and engineering workforce, both directly and indirectly.
• Factors leading to priority for allocation of resources to directorates include potential for impact on the science and engineering workforce.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 2000 plans and use of resources are described in the main text of the performance plan. Appendix 2 lays out performance goals for investment processes that support these strategies.
• Means and strategies supporting integration of research and education and increasing diversity in the science and engineering workforce are described in Appendix 2. These strategies are particularly important in development of the workforce.
• NSF encourages participation of students on international projects. More targeted approaches to enhancing the global awareness of the science and engineering workforce are planned for future years.
• Impact on the developing workforce is an element in the merit review criteria used in the decision process.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff. Attention to all relevant merit review criteria is a factor in the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix.
• NSF provides very little of the overall investment of the development of the science and engineering workforce. Meeting the outcome goal for the long term implies a gradual change in process and philosophy of educating the workforce. Meeting the performance goal implies a commitment on the part of institutions and their faculties to enhancing the diversity of the science and engineering workforce and to providing a broader range of educational opportunities.

Cross-Cutting Areas with Other Agencies:

• Many other agencies support research and education activities in academic institutions, usually with a mission orientation. Their activities contribute to developing the workforce in science and engineering, both directly and indirectly. Most work indirectly through support of research assistants. The National Institutes of Health, Department of Education, and Department of Defense have dedicated fellowship or traineeship activities for graduate education.
• See Appendix 2 for a discussion of cross-cutting activities related to integration of research and education and enhancing diversity in the workforce.

Baseline: There is no formal baseline at present. Preliminary efforts to pilot the use of expert judgement in performance assessment either did not address this performance goal or did so in the context of a small base of program activity. The extensive reporting and evaluation elements for Education and Human Resources (EHR) programs provide information that indicates successful progress, but this information needs further assessment before yielding a valid baseline.

Exceptionally strong performance is characterized by awards that lead to high impact development, adoption, adaptation, and implementation of effective models, products, and practices that extend beyond the period of NSF investment or beyond the scope of NSF award sites; generate novel approaches with high potential to improve and change the education process; or are associated with significant improvement in student performance.

FY 2000 Resources Supporting Achievement of the Successful Level of Performance:

• This outcome goal is primarily supported through the Education and Training key function, which increases by $22 million to $704 million in FY 2000.⁹
• NSF’s activities under the theme of Educating for the Future (an increase of $25 million on a base of about $400 million) describe several areas of emphasis focused on improving achievement in mathematics and science skills.

FY 2000 Areas of Emphasis:

The FY 2000 government-wide performance plan contains a performance goal that is particularly relevant to the advisory committee assessments of performance in this area. It is related to NSF’s systemic activities in K-12 education. At the start of the decade, NSF initiated major programs for the systemic reform of science, mathematics, engineering, and technology education. Based on the belief that all students can learn and achieve in science and mathematics at much higher levels than then obtained, systemic projects treat whole systems and build much-needed educational capacity at state, urban, rural, school district, and school levels. These projects are unique in their involvement of broad partnerships and development of comprehensive goals, solutions, and actions.

Performance Goal: Over 80 percent of schools participating in a systemic initiative program will (1) implement a standards-based curriculum in science and mathematics; (2) further professional development of the instructional workforce; and (3) improve student achievement on a selected battery of tests, after three years of NSF support.

This performance goal has a quantitative form that could be addressed without the use of the advisory committee process. However, it will be addressed through the EHR advisory committee so that there will be an assessment of quality as well.

Through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences for at least 65,000 precollege teachers.

Research on Learning and Education, an element of EFF, was given high priority in the report of the President’s Committee of Advisors in Science and Technology on the Use of Technology to Strengthen K-12 Education in the United States (March 1997). NSF, in partnership with the Department of Education, will build on past investments in this area in FY 1999 and will continue joint activities in FY 2000. While all directorates will participate in this effort, the EHR advisory committee will be asked to provide an assessment of the new investments for NSF as a whole.

The NSF Graduate Teaching Fellows in K-12 Education places graduate and undergraduate students in K-12 schools to serve as science and mathematics content resources for teachers. This continues a FY 1999 innovative pilot effort that provides much needed expertise to support high-quality learning and also exposes graduate and undergraduate students to the needs of K-12 education. This is part of a comprehensive approach to workforce development that reaches from grade school through graduate school.

The development of the National Science, Mathematics, Engineering, and Technology Education Digital Library will be accelerated in FY 2000. This national resource will increase the quality, quantity, and comprehensiveness of internet-based K-16 educational resources. This virtual facility will link students, teachers, and faculty, and provide broad access to standards-based educational materials and learning tools for schools and academic institutions nationwide.

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that influence math and science achievement, both directly and indirectly.
• Factors leading to priority for allocation of resources to directorates include potential to improve achievement in math and science skills.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 2000 plans include systemic approaches, attention to teacher preparation and development, partnership with other agencies, digital libraries, graduate teaching fellows as content resources in K-12 schools, and developing a strong research base for use by practitioners.
• NSF activities must be developmental and catalytic, given the small fraction of total resources in K-12 education represented by NSF’s funding.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff. Attention to all relevant merit review criteria is a factor in the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.
• Rigorous evaluation activities support management of all K-12 education programs.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix.
• NSF provides very little of the overall investment in K-12 education. Meeting the outcome goal for the long term implies a gradual change in structure of education in math and science. Meeting the performance goal implies a commitment on the part of school districts, schools, and their faculty to modifying their approaches to education in order to enhance achievement and availability of resources to do so.

Cross-Cutting Areas with Other Agencies:

• The President’s Council of Advisors in Science and Technology has made recommendations with respect to establishing a strong research base for education and learning, particularly in investigation of the role of learning technologies. NSF and the Department of Education have worked to establish a joint research activity to address those recommendations.
• Results from the TIMSS report led to a call for developing an interagency action strategy to optimize the effectiveness of federal funding aimed at increasing achievement in math and science. NSF and the Department of Education shared responsibility for developing and implementing the strategy. Other agencies participated in the process, but none have extensive programmatic activities aimed at K-12 education.

Outcome Goal 5: Timely and relevant information on the national and international science and engineering enterprise.

NSF's provision of information on the national and international science and engineering enterprise is a customer-oriented activity, centered in the Division of Science Resources Studies (SRS). Funding is provided through the Education and Training key function. The performance goals for this activity aim for improved quality through enhanced timeliness and enhanced attention to data quality measures.    

Timeliness

In a recent survey, a sample of the science and engineering policy community indicated that improving timeliness of data was high priority for them.

Performance Indicator: Average time interval between reference period (the time to which the data refer) and reporting of data.

FY 2000 Performance Goal: 486 days.      Baseline: 540 days in 1995-1996.

Comment:

• FY 1999 performance goal was identical
• Performance is measured as a two-year moving average of the number of days between the end of the data reference period and the public availability of data (usually electronic dissemination) for surveys SRS supports. In FY 2000, performance for FY 1999-2000 will be reported.
• Means for achieving success: Taking advantage of advances in information and communications technologies; regular reporting of status to give ample time to take action to improve performance.
• Cross-cutting areas with other agencies: Many other agencies provide statistical information. At intersection of responsibilities, NSF cooperates with other agencies, rather than duplicate surveys. The performance goal refers only to data reported through the eleven core surveys that SRS supports, but SRS will calculate and report comparison numbers for data it reports from surveys sponsored by other federal agencies and other organizations.
• Data are maintained by SRS.

Data Quality

The value of information on the science and engineering enterprise is highly dependent on its ability to address issues of importance to those who seek to use it in making policy decisions. Measures of data quality help users determine the reliability of the information and the extent of likely variance introduced by sampling processes.

Performance Indicator: Data quality measures and their use in SRS products.

FY 2000 Performance Goal: Establish a standard set of data quality measures for reporting of Science Resources Studies products. Prepare reports on these measures for all SRS surveys and publish them in electronic formats to inform users of SRS data quality.

Baseline: This is a new effort to provide standard measures. Their absence places limits on the usefulness of surveys.

Comment:

• Related FY 1999 performance goal dealt with customer measures of relevance. Data quality is one factor in addressing relevance.
• Means for achieving success: Staff are working to develop a standard set of data quality measures; once in place, procedures will be established to ensure the appropriate information is provided electronically for all surveys.
• Cross-cutting areas with other agencies: Value of the data quality measures is enhanced when similar data are handled in similar ways. Staff will contact relevant statistical arms in other agencies in development of the standard set of measures.
• Information to establish performance toward this goal will be maintained by SRS.

Data Collection, Verification, and Validation

Most of the information base that will underlie assessments of achievement on the descriptive performance standards comes from outside the agency, through two major grantee reporting systems: the Project Reporting System which will include annual and final project reports for all awards, and the Impact Data Base and project monitoring system, designed by the Directorate for Education and Human Resources and built and maintained by a contractor. Through these systems, output indicators such as the following will be available to program staff, third party evaluators, and advisory committees:

• Related to discoveries: Results, published and disseminated: journal publications, books, audio or video products; software, newly-developed instrumentation, and other inventions; data, samples, specimens, germ lines, and related products of awards placed in shared repositories; contributions within and across disciplines.
• Related to connections: organizations of participants and collaborators (including collaborations with industry); publications; contributions to other disciplines, infrastructure, and beyond science and engineering; use beyond the research group of specific products, instruments, and equipment.
• Related to S&E workforce: student participants; demographics of participants; descriptions of student involvement; education and outreach activities under grants; and demographics of science and engineering students and workforce.
• Related to math and science skills: numbers and quality of educational models, products and practices and their use in the math and science community; volume and quality of teachers trained and faculty stimulated; student outcomes including enrollments in math and science courses, retention, achievement, and science and math degrees received.

This information is largely provided to NSF from external sources. When the data are gathered through a contractor, NSF staff will include data verification capacity in its evaluation of competing bids. The Impact Data Base and project monitoring system of the Education and Human Resources directorate is an example of a contractor-supported system.

Project reports for NSF’s research activities have been submitted largely on paper. They are subject to general review by the relevant program officer, but would be cumbersome in a wide-ranging assessment process. Thus, NSF is implementing an electronic project reporting system that will permit organized reporting of aggregate information. We anticipate that the reliability of the information in the system will improve over time, as investigators and institutions become comfortable with its use. FY 1999 will be the first year of its full implementation. Electronic submission will be required in FY 2000.

The scientific data from the reporting system will be tested for plausibility as a natural part of the external assessment process. In addition, data from the reporting system will be used to address progress under prior support when investigators reapply to NSF. Thus, the investigators have a strong interest in providing accurate information that reviewers may rely upon. NSF will work with institutions in order to establish some basic guidelines for reporting information on the demographics of participants in a reliable way.

NSF has worked with the university community to ensure that the added reporting and assessment burden will be minimal. This is important to having a viable performance plan for NSF. More direct efforts to verify and validate information in the project reporting system would add significantly to the cost and to the burden on the grantee community.

For the outcome goal on policy information, the Division of Science Resources Studies will calculate the time between collection and reporting of policy information, and their data will be open for verification and validation. The same is true for reporting on data quality measures.

NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection methods.

[Return to Table of Contents]

APPENDIX 2

Performance Goals for the NSF Investment Process

The following sections provide information about the means and strategies NSF uses in support of its outcome goals and articulates performance goals for the investment process by which NSF shapes its portfolio of awards. In FY 2000, NSF will invest $3.9 billion in these areas.

Proposal and Award Processes

NSF’s role in achieving its outcome goals for research and education is implemented through program officer selection of projects from among those submitted by the scientific community. The scientists and engineers comprising NSF’s program staff take NSF priorities and the advice of the external reviewers into account in developing their portfolio of awards.

Means and strategies for high quality proposal and award processes that support achievement of the outcome goals and meet customer expectations:

• Provide staff resources needed to manage proposal and award processes (see table on page 5 in the main document for planned staff resources in FY 2000).
• Provide electronic information systems that support the processes (systems enhancements expected in FY 2000, as per budget request).
• Provide administrative guidance/requirements that reflect the imperatives of high quality processes.
• Provide needed oversight of management to ensure guidance/requirements are met.
• Provide needed operating expenses to ensure credible processes (see budget request).
• Work with the science and engineering community to provide high quality external review of NSF proposals.

Cross-cutting areas with other agencies and institutional partners:

• NSF is an active participant with other agencies in the Federal Demonstration Partnership, a joint effort of government and academe to address commonality of processes and reporting requirements to facilitate federally funded research and education activities in academe.

Use of merit review.

Performance Indicator: Percent of NSF funds allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.

FY 2000 Performance Goal: 90%.     Baseline: 89% in FY 1997; 90% in FY 1998.

Comments:

• Performance goal is identical to that in FY 1999.
• Projects meeting the criteria for inclusion within the 90% include contracts selected through merit-based, competitive contracting procedures; FFRDCs that undergo external merit review in the process of renewal, even though they may not have direct competition; and continuing grant increments and supplements for projects reviewed in prior years.
• NSF makes a few exceptions to its general requirement for external merit review. These include situations in which objective external reviewers may be difficult to find, when natural phenomena such as volcanic eruptions or earthquakes make an external review process for proposals to study them too lengthy, or when researchers are proposing such new ideas that knowledgeable external reviewers may not exist.
• Data come from NSF’s Executive Information Systems.

Implementation of Merit Review Criteria

NSF’s generic review criteria were recently reviewed and revised by the National Science Board. Implementation of those criteria is an important, immediate goal in the merit review/project selection process, in order to better support the outcome goals for research and education, as noted in NSF’s GPRA strategic plan.

Performance Indicator: Use of merit review criteria by reviewers and program staff.

FY 2000 Performance Goal: NSF performance in implementation of the new merit review criteria is

• Successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards.
• Minimally effective when reviewers consistently use only a few of the suggested elements of the generic review criteria although others might be applicable.

Baseline: New criteria went into effect in FY 1998. External expert judgment will assess performance. The process will be used for the first time during FY 1999.

Comment:

• Data: NSF will use the directorate¹advisory committees to monitor this performance goal, as they will the performance goals for research and education outcomes. The advisory committees and their subcommittees will address questions on implementation of the new merit review criteria using the samples of reviews and staff analysis that they routinely examine in their judgment of the effectiveness and fairness of the review process. As part of the request to these committees, NSF will ask that they describe in their reports the samples and evidence they used to come to their judgments.
• Means for achieving success: NSF is modifying program announcements to encourage proposers to provide information on all relevant aspects of the merit review criteria. Program staff will work with reviewers to stress the importance of addressing both merit review criteria. Performance plans for program staff will include elements related to implementation of merit review criteria.

Customer Service

In 1995, NSF adopted a set of customer service standards, primarily related to the proposal review process, treating grantees and potential grantees (applicants) as the primary customers for NSF’s administrative processes. In a survey, applicants valued three standards most highly: (1) clear guidelines for proposal content and preparation, (2) a minimum of three months between program announcements and proposal deadlines, and (3) notification of proposal funding recommendation within six months of proposal submission. The survey measured baseline levels of customer satisfaction, with reference to FY 1995 experiences. The survey will be repeated in FY 1999, referring to FY 1998 experiences. Customer service has a potential impact on the number and quality of proposals received and thus on NSF’s ability to meet all outcome goals.

For this performance plan, we focus on two of these standards; ones NSF staff have devoted special attention to since the standards were adopted.

Customer Service - Time to Prepare Proposals

Customer Service Standard: Make program announcements and solicitations available to relevant individuals and organizations at least three months prior to the proposal deadline or target date.

Performance Indicator: Percent of program announcements and solicitations available at least three months prior to proposal deadlines or target dates.

FY 1999 Performance Goal: 95%      Baseline: 80% in FY 1997; 66% in FY 1998²

Comment:

• Means for achieving success: NSF staff work toward this performance goal by limiting the number of special competitions requiring individual program announcements and solicitations, planning for such competitions as far in advance as possible, and initiating clearance processes at least six months prior to the anticipated proposal deadlines.
• Data: NSF maintains records of timing between announcement and deadline. Timing begins when the announcement is placed on the Web for public information.

Customer Service - Time to Decision

Customer Service Standard:For 95 percent of proposals, be able to tell applicants whether their proposals have been declined or recommended for funding within six months of receipt.

Performance Indicator: Percent of proposals processed within six months of receipt.

FY 2000 Performance Goal: 75%      Baseline: 61% in FY 1997; 59% in FY 1998

Comment:

• FY 1999 target is 70%.
• For activities with deadline or target dates for receipt of proposals, the stated deadline or target date is set as the date of receipt for purposes of evaluating performance toward the standard.
• Processing proposals within six months of receipt has been a challenging goal for NSF, and we are therefore scaling annual performance targets to reach the customer service standard by FY 2002.
• This customer service standard is applicable to research project support and education and training. The proposal process for research facilities is more extended due to the nature of the projects.
• Means for achieving success: NSF staff work toward this performance goal by making effective use of electronic mechanisms in conducting the review, working cooperatively to eliminate overloads and bottlenecks, and keeping careful track of the stage of processing and age of all proposals.
• Data: Maintained in NSF’s proposal and award systems.

Maintaining openness in the system.

NSF must continually keep the proposal and award process open to new people and new ideas.

Performance Indicator: Percent of competitive research grants going to new investigators

FY 2000 Performance Goal: 30% Baseline: 27% for FY 1997; 27% for FY 1998

Comment:

• In the early 1990’s, NSF had percentages approximating 30 percent of all competitive research grants³ going to new investigators. The percentage dropped over the mid-1990’s and is now rising slightly. NSF’s FY 1999 performance goal is to raise that to 30%. The FY 2000 goal is to maintain that level.
• Means for achieving success: Program staff will examine trends over time for both their own programs and the Foundation and establish target levels for the projects they support that are appropriate to the Foundation meeting its performance goal.
• Data: Maintained in NSF’s proposal and award systems.

Integration of Research and Education

Integrating research and education appears as part of the investment strategies supporting all of the outcome goals for education and research as described in NSF’s GPRA strategic plan. NSF expects to see continuous improvement in the extent to which its research and education functions are accomplished jointly. The long term objective is two-fold: (1) to renew the strong interaction between federally-funded academic research and the development of the science and technology workforce that has characterized the U.S. science and engineering enterprise; and (2) to draw academic scientists and engineers into the challenge of improving K-12 education. We want to see all our awardees pay conscious attention to their effectiveness as both researchers and educators.

Means and strategies for reaching these objectives include:

• Including attention to integration of research and education as one of the elements of merit review criteria.
• Developing and increasing funding for specialized programs designed to effect the integration of research and education (Research Experiences for Undergraduates, Integrative Graduate Education and Research Training, and Faculty Early Career Development).
• Recognition awards to create a set of exemplary activities (FY 1997 and FY 1998).
• Requiring that centers and facilities use their increased visibility and size to enhance the integration of research and education.
• Meetings of awardees to obtain information on what works to share with the community.
• Encouraging all proposers to include information on plans for integrating research and education in their proposals, using outreach efforts and program announcements (FY 1999 performance goal).
• Working with reviewers to ensure they address these issues in reviews.
• Working with staff to ensure they address these issues in decision analyses and in developing award budgets.
• Measuring progress in proposals and reviews; in decision analyses; and in award budgets.
• Providing resources needed to undertake activities aimed at integration of research and education.

Cross-cutting activities with other agencies:

• The President issued a Presidential Review Directive in 1996 asking the National Science and Technology Council to review the government/university partnership. One preliminary conclusion of this review is the importance of strengthening the integration of research and education. NSF expects to work with other agencies as appropriate to implement the associated recommendations.
• NSF selects its candidates for the Presidential Early Career Award for Scientists and Engineers from among awardees in the Faculty Early Career Development program, one of those designed to increase integration of research and education.
• Many NSF centers and facilities have partnerships with related activities at other agencies. This provides an opportunity for other agencies to assist in and benefit from integration of research and education.

Proposer Attention to Integration of Research and Education

NSF’s generic merit review criteria address the extent to which proposals will advance discovery and understanding while promoting teaching, training, and learning and vice versa. At present, many proposers do not include information that would permit reviewers to address this element. To embed integration of research and education in all aspects of NSF activities NSF now has language in all program announcements encouraging proposers to discuss this aspect of their activity in proposals (FY 1999 performance goal).

Performance Indicator: Percent of research grant proposals explicitly addressing integration of research and education.

FY 2000 Performance Goal: 25%     Baseline: Under development.

Comment:

• This performance goal is applicable to the Research Project Support key program function.
• Means and strategies for success: NSF will continue working with the scientific community to encourage their attention on this area.
• Data: In order to minimize the burden of collecting information, NSF will sample the data using proposals submitted electronically to enable the use of search software.
• Baseline: FY 2000 is the first time we have attempted to address this area quantitatively. We have developed some key words for search purposes and will begin testing during FY 1999 to establish a baseline, revising the goal, if necessary.

Reviewer Attention to Integration of Research and Education

Once proposers include information on their plans for integration of research and education, reviewers will be able to address those plans in their reviews.

Performance Indicator: Percent of reviews of research grant proposals that explicitly address the merits of proposed activities addressing the integration of research and education.

FY 2000 Performance Goal: 25%     Baseline: Under development.

Comment:

• This performance goal is applicable to the Research Project Support key program function.
• Means and strategies for success: NSF will continue working with the scientific community to encourage their attention on this area. This will give reviewers more reason to address the issue.
• Data: In order to minimize the burden of collecting information, NSF will sample the data using reviews submitted electronically to enable the use of search software.
• Baseline: FY 2000 is the first time we have attempted to address this area quantitatively. We have developed some key words for search purposes and will begin testing during FY 1999 to establish a baseline, revising the goal, if necessary.

Diversity

In 1980, legislation gave NSF explicit responsibility for addressing issues of equal opportunity in science and engineering. This assignment of responsibility reflected the serious underrepresentation of women, minorities, and persons with disabilities in the science and engineering workforce, underrepresentation that persists to this day, although some progress has been made. Recognizing that progress toward all outcome goals for research and education requires maximal diversity of intellectual thought, NSF is emphasizing attention to enhancing the participation of groups currently underrepresented in science and engineering, including women, underrepresented minorities, and persons with disabilities, in all its programs. The long-term objective is to have a science and engineering workforce that mirrors the U.S. population. This performance goal applies to all key program functions.

Means and Strategies:

• Including attention to promoting diversity as one of the elements of merit review criteria.
• Developing and increasing funding for specialized programs designed to promote diversity.
• Work to embed enhancing diversity in all NSF programs.
• Focus on developing new entrants to the workforce.
• Creating a set of exemplary activities.
• Asking that centers and facilities use their increased visibility and size to promote participation of underrepresented groups and to create ties to minority serving institutions.
• Meetings of awardees to obtain information on what works to share with the community.
• Encouraging all proposers to include information on plans to promote diversity in their proposals, using outreach efforts and program announcements (FY 1999 performance goal).
• Working with reviewers to ensure they address these issues in reviews.
• Working with staff to ensure they address these issues in decision analyses and in developing award budgets.
• Measuring progress in proposals and reviews; in decision analyses; and in award budgets.
• Providing resources needed to undertake activities aimed at mentoring or other methods of increasing diversity.

Cross-cutting activities with other agencies:

• NSF manages the Presidential Award for Excellence in Mentoring in Science and Engineering on behalf of the Executive Office of the President. Award ceremonies involve other agencies, and many awardees draw resources from other agencies.
• NSF plays a leading role in the Interagency Working Group on the U.S. Science and Technology Workforce of the Future of the National Science and Technology Council’s Committee on Science. The group is looking at mechanisms to optimize the effect of the federal investment in the developing workforce, with emphasis on increasing participation of underrepresented groups.
• NSF and all other agencies participate in government-wide efforts to address the needs of minority-serving institutions.
• Many NSF centers and facilities have partnerships with related activities at other agencies. This provides an opportunity for other agencies to assist in promoting diversity.

Diversity Baseline:

The 1998 Science and Engineering Indicators* provides the following data that can serve as a baseline for assessing long term progress on diversity issues. In 1995:

• Women accounted for a bit over 22% of the science and engineering workforce, 52% of the population;
• Blacks made up 3.4% of the science and engineering workforce, 12% of the population;
• Hispanics made up 2.8% of the science & engineering workforce, 9% of the population.

*Pages 3-15, 3-16  

Diversity in Programs to Integrate Research and Education

Student-oriented programs designed to integrate research and education provide a focused opportunity to embed consideration of diversity in core NSF programming. Both REU sites and IGERT projects involve several students, so principal investigators can plan to develop a pool of potential participants that includes members of underrepresented groups.  

Performance Indicator: Participation of students from underrepresented groups on REU and IGERT awards.

FY 2000 Performance Goal: At least proportional to their representation in the relevant student population, exceeding that proportion in at least 25% of the projects.

Baseline: In preparation.

Comment:

• This goal is relevant to portions of the Research Project Support and Education and Training key program functions.
• Means and strategies for success: Proposers for REU sites and IGERT awards are required to describe plans for increasing participation of underrepresented groups. Reviewers take these plans into account. Program officers include it as an aspect of decision analysis and follow-up to see how well the plans are followed by awardees.
• Data: Data will come from principal investigators via the project reporting system. Such data has only limited accuracy, as we cannot require that principal investigators provide information on gender and ethnic or racial background. Data on the relevant student population comes from Science Resources Studies surveys.
• Baseline: FY 2000 will mark the first time we have focused attention on this data. We are gathering data where available for FY 1998 to create a baseline.

Baseline: In preparation.

Comment:

• This performance goal is relevant to the centers portion of Research Project Support and the Research Facilities Key Program Function.
• Means and strategies for success: Proposers for centers are encouraged to describe plans for increasing participation of underrepresented groups. Reviewers take these plans into account. Program officers include it as an aspect of decision analysis and follow-up to see how well the plans are followed by awardees. NSF is currently providing additional resources to link centers and facilities with minority serving institutions to increase participation in centers from underrepresented groups.
• Data: Data will come from principal investigators via the project reporting system. Such data has only limited accuracy, as we cannot require that principal investigators provide information on gender and ethnic or racial background. Data on the relevant student population comes from Science Resources Studies surveys.
• Baseline: FY 2000 will mark the first time we have focused attention on this data. We are gathering data where available for FY 1998 to create a baseline.

Facilities Oversight

The performance goals listed under this heading are applicable to the Research Facilities key program function.

The following comments are relevant to all goals in this section.

Means and Strategies:

• Careful planning ensures that construction and operating plans are realistic and contain needed contingency funds.
• NSF program officers will work closely with the project directors to ensure that the performance goals can be met.
• Where potential problems are identified, the program officer will immediately inform the NSF-wide team assigned to that project so that all appropriate actions can be taken to keep construction projects within cost and schedule and to maintain operating schedules to the extent possible.
• External factors such as extremely adverse weather or failure of partners to act as planned can have a significant effect on construction projects and operating plans.

Data:

• NSF staff has developed a common template for reporting needed information that is being used to develop baselines.
• This will be linked to the Project Reporting System, with project managers entering the information.

Cross-cutting Activities with Other Agencies and Countries:

• NSF consults with other agencies to optimize capabilities available to American researchers with no inappropriate duplication and cooperates with other agencies in construction of facilities where it will facilitate use across broad communities of researchers.
• NSF manages facilities in the Antarctic that are used by all federal agencies for selected projects.
• Many major facilities involve international cooperation.

Construction and Upgrade of Facilities

NSF puts a high premium on professional initial planning for construction and upgrade of facilities. But any planning for unique, state-of-the-art facilities must take into account the exploratory nature of the facilities themselves. Such facilities stretch the limits of technological capability.

Every year, in the President’s Budget Request, NSF sets out a cost plan and schedule for major construction and upgrade projects currently underway or planned for initiation in the Major Research Equipment account. NSF has established performance goals and measurements with respect to these plans and expects each construction and upgrade activity to meet these performance goals.  

Performance Indicator: Comparison with planned annual cost

FY 2000 Performance Goal: Keep construction and upgrades within annual expenditure plan, not to exceed 110 percent of estimates.

Baseline: In preparation.    

Performance Indicator: Comparison with planned annual schedule

FY 2000 Performance Goal: Keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110 percent of estimates.

Baseline: In preparation.  

Performance Indicator: Comparison with planned total cost.

Performance Goal: For all construction and upgrade projects initiated after 1996, when current planning processes were put in place, keep total cost within 110 percent of estimates made at the initiation of construction.

Baseline: In preparation.

Operations and Management of Facilities

Facilities must operate efficiently and reliably and must offer appropriate opportunities, if they are to be valuable to those they serve. NSF program officers work closely with facilities’ directors to ensure that facilities have appropriate resources to conduct operations and to provide maintenance that ensures reliable operations.

Performance Indicator: Comparison to scheduled operating time.

Performance Goal: Keep operating time lost due to unscheduled downtime to less than 10 percent of the total scheduled operating time.

Baseline: In preparation.

Comment:

• Data: Performance is measured as the average percentage among all facilities of full capacity "user units" lost during the year to breakdowns or other circumstances considered within the control of the facilities. The average across facilities is used in this instance because, although there should be latitude for some facilities to be run at greater failure rates with good reason, those facilities should be balanced by others operating more reliably. User units are defined separately for each facility, and are typically user-hours or something similar.
• Preliminary baseline performance for FY96, based on data from 26 facilities, is that an average of 6% of possible user units was lost. Facilities such as computer and communications networks where the concept of "full capacity user units" does not make sense are excluded from the calculation.

Data Collection, Verification, and Validation

Data in support of performance goals under the heading Proposal and Award Processes are maintained in NSF’s proposal and award systems. These systems are subject to regular checks for accuracy and reliability. One exception is the performance goal on time to prepare proposals. The budget division will maintain records for this performance goal that will be open to public scrutiny. Another exception is the performance goal on use of merit review criteria. Advisory committees will be provided with summary information developed from random samples of review records as they make their assessments. Background information to validate the accuracy of the summaries will be available upon request.

The performance goals in the categories of Integration of Research and Education, and Diversity will be supported by examination of materials in the Project Reporting System and, in the case of Diversity, comparison to reliable information on the student population in science and engineering.

Data in support of the performance goals under the heading Facilities Oversight are currently maintained by the directorates supporting the facilities, based on input from the facilities’ leadership. They will be linked to the Project Reporting System shortly. NSF verifies the accuracy and completeness of the information through constant interaction between NSF staff and the management of the facilities.

NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection methods.

[Return to Table of Contents]

APPENDIX 3

[Return to Table of Contents]