² This version reflects the enacted FY 1999 Budget.

I. INTRODUCTION

This is NSF’s first GPRA performance plan. It is based on NSF’s GPRA strategic plan, submitted to Congress in September, 1997. The mission, outcome goals, and critical factors for success from that plan are outlined below.

Mission

NSF’s continuing mission is set out in the preamble to the National Science Foundation Act of 1950 (Public Law 810507):

To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense; and for other purposes.

The Act authorizes and directs NSF to initiate and support:

• basic scientific research and research fundamental to the engineering process,
• programs to strengthen scientific and engineering research potential,
• science and engineering education programs at all levels and in all the various fields of science and engineering,
• an information base for science and engineering appropriate for development of national and international policy.

NSF carries out its mission primarily by making merit-based grants to and making cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S.

Outcome Goals

The outcomes of NSF investments, the results stemming from the grants and cooperative agreements we make, provide the evidence for NSF’s success as an investment agent. NSF staff pursue the following outcome goals, the general goals of the strategic plan, as they develop the NSF award portfolio:

1. Discoveries at and across the frontier of science and engineering;
2. Connections between discoveries and their use in service to society;
3. A diverse, globally-oriented workforce of scientists and engineers;
4. Improved achievement in mathematics and science skills needed by all Americans; and
5. Relevant, timely information on the national and international science and engineering enterprise.

Critical Factors for Success: Goals for Management

Excellence in managing the agency’s processes is an NSF goal on a par with our mission-oriented outcome goals. In the GPRA strategic plan, NSF articulated four critical factors in managing for excellence that provide the framework for annual performance goals.

• Operating a viable, credible, efficient merit review system;
• Exemplary use of and broad access to new and emerging technologies;
• A diverse, capable, motivated staff that operates with integrity; and
• Implementation of mandated performance assessment and management reforms in line with agency needs.

Means and Strategies

NSF’s primary business is to provide merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S. By providing these resources, NSF contributes to the health and vitality of the U.S. research and education system, which enables and enhances the nation’s capacity for sustained growth and prosperity. The individuals and organizations in which NSF invests conduct the work that ultimately determines the outcomes of the investment process that NSF manages.

NSF uses merit review with external peer evaluation to select about 10,000 new awards each year from about 30,000 competitive proposals submitted by the science and engineering community. Work continues through another 10,000 awards made in competitions of previous years. NSF's role in the fabric of federal funding of science and engineering is defined by the fundamental nature of the problems our grantees propose and explore, the innovative nature of the research and education we support, and our integrative approach to research and education.

NSF’s GPRA strategic plan outlines key investment strategies and an action plan for achievement of each of the outcome goals. There are common themes running through these investment strategies, and this performance plan reflects the importance of emphasizing activities that influence achievement of multiple objectives. Common strategies include: (1) broad support for activities across science and engineering research and education using competitive merit review with peer evaluation to identify the most promising ideas from the strongest researchers and educators; (2) integrating research and education to strengthen both; (3) extending NSF’s reach to underserved communities, including enhancing the diversity of the human resource base for science and engineering; (4) emphasis on emerging opportunities, particularly those that drive science and engineering forward at disciplinary interfaces while adding to the knowledge base in areas of national interest; (5) building partnerships with other agencies and other sectors; and (6) assuring that both NSF and the research and education communities reap optimal benefit from the revolution in information, communications, and computing technologies. In addition, NSF is committed to using committees and panels of external experts to assess its effectiveness and directions on a regular basis.

NSF’s management uses information on past performance (where available) and applies the strategies for achieving outcomes to allocate available resources. NSF staff, advised by the merit review process, select the individual projects to be supported, managing toward the optimal mix of outcomes, given the available resources. At the NSF level, response to these common strategic elements has led to the identification of broad thematic areas for investment. Three thematic areas are emphasized in the FY 1999 Budget: Knowledge and Distributed Intelligence, Life and Earth’s Environment, and Educating for the Future. These areas for enhanced investment stimulate coordination and synergy outside the usual budget lines, that is, across NSF organizations and with other agencies.

Deployment of Resources

Proposals and awards are managed through eight programmatic organizations, seven directorates and the Office of Polar Programs.³

• The Biological Sciences (BIO) directorate supports research ranging from the study of the structure and dynamics of biological molecules, such as proteins and nucleic acids, through cells, organs, and organisms, to studies of populations and ecosystems. (FY 1999 Budget: $390.86 million)

• The Computer and Information Science and Engineering (CISE) directorate supports research on the theory and foundations of computing, system software and computer system design, and human-computer interactions, as well as prototyping, testing and development of cutting-edge computing and communications systems. (FY 1999 Budget: $298.68 million)

• The Engineering (ENG) directorate supports research in bioengineering and environmental systems; chemical and transport systems; civil and mechanical systems; electrical and communications systems; design, manufacture, and industrial innovation; and engineering education activities. (FY 1999 Budget: $368.55 million)

• The Geosciences (GEO) directorate supports research in the atmospheric, earth, and ocean sciences. (FY 1999 Budget: $472.98 million)

• The Mathematical and Physical Sciences (MPS) directorate supports research in mathematics, astronomy, physics, chemistry, and materials science. (FY 1999 Budget: $734.40 million)

• The Social, Behavioral and Economic Sciences (SBE) directorate supports research on human characteristics and behavior of individuals and groups, including research on economic systems; coordinates and supports the Foundation’s international activities; and provides informational tools for tracking the human and institutional resources that make up the nation’s science and engineering infrastructure. (FY 1999 Budget: $137.23 million)

• The Office of Polar Programs (OPP) supports multidisciplinary research in Arctic and Antarctic regions. (FY 1999 Budget: $245.07 million)

• The Education and Human Resources directorate (EHR) supports education and training activities at every level, informal education, research on teaching and learning, human resource development, and development of research capabilities in every region of the country. The FY 1999 EHR appropriation is $662.00 million. In addition, the American Competitiveness and Workforce Improvement Act provides an estimated $27 million from the collection of fees from the H-1B non-immigrant program. (FY 1999 Budget: $689.00 million)

• In FY 1999 NSF has established a new budget activity entitled Integrative Activities. These funds will be managed on an agency-wide basis and include support for efforts such as a focused competition for Knowledge and Distributed Intelligence (KDI), Major Research Instrumentation (MRI), and the Opportunity Fund. It will also include the one-time $39.2 million from the intellectual infrastructure fund, and $4 million for the Science and Technology Policy Institute (formerly the Critical Technologies Institute). (FY 1999 Budget: $161.43 million)

NSF directorates make investments in three functional categories: research projects, research facilities, and education and training. Approximately 95% of NSF’s budget goes directly to these investments. A fourth function, administration and management, provides support for the immediate activities of the agency.

The FY 1999 Budget leads to the following distribution of NSF budget resources across the key functions, with a total budget of $3.737 billion.

Investments in Research Project Support fund the cutting edge research that yields new discoveries. These investments help to maintain the nation’s capacity to excel in science and engineering, particularly in academic institutions. The store of knowledge produced by NSF-funded research projects also provides a rich foundation for broad and useful applications of knowledge and the development of new technologies. Research projects contribute to the education and training of the next generation of scientists and engineers by giving them the opportunity to participate in discovery-oriented projects. NSF centers provide a particularly rich environment for broad interdisciplinary research and education at all levels. The FY 1999 Budget for Research Project Support is $2.144 billion.

NSF provides support for large, multi-user Research Facilities that meet the need for access to state-of-the-art research facilities that would otherwise be unavailable to academic scientists and engineers. Support includes funding for construction, upgrade, operations, and maintenance of facilities, and for staff and support personnel to assist scientists and engineers in conducting research at the facilities. Support for these unique national facilities is essential to advancing U.S. research and education capabilities, and is driven predominantly by research opportunities and priorities. Planning for facilities involves selecting facilities for initiation or upgrade from among many exciting ideas possibilities. It also involves phasing down or terminating those components of facilities that no longer support cutting edge research. The FY 1999 Budget for Research Facilities totals $730 million.

Investments in Education and Training help ensure an adequate, well-prepared workforce of scientists and engineers that can maintain U.S. leadership in science and technology, both now and in the future, and help all students to acquire the mathematics and science skills needed to thrive in an increasingly technological society. NSF’s programs produce scientists and engineers knowledgeable about the most recent scientific and technical advances. These highly educated people then reach every sector of society and actively disseminate and use that knowledge in the service of innumerable social goals. They also provide well-prepared teachers and instructional materials and technologies that influence mathematics and science education at all levels. The FY 1999 Budget for Education and Training totals $683 million.

The FY 1999 level for Administration and Management of $181 million provides support for salaries and benefits of persons employed at the NSF; general operating expenses, including key activities to advance the agency’s information systems technology and to enhance staff training; and audit and Inspector General activities.

The following table describes the relationship between NSF directorates and key functions. This balance of funds may change slightly as plans continue to develop and programs are implemented.

Just as several investment strategies support progress toward more than one outcome goal, so do the directorates and key functions. The table below provides a picture of the interactions between outcome goals and key functions. The most heavily shaded areas for each outcome goal indicate the key functions that most directly support it; the more lightly shaded areas show the key functions that influence progress toward the outcome goal less directly. By comparing this table with the crosswalk above, it is possible to determine how directorates exert their influence on attaining the outcome goals.

⁵ FTE and IPA counts for Integrative Activities are spread throughout NSF. Cross-agency teams are responsible for conducting this work.

⁶ Other items include Major Research Equipment ($90 million, Research Facilities); Salaries and Expenses ($144 million, Administration and Management); and Office of the Inspector General ($5 million, Administration and Management).

⁷ Numbers may not add due to rounding.

Cross-Cutting Areas with Other Agencies

Many other agencies support or conduct research and education activities in science and engineering in support of their missions. Frequently they will define outcome and performance goals that are similar to those NSF has defined. However, an agency’s mission will have an impact on the nature of the outcome and performance goals, so, in general, they are not identical. The FY 1999 government-wide performance plan for research and development contains a number of common performance goals related to like sets of activities. These relate to use of merit review in the awarding of funds for research and to construction and operation of research facilities.

Appendix 4 in NSF’s GPRA strategic plan discusses cross-cutting areas with other agencies in programs and processes in support of research, education, and development of information systems. It describes the mechanisms NSF and other agencies use to cooperate in addressing the broad spectrum of activities needed to manage the federal science and technology programs while avoiding inappropriate overlap and duplication.

External Factors Bearing on Success

Appendix 1 of NSF’s GPRA strategic plan discusses external factors bearing on NSF’s ability to achieve the outcome goals. In general, these factors result from changes in the environment for the conduct of research and education activities in the federal sector, the private sector, and in academe. Similar factors have a bearing on success in meeting NSF’s performance goals. They stem largely from the fact that NSF does not conduct the research and education activities directly and, therefore, influences outcomes rather than controls them. Where particular factors could have exceptional impact on individual performance goals, they are discussed in the appendices that accompany this performance plan.

Assessing Agency Progress toward Outcome Goals

The challenge of performance assessment for NSF is that both the substance and the timing of outcomes from research and education activities are largely unpredictable. Moreover, NSF staff do not conduct the research and education projects. They provide support for others to undertake these activities based on proposals for the work to be done, using the best information available as to the likely outputs and outcomes and their knowledge of NSF’s outcome goals and the strategies for achieving them. They influence rather than control the outputs and outcomes.

Annual outputs of NSF awards, while in a sense quantifiable and certainly important to the success of individual projects, frequently shed little light on and consistently understate progress toward outcomes. NSF staff and external experts assess the performance of individual projects in the merit review process when the principal investigator requests funds for future activities. Assessment of agency performance must integrate across the outputs of many projects and address the impact of those outputs on meeting agency outcome goals, including information on NSF’s contribution as compared to that of many other possible factors. Whether assessing an individual project for future funding or assessing performance of the agency, the collective quality of the outputs is generally more important than the quantities. Assessing quality requires expert judgment.

Thus, NSF has developed performance goals for results of NSF’s investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format. The descriptive standards characterize successful and minimally effective performance. The stream of data and information on the products of NSF’s awards provides the basis for assessing NSF’s performance through annual reporting processes and the expert judgment of independent external panels.

Much of this performance assessment is retrospective, addressing investments made at some point in the past. In order to tie this effectively to current issues in management of the programs, the assessments must also address the quality of the set of awards made in the fiscal year under consideration. The focus of this portfolio assessment is the likelihood that the package of awards will produce strong results in the future. Special emphases within the plans for the fiscal year merit special attention in the assessment process.

Another way to tie performance to current issues in management of the programs is to address NSF performance in implementing the strategies articulated in the GPRA strategic plan. NSF staff have control over budget allocations and the decision processes that determine the set of awards. NSF performance goals for investment processes, along with those for management of the agency, are generally quantitative. They refer to processes conducted during the fiscal year that are captured in NSF systems.

[Return to Table of Contents]

II. SUMMARY TABLE OF ANNUAL PERFORMANCE GOALS

NSF’s annual performance goals for FY 1999 fall into three categories:

• Results (rationale, description of measurement approach, means and strategies, and baseline information found in Appendix 1);
• Investment process (rationale, description of measurement approach, means and strategies, and baseline information found in Appendix 2); and
• Management (rationale, description of measurement approach, means and strategies, and baseline information found in Appendix 3).

¹⁰ Elements in italics are highlighted in the government-wide performance plan.

¹¹ NSF's decision making process is based on merit review and requires qualitative assessments by experts. Similarly, the language used to describe performance goals needs to be flexible so that external reviewers may apply it to different disciplines and activities.

¹³ Additional information of performance goals in this section is found in Appendix 2.

¹⁴ Performance goals in italics are highlighted in the government-wide performance plan.

¹⁵ This performance goal is stated in the alternative format provided for in GPRA legislation. See Appendix 2 for the complete statement of the performance goal, including the minimally effective standard.

[Return to Table of Contents]

APPENDIX 1

PERFORMANCE GOALS FOR RESULTS

The main text of this performance plan discusses the challenges to performance assessment of results of NSF research and education programs. In response to these challenges, NSF has developed performance goals for results of NSF’s investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format. The descriptive standards characterize successful and minimally effective performance. For each of the research and education outcome goals, there is a single alternative format performance goal that covers several elements of performance that NSF regards as most important in the current environment. In some instances there are a few related performance goals stated in a more standard format. The stream of data and information on the products of NSF’s awards provides the basis for assessing NSF’s performance through annual reporting processes and the expert judgment of independent external panels.

Much of this performance assessment is retrospective, addressing investments made at some point in the past. In order to tie this effectively to current issues in management of the programs, the assessments must also address the quality of the set of awards made in the fiscal year under consideration. The focus of this portfolio assessment is the likelihood that the package of awards will produce strong results in the future. Budget emphases for FY 1999 merit special attention in the assessment process. These special emphases are noted in the table of the main text and there is added discussion in this appendix. In most instances they refer to assessment of the likelihood that the current portfolio of awards will achieve the objectives of the special emphasis. For a few, NSF has highlighted the activity for several years, and it is possible to include some assessment of results.

Use of Advisory Committees¹ in Addressing Performance Goals for Outcomes in Research and Education

Use of the alternative format performance goals for outcomes in research and education allows for human judgment to consider both quantitative and qualitative information on performance and weigh it in a balanced assessment. The descriptive performance goals are designed to be used as tools in NSF’s management process through a combination of internal self-assessment and review by external panels managed through the directorate advisory committees. Members of these committees are highly credible experts in their fields. Because of the need for the judgments of potential users and international perspectives on quality, committee membership will evolve to provide enhanced independence of judgements.

In FY 1997 and early FY 1998, NSF has run a series of experiments with advisory committees and their subcommittees to determine how to adapt existing processes² to obtain effective assessments of outcomes. For FY 1999, all advisory committees will give judgments of program effectiveness, describing strengths and weaknesses using the target levels of performance. The credibility of these reports will rest on the provision of qualitative detail about program results, to demonstrate which of the levels of performance are actually achieved. Advisory committees will have full access to a variety of data sources.

² NSF has a long-standing practice of reviewing all programs on a three-year cycle for their performance in administering the merit review process. The review is performed by a Committee of Visitors (COV), usually set up as a subcommittee of a directorate advisory committee. The COV members form an independent group of external experts. COV reports are routed through the Advisory Committees to the directorates and NSF's Director. Once approved by the advisory committee, they become public documents and are available upon request. NSF is experimenting with using COVs to assess results as well as processes.

Results from NSF awards appear over time. Thus, in assessing performance toward the outcome goals during a given year, NSF must look carefully at (1) the noteworthy achievements of the year based on NSF awards, (2) the ways in which projects collectively affect progress, and (3) expectations for future performance based on the current set of awards.

As a basis for NSF’s annual performance report, each directorate will develop an annual self-assessment that addresses the three elements described above, articulating achievements, collective progress, and expectations for the future in understandable terms. The first two of these elements will provide means of assessing the performance of past NSF investments; the last, a means of assessing the characteristics of the current investments, with a focus on the likelihood of strong performance in the future. Directorate advisory committees, panels composed of experts external to NSF, will take the self-assessment and other available sources of information³ into account in benchmarking the directorate’s performance against the performance standards. The advisory committees will be expected to address both strengths and weaknesses of the directorates, including ways in which the performance is exceptionally strong. NSF management, working with the advice and guidance of the National Science Board, will integrate the advisory committee recommendations into the GPRA performance report. All of the performance goals are addressed to aggregate activities, and both advisory committee and NSF-wide reports will assess progress toward them based on the performance of collections of awards within key program functions.

Outcome Goal 1: Discoveries at and across the frontier of science and engineering

Performance Indicators: level of outputs, quality of outputs, importance of discoveries, introduction of new ideas, development of new tools and technologies, interplay of disciplinary and interdisciplinary research, balance of the portfolio.

FY 1999 Performance Goal: NSF’s performance toward this outcome goal is

• Successful when NSF awards lead to important discoveries; new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and high-potential links across these boundaries;
• Minimally effective when there is a steady stream of outputs of good scientific quality.

Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.⁴

Elements of exceptionally strong performance include outcomes where NSF awards lead to advances of major scientific or engineering importance. These might, for instance: create new paradigms, stimulate important new areas of inquiry, or otherwise excite national and international attention.

⁴ During FY 1998 11 of the 14 COV reviews held included new processes to assess NSF's performance. While each addressed GPRA they did not use the same mechanism and a number of reports are still outstanding. However, of the five reports reviewed to-date, performance for this goal was deemed as exceptionally strong.

FY 1999 Resources Supporting Achievement of the Successful Level of Performance:

• Investments made through the Research Project Support and Research Facilities key functions are the principal contributors toward reaching this goal. Resources are allocated to these key functions to ensure they support attainment of the outcome goal.
• Research on education is included in the Education and Training key function. It also contributes to discovery, and results of those activities will be assessed under this performance goal.
• The FY 1999 Budget increased by $156 million in Research Project Support, $78 million in Research Facilities, and $16 million in research on education to respective totals of $2,144, $730, and $49 million.
• Focused increases for Knowledge and Distributed Intelligence ($97 million on a base of approximately $500 million) and Life and Earth’s Environment ($55 million on a base of approximately $500 million) provide the basis for describing areas of emphasis for consideration within the total portfolio of awards in the assessment process.

Capital Investment to Enhance Future Performance:

• NSF’s capital investment is funded through the Major Research Equipment account. The FY 1999 Budget is $90 million.
• Continuing projects include South Pole Station Modernization and Phase I of the Millimeter Wave Array.
• New projects include upgrade of two LC-130s for polar air support and initiation of construction of detectors for the Large Hadron Collider.
• (Performance goals for these capital construction projects are found in Appendix 2.)

Areas of Emphasis in FY 1999:

The FY 1999 government-wide performance plan contains a performance goal that is particularly relevant to the advisory committee assessments of directorate performance.

Performance Goal: An independent assessment will judge NSF research programs to have the highest scientific quality and an appropriate balance of projects characterized as high-risk, multidisciplinary, or innovative.

NSF will ask all directorate advisory committees to examine the directorate’s entire FY 1999 portfolio of research project support activities, identify activities they would characterize as high-risk, multidisciplinary, or innovative, and make an assessment of overall scientific quality and balance with respect to these specific characteristics. The focus of this review will be to ensure that NSF is positioned well to attain the first outcome goal: discovery at and across the frontier of science and engineering. Directorates will structure their self-assessments to facilitate this review.

All directorate advisory committees will also be asked to examine the extent and appropriateness of investment in new types of scientific databases and the tools to use them. This is a critical component of activity under Knowledge and Distributed Intelligence, one of NSF’s areas of emphasis in FY 1999.

Life in Extreme Environments, begun as a focused investment theme in FY 1997, is a part of Life and Earth’s Environment. Preliminary results coming from awards made then may be available for assessment in FY 1999. Advisory committees will also be able to examine the active portfolio of awards for their potential influence on progress in this exciting area. Relevant directorates are BIO, GEO, MPS, and OPP.

In addition, Nanoscience and Engineering is just beginning to develop as an area for coordinated investment. Advisory committees in ENG and MPS will be asked to examine the set of awards in this area for their potential influence on progress.

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that focus on discovery.
• Factors leading to priority for allocation of resources to directorates include emerging opportunities for major advances.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 1999 plans and use of resources are described in the main text of the performance plan. Appendix 2 lays out performance goals for investment processes that support these strategies.
• The most important single factor in attaining this outcome goal is the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix. There are no specific factors relative to this performance goal that merit additional attention.

Cross-Cutting Areas with Other Agencies:

• Facilities, with their large capital construction base and continuing operating costs, are particularly important elements of interagency planning. NSF has both formal and informal agreements with several other agencies to ensure that needed infrastructure is available to U.S. researchers without inappropriate overlap or duplication. For example, NSF and the Department of Energy cooperate in support for U.S. involvement with the Large Hadron Collider at the European Organization for Nuclear Research (CERN). This is a formal agreement between the two agencies on behalf of the U.S. and with the international partnership building the Collider. On a more informal level, generally NSF develops and supports ground-based astronomy facilities, while NASA does the same for space-based facilities.

• Many other agencies support basic research in academic institutions, usually with a mission orientation. Their activities contribute to developing discoveries at and across the frontier of science and engineering. The introduction to this appendix describes NSF’s general approach to working with other agencies to ensure complementary sets of activities. Certain interactions are particularly important for fundamental research: National Institutes of Health (biosciences, genomics, biomedical research, chemistry); Department of Energy (high energy and nuclear physics, materials, high end computing, genomics); Department of Defense (engineering, computer and information science and engineering, mathematics); Department of Commerce (ocean and atmospheric sciences, global climate change, meteorology, atomic and molecular physics); National Aeronautics and Space Administration (astronomical sciences, global climate change); Department of Agriculture (biosciences, plant genomics); Department of Education (research on education); Environmental Protection Agency (environmental research).

Outcome Goal 2: Connections between discoveries and their use in service to society

Performance Indicators: availability of results of NSF awards to potential users, use of results of NSF awards by researchers and practitioners, role of NSF-sponsored activities in stimulating innovation and policy development.

FY 1999 Performance Goal: NSF’s performance toward this outcome goal is

• Successful when the results of NSF awards are rapidly and readily available and feed, as appropriate, into education, policy development, or use by other federal agencies or the private sector.
• Minimally effective when results of NSF awards show the potential for use in service to society, and when activities designed to enhance connections between discoveries and their use in service to society meet the successful standard.

Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.⁵

Exceptionally strong performance is characterized by NSF staff and grantees actively reaching out to potential users, and NSF-supported work playing critical roles in important innovation or problem solving for society.

FY 1999 Resources Supporting Achievement of the Successful Level of Performance:

• Investments made through the Research Project Support and Research Facilities key functions are the principal contributors toward reaching this goal. Some elements of these key functions lend themselves to fairly immediate connections between discoveries and their use in service to society. Resources may be allocated to these areas to ensure they support attainment of the outcome goal. For other elements, the connections may be less apparent, and the identification of resources supporting those connections less clear.
• Research on education is included in the Education and Training key function. Such activities result in discoveries with potential for almost immediate impact on practitioners, and impact of results of research on the practice of education will be assessed under this performance goal.

FY 1999 Areas of Emphasis:

NSF’s FY 1999 Budget includes a number of multidisciplinary areas of research that are ripe for progress and of particular importance for their potential connections to use in service to society. Generally speaking, these fit within the Foundation’s broad themes of Knowledge and Distributed Intelligence (KDI), Life and Earth’s Environment (LEE), and Educating for the Future (EFF). In implementing focused research activities in these areas, NSF works in partnership with other agencies. Relevant directorate advisory committees will be asked to pay particular attention to these areas in their assessments, examining results of past investments, where results are available, and the contents of the current portfolio for (1) appropriateness in meeting the goals of the interagency program; (2) quality of the NSF research and infrastructure activities; and (3) balance among related areas of activity within NSF.

Global Change, fitting within the broad theme of LEE, has been an important area of focused research investment at NSF for several years, in conjunction with NSF’s participation in the U.S. Global Change Research Program. Advisory committees in BIO, GEO, OPP, and SBE will be asked to address performance in global change, addressing both past results and the current portfolio. (FY 1999 Budget: $182 million)

Research on Learning and Education, an element of EFF, was given high priority in the report of the President’s Committee of Advisors in Science and Technology on the Use of Technology to Strengthen K-12 Education in the United States (March 1997). NSF, in partnership with the Department of Education, will build on past investments in this area with a major initiative in FY 1999. While all directorates will participate in this effort, the EHR advisory committee will be asked to provide an assessment of the new investments for NSF as a whole. ( FY 1999 Budget: $22 million)

Plant Genome Research received a major funding increase in FY 1998 and an additional $10 million increase in FY 1999. The BIO directorate has initiated a program of support for research and infrastructure development, consistent with the recommendations of the 1998 report of the National Science and Technology Council entitled National Plant Genome Initiative. In FY 1999, the BIO advisory committee will review progress in developing a strong portfolio in this area, including interactions with other agencies, other nations, and the private sector. They will pay particular attention to completing sequencing of the model plant Arabidopsis. (FY 1999 Budget: $50 million)

The Next Generation Internet (NGI) is an interagency activity aimed at increasing hundred-fold the speed and capacity of today’s Internet. NSF’s role is to facilitate high-performance links among academic institutions and national centers and to conduct research on networking. NSF activities are built on past investments in similar areas in networking research and infrastructure. The CISE advisory committee will be asked to assess NSF’s performance in NGI, with the focus on the current portfolio. (FY 1999 Budget: $25 million)

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that focus on discovery and that create or have the potential for connections with use in service to society.
• Factors leading to priority for allocation of resources to directorates include potential for connections with use in service to society.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 1999 plans and use of resources are described in the main text of the performance plan. Appendix 2 lays out performance goals for investment processes that support these strategies.
• Potential for use in service to society is an element in the merit review criteria used in the decision process.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff. Attention to all relevant merit review criteria is a factor in the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards. Staff outreach efforts accelerate with activities with strong potential for connections with use in service to society, frequently including specialized program announcements.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix.
• Many investigators do not think about the possible connections their work might have with use in service to society. Many potential users are not aware that results from NSF awards could be useful to them. NSF can conduct outreach and awareness efforts, but, generally, cannot force the connections.

Cross-Cutting Areas with Other Agencies:

• Many other agencies support basic research in academic institutions, usually with a mission orientation. Their missions contribute to developing connections between discoveries and their use in service to society. The introduction to this appendix describes NSF’s general approach to working with other agencies to ensure complementary sets of activities. Certain interactions are particularly important for fundamental research linked to missions of other agencies: National Institutes of Health (biosciences, genomics, biomedical engineering, chemistry); Department of Energy (high end and nuclear physics, materials, high end computing, genomics); Department of Defense (engineering, computer and information science and engineering, mathematics); Department of Commerce (ocean and atmospheric sciences, global climate change, meteorology, atomic and molecular physics); National Aeronautics and Space Administration (astronomical sciences, global climate change, engineering); Department of Agriculture (biosciences, plant genomics); Department of Education (research on education); Environmental Protection Agency (environmental research).

• NSF is an active participant in many interagency planning activities coordinated through the National Science and Technology Council (NSTC) that link fundamental research to national priorities. These include: U.S. Global Change Research Program, High Performance Computing and Communications, Program for the Next Generation of Vehicles, Education Research, Integrative Science for Ecosystem Challenges, Children’s Research and Plant Genome Research among others. In all these activities, NSF’s role is at the fundamental end of the research and development spectrum. These interagency planning efforts are coordinated among agencies to reap optimal benefit from the federal investment.

Outcome Goal 3: A diverse, globally-oriented workforce of scientists and engineers

Performance Indicators: demographic data on participants in NSF-funded activities and in the workforce; character of experiences in NSF-funded activities aimed at educating the next generation of the workforce.

FY 1999 Performance Goal: NSF’s performance toward this outcome goal is

• Successful when participants in NSF activities experience world-class professional practices in research and education, using modern technologies and incorporating international points of reference; when academia, government, business, and industry recognize their quality; and when the science and engineering workforce shows increased participation of underrepresented groups.
• Minimally effective when opportunities and experiences of students in NSF-sponsored activities are comparable to those of most other students in their fields; and when the participation of underrepresented groups in NSF-sponsored science and engineering projects and programs increases.

Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.⁶

Exceptionally strong performance is characterized by recognition of scientists or engineers who received NSF support during their training; and when the production of degree recipients in science, mathematics, and engineering increases markedly for underrepresented groups.

FY 1999 Resources Supporting Achievement of the Successful Level of Performance:

• The Research Project Support, Research Facilities, and Education and Training key functions all contribute to attaining this outcome goal.⁷

⁷ This is an example of how NSF resources may serve multiple purposes. In general, resources provided through the Research Project Support and Research Facilities key functions are not directed at serving this goal, but they influence its achievement. Likewise, the higher education components of Education and Training serve this goal most directly, but the K-12 and informal education activities also have an impact in the quality of the students entering higher education.

• Within the Research Project Support the Research Experiences for Undergraduates (REU) and Faculty Early Career Development (CAREER) programs are designed to address this goal. Within the FY 1999 Budget, REU will increase by $3 million to $36 million, and CAREER will increase by $12 million to $93 million.
• Funds supporting undergraduate and graduate students and postdoctoral researchers have a less direct, but noticeable effect on this performance goal. In FY 1998, approximately $24 million went to support undergraduates, $310 million to graduate students, and $117 million to postdoctoral researchers.
• Within Education and Training, funding for undergraduate education activities will increase by $6 million to $160 million and funding for graduate and postdoctoral activities will increase by $22 million to $135 million within the FY 1999 Budget, including an increase of $7 million to $27 million for the Integrative Graduate Education and Research program (IGERT).
• NSF investments in support of workforce development are based on good information on demographics of the current workforce, the future needs of the public and private sectors for the workforce, and, where appropriate, on valid research on teaching and learning.

FY 1999 Areas of Emphasis:

NSF’s activities under the theme of Educating for the Future (an increase of $74 million on a base of about $400 million within the FY 1999 Budget) describe several areas of emphasis focused on developing the science and engineering workforce.

Providing opportunities for participation in integrative research and education experiences is one of NSF’s key investment strategies for addressing this outcome goal. In order to influence the development of integrated approaches, NSF has developed a number of programs intended to synergize the integration of research and education. The most prominent of these programs are the CAREER, REU, and IGERT programs. Each of these is aimed at using NSF’s close interaction with academic science and engineering to support on-going efforts within academe to draw research and education together. They are supported across the Foundation. All advisory committees will assess the activities of the directorates they advise in the CAREER and REU programs, examining results of past investments and the level and quality of current investments. IGERT is a newer program that was undertaken in response to a National Science Board recommendation that NSF experiment with alternative forms of graduate student support and training. It builds on earlier investments in the BIO and EHR directorates. All directorates will address the quality of the current set of awards. The BIO and EHR advisory committees will also assess the results of past related investments and provide guidance on NSF’s implementation of IGERT.

The three programs addressed above, along with NSF-supported centers, are key elements of NSF’s activities to increase the participation of underrepresented groups in integrative research and education experiences. Their program announcements ask proposers to address how the activity proposed will impact diversity in the science and engineering workforce, and participation by members of underrepresented groups is strong. All directorate advisory committees will be asked to include an assessment of the impact of past activities on participation of underrepresented groups and the extent to which the current portfolio can be expected to have an impact on increasing participation.

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that influence development of the science and engineering workforce, both directly and indirectly.
• Factors leading to priority for allocation of resources to directorates include potential for impact on the science and engineering workforce.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 1999 plans and use of resources are described in the main text of the performance plan. Appendix 2 lays out performance goals for investment processes that support these strategies.
• Means and strategies supporting integration of research and education and increasing diversity in the science and engineering workforce are described in Appendix 2. These strategies are particularly important in development of the workforce.
• NSF encourages participation of students on international projects. More targeted approaches to enhancing the global awareness of the science and engineering workforce are planned for future years.
• Impact on the developing workforce is an element in the merit review criteria used in the decision process.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff. Attention to all relevant merit review criteria is a factor in the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix.
• NSF provides very little of the overall U.S. investment of the development of the science and engineering workforce. Meeting the outcome goal for the long term implies a gradual change in process and philosophy of educating the workforce. Meeting the performance goal implies a commitment on the part of institutions and their faculties to enhancing the diversity of the science and engineering workforce and to providing a broader range of educational opportunities.

Cross-Cutting Areas with Other Agencies:

• Many other agencies support research and education activities in academic institutions, usually with a mission orientation. Their activities contribute to developing the workforce in science and engineering, both directly and indirectly. Most work indirectly through support of research assistants. The National Institutes of Health, Department of Education, and Department of Defense have dedicated fellowship or traineeship activities for graduate education.

Outcome Goal 4: Improved achievement in mathematics and science skills needed by all Americans

Performance Indicators: models and practices to improve achievement, teacher training, teacher classroom work, student achievement.

FY 1999 Performance Goal: NSF’s performance toward this outcome goal is

• Successful if NSF awards lead to the development, adoption, adaptation, and implementation of effective models, products, and practices that address the needs of all students; well-trained teachers who implement standards-based approaches in their classrooms; and improved student performance in participating schools and districts.
• Minimally effective if NSF awards lead to the development and adaptation of effective educational models, products, and practices; train and further develop teachers in the standards-based approaches; and prevent further deterioration in student achievement in participating schools and districts.

Baseline: There is no formal baseline at present. Preliminary efforts to pilot the use of expert judgement in performance assessment either did not address this performance goal or did so in the context of a small base of program activity. The extensive reporting and evaluation elements for Education and Human Resources (EHR) programs provide information that indicates successful progress, but this information needs further assessment before yielding a valid baseline.

Exceptionally strong performance is characterized by awards that lead to high impact development, adoption, adaptation, and implementation of effective models, products, and practices that extend beyond the period of NSF investment or beyond the scope of NSF award sites; generate novel approaches with high potential to improve and change the education process; or are associated with significant improvement in student performance.

FY 1999 Resources Supporting Achievement of the Successful Level of Performance:

• This outcome goal is primarily supported through the Education and Training key function, which increases by $66 million to $683 million in FY 1999.⁸
• NSF’s activities under the theme of Educating for the Future (an increase of $74 million on a base of about $400 million within the FY 1999 Budget) describe several areas of emphasis focused on improving achievement in mathematics and science skills.

FY 1999 Areas of Emphasis:

The FY 1999 government-wide performance plan contains two performance goals that are particularly relevant to the advisory committee assessments of performance in this area. Both are related to NSF’s systemic activities in K-12 education. At the start of the decade, NSF initiated major programs for the systemic reform of science, mathematics, engineering, and technology education. Based on the belief that all students can learn and achieve in science and mathematics at much higher levels than then obtained, systemic projects treat whole systems and build much-needed educational capacity at state, urban, rural, school district, and school levels. These projects are unique in their involvement of broad partnerships and development of comprehensive goals, solutions, and actions.

Performance Goal: Over 80 percent of schools participating in a systemic initiative program will (1) implement a standards-based curriculum in science and mathematics; (2) further professional development of the instructional workforce; and (3) improve student achievement on a selected battery of tests, after three years of NSF support.

Performance Goal: Through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences for at least 65,000 precollege teachers.

Baseline: 61,750 in 1997-98 academic year.

Both of these performance goals have quantitative forms that could be addressed without the use of the advisory committee process. However, they will be addressed through the EHR advisory committee so that there will be an assessment of quality as well.

Means and Strategies for Successful Performance:

• NSF works toward this outcome goal by using the merit review process to make awards for research and education activities that influence math and science achievement, both directly and indirectly.
• Factors leading to priority for allocation of resources to directorates include potential to improve achievement in math and science skills.
• Key investment strategies for addressing the outcome goal underlying this performance goal are found in the relevant section of the NSF GPRA strategic plan. Those particularly important to development of FY 1999 plans include systemic approaches, attention to teacher preparation and development, partnership with other agencies, and developing a strong research base for use by practitioners.
• NSF activities must be developmental and catalytic, given the small fraction of total resources in K-12 education represented by NSF’s funding.
• Quality of decision process and award portfolio is fundamental to the performance evaluation of program staff. Attention to all relevant merit review criteria is a factor in the quality of the decision process.
• NSF communicates with staff and the community with respect to outcome goals, investment strategies, and expectations for the set of awards.
• Regular reporting requirements for all awards help program staff understand the outputs and outcomes of their portfolio and provide context for decisions on new awards.
• Rigorous evaluation activities support management of all K-12 education programs.

External Factors that Affect Performance:

• General factors affecting performance are described in the introduction to this appendix.
• NSF provides very little of the overall U.S. investment in K-12 education. Meeting the outcome goal for the long term implies a gradual change in structure of education in math and science. Meeting the performance goal implies a commitment on the part of school districts, schools, and their faculty to modifying their approaches to education in order to enhance achievement and availability of resources to do so.

Cross-Cutting Areas with Other Agencies:

• The President’s Council of Advisors in Science and Technology has made recommendations with respect to establishing a strong research base for education and learning, particularly in investigation of the role of learning technologies. NSF and the Department of Education have worked to establish a joint research activity to address those recommendations.

• Results from the TIMSS report led to a call for developing an interagency action strategy to optimize the effectiveness of federal funding aimed at increasing achievement in math and science. NSF and the Department of Education shared responsibility for developing and implementing the strategy. Other agencies participated in the process, but none have extensive programmatic activities aimed at K-12 education.

Outcome Goal 5: Timely and relevant information on the national and international science and engineering enterprise

NSF's provision of information on the national and international science and engineering enterprise is a customer-oriented activity, centered in the Division of Science Resources Studies (SRS). Funding is provided through the Education and Training key function. The performance goals for this activity aim for improved quality through enhanced timeliness and enhanced attention to data quality measures. The FY 1999 Budget includes $14.50 million for SRS, an increase of $1 million over FY 1998.

Timeliness

In a recent survey, a sample of the science and engineering policy community indicated that improving timeliness of data was high priority for them.

Performance Indicator: Average time interval between reference period (the time to which the data refer) and reporting of data.

FY 1999 Performance Goal: 486 days.     Baseline: 540 days in 1995-1996.

Comment:

• Performance is measured as a two-year moving average of the number of days between the end of the data reference period and the public availability of data (usually electronic dissemination) for surveys SRS supports.
• Means for achieving success: Taking advantage of advances in information and communications technologies; regular reporting of status to give ample time to take action to improve performance.
• Cross-cutting areas with other agencies: Many other agencies provide statistical information. At intersection of responsibilities, NSF cooperates with other agencies, rather than duplicates surveys. The performance goal refers only to data reported through the eleven core surveys that SRS supports, but SRS will calculate and report comparison numbers for data it reports from surveys sponsored by other federal agencies and other organizations.
• Data are maintained by SRS.

Relevance

The value of information on the science and engineering enterprise is highly dependent on its relevance to those who seek to use it in making policy decisions. Different users are interested in different aspects of the enterprise.

Performance Indicator: Customer satisfaction ratings for the relevance of products offered.

FY 1999 Performance Goal: At least 45% "excellent" and at least 90% "excellent" or "good".

Baseline: 38% "excellent" and 88% "excellent" or "good" in a 1996 customer service survey.

Comment:

• Performance is measured as relevance ratings by users of SRS data. In 1999, information from a follow-up to the 1996 survey will form the basis for establishing achievement of this performance goal.
• Other information from key customers (science and technology policy makers and other influential data users) will permit a more complete assessment of customer satisfaction with the relevance of SRS products and will form a basis for future performance goals.
• Means for achieving success: Information from the 1996 customer service survey provides a basis for determining where improvements in relevance are needed and for modifying existing surveys as needed to gain customer satisfaction.
• Data: The customer survey will be administered by an external contractor on behalf of SRS.

Data Collection, Verification, and Validation

Most of the information base that will underlie assessments of achievement on the descriptive performance standards comes from outside the agency, through two major grantee reporting systems: the Project Reporting System which will include annual and final project reports for all awards, and the Impact Data Base and project monitoring system, designed by the Directorate for Education and Human Resources and built and maintained by a contractor. Through these systems, output indicators such as the following will be available to program staff, third party evaluators, and advisory committees:

• Related to discoveries: Results, published and disseminated: journal publications, books, audio or video products; software, newly-developed instrumentation, and other inventions; data, samples, specimens, germ lines, and related products of awards placed in shared repositories; contributions within and across disciplines.
• Related to connections: organizations of participants and collaborators (including collaborations with industry); publications; contributions to other disciplines, infrastructure, and beyond science and engineering; use beyond the research group of specific products, instruments, and equipment.
• Related to S&E workforce: student participants; demographics of participants; descriptions of student involvement; education and outreach activities under grants; and demographics of science and engineering students and workforce.
• Related to math and science skills: numbers and quality of educational models, products and practices and their use in the math and science community; volume and quality of teachers trained and faculty stimulated; student outcomes including enrollments in math and science courses, retention, achievement, and science and math degrees received.

This information is largely provided to NSF from external sources. When the data are gathered through a contractor, NSF staff will include data verification capacity in its evaluation of competing bids. The Impact Data Base and project monitoring system of the Education and Human Resources directorate is an example of a contractor-supported system.

Project reports for NSF’s research activities have been submitted largely on paper. They are subject to general review by the relevant program officer, but would be cumbersome in a wide-ranging assessment process. Thus, NSF is implementing an electronic project reporting system that will permit organized reporting of aggregate information. We anticipate that the reliability of the information in the system will improve over time, as investigators and institutions become comfortable with its use. FY 1999 will be the first year of its full implementation. Electronic submission will be required in FY 2000.

The scientific data from the reporting system will be tested for plausibility as a natural part of the external assessment process. In addition, data from the reporting system will be used to address progress under prior support when investigators reapply to NSF. Thus, the investigators have a strong interest in providing accurate information that reviewers may rely upon. NSF will work with institutions in order to establish some basic guidelines for reporting information on the demographics of participants in a reliable way.

NSF has worked with the university community to ensure that the added reporting and assessment burden will be minimal. This is important to having a viable performance plan for NSF. More direct efforts to verify and validate information in the project reporting system would add significantly to the cost and to the burden on the grantee community.

For the outcome goal on policy information, the Division of Science Resources Studies will calculate the time between collection and reporting of policy information, and their data will be open for verification and validation. The same is true for reporting on data quality measures.

NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection methods.

[Return to Table of Contents]

APPENDIX 2

Performance Goals for the NSF Investment Process

The following sections provide information about the means and strategies NSF uses in support of its outcome goals and articulates performance goals for the investment process by which NSF shapes its portfolio of awards. In FY 1999, NSF will invest $3.7 billion in these areas.

Proposal and Award Processes

NSF’s role in achieving its outcome goals for research and education is implemented through program officer selection of projects from among those submitted by the scientific community. This is a competitive process based on merit review by external peer reviewers, using criteria established by NSF. The scientists and engineers comprising NSF’s program staff take NSF priorities and the advice of the external reviewers into account in developing their portfolio of awards.

Means and strategies for high quality proposal and award processes that support achievement of the outcome goals and meet customer expectations:

• Provide staff resources needed to manage proposal and award processes (see table on page 5 in the main document for planned staff resources in FY 1999).
• Provide electronic information systems that support the processes (systems enhancements expected in FY 1999, as per FY 1999 Budget and Current Plan).
• Provide administrative guidance/requirements that reflect the imperatives of high quality processes.
• Provide needed oversight of management to ensure guidance/requirements are met.
• Provide needed operating expenses to ensure credible processes (see FY 1999 Budget and Current Plan).
• Work with the science and engineering community to provide high quality external review of NSF proposals.

Cross-cutting areas with other agencies and institutional partners:

• NSF is an active participant with other agencies in the Federal Demonstration Partnership, a joint effort of government and academe to address commonality of processes and reporting requirements to facilitate federally-funded research and education activities in academe.

Use of merit review.

Performance Indicator: Percent of NSF funds allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.

FY 2000 Performance Goal: 90%.      Baseline: 89% in FY 1997; 90% in FY 1998

Comments:

• Projects meeting the criteria for inclusion within the 90% include contracts selected through merit-based, competitive contracting procedures; FFRDCs that undergo external merit review in the process of renewal, even though they may not have direct competition; and continuing grant increments and supplements for projects reviewed in prior years.
• NSF makes a few exceptions to its general requirement for external merit review. These include situations in which objective external reviewers may be difficult to find, when natural phenomena such as volcanic eruptions or earthquakes make an external review process for proposals to study them too lengthy, or when researchers are proposing such new ideas that knowledgeable external reviewers may not exist.
• Data come from NSF’s Executive Information Systems.

Implementation of Merit Review Criteria

NSF’s generic review criteria were reviewed and revised by the National Science Board in FY 1997 for implementation in FY 1998. Implementation of those criteria is an important, immediate goal in the merit review/project selection process, in order to better support the outcome goals for research and education, as noted in NSF’s GPRA strategic plan.

Performance Indicator: Use of merit review criteria by reviewers and program staff.

FY 1999 Performance Goal: NSF performance in implementation of the new merit review criteria is

• Successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards.

• Minimally effective when reviewers consistently use only a few of the suggested elements of the generic review criteria although others might be applicable.

Comment:

• Data: NSF will use the directorates’¹ advisory committees to monitor this performance goal as they will the performance goals for research and education outcomes. The advisory committees and their subcommittees will address questions on implementation of the new merit review criteria using the samples of reviews and staff analysis that they routinely examine in their judgment of the effectiveness and fairness of the review process. As part of the request to these committees, NSF will ask that they describe in their reports the samples and evidence they used to come to their judgments.
• Means for achieving success: NSF is modifying program announcements to encourage proposers to provide information on all relevant aspects of the merit review criteria. Program staff will work with reviewers to stress the importance of addressing both merit review criteria. Performance plans for program staff will include elements related to implementation of merit review criteria.

Customer Service

In 1995, NSF adopted a set of customer service standards, primarily related to the proposal review process, treating grantees and potential grantees (applicants) as the primary customers for NSF’s administrative processes. In a survey, applicants valued three standards most highly: (1) clear guidelines for proposal content and preparation, (2) a minimum of three months between program announcements and proposal deadlines, and (3) notification of proposal funding recommendation within six months of proposal submission. The survey measured baseline levels of customer satisfaction, with reference to FY 1995 experiences. The survey will be repeated in FY 1999, referring to FY 1998 experiences. Customer service has a potential impact on the number and quality of proposals received and thus on NSF’s ability to meet all outcome goals.

For this performance plan, we focus on two of these standards, ones NSF staff have devoted special attention to since the standards were adopted.

Customer Service - Time to Prepare Proposals

Customer Service Standard: Make program announcements and solicitations available to relevant individuals and organizations at least three months prior to the proposal deadline or target date.

Performance Indicator: Percent of program announcements and solicitations available at least three months prior to proposal deadlines or target dates.

FY 1999 Performance Goal: 95%      Baseline: 80% in FY 1997; 66% in FY 1998²

Comment:

• Means for achieving success: NSF staff work toward this performance goal by limiting the number of special competitions requiring individual program announcements and solicitations, planning for such competitions as far in advance as possible, and initiating clearance processes at least six months prior to the anticipated proposal deadlines.
• Data: NSF maintains records of timing between announcement and deadline. Timing begins when the announcement is posted on NSF’s web-based Online Document System.

Customer Service - Time to Decision

Customer Service Standard: For 95 percent of proposals, be able to tell applicants whether their proposals have been declined or recommended for funding within six months of receipt.

Performance Indicator: Percent of proposals processed within six months of receipt.

FY 1999 Performance Goal: 70%     Baseline: 61% in FY 1997; 59% in FY 1998

Comment:

• For activities with deadline or target dates for receipt of proposals, the stated deadline or target date is set as the date of receipt for purposes of evaluating performance toward the standard.
• Processing proposals within six months of receipt has been a challenging goal for NSF, and we are therefore scaling annual performance targets to reach the customer service standard by FY 2002.
• This customer service standard is applicable to research project support and education and training. The proposal process for research facilities is more extended due to the nature of the projects.
• Means for achieving success: NSF staff work toward this performance goal by making effective use of electronic mechanisms in conducting the review, working cooperatively to eliminate overloads and bottlenecks, and keeping careful track of the stage of processing and age of all proposals.
• Data: Maintained in NSF’s proposal and award systems.

Award Duration

Many scientists and engineers contribute to NSF’s proposal and award system -- writing proposals, reviewing proposals, and conducting NSF-funded research. Their most productive time is the conduct of research. This would argue for providing needed resources for a sufficient duration to complete the work, thus requiring successful researchers to write fewer proposals.

Performance Indicator: Average duration of awards for research projects.

Performance Goal for FY 1999: 2.8 years      Baseline: 2.8 years in FY 1997
                                                                                           2.7 years in FY 1998

Comment:

• This performance goal is applicable only to competitive research grants.³ Originally the dataset was the key program area of research project support, but it was changed in order to be consistent with other goals.
• Adequate award size and duration are important both to getting high quality proposals and to ensuring that proposed work can be accomplished as planned. They affect progress toward all outcome goals addressed through Research Project Support.
• Means for achieving success: NSF staff work toward this performance goal by carefully examining the amount of time needed to complete the work proposed, using the guidance of reviewers as needed. They are also able to use electronic monitoring systems to keep track of average award duration and to modify their funding strategies as needed.
• Data: Maintained in NSF’s proposal and award systems.

Maintaining openness in the system.

NSF must continually keep the proposal and award process open to new people and new ideas.

Performance Indicator: Percent of competitive research grants going to new investigators

FY 1999 Performance Goal: 30%      Baseline: 27% for FY 1997; 27% for FY 1998

Comment:

• In the early 1990’s, NSF had percentages approximating 30 percent of all competitive research grants³ going to new investigators. The percentage dropped over the mid-1990’s and is now rising slightly. NSF’s FY 1999 performance goal is to raise that to 30%. Current budget levels reduce the flexiblity in meeting this goal, but NSF is committed to moving towards this goal in any case.
• Means for achieving success: Program staff will examine trends over time for both their own programs and the Foundation and establish target levels for the projects they support that are appropriate to the Foundation meeting its performance goal.
• Data: Maintained in NSF’s proposal and award systems.

Emerging Opportunities

Identifying and acting upon emerging opportunities in science and engineering is another basic strategy that supports all of the outcome goals for research and education. NSF uses many mechanisms to assist in the identification of areas of emerging opportunity: advisory committees, special workshops, interaction with elements of the National Research Council, proposal review panels, and even the proposals themselves.

Identifying emerging opportunities.

In FY 1999, NSF will introduce a mechanism for identifying emerging opportunities that takes advantage of newly expanded computer networking capabilities.

Performance Indicator: Active web sites that seek input on emerging opportunities.

FY 1999 Performance Goal: All directorates will establish Web sites for the science and engineering research and education community to provide suggestions for and comment upon emerging opportunities for significant advances in disciplinary and interdisciplinary fields of exploration; innovative research facilities; and new approaches to education and training.

Baseline: NSF has not explicitly sought input on emerging opportunities through Web sites in the past.

Comment:

• As NSF undertakes its annual planning processes, this information will help establish priorities for future funding.
• The performance goal is applicable in all key functions.
• Means for achieving success: NSF will develop common language on emerging opportunities that will be placed on all Web sites linked to the directorates. NSF will develop mechanisms to ensure that the information generated is considered in planning processes.
• Data: NSF provides quality control and maintains information on active Web sites that will be used to verify that this goal is met.

Integration of Research and Education

Integrating research and education appears as part of the investment strategies supporting all of the outcome goals for education and research as described in NSF’s GPRA strategic plan. NSF expects to see continuous improvement in the extent to which its research and education functions are accomplished jointly. The long term objective is two-fold: (1) to renew the strong interaction between federally-funded academic research and the development of the science and technology workforce that has characterized the U.S. science and engineering enterprise; and (2) to draw academic scientists and engineers into the challenge of improving K-12 education. We want to see all our awardees pay conscious attention to their effectiveness as both researchers and educators.

Means and strategies for reaching these objectives include:

• Including attention to integration of research and education as one of the elements of merit review criteria.
• Developing and increasing funding for specialized programs designed to effect the integration of research and education (Research Experiences for Undergraduates, Integrative Graduate Education and Research Training, and Faculty Early Career Development).
• Recognition awards to create a set of exemplary activities (FY 1997 and FY 1998).
• Requiring that centers and facilities use their increased visibility and size to enhance the integration of research and education.
• Meetings of awardees to obtain information on what works to share with the community.
• Working with reviewers to ensure they address these issues in reviews.
• Working with staff to ensure they address these issues in decision analyses and in developing award budgets.
• Measuring progress in proposals and reviews; in decision analyses; and in award budgets.
• Providing resources needed to undertake activities aimed at integration of research and education.

Cross-cutting activities with other agencies:

• The President issued a Presidential Review Directive in 1996 asking the National Science and Technology Council to review the government/university partnership. One preliminary conclusion of this review is the importance of strengthening the integration of research and education. NSF expects to work with other agencies as appropriate to implement the associated recommendations.
• NSF selects its candidates for the Presidential Early Career Award for Scientists and Engineers from among awardees in the Faculty Early Career Development program, one of those designed to increase integration of research and education.
• Many NSF centers and facilities have partnerships with related activities at other agencies. This provides an opportunity for other agencies to assist in and benefit from integration of research and education.

Encouraging Integration of Research and Education

Performance Indicator: Coverage of integration of research and education in NSF announcements of opportunities and proposal solicitations.

FY 1999 Performance Goal: NSF will ensure that all of its new announcements of opportunities and proposal solicitations will contain an explicit statement encouraging proposers to integrate research activities with improving education or public understanding of science.

Baseline: Selected NSF activities require statements about integration of research and education as part of the proposal. All renewal proposals are required to address the impact of the previously funded project on education. NSF has not previously explicitly addressed integration of research and education in all its activities.

Comment:

• NSF’s generic merit review criteria address the extent to which proposals will advance discovery and understanding while promoting teaching, training, and learning and vice versa. At present, many proposers do not include information that would permit reviewers to address this element.
• This is the first step in the broader strategy to accomplish the longer-term goal of embedding integration of research and education in all NSF activities.
• This performance goal is relevant to all key functions.
• Means to achieve success: NSF has developed standard language for inclusion in all announcements of opportunities and proposal solicitations. The language now appears in the Grant Proposal Guide. Clearance mechanisms will ensure that the language appears in all electronic and hard copy publications of this sort.
• Data: All proposal solicitations and announcements of opportunities are public documents. It will be possible to verify that the standard language is included.

Diversity

In 1980, legislation gave NSF explicit responsibility for addressing issues of equal opportunity in science and engineering. This assignment of responsibility reflected the serious underrepresentation of women, minorities, and persons with disabilities in the science and engineering workforce, underrepresentation that persists to this day, although some progress has been made. Recognizing that progress toward all outcome goals for research and education requires maximal diversity of intellectual thought, NSF is emphasizing attention to enhancing the participation of groups currently underrepresented in science and engineering, including women, underrepresented minorities, and persons with disabilities, in all its programs. The long term objective is to have a science and engineering workforce that mirrors the U.S. population. This performance goal applies to all key program functions.

Means and Strategies:

• Including attention to promoting diversity as one of the elements of merit review criteria.
• Developing and increasing funding for specialized programs designed to promote diversity.
• Work to embed enhancing diversity in all NSF programs.
• Focus on developing new entrants to the workforce.
• Creating a set of exemplary activities.
• Asking that centers and facilities use their increased visibility and size to promote participation of underrepresented groups and to create ties to minority serving institutions.
• Meetings of awardees to obtain information on what works to share with the community.
• Working with reviewers to ensure they address these issues in reviews.
• Working with staff to ensure they address these issues in decision analyses and in developing award budgets.
• Measuring progress in proposals and reviews; in decision analyses; and in award budgets.
• Providing resources needed to undertake activities aimed at mentoring or other methods of increasing diversity.

Cross-cutting activities with other agencies:

• NSF manages the Presidential Award for Excellence in Mentoring in Science and Engineering on behalf of the Executive Office of the President. Award ceremonies involve other agencies, and many awardees draw resources from other agencies.
• NSF plays a leading role in the Interagency Working Group on the U.S. Science and Technology Workforce of the Future of the National Science and Technology Council’s Committee on Science. The group is looking at mechanisms to optimize the effect of the federal investment in the developing workforce, with emphasis on increasing participation of underrepresented groups.
• NSF and all other agencies participate in government-wide efforts to address the needs of minority-serving institutions.
• Many NSF centers and facilities have partnerships with related activities at other agencies. This provides an opportunity for other agencies to assist in promoting diversity.

Diversity Baseline:

The 1998 Science and Engineering Indicators* provides the following data that can serve as a baseline for assessing long term progress on diversity issues. In 1995:

• Women accounted for a bit over 22% of the science and engineering workforce, 52% of the population;
• Blacks made up 3.4% of the science and engineering workforce, 12% of the population;
• Hispanics made up 2.8% of the science & engineering workforce, 9% of the population.

*pages 3-15, 3-16

Encouraging Diversity

Performance Indicator: Coverage of diversity in NSF announcements of opportunities and proposal solicitations.

FY 1999 Performance Goal: NSF will ensure that all of its new announcements of opportunities and proposed solicitations will include a statement encouraging proposers to address improving the participation of underrepresented groups in science and engineering in the course of their research and education activities.

Baseline: Selected NSF activities require statements about diversity of participation as part of the proposal. NSF has not previously addressed diversity in all its activities.

Comment:

• NSF’s generic merit review criteria address the extent to which proposals will advance diversity in the science and engineering workforce. At present, many proposers do not include information that would permit reviewers to address this element.
• NSF is working to embed attention to diversity of participation in all aspects of NSF activities. This is the first step in the broader strategy to accomplish a longer-term goal.
• This performance goal is relevant to all key functions.
• Means to achieve success: NSF has developed standard language for inclusion in all announcements of opportunities and proposal solicitations. The language now appears in the Grant Proposal Guide. Clearance mechanisms will ensure that the language appears in all electronic and hard copy publications of this sort.
• Data: All proposal solicitations and announcements of opportunities are public documents. It will be possible to verify that the standard language is included.

Facilities Oversight

The performance goals listed under this heading are applicable to the Research Facilities key program function.

The following comments are relevant to all goals in this section.

Means and Strategies:

• Careful planning ensures that construction and operating plans are realistic and contain needed contingency funds.
• NSF program officers will work closely with the project directors to ensure that the performance goals can be met.
• Where potential problems are identified, the program officer will immediately inform the NSF-wide team assigned to that project so that all appropriate actions can be taken to keep construction projects within cost and schedule and to maintain operating schedules to the extent possible.
• External factors such as extremely adverse weather or failure of partners to act as planned can have a significant effect on construction projects and operating plans.

Data:

• NSF staff have developed a common template for reporting needed information on faciities.
• This will be linked to the Project Reporting System, with project managers entering the information.

Cross-cutting Activities with Other Agencies and Countries:

• NSF consults with other agencies to optimize capabilities available to American researchers with no inappropriate duplication and cooperates with other agencies in construction of facilities where it will facilitate use across broad communities of researchers.

• NSF manages facilities in the Antarctic that are used by all federal agencies for selected projects.

• Many major facilities involve international cooperation.

Construction and Upgrade of Facilities

NSF puts a high premium on professional initial planning for construction and upgrade of facilities. But any planning for unique, state-of-the-art facilities must take into account the exploratory nature of the facilities themselves. Such facilities stretch the limits of technological capability.

Every year, in the President’s Budget Request, NSF sets out a cost plan and schedule for major construction and upgrade projects currently underway or planned for initiation in the Major Research Equipment account. NSF has established performance goals and measurements with respect to these plans and expects each construction and upgrade activity to meet these performance goals.

Performance Indicator: Comparison with planned annual cost

FY 1999 Performance Goal: Keep construction and upgrades within annual expenditure plan, not to exceed 110 percent of estimates.

Baseline: In preparation with preliminary information available Spring 1999.

Performance Indicator: Comparison with planned annual schedule

FY 1999 Performance Goal: Keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110 percent of estimates.

Baseline: In preparation with preliminary information available Spring 1999.

Performance Indicator: Comparison with planned total cost.

FY 1999 Performance Goal: For all construction and upgrade projects initiated after 1996, when current planning processes were put in place, keep total cost within 110 percent of estimates made at the initiation of construction.

Baseline: In preparation with preliminary information available Spring 1999.

Operations and Management of Facilities

Facilities must operate efficiently and reliably and must offer appropriate opportunities, if they are to be valuable to those they serve. NSF program officers work closely with facilities’ directors to ensure that facilities have appropriate resources to conduct operations and to provide maintenance that ensures reliable operations.

Performance Indicator: Comparison to scheduled operating time.

FY 1999 Performance Goal: Keep operating time lost due to unscheduled downtime to less than 10 percent of the total scheduled operating time.

Baseline: In preparation.

Comment:

• Data: Performance is measured as the average percentage among all facilities of full capacity "user units" lost during the year to breakdowns or other circumstances considered within the control of the facilities. The average across facilities is used in this instance because, although there should be latitude for some facilities to be run at greater failure rates with good reason, those facilities should be balanced by others operating more reliably. User units are defined separately for each facility, and are typically user-hours or something similar.
• Preliminary baseline performance for FY 1996, based on data from 26 facilities, is that an average of 6% of possible user units was lost. Facilities such as computer and communications networks where the concept of "full capacity user units" does not make sense are excluded from the calculation.

Data Collection, Verification, and Validation

Data in support of performance goals under the heading Proposal and Award Processes are maintained in NSF’s proposal and award systems. These systems are subject to regular checks for accuracy and reliability. One exception is the performance goal on time to prepare proposals. The budget division will maintain records for this performance goal that will be open to public scrutiny. Another exception is the performance goal on use of merit review criteria. Advisory committees will be provided with summary information developed from random samples of review records as they make their assessments. Background information to validate the accuracy of the summaries will be available upon request.

The performance goals in the categories of Integration of Research and Education, and Diversity will be supported by examination of materials in the Project Reporting System and, in the case of Diversity, comparison to reliable information on the student population in science and engineering.

Data in support of the performance goals under the heading Facilities Oversight are currently maintained by the directorates supporting the facilities, based on input from the facilities’ leadership. They will be linked to the Project Reporting System shortly. NSF verifies the accuracy and completeness of the information through constant interaction between NSF staff and the management of the facilities.

NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection methods.

[Return to Table of Contents]

APPENDIX 3

PERFORMANCE GOALS FOR MANAGEMENT

Excellence in managing the agency’s processes is an NSF goal on a par with our mission-oriented outcome goals. In the GPRA strategic plan, NSF articulated four critical factors in managing for excellence that provide the framework for annual performance goals.

• Operating a viable, credible, efficient merit review system;
• Exemplary use of and broad access to new and emerging technologies;
• A diverse, capable, motivated staff that operates with integrity; and
• Implementation of mandated performance assessment and management reforms in line with agency needs.

Performance goals related to the merit review system, given their role in NSF investment processes, have been addressed in Appendix 2. The following performance goals for FY 1999 represent key indicators that NSF is managing its centrally funded and coordinated administrative activities efficiently and effectively in support of its mission. NSF has chosen to emphasize performance goals related to implementation of information technologies and human resources development in FY 1999. These performance goals are largely accomplished through the Administration and Management key function.

New and emerging technologies

A state-of-the-art communications and technology infrastructure has been essential to NSF’s success in managing an increasing workload with approximately level resources. This investment also provides incentives in recruitment and retention of high quality employees.

Means and strategies:

• Continuing experimentation with new means to do business electronically;
• Active leadership among federal agencies in doing business electronically;
• Active leadership in government/university forums for addressing business practices;
• Appropriate use of contractors to bring needed expertise to bear on systems development;
• Testing in contained situations;
• Training for staff;
• Development of implementation plans involving all parts of NSF and issuance of important notices to institutions;
• Movement from mainframe to client server operation; and
• Movement to full implementation in cooperation with proposers, reviewers, staff.

Cross-cutting activities with other agencies:

• NSF participates in interagency activities to develop a common approach to doing business electronically.

• NSF cooperates with other agencies in the Federal Demonstration Partnership.

Electronic proposal submission

The research and education communities have worked with NSF staff to build our Web-based interface with grantee institutions, called FastLane. Each FastLane module has gone through a phase of expanding use. The most complex use of FastLane is for the submission of full technical proposals. NSF is the only federal research agency currently receiving proposals electronically on a production basis. In fact, among those proposing entities capable of it, electronic submission is the preferred mode at NSF.

Performance Indicator: Percent of full proposal submissions received electronically through FastLane.

FY 1999 Performance Goal: 25%Baseline:4.4% in FY 1997;

17.5% in FY 1998

Comment:

• Means for achieving success: The performance goal will be met through a combination of increasing user friendliness of the system and programs requiring submission through FastLane and stronger efforts to encourage such submission. Enhanced staff training will enable NSF staff to use the FastLane modules effectively. NSF will continue infrastructure investments during FY 1999 with the goal of making Fastlane fully operational as soon as possible.
• Beginning in FY 2001, NSF will require proposal submission through FastLane.
• Approximately 7,500 proposals need to be handled through this route to meet the FY 1999 goal.
• Data: The FastLane system automatically yields counts on the numbers of proposals submitted through the system. Other proposal and award systems track the total number of submissions.
• External factors: Not all NSF grantees are academic institutions. NSF works with all grantees to ensure they can interface with electronic systems. However, we must also make alternative arrangements available for grantees without the needed interface.

NSF staff

Diversity

In order to increase the diversity of the U.S. science and engineering workforce, it is particularly important that the program officers at NSF exemplify that diversity. Yet this is the segment of the staff at NSF that shows the highest levels of under-representation of women, those minority groups under-represented in the science and engineering careers, and persons with disabilities. Realistic goals for closing that gap vary from one area to another across research and education. The Division of Human Resources Management and the Office of Equal Opportunity are working with Directorates on outreach and recruiting strategies.

Performance Indicator: Efforts to attract applications from members of underrepresented groups.

FY 1999 Performance Goal: In FY 1999, as all appointments for scientists and engineers are considered, the recruiting organization will demonstrate efforts to attract applications from groups that are underrepresented in the science and engineering staff as compared to their representation among Ph.D. holders in their fields.

Comment:

• Means for achieving success: NSF will stimulate members of under-represented groups to apply for NSF’s science and engineering positions through increased outreach efforts including targeted advertising, attendance at job fairs, and site visits to minority institutions and organizations. The Division of Human Resources Management is developing guidance for outreach activities.
• Data: The recruiting organization will document their recruiting efforts or the presence of members of underrepresented groups on their lists of finalists under consideration to the Division of Human Resources Management, which will maintain the data in support of this performance goal under the guidance of the Chief Information Officer.

Capability to work with information technology

Electronic communication is changing the character of work for support, administrative, and science and engineering staff. Everyone at NSF must have good computer skills and be able to master new ones on a continuing basis. Since so much of the Foundation’s business will be done through FastLane in the future, our goal for FY 1999 focuses on that system.

Performance Indicator: Staff training on FastLane systems.

FY 1999 Performance Goal: By the end of FY 1999, all staff will receive an orientation to FastLane, and at least 95% of program and program support staff will receive practice in using its key modules.

Comment:

• Means for achieving success: A range of formal and individual orientation opportunities will be developed to support achievement of this goal as the technological ability advances. Those staff involved in testing of systems will serve as a resource for others.
• Data: The Division of Human Resources Management will maintain the data in support of this performance goal under the guidance of the Chief Information Officer.

Implementation of management reforms

Year 2000

In order to fully support its mission, NSF’s information systems must be able to withstand the problems predicted for many systems at the turn of the century. Based on guidance from OMB, NSF has developed and submitted a plan (May 1997) for evaluating, correcting, and testing its systems. Performance Indicator: Performance against the plan.

FY 1999 Performance Goal: NSF will complete all activities needed to address the Year 2000 problem for its information systems according to plan, on schedule and within budget, during FY 1999.

Comment:

• Means for achieving success: NSF’s plans comply with OMB guidelines and milestones for assessment, renovation, validation and implementation. There is a regular schedule of testing that should preclude problems. $100,000 is included in NSF’s Salaries and Expenses appropriation to complete conversion and validation of NSF’s information systems. OMB has consistently classified NSF as making satisfactory progress (Tier 3 – the highest classification given) on its Y2K activities. NSF received a grade of A from the House Committee on Government Reform and Oversight, Subcommittee on Government Management, Information, and Technology in November 1998.

• Data: External validation of NSF systems’ compliance with Y2K guidance is underway.

Project Reporting

Assessing results for NSF’s outcome goals requires a more accessible database of project results than NSF has previously maintained. A new project reporting system goes into full implementation at the beginning of FY 1999. During FY 1999, NSF will monitor the use of the system and the quality of the information gathered, and take appropriate steps to address problems.

Performance Indicator: Percent of project reports submitted through the new Project Reporting System.

FY 1999 Performance Goal: 70% Baseline: This system was not in an operational stage in FY 1998.

Comment:

• NSF must move quickly to full implementation of this system (at least 95 percent of all project reports submitted electronically -- and the remainder entered into the electronic database at NSF -- with high degrees of accuracy and completeness) so that the database represents faithfully the outputs and outcomes of NSF-funded activities.
• Means for achieving success: NSF has issued changes to its Grant Proposal Guide and Grant Policy Manual prescribing use of the format of the Project Reporting System and strongly encouraging use of the electronic version in FY 1999. We have also issued an Important Notice to grantees requiring that all project reports be submitted electronically in FY 2000.
• Data: The project reporting system and data on its use will be jointly monitored by the Division of Information Systems and the Budget Division. The information reported in the system will be used in reporting on the results of prior support when investigators reapply to NSF for further funding. This will help address issues of validity in terms of scientific elements, as will program officer perusal of annual and final progress reports and their use in annual assessment processes. We will work with universities and investigators to ensure consistent reporting of demographic information on participants.

Data Collection, Verification, and Validation

All of these performance goals are supported by internal NSF management systems.

The Division of Information Services (DIS/IRM) maintains an extensive database concerned with use of FastLane. They will continue to maintain statistics on submission of full proposals through FastLane and on submission of project reports through the still-developing project reporting system. DIS will also monitor performance on the Year 2000 performance goal. The Division of Human Resources Management (HRM/IRM) maintains information related to staff recruitment and staff training, under the guidance of the Chief Information Officer.

NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection mechanisms.

[Return to Table of Contents]