I. Introduction |
Mission |
Outcome Goals |
Critical Factors for Success |
Means and Strategies |
Deployment of Resources |
Cross-Cutting Areas with Other Agencies |
External Factors Bearing on Success |
Assessing Agency Progress toward Outcome Goals |
II. Summary Table of Performance Goals |
Appendix 1 |
Performance Goals for Results |
Appendix 2 |
Performance Goals for NSF's Investment Process |
Appendix 3 |
Performance Goals for Management |
This is NSF’s second GPRA performance plan. It is based on NSF’s GPRA strategic plan, submitted to Congress in September, 1997, and the FY 1999 performance plan. The mission, outcome goals, and critical factors for success from the strategic plan are outlined below.
NSF’s continuing mission is set out in the preamble to the National Science Foundation Act of 1950 (Public Law 810507):
To promote the progress of science; to advance the national health, prosperity, and welfare; to secure the national defense; and for other purposes.
The Act authorizes and directs NSF to initiate and support:
NSF carries out its mission primarily by making merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S.
The outcomes of NSF investments, the results stemming from the grants and cooperative agreements we make, provide the evidence for NSF’s success as an investment agent. NSF staff pursue the following outcome goals, the general goals of the strategic plan, as they develop the NSF award portfolio:
Excellence in managing the agency’s processes is an NSF goal on a par with our mission-oriented outcome goals. In the GPRA strategic plan, NSF articulated four critical factors in managing for excellence that provide the framework for annual performance goals.
NSF’s primary business is to make merit-based grants and cooperative agreements to individual researchers and groups, in partnership with colleges, universities, and other institutions -- public, private, state, local, and federal -- throughout the U.S. By providing these resources, NSF contributes to the health and vitality of the U.S. research and education system, which enables and enhances the nation’s capacity for sustained growth and prosperity. The individuals and organizations in which NSF invests conduct the work that ultimately determines the outcomes of the investment process that NSF manages.
NSF uses merit review with external peer evaluation to select about 10,000 new awards each year from about 30,000 competitive proposals submitted by the science and engineering community for its consideration. Work continues through another 10,000 awards made in competitions of previous years. NSF's role in the fabric of federal funding of science and engineering is defined by the fundamental nature of the problems our grantees propose and explore, the innovative nature of the research and education we support, and our integrative approach to research and education.
NSF’s GPRA strategic plan outlines key investment strategies and an action plan for achievement of each of the outcome goals. There are common themes running through these investment strategies, and this performance plan reflects the importance of emphasizing activities that influence achievement of multiple objectives. Common strategies include: (1) broad support for activities across science and engineering research and education using competitive merit review with peer evaluation to identify the most promising ideas from the strongest researchers and educators; (2) integrating research and education to strengthen both; (3) extending NSF’s reach to underserved communities, including enhancing the diversity of the human resource base for science and engineering; (4) emphasis on emerging opportunities, particularly those that drive science and engineering forward at disciplinary interfaces while adding to the knowledge base in areas of national interest; (5) building partnerships with other agencies and other sectors; and (6) assuring that both NSF and the research and education communities reap optimal benefit from the revolution in information, communications, and computing technologies. In addition, NSF is committed to using committees and panels of external experts to assess its effectiveness and directions on a regular basis.
NSF’s management uses information on past performance (where available) and applies the strategies for enhancing outcomes to allocate available resources. NSF staff, advised by the merit review process, select the individual projects to be supported, managing toward the optimal mix of outcomes, given the available resources. At the NSF level, response to these common strategic elements has led to the identification of broad thematic areas for investment. Three thematic areas are emphasized in the FY 2000 Request: Information Technologies, Biocomplexity in the Environment, and Educating for the Future. These areas for enhanced investment stimulate coordination and synergy outside the usual budget lines, that is, across NSF organizations and with other agencies.
Proposals and awards are managed through eight programmatic organizations, seven directorates and the Office of Polar Programs.2
NSF directorates make investments in three functional categories of activity: research projects, research facilities, and education and training. Approximately 95% of NSF’s budget goes directly to these investments. A fourth function, administration and management, provides support for the immediate activities of the agency. Each of these functions supports progress toward multiple outcome goals and is distinct in how that support is structured.
The FY 2000 Request leads to the following distribution of NSF budget resources across the key functions, with a total Request of $3.954 billion.
58% | 19% | 18% | 5% |
Research Project Support | Research Facilities |
Education & Training |
Adm. & Mgmt. |
Investments in Research Project Support fund the cutting edge research that yields new discoveries. These investments help to maintain the nation’s capacity to excel in science and engineering, particularly in academic institutions. The store of knowledge produced by NSF-funded research projects also provides a rich foundation for broad and useful applications of knowledge and the development of new technologies. Research projects contribute to the education and training of the next generation of scientists and engineers by giving them the opportunity to participate in discovery-oriented projects. NSF centers provide a particularly rich environment for broad interdisciplinary research and education at all levels. The FY 2000 Request for Research Project Support is $2,316 million.
NSF provides support for large, multi-user Research Facilities that meet the need for access to state-of-the-art research facilities that would otherwise be unavailable to academic scientists and engineers. Support includes funding for construction, upgrade, operations, and maintenance of facilities, and for staff and support personnel to assist scientists and engineers in conducting research at the facilities. Support for these unique national facilities is essential to advancing U.S. research and education capabilities, and is driven predominantly by research opportunities and priorities. Planning for facilities involves selecting from among many exciting ideas those where facilities will be initiated or upgraded. It also involves phasing down or terminating those components of facilities that no longer support cutting edge research. The FY 2000 Request for Research Facilities totals $744 million.
Investments in Education and Training help ensure an adequate, well-prepared workforce of scientists and engineers that can maintain leadership in science and technology, both now and in the future, and help all students to achieve the mathematics and science skills needed to thrive in an increasingly technological society. NSF’s programs produce scientists and engineers knowledgeable about the most recent scientific and technical advances. These highly educated people then reach every sector of society and actively disseminate and use that knowledge in the service of innumerable social goals. They also provide well-prepared teachers and instructional materials and technologies that influence mathematics and science education at all levels. The FY 2000 Request for Education and Training totals $704 million.
The FY 2000 Request for Administration and Management of $190 million provides support for salaries and benefits of persons employed at the NSF; general operating expenses, including key activities to advance the agency’s information systems technology and to enhance staff training; and audit and Inspector General activities.
NSF is exploring revision of its budget account structure to track more closely with this functional approach to use of funds. The following table describes a crosswalk for the directorates and Integrative Activities with respect to key program function and staffing levels.
Programmatic Organization / Staffing Levels / Key Function Crosswalk
(Estimated Millions of dollars)
Staff |
Key Function |
Total |
||||
FTEs & |
Research |
Research |
Education |
Administration |
||
BIO |
111 |
$ 387 |
$ 1 |
$ 16 |
$ 4 |
$ 409 |
CISE |
75 |
289 |
122 |
7 |
4 |
423 |
ENG |
129 |
345 |
3 |
25 |
5 |
379 |
GEO |
99 |
298 |
178 |
7 |
2 |
485 |
MPS |
128 |
542 |
170 |
39 |
4 |
754 |
SBE |
141 |
123 |
0 |
18 |
2 |
143 |
OPP |
52 |
62 |
185 |
1 |
3 |
251 |
IA |
0 |
161 |
0 |
0 |
0 |
161 |
EHR |
148 |
108 |
0 |
592 |
11 |
711 |
Other4 |
439 |
0 |
85 |
0 |
154 |
239 |
TOTAL5 |
1,322 |
$2,316 |
$744 |
$704 |
$190 |
$3,954 |
4 Other staff include all central administration and management. Other budget items include Major Research Equipment ($85 million, Research Facilities); Salaries and Expenses ($149 million, Adminstration and Management); and Office of Inspector General ($5 million, Administration and Management).
5 Numbers may not add due to rounding.
Just as several investment strategies support progress toward more than one outcome goal, so do the directorates and key functions. The table below provides a picture of the interactions between outcome goals and key functions. The most heavily shaded areas for each outcome goal indicate the key functions that most directly support it; the more lightly shaded areas show the key functions that influence progress toward the outcome goal less directly. By comparing this table with the crosswalk above, it is possible to determine how directorates exert their influence on attaining the outcome goals.
OUTCOME GOAL |
Research |
Research |
Education & |
Admin. & |
Discoveries at and across the frontier of science and engineering |
||||
Connections between discoveries and their use in service to society |
||||
Diverse, globally-oriented science and engineering workforce |
||||
Improved achievement in mathematics and science skills needed by all Americans |
||||
Timely and relevant information on the national and international science and engineering enterprise6 |
Many other agencies support or conduct research and education activities in science and engineering in support of their missions. Frequently they will define outcome and performance goals that are similar to those NSF has defined. However, an agency’s mission will have an impact on the nature of the outcome and performance goals, so, in general, they are not identical. The FY 2000 government-wide performance plan for research and development contains a number of common performance goals related to like sets of activities. These relate to use of merit review in the awarding of funds for research and to construction and operation of research facilities.
Appendix 4 in NSF’s GPRA strategic plan discusses cross-cutting areas with other agencies in programs and processes in support of research, education, and development of information systems. It describes the mechanisms NSF and other agencies use to cooperate in addressing the broad spectrum of activities needed to manage the federal science and technology programs while avoiding inappropriate overlap and duplication.
Appendix 1 of NSF’s GPRA strategic plan discusses external factors bearing on NSF’s ability to achieve the outcome goals. In general, these factors result from changes in the environment for the conduct of research and education activities in the federal sector, the private sector, and in academe. Similar factors have a bearing on success in meeting NSF’s performance goals. They stem largely from the fact that NSF does not conduct the research and education activities directly and, therefore, influences outcomes rather than controlling them. Where particular factors could have exceptional impact on individual performance goals, they are discussed in the appendices that accompany this performance plan.
The challenge of performance assessment for NSF is that both the substance and the timing of outcomes from research and education activities are largely unpredictable. Moreover, NSF staff do not conduct the research and education projects. They provide support for others to undertake these activities based on proposals for the work to be done, the best information available as to the likely outputs and outcomes. and their knowledge of NSF’s outcome goals and the strategies for achieving them. They influence rather than control the outputs and outcomes.
Annual outputs of NSF awards, while in a sense quantifiable and certainly important to the success of individual projects, frequently shed little light on and consistently understate progress toward outcomes. NSF staff and external experts assess the performance of individual projects in the merit review process when the principal investigator requests funds for future activities. Assessment of agency performance must integrate across the outputs of many projects and address the impact of those outputs on meeting agency outcome goals, including information on NSF’s contribution as compared to that of many other possible factors. Whether assessing an individual project for future funding or assessing performance of the agency, the collective quality of the outputs is generally more important that the quantities. Assessing quality requires expert judgment.
Thus, NSF has developed performance goals for results of NSF’s investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format. The descriptive standards characterize successful and minimally effective performance. The stream of data and information on the products of NSF’s awards provides the basis for assessing NSF’s performance through annual reporting processes and the expert judgment of independent external panels.
Much of this performance assessment is retrospective, addressing investments made at some point in the past. In order to tie this effectively to current issues in management of the programs, the assessments must also address the quality of the set of awards made in the fiscal year under consideration. The focus of this portfolio assessment is the likelihood that the package of awards will produce strong results in the future. Special emphases within the plans for the fiscal year merit special attention in the assessment process.
Another way to tie performance to current issues in management of the programs is to address NSF performance in implementing the strategies articulated in the GPRA strategic plan. NSF staff have control over budget allocations and the decision processes that determine the set of awards. NSF performance goals for investment processes, along with those for management of the agency, are generally quantitative. They refer to processes conducted during the fiscal year that are captured in NSF systems.
NSF’s annual performance goals for FY 2000 fall into three categories:
Outcome Goal | FY 2000 Annual Performance Goal | FY 2000 Areas of Emphasis across NSF |
|
NSF is successful when | NSF is minimally effective when | ||
Discoveries at and across the frontier of science and engineering | NSF awards lead to important discoveries; new knowledge and techniques, both expected and unexpected, within and across traditional disciplinary boundaries; and high-potential links across these boundaries. | there is a steady stream of outputs of good scientific quality. |
|
Connections between discoveries and their use in service to society | the results of NSF awards are rapidly and readily available and feed, as appropriate, into education, policy development, or use by other federal agencies or the private sector. | Results of NSF awards show the potential for use in service to society, and when activities designed to enhance connections between discoveries and their use in service to society meet the successful standard. |
|
A diverse, globally-oriented workforce of scientists and engineers | participants in NSF activities experience world-class professional practices in research and education, using modern technologies and incorporating international points of reference; when academia, government, business, and industry recog-nize their quality; and when the science and engineering workforce shows increased participation of underrepresented groups. | Opportunities and experiences of students in NSF-sponsored activities are comparable to those of most other students in their fields; and when the participation of underrepresented groups in NSF-sponsored science and engineering projects and programs increases. |
|
Improved achievement in mathematics and science skills needed by all Americans | NSF awards lead to the development, adoption, adaptation, and implementation of effective models, products, and practices that address the needs of all students; well-trained teachers who implement standards-based approaches in their classrooms; and improved student performance in participating schools and districts. | NSF awards lead to the development and adaptation of effective educational models, products, and practices; train and further develop teachers in the standards-based approaches; and prevent further deterioration in student achievement in participating schools and districts. |
|
Outcome Goal | FY 2000 Annual Performance Goal |
Timely and relevant information on the national and international science and engineering enterprise. | Maintain FY 1999 gains in timeliness for an average of 486 days the time interval between reference period (the time to which the data refer) and reporting of data. Baseline: 540 days in 1995-1996. |
Establish a standard set of data quality measures for reporting of Science Resource Studies products. Prepare reports on these measures for all SRS surveys and publish them in electronic formats to inform users of SRS data quality. |
8 Elements in italics are highlighted in the FY 2000 government-wide performance plan.
9 Additional information on these quantitative annual performance goals is found in Appendix 1.
Performance Area | FY 2000 Annual Performance Goal |
Proposal and Award Processes | |
Use of Merit Review | At least 90 percent of NSF funds will be allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process, maintaining the FY 1998 baseline and FY 1999 goal of 90%. |
Review Criteria 12 | NSF performance in implementation of the new merit review criteria is successful when reviewers address the elements of both generic review criteria appropriate to the proposal at hand and when program officers take the information provided into account in their decisions on awards; minimally effective when reviewers consistently use only a few of the suggested elements of the generic review criteria although others might be applicable. |
Customer service -- Time to prepare proposals |
95% of program announcements and solicitations will be available at least three months prior to proposal deadlines or target dates improving upon the FY 1998 baseline of 66%. |
Customer service -- Time to decision |
Process 75% of proposals within six months of receipt improving upon the FY 1998 baseline of 59% and FY 1999 goal of 70%. |
Maintaining Openness in the System |
NSF will maintain the percentage of competitive research grants going to new investigators to be at least 30%, as provided for in the FY 1999 Performance Plan and 3% over the FY 1998 baseline of 27%. |
11 Performance goals comparable to those in italics are highlighted in the FY 2000 government-wide performance plan.
12 This performance goal is stated in the alternative format provided for in GPRA legisltation. See Appendix 2 for the complete statement of the performance goal, including the minimally effective standard.
Performance Area | FY 2000 Annual Performance Goal |
Integration of Research and Education | |
In Proposals | At least 25% of all research grant proposals explicitly addess integration of research and education. |
In Reviews | At least 25% of all reviews of research grant proposals explicitly address the merits of proposed activities addressing the integration of research and education. |
Diversity | |
REU and IGERT awards | Participation of students from underrepresented groups in REU and IGERT awards is at least proportional to their representation in the relevant student population, and exceeds that proportion in at least 25% of the projects. |
Center activities | Participation of students from underrepresented groups in NSF centers activities is at least proportional to their representation in the relevant student population, and exceeds that proportion in at least 25% of the projects. |
Facilities Oversight | |
Construction and upgrade | Keep construction and upgrades within annual expenditure plan, not to exceed 110 percent of estimates. |
Keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110 percent of estimates. | |
For all construction and upgrade project initiated after 1996, keep total cost within 110 percent of estimates made at the initiation of construction. | |
Operations | Keep operating time lost due to unscheduled downtime to less than 10 percent of the total scheduled operating time. |
Critical Factor for Success | Performance Goal |
New and emerging technologies | |
Electronic proposal processing | NSF will receive at least 35% of full proposal submissions electronically through FastLane improving upon the FY 1998 baseline of 17%. |
By the end of FY 2000, NSF will have the technological capability of taking competitive proposals submitted electronically through the entire proposal and award/declination process without generating paper within NSF. | |
NSF Staff | |
Diversity | In FY 2000, as all appointments for scientists and engineers are considered, the recruiting organization will demonstrate efforts to attract applications from groups that are underrepresented in the science and engineering staff as compared to their representation among Ph.D. holders in their fields. |
Capability in use of information technology | By FY 2000, all appropriate staff will receive preliminary training on use of electronic proposal/award jackets. |
Implementation of management reforms | |
Year 2000 | NSF will complete all activities needed to address the Year 2000 problem for its information systems according to plan, on schedule and within budget, during FY 1999. On January 1, 2000, all mission critical systems will perform as required. |
Project Reporting System | During FY 2000, at least 85% of all project reports will be submitted through the new electronic Project Reporting System. |
The main text of this performance plan discusses the challenges to performance assessment of results of NSF research and education programs. In response the these challenges, NSF has developed performance goals for results of NSF’s investments in research and education as descriptive standards, under the GPRA option to set performance goals in an alternative format. The descriptive standards characterize successful and minimally effective performance. For each of the research and education outcome goals, there is a single alternative format performance goal that covers several elements of performance that NSF regards as most important in the current environment. In some instances there are a few related performance goals stated in a more standard format. The stream of data and information on the products of NSF’s awards provides the basis for assessing NSF’s performance through annual reporting processes and the expert judgment of independent external panels.
Much of this performance assessment is retrospective, addressing investments made at some point in the past. In order to tie this effectively to current issues in management of the programs, the assessments must also address the quality of the set of awards made in the fiscal year under consideration. The focus of this portfolio assessment is the likelihood that the package of awards will produce strong results in the future. Budget emphases for FY 2000 merit special attention in the assessment process. These special emphases are noted in the table of the main text and there is added discussion in this appendix. In most instances they refer to assessment of the likelihood that the current portfolio of awards will achieve the objectives of the special emphasis. For a few, NSF has highlighted the activity for several years, and it is possible to include some assessment of results.
Use of the alternative format performance goals for outcomes in research and education allows for human judgment to consider both quantitative and qualitative information on performance and weigh it in a balanced assessment. The descriptive performance goals are designed to be used as tools in NSF’s management process through a combination of internal self-assessment and review by external panels managed through the directorate advisory committees. Members of these committees are highly credible experts in their fields. Because of the need for the judgments of potential users and international perspectives on quality, committee membership will evolve to provide enhanced independence of judgements.
In FY 1997 and early FY 1998, NSF has run a series of experiments with advisory committees and their subcommittees to determine how to adapt existing processes2 to obtain effective assessments of outcomes. For FY 1999, all advisory committees will give judgments of program effectiveness, describing strengths and weaknesses using the target levels of performance. The credibility of these reports will rest on the provision of qualitative detail about program results, to demonstrate which of the levels of performance are actually achieved. Advisory committees will have full access to a variety of data sources.
2 NSF has a long-standing proactice of reviewing all programs on a three-year cycle for their performance in administering the merit review process. The review is performed by a Committee of Visitors (COV), usually set up as a subcommittee of a directorate advisory committee. The COV members form an independent group of external experts. COV reports are routed through the Advisory Committees to the directorates and NSF's Director. Once approved by the advisory committee, they become public documents and are available upon request. NSF is experimenting with using COVs to assess results as well as processes.
Results from NSF awards appear over time. Thus, in assessing performance toward the outcome goals during a given year, NSF must look carefully at (1) the noteworthy achievements of the year based on NSF awards, (2) the ways in which projects collectively affect progress, and (3) expectations for future performance based on the current set of awards.
As a basis for NSF’s annual performance report, each directorate will develop an annual self-assessment that addresses the three elements described above, articulating achievements, collective progress, and expectations for the future in understandable terms. The first two of these elements will provide means of assessing the performance of past NSF investments; the last, a means of assessing the characteristics of the current investments, with a focus on the likelihood of strong performance in the future. Directorate advisory committees, panels composed of experts external to NSF, will take the self-assessment and other available sources of information3 into account in benchmarking the directorate’s performance against the performance standards. The advisory committees will be expected to address both strengths and weaknesses of the directorates, including ways in which the performance is exceptionally strong. NSF management, working with the advice and guidance of the National Science Board, will integrate the advisory committee recommendations into the GPRA performance report. All of the performance goals are addressed to aggregate activities, and both advisory committee and NSF-wide reports will assess progress toward them based on the performance of collections of awards within key program functions.
Outcome Goal 1: Discoveries at and across the frontier of science and engineering
Performance Indicators: level of outputs, quality of outputs, importance of discoveries, introduction of new ideas, development of new tools and technologies, interplay of disciplinary and interdisciplinary research, balance of the portfolio.
FY 2000 Performance Goal: NSF’s performance toward this outcome goal is
Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.4
Elements of exceptionally strong performance include outcomes where NSF awards lead to advances of major scientific or engineering importance. These might, for instance: create new paradigms, stimulate important new areas of inquiry, or otherwise excite national and international attention.
4 During FY 1998, 11 of the 14 COV reviews held included new processes to assess NSF's performance. While each addressed GPRA they did not use the same mechanism and a number of reports are still outstanding. However, of the five reports reviewed to-date, performance for this goal was deemed as exceptionally strong.
FY 2000 Resources Supporting Achievement of the Successful Level of Performance:
Capital Investment to Enhance Future Performance:
Areas of Emphasis in FY 2000:
NSF will ask all directorate advisory committees to examine the directorate’s entire FY 2000 portfolio of research project support activities, identify activities they would characterize as high risk, multidisciplinary, or innovative, and make an assessment of overall scientific quality and balance with respect to these specific characteristics. The focus of this review will be to ensure that NSF is positioned well to attain the discovery-oriented performance goal. Directorates will structure their self-assessments to facilitate this review.
All directorate advisory committees will also be asked to examine the extent and appropriateness of investment in new types of scientific databases and the tools to use them. This is a critical component of activity under Knowledge and Distributed Intelligence, one of NSF’s areas of emphasis in FY 1999 and related to the IT emphasis in FY 2000.
Life in Extreme Environments, begun as a focused investment theme in FY 1997, reflects an aspect of Biocomplexity in the Environment. Preliminary results coming from awards made then may be available for assessment in FY 2000. Advisory committees will also be able to examine the active portfolio of awards for their potential influence on progress in this exciting area. Relevant directorates are BIO, GEO, MPS, and OPP.
Biocomplexity represents a new focused investment opportunity in FY 2000. An assessment of the resulting portfolio of investments will be part of the FY 2000 performance report.
In addition, Nanoscience and Engineering is just beginning to develop as an area for coordinated investment. Advisory committees in ENG and MPS will be asked to examine the set of awards in this area for their potential influence on progress.
Comment:
Means and Strategies for Successful Performance:
External Factors that Affect Performance:
Cross-Cutting Areas with Other Agencies:
Outcome Goal 2: Connections between discoveries and their use in service to society
Performance Indicators: availability of results of NSF awards to potential users, use of results of NSF awards by researchers and practitioners, role of NSF-sponsored activities in stimulating innovation and policy development.
FY 2000 Performance Goal: NSF’s performance toward this outcome goal is
Baseline: Two years of experiments in the use of expert judgment in performance assessment through Committees of Visitors indicate NSF is successful in meeting this goal.5
Exceptionally strong performance is characterized by NSF staff and grantees actively reaching out to potential users, and NSF-supported work playing critical roles in important innovation or problem solving for society.
FY 2000 Resources Supporting Achievement of the Successful Level of Performance:
Capital Investment to Enhance Future Performance:
FY 2000 Areas of Emphasis:
Within NSF’s FY 2000 guidance level, a number of multidisciplinary areas of research are identified as ripe for progress and of particular importance for their potential connections to use in service to society. Generally speaking, these fit within the Foundation’s broad themes of Information Technology (IT), Biocomplexity in the Environment (BE), and Educating for the Future (EFF). In implementing focused research activities in these areas, NSF works in partnership with other agencies. Relevant directorate advisory committees will be asked to pay particular attention to these areas in their assessments, examining results of past investments, where results are available, and the contents of the current portfolio for (1) appropriateness in meeting the goals of the interagency program; (2) quality of the NSF research and infrastructure activities; and (3) balance among related areas of activity within NSF.
6 This is an example of how NSF resources may serve multiple purposes. Achievement of this performance goal is dependent on a continuing stream of discoveries. The resources address discovery and connections simultaneously.
Providing balanced, terascale computing resources access to the academic science and engineering community is critical to educating the next generation of computer and computational scientists. As part of the Foundation’s IT theme and the Administration’s initiative, Information Technology for the 21st Century (IT2), all parts of NSF will address relevant issues in these areas. Advisory committees will review progress in developing a strong portfolio of activity.
Global Change, fitting within the broad theme of BE, has been an important area of focused research investment at NSF for several years, in conjunction with NSF’s participation in the U.S. Global Change Research Program. Advisory committees in BIO, GEO, OPP, and SBE will be asked to address performance in global change, addressing both past results and the current portfolio.
Research on Learning and Education, an element of EFF, was given high priority in the report of the President’s Committee of Advisors in Science and Technology on the Use of Technology to Strengthen K-12 Education in the United States (March 1997). NSF, in partnership with the Department of Education, will build on past investments in this area with an initiative in FY 1999 that will continue in FY 2000. While all directorates will participate in this effort, the EHR advisory committee will be asked to provide an assessment of the new investments for NSF as a whole.
Plant Genome Research received a major funding increase in FY 1998. The BIO directorate has initiated a program of support for research and infrastructure development, consistent with the recommendations of the 1998 report of the National Science and Technology Council entitled National Plant Genome Initiative. In FY 2000, the BIO advisory committee will review progress in developing a strong portfolio in this area, including interactions with other agencies, other nations, and the private sector. They will pay particular attention to completing sequencing of the model plant Arabidopsis.
Urban Communities incorporate a mix of natural and built environments. They are frequently neglected in efforts to obtain baseline information in natural, pristine environments. Yet their impact on quality of life in the U.S. is evident. BIO, ENG, and SBE play the leading roles in addressing issues related to urban environments as part of LEE.
Science and Technology Centers: Integrative Partnerships have responsibilities in the area of knowledge transfer that go beyond those of individual investigators by design. The first new set of STCs in almost a decade will be awarded in FY 2000. Assessments will address both the effectiveness of existing centers in establishing connections and the potential for strong performance in the new class.
Comment:
Means and Strategies for Successful Performance:
External Factors that Affect Performance:
Cross-Cutting Areas with Other Agencies:
Outcome Goal 3: A diverse, globally-oriented workforce of scientists and engineers
Performance Indicators: demographic data on participants in NSF-funded activities and in the workforce; character of experiences in NSF-funded activities aimed at educating the next generation of the workforce.
FY 2000 Performance Goal: NSF’s performance toward this outcome goal is
Baseline: There is no formal baseline at present. Preliminary efforts to pilot the use of expert judgment in performance assessment indicate NSF is successful in meeting this goal.7
Exceptionally strong performance is characterized by recognition of scientists or engineers who received NSF support during their training; and when the production of degree recipients in science, mathematics, and engineering increases markedly for underrepresented groups.
FY 2000 Resources Supporting Achievement of the Successful Level of Performance:
8 This is an example how NSF resources may serve multiple purposes. In general, resources provided through the Research Project Support and Research Facilities key functions are not directed at serving this goal, but they influence its achievement. Likewise, the higher education components of Education and Training serve this goal most directly, but the K-12 and informal education activities also have an impact in the quality of the students entering higher education.
FY 2000 Areas of Emphasis:
NSF’s activities under the theme of Educating for the Future (an increase of $25 million on a base of about $400 million) describe several areas of emphasis focused on developing the science and engineering workforce.
Providing opportunities for participation in integrative research and education experiences is one of NSF’s key investment strategies for addressing this outcome goal. In order to influence the development of integrated approaches, NSF has developed a number of programs intended to synergize the integration of research and education. The most prominent of these programs are the CAREER, REU, and IGERT programs. Each of these is aimed at using NSF’s close interaction with academic science and engineering to support on-going efforts within academe to draw research and education together. They are supported across the Foundation. All advisory committees will assess the activities of the directorates they advise in the CAREER and REU programs, examining results of past investments and the level and quality of current investments. IGERT is a newer program that was undertaken in response to a National Science Board recommendation that NSF experiment with alternative forms of graduate student support and training. It builds on earlier investments in the BIO and EHR directorates. All directorates will address the quality of the current set of awards. The BIO and EHR advisory committees will also assess the results of past related investments and provide guidance on NSF’s implementation of IGERT.
The three programs addressed above, along with NSF-supported centers, are key elements of NSF’s activities to increase the participation of underrepresented groups in integrative research and education experiences. Their program announcements ask proposers to address how the activity proposed will impact diversity in the science and engineering workforce, and participation by members of underrepresented groups is strong. All directorate advisory committees will be asked to include an assessment of the impact of past activities on participation of underrepresented groups and the extent to which the current portfolio can be expected to have an impact on increasing participation.
See Appendix 2 for additional information and performance goals relative to these two areas of emphasis.
Means and Strategies for Successful Performance:
External Factors that Affect Performance:
Cross-Cutting Areas with Other Agencies:
Outcome Goal 4: Improved achievement in mathematics and science skills needed by all Americans
Performance Indicators: models and practices to improve achievement, teacher training, teacher classroom work, and student achievement.
FY 2000 Performance Goal: NSF’s performance toward this outcome goal is
Baseline: There is no formal baseline at present. Preliminary efforts to pilot the use of expert judgement in performance assessment either did not address this performance goal or did so in the context of a small base of program activity. The extensive reporting and evaluation elements for Education and Human Resources (EHR) programs provide information that indicates successful progress, but this information needs further assessment before yielding a valid baseline.
Exceptionally strong performance is characterized by awards that lead to high impact development, adoption, adaptation, and implementation of effective models, products, and practices that extend beyond the period of NSF investment or beyond the scope of NSF award sites; generate novel approaches with high potential to improve and change the education process; or are associated with significant improvement in student performance.
FY 2000 Resources Supporting Achievement of the Successful Level of Performance:
FY 2000 Areas of Emphasis:
The FY 2000 government-wide performance plan contains a performance goal that is particularly relevant to the advisory committee assessments of performance in this area. It is related to NSF’s systemic activities in K-12 education. At the start of the decade, NSF initiated major programs for the systemic reform of science, mathematics, engineering, and technology education. Based on the belief that all students can learn and achieve in science and mathematics at much higher levels than then obtained, systemic projects treat whole systems and build much-needed educational capacity at state, urban, rural, school district, and school levels. These projects are unique in their involvement of broad partnerships and development of comprehensive goals, solutions, and actions.
Performance Goal: Over 80 percent of schools participating in a systemic initiative program will (1) implement a standards-based curriculum in science and mathematics; (2) further professional development of the instructional workforce; and (3) improve student achievement on a selected battery of tests, after three years of NSF support.
This performance goal has a quantitative form that could be addressed without the use of the advisory committee process. However, it will be addressed through the EHR advisory committee so that there will be an assessment of quality as well.
Through systemic initiatives and related teacher enhancement programs, NSF will provide intensive professional development experiences for at least 65,000 precollege teachers.
Research on Learning and Education, an element of EFF, was given high priority in the report of the President’s Committee of Advisors in Science and Technology on the Use of Technology to Strengthen K-12 Education in the United States (March 1997). NSF, in partnership with the Department of Education, will build on past investments in this area in FY 1999 and will continue joint activities in FY 2000. While all directorates will participate in this effort, the EHR advisory committee will be asked to provide an assessment of the new investments for NSF as a whole.
The NSF Graduate Teaching Fellows in K-12 Education places graduate and undergraduate students in K-12 schools to serve as science and mathematics content resources for teachers. This continues a FY 1999 innovative pilot effort that provides much needed expertise to support high-quality learning and also exposes graduate and undergraduate students to the needs of K-12 education. This is part of a comprehensive approach to workforce development that reaches from grade school through graduate school.
The development of the National Science, Mathematics, Engineering, and Technology Education Digital Library will be accelerated in FY 2000. This national resource will increase the quality, quantity, and comprehensiveness of internet-based K-16 educational resources. This virtual facility will link students, teachers, and faculty, and provide broad access to standards-based educational materials and learning tools for schools and academic institutions nationwide.
Means and Strategies for Successful Performance:
External Factors that Affect Performance:
Cross-Cutting Areas with Other Agencies:
Outcome Goal 5: Timely and relevant information on the national and international science and engineering enterprise.
NSF's provision of information on the national and international science and engineering enterprise is a customer-oriented activity, centered in the Division of Science Resources Studies (SRS). Funding is provided through the Education and Training key function. The performance goals for this activity aim for improved quality through enhanced timeliness and enhanced attention to data quality measures.
Timeliness
In a recent survey, a sample of the science and engineering policy community indicated that improving timeliness of data was high priority for them.
Performance Indicator: Average time interval between reference period (the time to which the data refer) and reporting of data.
FY 2000 Performance Goal: 486 days. Baseline: 540 days in 1995-1996.
Comment:
Data Quality
The value of information on the science and engineering enterprise is highly dependent on its ability to address issues of importance to those who seek to use it in making policy decisions. Measures of data quality help users determine the reliability of the information and the extent of likely variance introduced by sampling processes.
Performance Indicator: Data quality measures and their use in SRS products.
FY 2000 Performance Goal: Establish a standard set of data quality measures for reporting of Science Resources Studies products. Prepare reports on these measures for all SRS surveys and publish them in electronic formats to inform users of SRS data quality.
Baseline: This is a new effort to provide standard measures. Their absence places limits on the usefulness of surveys.
Comment:
Most of the information base that will underlie assessments of achievement on the descriptive performance standards comes from outside the agency, through two major grantee reporting systems: the Project Reporting System which will include annual and final project reports for all awards, and the Impact Data Base and project monitoring system, designed by the Directorate for Education and Human Resources and built and maintained by a contractor. Through these systems, output indicators such as the following will be available to program staff, third party evaluators, and advisory committees:
This information is largely provided to NSF from external sources. When the data are gathered through a contractor, NSF staff will include data verification capacity in its evaluation of competing bids. The Impact Data Base and project monitoring system of the Education and Human Resources directorate is an example of a contractor-supported system.
Project reports for NSF’s research activities have been submitted largely on paper. They are subject to general review by the relevant program officer, but would be cumbersome in a wide-ranging assessment process. Thus, NSF is implementing an electronic project reporting system that will permit organized reporting of aggregate information. We anticipate that the reliability of the information in the system will improve over time, as investigators and institutions become comfortable with its use. FY 1999 will be the first year of its full implementation. Electronic submission will be required in FY 2000.
The scientific data from the reporting system will be tested for plausibility as a natural part of the external assessment process. In addition, data from the reporting system will be used to address progress under prior support when investigators reapply to NSF. Thus, the investigators have a strong interest in providing accurate information that reviewers may rely upon. NSF will work with institutions in order to establish some basic guidelines for reporting information on the demographics of participants in a reliable way.
NSF has worked with the university community to ensure that the added reporting and assessment burden will be minimal. This is important to having a viable performance plan for NSF. More direct efforts to verify and validate information in the project reporting system would add significantly to the cost and to the burden on the grantee community.
For the outcome goal on policy information, the Division of Science Resources Studies will calculate the time between collection and reporting of policy information, and their data will be open for verification and validation. The same is true for reporting on data quality measures.
NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection methods.
The following sections provide information about the means and strategies NSF uses in support of its outcome goals and articulates performance goals for the investment process by which NSF shapes its portfolio of awards. In FY 2000, NSF will invest $3.9 billion in these areas.
Proposal and Award Processes
NSF’s role in achieving its outcome goals for research and education is implemented through program officer selection of projects from among those submitted by the scientific community. The scientists and engineers comprising NSF’s program staff take NSF priorities and the advice of the external reviewers into account in developing their portfolio of awards.
Means and strategies for high quality proposal and award processes that support achievement of the outcome goals and meet customer expectations:
Cross-cutting areas with other agencies and institutional partners:
Use of merit review.
Performance Indicator: Percent of NSF funds allocated to projects reviewed by appropriate peers external to NSF and selected through a merit-based competitive process.
FY 2000 Performance Goal: 90%. Baseline: 89% in FY 1997; 90% in FY 1998.
Comments:
Implementation of Merit Review Criteria
NSF’s generic review criteria were recently reviewed and revised by the National Science Board. Implementation of those criteria is an important, immediate goal in the merit review/project selection process, in order to better support the outcome goals for research and education, as noted in NSF’s GPRA strategic plan.
Performance Indicator: Use of merit review criteria by reviewers and program staff.
FY 2000 Performance Goal: NSF performance in implementation of the new merit review criteria is
Baseline: New criteria went into effect in FY 1998. External expert judgment will assess performance. The process will be used for the first time during FY 1999.
Comment:
Customer Service
In 1995, NSF adopted a set of customer service standards, primarily related to the proposal review process, treating grantees and potential grantees (applicants) as the primary customers for NSF’s administrative processes. In a survey, applicants valued three standards most highly: (1) clear guidelines for proposal content and preparation, (2) a minimum of three months between program announcements and proposal deadlines, and (3) notification of proposal funding recommendation within six months of proposal submission. The survey measured baseline levels of customer satisfaction, with reference to FY 1995 experiences. The survey will be repeated in FY 1999, referring to FY 1998 experiences. Customer service has a potential impact on the number and quality of proposals received and thus on NSF’s ability to meet all outcome goals.
For this performance plan, we focus on two of these standards; ones NSF staff have devoted special attention to since the standards were adopted.
Customer Service - Time to Prepare Proposals
Customer Service Standard: Make program announcements and solicitations available to relevant individuals and organizations at least three months prior to the proposal deadline or target date.
Performance Indicator: Percent of program announcements and solicitations available at least three months prior to proposal deadlines or target dates.
FY 1999 Performance Goal: 95% Baseline: 80% in FY 1997; 66% in FY 19982
Comment:
Customer Service - Time to Decision
Customer Service Standard:For 95 percent of proposals, be able to tell applicants whether their proposals have been declined or recommended for funding within six months of receipt.
Performance Indicator: Percent of proposals processed within six months of receipt.
FY 2000 Performance Goal: 75% Baseline: 61% in FY 1997; 59% in FY 1998
Comment:
Maintaining openness in the system.
NSF must continually keep the proposal and award process open to new people and new ideas.
Performance Indicator: Percent of competitive research grants going to new investigators
FY 2000 Performance Goal: 30% Baseline: 27% for FY 1997; 27% for FY 1998
Comment:
Integration of Research and Education
Integrating research and education appears as part of the investment strategies supporting all of the outcome goals for education and research as described in NSF’s GPRA strategic plan. NSF expects to see continuous improvement in the extent to which its research and education functions are accomplished jointly. The long term objective is two-fold: (1) to renew the strong interaction between federally-funded academic research and the development of the science and technology workforce that has characterized the U.S. science and engineering enterprise; and (2) to draw academic scientists and engineers into the challenge of improving K-12 education. We want to see all our awardees pay conscious attention to their effectiveness as both researchers and educators.
Means and strategies for reaching these objectives include:
Cross-cutting activities with other agencies:
Proposer Attention to Integration of Research and Education
NSF’s generic merit review criteria address the extent to which proposals will advance discovery and understanding while promoting teaching, training, and learning and vice versa. At present, many proposers do not include information that would permit reviewers to address this element. To embed integration of research and education in all aspects of NSF activities NSF now has language in all program announcements encouraging proposers to discuss this aspect of their activity in proposals (FY 1999 performance goal).
Performance Indicator: Percent of research grant proposals explicitly addressing integration of research and education.
FY 2000 Performance Goal: 25% Baseline: Under development.
Comment:
Reviewer Attention to Integration of Research and Education
Once proposers include information on their plans for integration of research and education, reviewers will be able to address those plans in their reviews.
Performance Indicator: Percent of reviews of research grant proposals that explicitly address the merits of proposed activities addressing the integration of research and education.
FY 2000 Performance Goal: 25% Baseline: Under development.
Comment:
Diversity
In 1980, legislation gave NSF explicit responsibility for addressing issues of equal opportunity in science and engineering. This assignment of responsibility reflected the serious underrepresentation of women, minorities, and persons with disabilities in the science and engineering workforce, underrepresentation that persists to this day, although some progress has been made. Recognizing that progress toward all outcome goals for research and education requires maximal diversity of intellectual thought, NSF is emphasizing attention to enhancing the participation of groups currently underrepresented in science and engineering, including women, underrepresented minorities, and persons with disabilities, in all its programs. The long-term objective is to have a science and engineering workforce that mirrors the U.S. population. This performance goal applies to all key program functions.
Means and Strategies:
Cross-cutting activities with other agencies:
Diversity Baseline:
The 1998 Science and Engineering Indicators* provides the following data that can serve as a baseline for assessing long term progress on diversity issues. In 1995:
*Pages 3-15, 3-16
Diversity in Programs to Integrate Research and Education
Student-oriented programs designed to integrate research and education provide a focused opportunity to embed consideration of diversity in core NSF programming. Both REU sites and IGERT projects involve several students, so principal investigators can plan to develop a pool of potential participants that includes members of underrepresented groups.
Performance Indicator: Participation of students from underrepresented groups on REU and IGERT awards.
FY 2000 Performance Goal: At least proportional to their representation in the relevant student population, exceeding that proportion in at least 25% of the projects.
Baseline: In preparation.
Comment:
Performance Indicator: Participation of students from underrepresented groups in center awards.
FY 2000 Performance Goal: At least proportional to their representation in the relevant student population, exceeding that proportion in at least 25% of the individual centers.
Baseline: In preparation.
Comment:
Facilities Oversight
The performance goals listed under this heading are applicable to the Research Facilities key program function.
The following comments are relevant to all goals in this section.
Means and Strategies:
Data:
Cross-cutting Activities with Other Agencies and Countries:
Construction and Upgrade of Facilities
NSF puts a high premium on professional initial planning for construction and upgrade of facilities. But any planning for unique, state-of-the-art facilities must take into account the exploratory nature of the facilities themselves. Such facilities stretch the limits of technological capability.
Every year, in the President’s Budget Request, NSF sets out a cost plan and schedule for major construction and upgrade projects currently underway or planned for initiation in the Major Research Equipment account. NSF has established performance goals and measurements with respect to these plans and expects each construction and upgrade activity to meet these performance goals.
Performance Indicator: Comparison with planned annual cost
FY 2000 Performance Goal: Keep construction and upgrades within annual expenditure plan, not to exceed 110 percent of estimates.
Baseline: In preparation.
Performance Indicator: Comparison with planned annual schedule
FY 2000 Performance Goal: Keep construction and upgrades within annual schedule, total time required for major components of the project not to exceed 110 percent of estimates.
Baseline: In preparation.
Performance Indicator: Comparison with planned total cost.
Performance Goal: For all construction and upgrade projects initiated after 1996, when current planning processes were put in place, keep total cost within 110 percent of estimates made at the initiation of construction.
Baseline: In preparation.
Operations and Management of Facilities
Facilities must operate efficiently and reliably and must offer appropriate opportunities, if they are to be valuable to those they serve. NSF program officers work closely with facilities’ directors to ensure that facilities have appropriate resources to conduct operations and to provide maintenance that ensures reliable operations.
Performance Indicator: Comparison to scheduled operating time.
Performance Goal: Keep operating time lost due to unscheduled downtime to less than 10 percent of the total scheduled operating time.
Baseline: In preparation.
Comment:
Data in support of performance goals under the heading Proposal and Award Processes are maintained in NSF’s proposal and award systems. These systems are subject to regular checks for accuracy and reliability. One exception is the performance goal on time to prepare proposals. The budget division will maintain records for this performance goal that will be open to public scrutiny. Another exception is the performance goal on use of merit review criteria. Advisory committees will be provided with summary information developed from random samples of review records as they make their assessments. Background information to validate the accuracy of the summaries will be available upon request.
The performance goals in the categories of Integration of Research and Education, and Diversity will be supported by examination of materials in the Project Reporting System and, in the case of Diversity, comparison to reliable information on the student population in science and engineering.
Data in support of the performance goals under the heading Facilities Oversight are currently maintained by the directorates supporting the facilities, based on input from the facilities’ leadership. They will be linked to the Project Reporting System shortly. NSF verifies the accuracy and completeness of the information through constant interaction between NSF staff and the management of the facilities.
NSF management will consult with the Office of Inspector General on appropriate mechanisms to verify the data and validate the collection methods.