Title: A Report on the Evaluation of the National Science Foundation’s Experimental Program to Stimulate Competitive Research Date: June 4, 1999 A Report on the Evaluation of the National Science Foundation’s Experimental Program to Stimulate Competitive Research Evaluation of the EPSCoR Program Prepared under Contract RED 94-52970 by COSMOS Corporation MAY 1999 Mary Sladek NSF Program Officer ----------------------------------------------------------------------- Any opinions, findings, conclusions, or recommendations expressed in this report are those of the participants, and do not necessarily represent the official views, opinions, or policy of the National Science Foundation. The National Science Foundation Directorate for Education and Human Resources AN REC-SPONSORED Division of Research, Evaluation and Communication REPORT ON EVALUATION PREFACE This document presents COSMOS Corporation’s final evaluation report of the National Science Foundation’s Experimental Program to Stimulate Competitive Research (EPSCoR). The evaluation was conducted under Task Order 1, Contract No. RED 94-52970. Conrad Katzenmeyer is the contracting officer’s technical representative. Dr. Susan Gross, succeeded by Mary Sladek, was the program officer for the evaluation, conducted as a task order under the contract. The evaluation team was led by Drs. Robert K. Yin and Irwin Feller, co-project directors. Other contributors included Cheryl Sattler, Dana Edwards, Lynne Adduci, Lee Carpenter, Meg Gwaltney, Tasanee Ross-Sheriff, Lynette Tucker, and Jennie Heard (editor). In addition, Professors Patricia Gumport (Associate Professor of Education, Stanford University) and Edward Hackett (Associate Professor, Department of Science and Technology Studies, Rensselaer Polytechnic Institute) prepared commissioned background papers to augment the evaluation design materials. A workgroup, comprising individuals with extensive participation in the several facets of the EPSCoR program, also provided insights to the evaluation team about the workings of the EPSCoR program. The members of this workgroup were Dr. Judith Bailey, University of Maine; Dr. Colin Bennett, University of South Carolina; Dr. Philip Boudjouk, North Dakota State University; Dr. Hans Brisch, Oklahoma State Regents for Higher Education; Dr. Daryle Busch, University of Kansas; Dr. Delwood Collins, University of Kentucky; Dr. Collis R. Geren, University of Arkansas; Dr. Randolph V. Lewis, University of Wyoming; Dr. Ralph Powe (deceased), Mississippi State University; and Dr. Barbara Wright, University of Montana. An advisory committee to the project also provided critical oversight and constructive recommendations to preliminary versions of the evaluation plan. The members of the committee were Dr. James R. Durig, University of Missouri at Kansas City; Dr. William Massy, Stanford University; Dr. Norine Noonan, Florida Institute of Technology; Dr. Karen Seashore Louis, University of Minnesota; and Dr. Reginald Wilson, American Council on Education. Dr. Daryl Chubin of NSF also participated in the committee meetings and provided policy insights and guidance concerning the evaluation from its inception. The evaluation team especially acknowledges Dr. Chubin’s contributions. Throughout the evaluation project, the COSMOS team benefited from the assistance of Richard Anderson, Bruce Reiss (retired), and Jim Hoehn, NSF-EPSCoR program officers who gave generously of their time and expertise in responding to numerous questions about EPSCoR’s history, current operations, and databases. Additional assistance has been provided by Brandon Cushing and Tom Trumbull, Quantum Research Corporation, who made available data on the proposal submission activity of EPSCoR faculty. Finally, an anonymous peer reviewer made substantial comments on an earlier version of the report, markedly helping to improve the presentation of the findings. The assistance of others does not remove the authors’ full and final responsibility for this report. EXECUTIVE SUMMARY The EPSCoR Program. NSF started EPSCoR--the Experimental Program to Stimulate Competitive Research--in 1978, with investments through 1996 totalling $182.2 million in grants and cooperative agreements. The program aimed at stimulating competitive research in those states traditionally receiving low percentages of federal R&D support. As found in NSF’s basic statutory language, the program goal was to "avoid the undue concentration" of R&D in the United States. At first, five states were the target of EPSCoR efforts, but by 1992 the program involved 18 states and Puerto Rico: Alabama, Arkansas, Idaho, Kansas, Kentucky, Louisiana, Maine, Mississippi, Montana, Nebraska, Nevada, North Dakota, Oklahoma, South Carolina, South Dakota, Vermont, West Virginia, and Wyoming. In each state, the bulk of EPSCoR’s funds was devoted to the support of research in the state’s university campuses. Because 53 of the 56 participating campuses were publicly supported universities--and, in part, because EPSCoR funding had a $1-to-$1 matching requirement--over the years EPSCoR has deliberately pursued long-term partnerships with state leaders in government, business, and higher education. The EPSCoR program has continued to this day. The EPSCoR evaluation mainly covers the period 1980 to 1994 but not programmatic developments since then. EPSCoR States’ Main Outcome: Shares of R&D. During the 1980-1994 period, the evaluation found that the EPSCoR states’ share of federal R&D did increase, from an average of .25 percent per state to .40 percent per state. (Overall, the EPSCoR states’ share of federal R&D funding represented 7.65 percent, totalling $960 million, by 1994.) The increase in share was a modest but positive outcome; in contrast, during the same period the non-EPSCoR states’ shares declined. Although the changes were small in absolute terms, they occurred in an era when universities in many (non-EPSCoR) states were dramatically expanding their R&D capabilities. Further, a similarly favorable outcome was found when examining changes in the EPSCoR states’ shares of NSF-funded research. The EPSCoR Program’s Contribution to the Outcome. Whether and how the EPSCoR program contributed to these increases in states’ shares was the subject of a major part of the evaluation. The evaluation found that a plausible argument can be made that EPSCoR contributed to the outcome. The program followed a deliberately crafted and multifaceted strategy, and did not just operate as a special "set-aside" of funds for investigators from the eligible states. Included in this strategy during some or all of the 1980-1994 period were the following features: * Building a state’s capabilities by supporting proposals from promising investigators whose proposals were rated "good" to "very good," and not just more senior investigators whose proposals were already scientifically excellent; * Requiring a $1-to-$1 match, satisfied only by newly-appropriated funds, not in-kind matches, and thereby making EPSCoR a visible and explicit investment by state government; * Requiring each state to form an EPSCoR steering committee, consisting of diverse members from academe, industry, and state government--including a state’s best and most senior scientific and technological personnel--and working to blend EPSCoR’s priorities with the state’s science and technology (S&T) goals; * Supporting groups of related research projects within and across universities (research clusters)--rather than single investigators--as a means of promoting long-term, sustainable increases in institutional research competitiveness; * Promoting the research capability of universities, by supporting the acquisition of state-of-the-art instrumentation and equipment and offering start-up funding for new faculty; and * Promoting cross-campus collaboration, thereby producing opportunities for distinctive lines of research that added to a state’s competitive capability. In short, the EPSCoR program sought to create systemic change within a state’s R&D and university infrastructure, thus building the capability for long-term and sustainable improvement in the state’s ability to compete for subsequent R&D funding. Based on in-depth studies of five representative EPSCoR states (and covering 14 separate campuses within these states), the evaluation tracked relevant changes in the states, such as a state’s ability to set R&D priorities and link these interests with strengthened within-state peer review processes. In each state, the evaluation showed how EPSCoR did stimulate such developments. Similarly, the evaluation investigated changes in the universities’ research capabilities. Illustrative accomplishments attributable to EPSCoR were: * The creation of one of the major DNA protein facilities in the world, with genetic materials and histories useful for forensics work; * New faculty in chemistry and a stronger optics research group; * University-industry collaboration on polymers; and * The building of a surface hydrology group that also involved a unique, statewide system for monitoring climatic conditions, soil moisture, water quality, and ultraviolet atmospheric conditions for research. The evaluation also made a special inquiry into the entire universe of research projects funded by EPSCoR from 1992 to 1996. These projects produced high numbers of subsequent publications and new R&D funding, in this sense yielding evidence of scientific productivity and therefore a plausible argument for increased competitiveness. However, from the broader systemic perspective of creating university-wide actions and policies, EPSCoR’s influence was found to be limited, with the major effects being the stimulation of cross-campus collaboration and the initiation of competitive start-up packages to attract top faculty, especially in the sciences. Implications for EPSCoR Policy and Program Operations. The evaluation results have the following implications for EPSCoR’s ongoing policy and program operations. First, a continuing challenge is for EPSCoR to "stimulate competitive research," and not necessarily to support research that already may be rated "excellent" by NSF reviewers. Meeting this challenge means that EPSCoR needs to fund proposals that may be rated "good" to "very good," but identifying these proposals is not easy. Reviewers sometimes rate proposals as "excellent" because they mean "excellent for EPSCoR," and not scientifically "excellent." Continued guidance on review criteria and how to use specific terms is, therefore, needed. Second, the evaluation has shown that states’ R&D competitiveness has improved, and that EPSCoR has contributed to this competitiveness. However, no criteria have been established for defining when a state has been deemed sufficiently successful that it should "graduate" from EPSCoR. One option is for the EPSCoR program to reassess the current eligibility of all EPSCoR states, using the existing eligibility criteria.* Such an assessment could be conducted on an annual or other cyclic basis. Third, a key EPSCoR strategy has been the program’s support of "clusters" of related research projects--especially across disciplines and campuses--rather than single and unrelated research projects. EPSCoR had at first supported such individual projects, only to find that many investigators were later recruited away from universities in EPSCoR states. The clusters, in contrast, have created institutional capabilities less vulnerable to the movement of individual investigators. Whether and how the cluster strategy can be continued, therefore, warrants ongoing attention. ____________________ *The criteria cover highly relevant conditions, such as a state’s ranking among all states and its amount of R&D funding per academic scientist and engineer in the state. TABLE OF CONTENTS Preface Executive Summary A. The EPSCoR Program and Its Evaluation 1. What Is the EPSCoR Program? 2. What Is the EPSCoR Evaluation? 3. What Were the Evaluation’s Main Conclusions? (A Preview) B. Evaluation Findings 4. What Was EPSCoR’s Funding Practice? 5. Was EPSCoR Associated with Changes in Academic R&D Funding? 6. To What Extent Did EPSCoR Influence University-Government-Industry Relationships, University Research Infrastructures, and Research Competitiveness? C. Policy and Program Implications 7. What Are the Evaluation’s Implications for EPSCoR Policy and Program Operations? Exhibits 1. Hypothesized Program Events in EPSCoR States 2. Percent Share of Academic R&D Expenditures, by Sector 3. Shares of Federal Obligations for Academic Research, by Agency 4. EPSCoR States’ Share of Federal Academic R&D Expenditures 5. EPSCoR States’ Share of NSF Obligations 6. Percentage Changes in Federal Academic R&D, by Individual EPSCoR States, 1980-1984 7. Scientific Publications, by Cluster 8. Distribution of New Award Dollars, by Cluster References SECTION A: The EPSCoR Program and Its Evaluation 1. What Is the Since its creation in 1950, the National Science EPSCoR Program? Foundation (NSF) has sought to advance research and education in science, mathematics, engineering, and technology throughout the United States. Explicit in its original mandate was a cautionary warning to avoid concentrating federal funding of academic research and development (R&D)1 on a geographic basis. Thus, NSF’s authorizing legislation, as amended (42 U.S.C. 1862, Sec. 3e), stated: In exercising the authority and discharging the functions referred to in the foregoing subsections, it shall be an objective of the Foundation to strengthen research and education in the sciences and engineering, including independent research by individuals, throughout the United States, and to avoid undue concentration of such research and education. [Emphasis added.] In 1978, congressional concern about the geographical concentration of federal funds for academic research led Congress to authorize the NSF to conduct the Experimental Program to Stimulate Competitive Research (EPSCoR) (Greenberg, 1967; U.S. Congress, Office of Technology Assessment, 1991; and Martino, 1992). Eligibility for EPSCoR participation was restricted to states that historically had received relatively low levels of federal R&D funding. Additionally, such states were required to demonstrate a commitment to improving the quality of university-based research in science, mathematics, engineering, and technology. EPSCoR’s Mission EPSCoR, as the program’s name reflects, ...enhancing was designed to stimulate competitive the research--and not to be a special capability set-aside program (EPSCoR Program of eligible Solicitation, 1989b). Through EPSCoR, NSF states to would become a partner with participant compete for state governments and universities. The research role of state government was seen as funds. especially important in achieving the EPSCoR program’s objectives because of the overwhelming importance of public universities within the higher education systems of EPSCoR states. In 1995, for example, 53 of the 56 universities participating in the EPSCoR program were publicly supported universities. Over time, NSF’s goals for the EPSCoR program broadened to include enhancement of educational and human resource opportunities for underrepresented faculty and student populations in science, mathematics, engineering, and technology, and the transfer of academic research to the private sector. EPSCoR’s Program Strategy EPSCoR’s program strategy is designed to EPSCoR: A increase the competitiveness of multifaceted merit-reviewed proposals from program investigators in states that have strategy historically received low percentages of federal R&D support. The assumption is that increased competitiveness will subsequently lead to increased R&D funding in these states. Further, on the premise that university science and engineering departments can positively influence a state’s economic and human resource development, EPSCoR pursues long-term partnerships with state leaders in government, business, and higher education (Malecki, 1991; and Luger and Goldstein, 1991). In pursuit of these goals, EPSCoR applies the following strategies: 1. Support research investigators in states with the lowest share of R&D funds per scientist and population (18 states and Puerto Rico are eligible, according to the criteria defined by the program in 1991); 2. Exclude, even in these states, proposals rated "excellent" and thus already considered competitive according to NSF’s traditional peer review rating system; 3. Support research projects and explicit capacity-building strategies aimed at increasing the research competitiveness of the whole state (the capacity-uilding aspect was critical because of the long-term expectation of increased research funding, going beyond support from the EPSCoR program itself); and 4. Support groups of related research projects within and across universities (research clusters)--rather than single investigators--as a means of promoting long-term, sustainable increases in institutional research competitiveness. Although the strategies appear to focus on...systemic individual or groups of research projects,and EPSCoR’s vision from the onset was sustainable systemic. The program always has sought tochange... strengthen the entire science and technology (S&T) infrastructure in an eligible state--thereby increasing the ability of EPSCoR researchers to compete for federal and private sector R&D funding and accelerating the movement of EPSCoR researchers and institutions into the mainstream of federal and private sector R&D support. Non-Federal Cost Sharing The EPSCoR partnership between NSF and eligible states requires investment by both partners. A one-to-one dollar match was required for the awards made from 1980 to 1994. Infrastructure Improvement In preparing to submit a proposal, an EPSCoR steering committee within each eligible state was expected to have undertaken a comprehensive analysis of the strengths and opportunities for developing its research institutions in support of the state’s overall R&D objectives. Examples of infrastructure activities EPSCoR has funded include the following: * Start-up funding for new faculty, including seed funding of faculty research leading to the submission of competitive grant proposals; * Faculty exchange programs with major research centers; * Acquisition of state-of-the-art research instrumentation and development of nationally competitive high-performance computing and networking capabilities; * Partnerships between the state’s research universities and the private sector; * Innovations in graduate education that will expand student career options and facilitate the entry of individuals from traditionally underrepresented groups (that is, African-Americans, Hispanics, Native Americans, Pacific Islanders, women, and the physically disabled) into high-demand S&T fields; * Funding of senior faculty to 1) work with newly developing S&T businesses; 2) serve as policy advisors for state legislatures and S&T agencies; 3) serve as senior postdoctoral associates in established department- or institution-wide research programs; 4) develop new educational technologies and delivery systems; or 5) create new career alternatives for young scientists; and * Creation of graduate research training groups, or similar appropriate mechanisms, that integrate education and research, encourage multi-disciplinary educational experiences, or establish links with industry and national laboratories. The NSF initiated the EPSCoR program in "...18 Fiscal Year (FY) 1978 with seven planning states and grants. From that time through FY1996, thePuerto program awarded $182.2 million in grants Rico..." and cooperative agreements. Participating in the program by the end of the period were 18 states and the Commonwealth of Puerto Rico. The states were Alabama, Arkansas, Idaho, Kansas, Kentucky, Louisiana, Maine, Mississippi, Montana, Nebraska, Nevada, North Dakota, Oklahoma, South Carolina, South Dakota, Vermont, West Virginia, and Wyoming. By 1994, EPSCoR had been operational for over 15 years and the NSF decided that, as part of its plan to evaluate the programs of the Directorate for Education and Human Resources, the time had come to look at whether the EPSCoR program was achieving its intended outcomes. This evaluation of the EPSCoR program covers the period 1980-1994, with 1994 as the last year for which R&D expenditures were available at the time of the evaluation. The evaluation thus does not cover changes or developments in the EPSCoR program after 1995. ____________________ 1 The term "academic R&D" is used throughout to distinguish research funding at universities--the main concern of NSF’s original congressional mandate--from a state’s entire R&D funding, which includes industry, government laboratories, and other activities well beyond NSF’s mandate or reach. Wherever "R&D" is used, the intention is to limit its coverage to academic R&D. 2. What Is the The EPSCoR evaluation--conducted by COSMOS 5 EPSCoR Corporation--was designed with two objectives in mind: Evaluation? 1. to determine whether EPSCoR had an influence on reducing the pre-existing geographical concentration of federal R&D funds--that is, whether participating states and universities had increased their share of federal academic research funds over a 14-year period (1980-1994) and, if so, 2. to identify the EPSCoR program strategies responsible for improving state government and university competitiveness in acquiring the additional research support. The Evaluation Team The design and conduct of the evaluation were carried out by a COSMOS research team in consultation with a variety of experts. First, the team convened an EPSCoR Workgroup, consisting of EPSCoR leaders, to provide advice to the evaluation. Next, the team met (twice) with an expert group of advisors, who discussed the design and preliminary findings. Finally, the team commissioned two special papers by two additional experts (Gumport,1996; and Hackett, 1996) to serve as background to the entire evaluation design. 2 The Evaluation Design EPSCoR had deliberately selected the states with the lowest R&D activity in the United States. Consequently, a control group design could not be implemented by the evaluation. An alternative design was to test a causal model of EPSCoR using a theory-based model. The theory-based approach could not yield the same degree of certainty as a control group design. However, the extent to which the model could be supported by empirical evidence and alternative explanations ruled out, the more confidence the NSF, Congress, and other stakeholders would have in the model’s specifications regarding claims for EPSCoR’s effects on R&D funding outcomes. The EPSCoR model investigated by COSMOS was specified as a series of hypothesized causal links depicted in Exhibit 1, which presupposes the following critical steps: * The EPSCoR-funded research projects should be at a level verging on national competitiveness but not yet nationally competitive (left part of Exhibit 1); * EPSCoR’s funds also should lead to improvements in university research infrastructures and university-state government relationships in support of R&D (middle part of Exhibit 1); * The funded research projects, together with improvements in university research infrastructures and state capabilities, should lead to more scientifically competitive research (right part of Exhibit 1); and * The increased research competitiveness should lead to an increased share of R&D funding for the state (right part of Exhibit 1). The evaluation sought the answers to four Four main questions, reflecting the presumed evaluation conditions in Exhibit 1: questions 1. Did EPSCoR target research investigators and groups that were nearly, but not yet, nationally competitive? Projects funded by EPSCoR should neither have been already nationally competitive (and thus able to compete without EPSCoR’s "stimulation") nor significantly below national peer review norms for acceptable research. Therefore, the evaluation reviewed funded proposals to confirm that these proposals fell within the "good" to "very good" range of NSF’s standard scoring system for peer review--but neither above nor below this range. 2. Did EPSCoR’s funded projects influence changes in university research infrastructures and university-state government relationships? In site visits to five EPSCoR states, the evaluation examined the organizational structures and policies of EPSCoR-supported universities for systemic changes that could be attributed to EPSCoR. Fourteen of 16 universities in the five states were visited in 1995 and 1996, during which time the evaluation team collected data from interviews and observations and reviewed documentary and archival evidence. The evaluation also examined the EPSCoR states’ financial and administrative policies toward university-sponsored research. They tried to determine, in particular, 1) the role of a state’s EPSCoR steering committee in policy changes (if any); 2) the effect of NSF’s matching requirement on participation; and 3) the influence of inter-institutional relationships among participating universities. The site visits, therefore, included collecting data (through interviews and reviews of documentary evidence) about the state university, state government, and industry in the five sampled states. 3. Did the quality of academic research performed by EPSCoR recipients become more scientifically competitive? To gauge whether the quality of academic research had improved in EPSCoR states, thus becoming more competitive, the evaluation used two measures: o Acceptance of papers on the EPSCoR-funded research by academic publications; and o The award of follow-on external funding to EPSCoR-funded researchers. The evaluation reviewed NSF monitoring report data on research productivity for 86 EPSCoR research clusters funded between 1992 and 1996. To the extent that EPSCoR-supported research projects could be associated with subsequent academic publications and external funding, it might plausibly be argued that EPSCoR was producing more competitive scientists--who would be better able to compete for merit-based federal research dollars. 4. Was there an increase in the EPSCoR states’ share of funded research? The evaluation charted changes in the proportion of federal research funds received by EPSCoR versus non-EPSCoR states over four points in time: 1980, 1985, 1990, and 1994. These changes were taken to be the main indicator of the EPSCoR program’s effects. The evaluation also looked at the EPSCoR states’ share of NSF funding for the same period to track any concomitant changes. To address these questions, the evaluation team collected data by using a variety of procedures. First, evidence about R&D funding, derived from NSF’s ongoing science resource studies, was analyzed. Second, information about EPSCoR’s funding practices was based on an analysis of NSF’s records of its individual research awards and interviews with former and current NSF program officers. Third, the evaluation team made multiple site visits, over a nine-month period, to five representative EPSCoR states (and 14 campuses in those states), to collect data about EPSCoR funded research projects, state S&T developments, and university research policies and practices. Many individuals were interviewed and numerous documents collected during these site visits. Finally, information about EPSCoR-related scientific publications and subsequent funding was derived from COSMOS’s monitoring work for the EPSCoR program conducted from 1994 to 1996, in which all 19 states were the subjects of site visits and in which principal investigators were asked to submit copies of their publications for the team’s review. The evaluation’s findings and conclusions responding to the four evaluation questions are presented in Section B. The results are organized around three major themes: 1. EPSCoR funding in practice (Question 1); 2. Changes in R&D funding (Question 4); and 3. EPSCoR’s influence on university infrastructure, university-government-industry relationships, and university competitiveness (Questions 2 and 3). In Section C, the evaluation presents the implications of the findings for EPSCoR policy and program operations. First, however, the main evaluation conclusions (based on the data in Section B) are previewed below. ____________________ 2 Copies of these papers are available from COSMOS Corporation, 3 Bethesda Metro Center, Suite 950, Bethesda, MD 20814, phone 301-215-9100. 3. What Were The EPSCoR program’s objective was to reduce the the geographic concentration of federal R&D funds in the Evaluation’s United States and its territories. The most direct Main measure of this desired impact was a comparison of Conclusions? (A changes in the share of federal R&D expenditures (and Preview) NSF obligations) across states between 1980 and 1994. For EPSCoR to demonstrate a positive impact, 1) the EPSCoR states’ share of R&D funding would have had to increase relative to the non-EPSCoR states’ share, and 2) EPSCoR’s program strategies would have had to show plausible influence in producing the observed increase in R&D funding share. EPSCoR States Increased Their Share of Federal R&D Funding From 1980 to 1994, EPSCoR states’ share ofEPSCoR federal R&D funding increased from .25 states’ percent to .40 percent per state, or from share did $10.1 million to $50.5 million, per state.increase. (The "per state" assessment is given because the program continually added newly eligible states during this period of time; overall, the EPSCoR states’ share of federal R&D funding represented 7.65 percent or $960 million by 1994). EPSCoR states’ shares of NSF funding showed a similar pattern of increase. Although the increases were small in absolute terms, the increases represent a successful outcome for the EPSCoR program (during the same period, the non-EPSCoR states’ shares decreased).3 Some of EPSCoR’s Program Strategies Appear to Have Been Responsible for Increasing Federal R&D Funding in EPSCoR States Within each EPSCoR state, the program EPSCoR’s required the formation of a statewide program steering committee, to represent the R&D strategies interests of key organizations and sectorsdid of the state (EPSCoR Program Solicitation,influence 1989a). The committee assumed a pre-reviewsome of the function over the component research results. projects that formed a state’s overall proposal to NSF. Thus, the steering committee engaged in a peer review process that involved out-of-state experts. In many states, such innovation in itself created a new S&T environment and a foundation for increased R&D competitiveness. EPSCoR Influenced the States’ S&T Environments through the Steering Committees and the State Funding Match Requirement The EPSCoR steering committees and the state funding match requirement promoted dialogues, planning, and new S&T initiatives. In particular, the steering committees opened communications between state university research officials and their counterparts in business, industry, and state government. This led to the identification of state research priorities that factored in the combined resources of a state’s university system with the unique opportunities offered by a state’s natural or institutional environment. EPSCoR’s one-to-one dollar match requirement helped to focus the states’ R&D priorities and to foster collaboration among universities and between universities and state government and industry. State governments’ willingness or ability to provide the required matching funds, however, has been a recurring problem in some states. The EPSCoR steering committees usually served as the initial opportunity for research collaboration among a state’s universities and between the universities and state government agencies and industry. These collaborations were institutionalized in many instances through the development of formal consortium relationships, state S&T plans, new S&T agencies, and state science advisory councils. EPSCoR Enhanced Some Aspects of Universities’ Orientation to Research Compared to its effects on the states’ S&T environments, EPSCoR’s influence on university policies or resources devoted to research was not significant. However, the EPSCoR awards did stimulate inter-university collaboration (in some cases overcoming strong, traditional rivalries), heighten university-industry research collaboration, install more rigorous standards of peer review, and create new interdisciplinary research facilities. EPSCoR also enabled state universities to offer larger and competitive startup packages to attract talented new faculty, which tended to increase the research orientation of state university faculties in the sciences. EPSCoR-supported research also showed evidence of scientific productivity and, hence, competitiveness. This was evident from the number of subsequent academic publications and external funding awards associated with the EPSCoR-supported research projects. ____________________ 3During the entire period from 1978 to 1996, NSF’s cumulative investment in the EPSCoR program amounted to $182.2 million, or less than 1 percent of the NSF budget during this period; other non-NSF EPSCoR programs had just started around 1994 and had not made significant investments during the 1978-1994 period. SECTION B: Evaluation Findings 4. What Was EPSCoR was charged with the mission of "improving" the EPSCoR’s quality of science, not just funding high-quality Funding science. Similarly, EPSCoR’s mission was to increase Practice? research "competitiveness," not just fund research that was already competitive. To achieve these aims, EPSCoR’s strategy was to fund proposals that were judged "good" or "very good" but not "excellent" under the peer review process. The expectation was that the research experience gained would help scientists become more competitive in obtaining external funding support in the future. An important part of the evaluation was, therefore, to verify that EPSCoR did indeed implement this atypical funding policy. The EPSCoR model assumed that the funded research projects were just short of being nationally competitive. It was further assumed that the resulting enhancement in research competitiveness would produce increased R&D funding. Findings on these issues follow. Evaluation of EPSCoR Proposals The evaluation analyzed a sample of 48 EPSCoR proposals from 10 states for the period 1991-1992.4 These proposals had been peer-reviewed using the NSF’s traditional proposal rating system of 5=excellent, 4=very good, 3=good, 2=fair, and 1=poor. To compute a mean reviewer score, the responses of approximately five reviewers for each proposal were averaged, classifying the proposals according to the following categories: * Category 1: Proposals with a mean score between 5.0 and 4.6 (those judged excellent, which should have excluded them from EPSCoR consideration and resulted in their investigators’ being referred directly to the pertinent NSF Directorate for peer review); * Category 2: Those with an average score between 4.5 and 3.0 (considered appropriate for EPSCoR support); and * Category 3: Those receiving an average score of 2.9 or below (which should have excluded them because of low quality). The evaluation compared the actual award decisions with the funding decisions that would have been predicted from the preceding criteria. Critically, the evaluation also examined reviewers’ comments about their ratings. Nearly 10 Percent of Scientifically Excellent Proposals Were Excluded (Rightfully) from EPSCoR Program Funding Had funding decisions been based solely on the peer reviewers’ scores, 11 of the 48 proposals that fell into Category 1 should have been excluded; the 34 that fell into Category 2 should have been funded; and the 3 proposals that fell into Category 3 should have been excluded. However, only 2 of the 11 proposals in Category 1 were excluded, while 2 of the 34 proposals in Category 2 were incorrectly excluded. The examination of the reviewers’ "Excellent": explanations of their rating revealed scientifically that the reviewers were using two excellent different definitions of the or "excellent" criterion. That is, the excellent EPSCoR reviewers rated a proposal as for EPSCoR? excellent either because they considered it to be "scientifically excellent" (in which case the proposal should have been excluded) or an "excellent fit" for the EPSCoR program (in which case the proposal should not have been excluded). When these critical differences in meaning were taken into account, only 4 of the 11 proposals originally defined as excellent were found to have been judged "excellent" in the sense of scientific excellence. Further, the 2 "good-to-very-good" proposals that were excluded had their ratings downgraded by reviewers because they thought that the proposed work was already of too high scientific quality and therefore not a good fit for the EPSCoR program. In effect, the actual funding decisions were much closer to the predicted pattern than the raw peer reviewer scores indicated. Overall, EPSCoR had excluded four scientifically excellent proposals, which constituted the top cohort of about ten percent of the 48 proposals. To the extent that this proportion may have existed in other EPSCoR competitions, the evaluation concluded that the EPSCoR program did implement its program mandate--"to stimulate competitive research"--and not to fund already-competitive research. Further, the program has had to accomplish its goals--whether in terms of increased share of R&D funds (discussed next) or improvements in scientific competitiveness (discussed later)--without the benefit of the most competitive and outstanding cohort of proposals from the EPSCoR states.5 ____________________ 4 EPSCoR’s competitions are not held annually, and the sample, therefore, represents one of only five cycles of EPSCoR competitions during the 1978-1994 period. 5 Whether the same funding practice has been continued since 1994 was beyond the scope of the evaluation. 5. Was EPSCoR To determine whether the EPSCoR program had achieved Associated with its primary objective of reducing the undue geographic Changes in concentration of R&D funds, the evaluation had to Academic R&D determine whether the EPSCoR states had increased Funding? their share of federal academic R&D funding, reflected by annual data on R&D expenditures. Also of interest was whether the EPSCoR states had increased their share of NSF R&D funding. However, state-by-state data for NSF are reported only for R&D obligations. Since the end of World War II, most of the academic R&D funding has come from the federal sector, although this federal share declined from about 68 to 60 percent between 1980 and 1995 (see Exhibit 2). Also, the NSF share of federal R&D obligations, historically, has been a distant second to that of the National Institutes of Health (NIH) and has declined somewhat over the past 20 years--from about 20 percent to about 17 percent (see Exhibit 3). For the EPSCoR program, the main findings were as follows. EPSCoR States Increased Their Share of R&D Funding The EPSCoR states increased their EPSCoR aggregate share of federal academic R&D states’ awards from 0.25 percent or $10.1 million shares of per state in 1980 to 0.40 percent or $50.5R&D funds million per state in 1994. A "per state" increased unit of analysis was used to assess from 1980 changes in R&D because the number of to 1994. states participating in the EPSCoR program changed over time, from 5 to 19 (18 states and Puerto Rico), as new cohorts of states were added to the program in 1988, 1990, and 1992. Overall, the EPSCoR states’ share of federal academic R&D funding represented a 7.65 percent share or $960 million by 1994 (see Exhibit 4). The observed increase varied by cohort, with the oldest EPSCoR cohorts showing the largest increase in share and the newer cohorts showing the smaller share increases. For NSF funding, the EPSCoR states’ share EPSCoR’s rose from .26 to .34 percent per state states’ between 1980 and 1994, ending with an share of overall share of 6.53 percent. Again, on NSF funds a per state basis, the increase was found also for every cohort (see Exhibit 5). increased... However, unlike the federal expenditures data, the NSF obligations data showed no relationship between cohort age and funding gains. Based on the observed changes in federal and NSF shares, it can be concluded that the EPSCoR states’ share of R&D funding did increase relative to the shares of the other states. To this extent, EPSCoR was associated with a lessening of the undue geographic concentration of R&D in the United States. Although the changes were small in absolute terms, this was a notable accomplishment in an era when research universities in non-EPSCoR states also were thriving and upgrading substantially. For instance, Carnegie rankings of universities’ research capabilities are based on the volume of doctoral degrees at a university. These rankings showed that, during the same 1980-1994 period, double the number of universities in non-EPSCoR states attained the top Carnegie ranking ("Research I" universities). In other words, EPSCoR’s accomplishments of relative increase in R&D share occurred in a highly competitive environment when non-EPSCoR universities were dramatically expanding their research capabilities. Other Conditions That May Have Accounted for EPSCoR’s Increased R&D Share Are Not Supported One might argue that the observed Rival increases in R&D funding in the EPSCoR explanations states could be attributable to are not conditions other than the EPSCoR supported.... program. An initial consideration was whether the increase in R&D funding was associated with growth in the population, number of students, or even the number of research investigators in the EPSCoR states. Many of the EPSCoR states have relatively small populations or are located in regions undergoing rapid population growth (e.g., Sunbelt states). Consequently, population growth-- accompanied by presumed growth in students and faculty--could partially account for the increased share in R&D funding. Examination of this possibility revealed that the average (aggregate) rate of population growth in the EPSCoR states was actually less than that of the non-EPSCoR states during the 1980- 1994 period (9 versus 18 percent respectively). Similarly, within the EPSCoR states as a group, there was no correlation between increases in the EPSCoR states’ R&D share and increases in their population. Thus, changes in population could not account for the increases in R&D share. A second consideration was whether a few of the EPSCoR states had accounted for most of the increase in R&D share, rather than the program as a whole. This possibility arises from a conjecture that the EPSCoR program only derived its apparent gain in share because of a few successful states whose unique circumstances--not the EPSCoR program--accounted for the results. The histogram in Exhibit 6 shows the distribution of share changes (in percent between 1980 and 1994) for each of the EPSCoR states compared to the nation as a whole and the non-EPSCoR states. Exhibit 6 fails to support the conjecture, by showing that over half of the EPSCoR states (10 of 19) exceeded the average share change for the nation-- and the average share change of the non-EPSCoR states as well. In sum, the lack of support for these two considerations provides added confidence that the increase in R&D share was a genuine outcome. 6. To What Extent Did EPSCoR In addressing whether EPSCoR’s Influence strategies influenced state University-Government-Industry relationships, university research Relationships, University capability, and scientific Research Infrastructures, and competitiveness, the evaluation had to Research Competitiveness? consider 1) the nature of a federal-state government partnership program as well as 2) the influence of state-level factors on the EPSCoR strategies used in individual states. Each state, moreover, had its own unique experience with the EPSCoR program. Reported below are findings drawn from five representative EPSCoR states on the possible causal linkages between EPSCoR’s program strategies and the previously discussed R&D funding increases, as conditioned by state-level influences. State-based EPSCoR Steering Committees Helped to Set R&D Priorities, Linking State Interests with Strengthened Within-State Peer Review Processes The EPSCoR program required each participating state to form a steering committee that represented key organizations and sectors within the state. From the beginning, NSF signaled the importance of these committees, insisting on participation by a diversity of members from academe, industry, and state government--including a state’s best and most senior scientific and technological personnel (Drew, 1985). Each committee generally had about 12 members, and the committee’s first responsibility was to apply for and implement the initial planning award from NSF. The committee also assumed a pre-review function over the research projects proposed to NSF as part of a state’s overall proposal and, in so doing, established preliminary peer review processes engaging out-of-state experts. In some states, this was the first occasion for using external experts on a large scale. Such innovation in itself created a new environment and one foundation for increased competitiveness. Through these steering committees--which became forums for S&T dialogues, planning, and the development of new R&D initiatives in science and technology--EPSCoR influenced the states’ S&T environments. This influence was commonly demonstrated in the opening of communication between state university research officials and a state’s government and industry representatives. One result was the identification of research priorities that combined state universities’ strengths with the unique opportunities offered by a state’s natural and institutional environments. The formation of the Steering steering committees was committees therefore one of the most were one of important features of the the most EPSCoR program. These important committees could influence features of a state to create or alter the EPSCoR its strategic S&T plan, to program. establish formal state S&T agencies, and to develop more formal relationships among state universities, federal and state-supported research laboratories, and industry. In short, variations of the following account (derived from a report on one of the five states the evaluation visited) were found across EPSCoR states: EPSCoR has helped the state’s universities to identify areas of research uniquely suited to the state’s conditions. Examples include agriculture, such as natural products and physical acoustics (e.g., catfish farmers using sonar to count fish); issues related to distinctive demographic conditions (the state now has the largest controlled study of arteriosclerosis ever attempted, with NIH calling the state the "perfect laboratory" for the study); economic issues of the delta region; and research on water. The research capability, stimulated in part by EPSCoR’s early support, is evidenced by new centers across the state, including a center for population studies, a mineral resources institute, and others.6 State Matching Requirements Helped to Tighten the Connection between EPSCoR and State Priorities As a direct reflection of EPSCoR’s EPSCoR’s federal-state matching partnership strategy, requirements EPSCoR called for a also $1-to-$1 matching helped. requirement, with the requirement only satisfied by newly appropriated funds, not in-kind matches (Feller, 1997; and Drew, 1985). Over the years, and for any given state, this requirement sometimes created great challenge, especially under fiscally constrained conditions. On several occasions, universities participating in a state’s EPSCoR program had to draw on institutional funds to augment state-appropriated funds to meet the state’s matching requirement in its proposal to NSF. The matching requirement, however, had its intended effect of forcing within-state dialogue over R&D priorities. That is, the matching requirement compelled universities to come together to seek state support, which in turn, influenced the development of common research agendas and collaborative, cross-university initiatives. The dialogue over state R&D priorities effectively forced EPSCoR-funded research projects to work within the larger S&T environment of a state, leading to closer integration among universities, state government, and industry. Such communication and collaboration may be contrasted with an oft-asserted posture of the scientific community, which, according to one EPSCoR project director, has been "detached from the people who pay its bills-- Congress, state legislators, and, ultimately, the public" (Strobel, 1996). States’ Science and Technology Capabilities Increased-- Stimulated in Part by EPSCoR’s Strategies Also important in the capacity-building process was the creation of state science and technology policies, investments, and support that promoted basic research. The evaluation investigated EPSCoR’s influence on the capacity-building process through site visits to the same five representative EPSCoR states. The findings were as follows. * Cross-Campus Collaboration: In four of the five EPSCoR states, EPSCoR had influenced clear influence in cross-campus creating distinctive collaboration and pervasive and S&T cross-campus planning. collaborations. The EPSCoR steering committee usually serves as the initial opportunity for cross-campus communications and collaboration in research. These collaborations have become institutionalized, with the creation of formal consortia or other inter-institutional ties. * Science & Technology (S&T) Planning: In four of the five states, the statewide EPSCoR steering committee was instrumental in creating the initial forum for collaboration among universities, state agencies, industry representatives, and other significant S&T participants, including state legislators. These forums contributed in various ways to the states’ S&T capacities and infrastructure by forming S&T plans, creating new S&T agencies, serving as the governor’s science advisory committee, fostering university-industry collaboration, and, as previously noted, mapping EPSCoR’s funding priorities onto state S&T priorities and strengths. * Inter-institutional Relationships: In addition to cross-campus collaboration, a state’s S&T capability also increases with the formation and solidification of other inter-institutional relationships. EPSCoR had an important influence over the relationships in three of the five states. In the first state, this led to collaboration among entities involved in basic research, applied research, and industry (in turn leading to increased industry-university research collaboration). In the second state, this led to a statewide S&T plan and the enactment of legislation to create a statewide S&T council. In the third state, the EPSCoR steering committee helped to form a research consortium on economic, science, and technology development and a public education forum on educational issues facing the state’s leaders. The overall impression of the evaluation team was that the five states did not yet have the rich and diverse array of S&T capabilities found in non-EPSCoR states. But significant portions of what had been put into place could be attributed to the EPSCoR program. EPSCoR Enhanced Some Aspects of Universities’ Increased Research Orientation University-wide actions and policies directly reflect a state’s research infrastructure and capability and, therefore, research competitiveness. Through site visits to 14 universities within the five representative EPSCoR states, the evaluation found that EPSCoR had enhanced some aspects of this infrastructure and capability. In large part, EPSCoR had enhanced participating universities’ orientation toward research by fostering cross-university collaborations (as discussed previously in connection with the EPSCoR steering committees). Single institutions with limited facilities and small faculties could combine forces in assembling a critical mass of research faculty, technical personnel, and facilities to compete for major federal research awards. This resulted, for example, in the development of interdisciplinary research centers and research teams. EPSCoR funding also helped to create new science faculty positions and laboratories that helped some universities shift from an exclusive emphasis on undergraduate teaching to one that also included the conduct of basic research. Further, the size and "EPSCoR was prestige of EPSCoR’s awardsabout could have had an impact, everything varying from new research the endeavors to the university introduction of more should have rigorous standards of peer been review to changes in about...." universities’ research cultures. As put by one vice-president for research, "EPSCoR was about everything the university should have been about: capacity, partnership, and long-term vision." Across universities, the impact showed up in specific research endeavors, including: * The creation of one of the major DNA protein facilities in the world, with genetic materials and histories useful to forensic work; * New faculty in chemistry and a stronger optics research group; * University-industry collaboration on polymers; * A "boom" in materials science and a five-university consortium that submitted a proposal for an NSF Engineering Research Center award; * The building of a surface hydrology group that also involved a unique, statewide system for monitoring climatic conditions, soil moisture, water quality, and ultraviolet atmospheric conditions for research; and * Cross-campus collaborations of all sorts, where there might previously have been strong rivalries. EPSCoR support also was used to increase faculty startup packages, which helped state universities compete for and attract top faculty in the sciences. Packages of sufficient size to attract leading young faculty were not widely offered in the EPSCoR states prior to EPSCoR funding. The evaluation found that EPSCoR was associated with increases in the packages at 6 of 11 universities (no information was available at three of them), leading to the later hiring of a strong group of research-oriented faculty. Little Evidence Exists that EPSCoR Initiatives Influenced University Research Capabilities in Other Ways EPSCoR’s influence on other aspects of internal changes in the participating universities has been modest or so interwoven with other influences that it was not possible to assign it anything other than a supportive or reinforcing role. EPSCoR’s presence was concurrent with several significant shifts in internal university practices and policies linked to the promotion of research. As discussed below, however, little evidence exists that EPSCoR’s initiatives directly caused these changes, four of which are outlined below. First, vice presidents for EPSCoR’s research existed or emergedinfluence in 6 of 14 universities on many (one university provided nouniversity information and one had a research director of research and policies economic development that was reported to the president).minimal. Research visibility, influence, and investments benefit when a university’s lead research officer reports directly to the university’s president. The evaluation tried to determine if EPSCoR had played a part in the upgrading of senior research administration positions at the EPSCoR states’ universities, finding that EPSCoR could be associated with only one upgrading: an assistant vice-president had been promoted to vice-president. Second, a practice found at some research-intensive universities is for the central administration to allocate a portion of the institution’s indirect cost recovery funds to research investigators or departments that receive external research awards, as an additional incentive for seeking such awards. Among the EPSCoR states, the 14 universities showed highly varied patterns of allocation policies, with EPSCoR having little or no influence on these policies. Third, of the 14 universities, only 2 had the minimum course loads typically available at research-intensive universities, 4 others had provisions for " buying out" teaching time to get to the minimum (no information was available for 2 of the universities). Many research-intensive universities require the teaching of only one course, with added allowance if the course has a high enrollment. EPSCoR had little or no influence on the emergence of like policies at the 14 universities. Fourth, the evaluation found that the 14 universities did not have very large centralized sponsored research offices (SROs), and only a few used a decentralized system. (At research-oriented universities, well-tooled and efficient SROs or well-staffed decentralized systems are usually present.) EPSCoR may have influenced the emergence of one of the SROs. Because the five states were eligible to participate in the EPSCoR program, the visited universities were not major performers of academic research. Only 2 of the 14 universities were classified as "Research I" universities in the 1994 Carnegie rankings (5 were Research II, 6 were Doctoral I or II, and 1 was Master’s I).7 EPSCoR-Supported Research Showed Evidence of Scientific Productivity, and Hence Competitiveness The evaluation examined whether EPSCoR-supported researchers showed evidence of increased scientific productivity and thus, presumably, increased competitiveness for research awards over time. Establishing evidence of increased scientific competitiveness would make plausible a link between improved university research capabilities and the observed increases in shares of R&D funding in the EPSCoR states (Stigler, 1994; Feller, 1996). The scientific competitiveness of EPSCoR-supported research, however, was impossible to assess directly, without intensive peer review on a project-by-project basis. Consequently, the evaluation looked instead at the "productivity" of EPSCoR-supported research, defined as scientific investigations that produced 1) scientific publications and 2) funding from external sources following the EPSCoR award. The evaluation examined the number of scientific publications and the amount of external funding that the EPSCoR principal investigators of 86 research clusters8 (the entire universe funded between 1992 and 1996) claimed as outgrowths of their EPSCoR funding. However, data to establish normative standards for interpreting the productivity of the EPSCoR-supported clusters were not obtainable. Consequently, the descriptive data presented below are useful primarily for establishing the plausibility of the assertion that the EPSCoR-funded research clusters could reasonably be considered scientifically productive. Exhibits 7 and 8 show the EPSCoR’s frequency of scientific research publications and led to many subsequent new awards, by subsequent these research clusters. publications Both exhibits show high and new numbers of publications or research award dollars and, more awards. important, an even distribution across the clusters, showing that these accomplishments were not limited to a small number. For instance, over two-thirds of the funded clusters (52 of 76) had 20 or more publications emanating from their EPSCoR-supported work, and 70 percent of the funded clusters (49 of 70) had $1 million or more in new awards. The publication dates of the research publications sample were consistent with research that could have been conducted with EPSCoR funds. The research reported in the publications also pertained to the scientific topics funded by EPSCoR. As to the continuation funding data, it should be noted that there was roughly a 2:1 ratio of external to EPSCoR dollars, again an observation consistent with the suggestion that the EPSCoR clusters were scientifically productive. Thus, it is plausible to conclude that EPSCoR’s influence on enhancing university research capabilities led to more productive scientific research, which in turn was sufficient to compete for, and win, additional future R&D funding. ____________________ 6 To retain the anonymity of individuals interviewed, the identity of the state is not cited. 7 Because the Carnegie rankings are based on the volume of doctoral degrees, it is worth noting that 10 of the 14 universities had graduate enrollments of 3,000 or more, and the 2 smallest universities had enrollments of about 1,700 students each. 8 EPSCoR urged states to propose groups of related research projects ("research clusters") to encourage capacity building and interdisciplinary research, in contrast to totally independent research projects. The clusters could be of differing sizes--e.g., covering from 3 to 8 related projects. Funding decisions would then be made for a cluster in its entirety, and the totality of the clusters as well as other related educational components would then comprise the EPSCoR award to an individual state. SECTION C: Policy and Program Implications 7. What Are the Overall the evaluation has shown a plausible case for Evaluation’s the EPSCoR program and progress in fulfilling its Implications mandate. The management of the EPSCoR program entails for EPSCoR a number of strategic operational choices. The Policy and implications of the evaluation’s findings for some of Program these choices are discussed below. Operations? EPSCoR Proposal Review EPSCoR’s program design is based on excluding from further funding consideration those proposals deemed scientifically excellent. The evaluation found that NSF reviewers had problems in judging whether EPSCoR proposals considered already scientifically excellent were appropriate for EPSCoR support--and that the reviewers commingled two different definitions of "excellent"--excellent for EPSCoR and scientifically excellent. Thus, there is a need to clarify the Need for EPSCoR definition of an excellent proposalorienting and the funding decision that should reviewers accompany such ratings. Peer reviewers of EPSCoR should receive explicit orientation, and proposals their review comments should be continually monitored to minimize confusion between the two definitions. Graduation from EPSCoR "Graduation" from the EPSCoR program When do occurs in principle when a state has EPSCoR become nationally competitive for academicstates R&D funds and thus no longer requires the "graduate" extra help provided by EPSCoR resources. from Although many EPSCoR clusters have EPSCoR? subsequently become fully competitive and no longer seek EPSCoR funding, no state has yet to graduate from EPSCoR. Since the evaluation has demonstrated that the program has been successful in improving the R&D competitiveness of participating states, consideration of graduation criteria and state or university transition from EPSCoR support appears relevant. One option is for the EPSCoR program to reassess the current eligibility of all EPSCoR states, using the current eligibility criteria. Such criteria include conditions relevant not only to eligibility but also to graduation--e.g., a state’s ranking among all states and its amount of R&D funding per academic scientist and engineer in the state. Such an assessment has not been conducted in recent years, much less on an annual or other cyclic basis. EPSCoR’s Cluster Research Strategy The cluster strategy has been a key operational mechanism for inducing increased research competitiveness at the faculty and institutional level. A cluster is a related group of research projects, often interdisciplinary, and awards are made to a cluster’s principal investigator as well as to the component research projects. This strategy has led to the development of laboratories or centers, and not just the recruitment of cadres of new, research-oriented faculty in a number of EPSCoR universities. The cluster strategy also has been used to promote interdisciplinary collaboration among universities and between universities and industry. For the universities in the EPSCoR states, the cluster strategy may continue to be a more effective way of increasing research capability than the alternative strategy attempted earlier in EPSCoR’s history--involving the funding of individual researchers and single research projects. The earlier strategy proved to be counterproductive when individual researchers were later recruited away from universities in EPSCoR states. How the cluster strategy is to be continued, therefore, warrants ongoing attention. REFERENCES Drew, David Eli, Strengthening Academic Science, Praeger, New York, 1985. Feller, Irwin, Matching Fund and Cost-Sharing Experiences of U.S. Research Universities, COSMOS Corporation, Bethesda, Maryland, 1997. Gumport, Patricia J., Improving Academic R&D In State Universities: Some Trends, Issues, and Strategies, 1996. Feller, Irwin, "The Determinants of Research Competitiveness Among Universities," in Albert Teich (ed.), Competitiveness in Academic Research, American Association for the Advancement of Science, Washington, D.C., 1996. Greenberg, Daniel, The Politics of Pure Science, The New American Library, Inc., Cleveland, Ohio, 1967. Hackett, Edward J., Understanding the Social Dynamics and Performance of Research Groups, 1996. Luger, Michael and Harvey Goldstein, Technology in the Garden, University of North Carolina Press, Chapel Hill, North Carolina, 1991. Malecki, Edward, Technology and Economic Development, Longman Scientific and Technical, Essex, England, 1991. Martino, Joseph, Science Funding, Transaction Publishers, New Brunswick, New Jersey, 1992. National Science Foundation, EPSCoR Program Solicitation: Guidelines for Preparation of Proposals, Planning Phase, Washington, D.C., 1989a. National Science Foundation, EPSCoR Program Solicitation: Guidelines for Preparation of Proposals, Planning Phase, Washington, D.C., 1989b. Stigler, Stephen, "Competition and the Research Universities," in Jonathan Cole, Elinor Barber, and Stephen Graubard (eds.), The Research University in a Time of Discontent, The Johns Hopkins University Press, Baltimore, Maryland, 1994, pp. 131-152. Strobel, Gary A., "Lessons from the EPSCoR States," editorial, Science, 31 May 1996, Vol. 272. U.S. Congress, Office of Technology Assessment, Federally Funded Research: Decisions for a Decade, OTA-SET-490, Washington, D.C., May 1991. EXHIBITS Expert Group of Advisors Dr. James R. Durig Dean, College of Arts and Sciences University of Missouri-Kansas City Dr. William Massy Stanford Institute for Higher Education Research Stanford University Dr. Norine Noonan Vice President for Research and of the Graduate School Florida Institute of Technology Dr. Karen Seashore Louis Associate Dean, College of Education University of Minnesota Dr. Reginald Wilson Senior Scholar American Council on Education EPSCoR Workgroup Dr. Judith Bailey (EPSCoR Committee Member) Vice President, Research University of Maine Dr. Colin Bennett (EPSCoR Principal Investigator) Department of Mathematics University of South Carolina Dr. Philip Boudjouk (EPSCoR Project Director) Distinguished Professor and Project Director North Dakota State University Dr. Hans Brisch (EPSCoR Committee Chair) Chancellor Oklahoma State Regents for Higher Education Dr. Daryle Busch (EPSCoR Principal Investigator) The Kansas Program for Molecular Design, Synthesis, and Applications of Macromolecular Materials and Supramolecular Systems University of Kansas Dr. Delwood Collins (EPSCoR Committee Member) Vice President for Research and Graduate Studies University of Kentucky Dr. Collis R. Geren (EPSCoR Project Director and Committee Chair) Dean of Graduate School Associate Vice Chancellor for Research University of Arkansas Dr. Randolph V. Lewis (EPSCoR Project Director and Committee Member) Department of Molecular Biology University of Wyoming Dr. Ralph Powe (EPSCoR Project Director and Committee Chair) Vice President for Research Mississippi State University Dr. Barbara Wright (EPSCoR Principal Investigator) Division of Biological Science University of Montana Exhibit 1 HYPOTHESIZED PROGRAM EVENTS IN EPSCoR STATES [DeviceRGB 3 bits] Source: Robert K. Yin and Irwin Feller, EPSCoR Evaluation (draft report), COSMOS Corporation, Bethesda, Maryland, 1997. Exhibit 2 PERCENT SHARE OF ACADEMIC R&D EXPENDITURES, BY SECTOR (selected years, 1980-1995) Year Federal State/Local Industry Academic All Other Government Government Institutions Sources 1980 67.6 8.1 3.9 13.8 6.6 1985 62.6 7.8 5.81 6.7 7.2 1990 59.2 8.1 6.9 18.5 7.3 1995* 60.2 7.4 6.9 18.1 7.4 *1995 figures are estimated. Source: Science & Engineering Indicators--1996, Appendix Table 5-2, p. 167. Exhibit 3 [DeviceRGB 3 bits] Exhibit 4 EPSCoR STATES’ SHARE OF FEDERAL ACADEMIC R&D EXPENDITURES (in percent) States/Cohort 1980 1985 1988 1990 1992 1994 Change 5 states/1979 1.25 1.26 1.14 1.16 1.3 1.58 +.33 8 new states/1988 3.42 3.54 3.46 3.57 +.15 4 new states/1990 1.62 1.61 1.62 -- 2 new states/1992 0.83 0.88 +.05 ----------------------------------------------------------------- Total for EPSCoR states 1.25 1.26 4.56 6.33 7.21 7.65 Per State 0.25 0.25 0.35 0.37 0.38 0.40 Source: NSF Survey of Scientific and Engineering Expenditures at Universities and Colleges (supported by NSF Division of Science Resources Studies), 1995. Exhibit 5 EPSCoR STATES’ SHARE OF NSF OBLIGATIONS (in percent) Cohort 1980 1985 1990 1994 Change 1 (5 states) 1.29 1.33 1.32 1.49 +.20 2 (8 states) 1.59 2.50 2.81 +1.22 3 (6 states)* 1.78 2.23 +.45 ------------------------------------------------------ Total for EPSCoR States 1.29 2.92 5.60 6.53 Per State .26 .24 .29 .34 *Cohort 3 combines the 1990 cohort (4 states) and the 1992 cohort (2 states) as shown in Exhibit 4. Source: NSF Survey of Federal Obligations to Universities, Colleges, and Selected Nonprofit Institutions, 1995. Exhibit 6 [DeviceRGB 3 bits] Exhibit 7 [DeviceRGB 3 bits] Exhibit 8 [DeviceRGB 3 bits]