The National Science Foundation (NSF) began conducting the Survey of Science and Engineering Research Facilities, a congressionally mandated biennial survey, in 1986 to provide data on the status of science and engineering (S&E) research facilities at research-performing colleges and universities in the United States. During the 1999 and 2001 survey cycles, NSF collected data from a population of approximately 600 colleges and universities as well as 250 nonprofit biomedical research institutions that receive funding from the National Institutes of Health (NIH).
NSF, in conjunction with NIH and Westat, decided to redesign the survey to improve both the quality and usefulness of the data and response rates, and also to reflect changes in the ways that S&E research is conducted. This document explains how this extensive survey redesign effort was carried out during the 2001-03 period.
Previously, the survey consisted of questions on the amount of S&E research and instructional space at an institution by field of science, the adequacy and condition of S&E research space, costs of capital projects, capital projects that were deferred, and availability of animal research facilities. In 2001, a very limited two-question survey on the amount and adequacy of S&E space was administered to limit the cost and eliminate the burden of asking questions that might change during the redesign effort.
The survey redesign was intended to completely reevaluate the full questionnaire, with every part of the questionnaire potentially open to change. In conducting the redesign, data were collected through numerous techniques to obtain multiple perspectives on how the questionnaire was working and on the information researchers and policymakers needed. Specifically, the following approaches were used.
These approaches did not take place independently or sequentially, but rather, were designed to support each other. For example, some initial cognitive interviews aided in the preparation of the agenda and issues for the primary expert panel and, consequently, identified topics examined in later cognitive interviews. The same process was also used for the cyberinfrastructure methodology workshop. Part of the literature review took place before the expert panel's meeting, and thus helped develop the agenda and issues. The literature review also was expanded as new topics of interest, primarily relating to cyberinfrastructure, arose. An example of how the different approaches fit together is how the cyberinfrastructure section developed. First, the literature review identified some topics concerning cyberinfrastructure, which were followed up in consultation with internal NSF experts. The expert panel also provided some advice on this topic. Based on this information, draft questions were developed and tested through cognitive interviews. The results were presented to participants in the methodology workshop, who further refined the scope of the section and the wording of the questions. These refinements were investigated through additional literature review and expert consultation and tested through supplemental cognitive interviews.
Once the section became relatively final, it was pretested to verify how well the questions worked. Thus, the general process was one of iteratively looking at the questionnaire from multiple perspectives, revising it, and testing the revisions. The pretests were the final step in verifying whether the changes in the questionnaire worked as intended. The extent to which these various procedures overlapped can be seen from the following schedule:
Throughout this process, the questionnaire was a continually evolving document, with changes occurring after a major event or cluster of site visits and after each of the many periodic internal reviews at NSF and Westat. Almost every line of the questionnaire was changed in some way. A complete historical account of the process would be lengthy and difficult to follow because of multiple simultaneous changes; instead, this report describes each method separately. The reader is cautioned to remember that these were interrelated, not discrete, events, and although major decision points may have occurred at various times, the activities often extended over some period of time. Even the expert panel, which met only once, had an involvement that extended beyond that time; the panel was polled shortly after the meeting to verify the degree of consensus on different issues, and individual members of the panel were consulted on particular issues either before or after the meeting.
This next section provides an overview of these methods and how they were used; later sections discuss the findings from the redesign study.
Cognitive interviews were conducted primarily in person (44 institutions), as well as by telephone (2 institutions). Their purpose was to determine how respondents interpreted the draft survey questions, to learn whether institutions had the requested data available and how they maintained data on facilities, and to elicit feedback on and suggested changes to the questionnaire.
Cognitive interviews are designed to show respondents' thought processes as they complete a questionnaire. Generally, this is done by asking respondents for a verbal account of their thinking either at the same time they are completing a survey question or immediately after they answer it. Their thought processes are then recorded in a verbal protocol.[1] However, the process of acquiring data and completing an establishment survey of this scope can be extremely complex and time consuming, as it involves multiple respondents and data sources. As a result, it was not feasible to observe respondents collecting the actual data for most questions while they were completing the formal survey. Instead, respondents were asked to read a question, think aloud about how they would gather the requested data, and answer the question. Respondents were then asked more specific questions relating to both the draft survey questions and their answers. For example, respondents might be asked how they would count research space shared between two S&E fields. This would be used to determine how they would decide where to include the space and to gauge the consistency of methods for prorating shared space across institutions. Respondents also might be asked to explain their understanding of NSF terminology (e.g., grid technology) to determine whether there is a shared understanding of the concept across institutions, to learn if different terminology should be used, and to determine if a definition needs to be added or clarified.
Cognitive interviews were conducted with staff members from 46 colleges, universities, and nonprofit biomedical institutions over a 17-month period between August 2001 and December 2002. NSF primarily prepared the protocol for the cognitive interviews, with input from NIH and Westat. Over the course of these interviews, the protocol evolved incrementally to address new issues and to focus attention on particularly problematic questions. During the first site visits to California in August 2001, respondents were given a copy of the questionnaire used in 1999 and asked to review the entire instrument and to discuss the process their institution used for identifying a coordinator and completing the questionnaire. The protocol for the interviews included topics such as the amount of instructional and research space at an institution, the condition and adequacy of research space, interdisciplinary research, collaborations on research with other organizations, budget data, construction, repair and renovation, animal research space, and general survey design issues. Site visits in Boston followed a similar protocol.
The findings from these site visits, along with a review of the literature, was a primary source of information presented to the expert panel in January 2000. After the expert panel meeting, the questionnaire was revised and the protocol became focused more narrowly on questions updated based on issues that the expert panel had identified as important and needing more research. These questions covered five topic areas: how to categorize different kinds of research space, how to measure the condition of research space (and whether the age of facilities is a good indicator of condition), how to measure the need for additional space (specifically, whether the institution's capital plan is a good indicator of its need), how to measure whether new construction replaces old space or provides new space, and how to use potential alternative measures of research capacity (i.e., computing capacity such as bandwidth or connectivity). For example, the expert panel recommended asking for more detail on research space such as how much was in wet labs, dry labs, and support space. Also, the expert panel considered the question on research space condition to be too subjective and suggested determining building and research space age as a possible replacement. They suggested talking to IT experts to determine how to structure a new section of the survey on network capacity/cyberinfrastructure because of the highly technical nature of the topic.
From that point on, the cognitive interviews were largely conducted in an iterative fashion, using the information obtained from one cluster of interviews to revise the questionnaire, then examining the changes in the next set of interviews. Also, the interview sessions (typically 2 hours long) were too short to cover all topics in depth, especially if the interviewees wanted to discuss other aspects of the questionnaire that were important to them. The protocols thus were highly flexible often focusing on the most critical questions. Researchers sometimes changed the emphasis from one interview to the next to ensure that collectively, they covered all of the topics in depth. They also sometimes adjusted the content of the protocols based on the qualifications of the people present (e.g., if someone from the financial office was not available, that topic received less emphasis).
The cyberinfrastructure section was often treated separately: sometimes it was not covered at all, at other times it was covered in combination with the rest of the questionnaire, and at still other times, it was the sole focus. The treatment depended on the individuals available for the interview and the status of the questionnaire development process. Work started later on the cyberinfrastructure section because it was entirely new and researchers needed to consult with experts on that topic to develop a draft questionnaire for testing. Therefore, there was one point in time when the section was not yet available, another when both the cyberinfrastructure section and base questionnaire were being tested, and another when the cyberinfrastructure section was still being tested while the rest of the questionnaire was largely finalized. In general, the site visits were performed in clusters, with four or five institutions in a the same city being visited over 2 days. This process allowed a large number of diverse sites to be included, and the visits were also conducted more efficiently.
Several criteria were used to select institutions for participation in the cognitive interviews. First, institutions had to have responded to both the 1999 and 2001 surveys. Among past respondents, academic institutions were then selected to ensure regional representation, institution type (public versus private) and size, and research expenditures (as determined using data from the 2000 Survey of Research and Development Expenditures at Universities and Colleges) (table 1). HBCUs also were specifically included because they have been an historic focus of the Facilities survey and also because their generally smaller size led them to approach the questionnaire differently than many larger institutions, which helped verify whether the questionnaire would work well across a broad range of institutions.
Table 1. Number of academic institutions that participated in cognitive interviews, by institution characteristic
|
||||||||||||||||||||||||||||||||||||||||||
Biomedical research institutions were selected using similar criteria on past participation and geographic representation (table 2). Often, only one or two eligible biomedical institutions were potentially available in a given city, so scheduling them concurrently with visits to academic institutions ensured a broad geographic representation but typically left little flexibility as to which institutions would be included. Sometimes no biomedical institutions could be scheduled in a particular city so researchers also conducted a supplemental cluster of site visits focused entirely on biomedical institutions in a city that happened to have multiple eligible institutions.
Table 2. Biomedical institutions that participated in cognitive interviews, by region
|
Some site visits were narrow in focus, designed to address very particular issues such as the new construction project sheet (1 institution) or the new section on cyberinfrastructure (12 institutions).[2] These topics were sometimes treated differently because of their different schedules for development and testing; the new construction project sheet was developed only after conducting discussions with the Office of Management and Budget (OMB), and cyberinfrastructure testing was delayed by the need to develop a new questionnaire section and consult with experts. Therefore, some of the cyberinfrastructure interviews took place after the base questionnaire had largely been finalized. In these cases, additional criteria were considered. For the new construction project sheet, telephone calls were made to the institutional coordinator to determine whether the institution had any new construction projects that met a $100,000 threshold. For cognitive interviews pertaining to cyberinfrastructure, an attempt was made to visit a range of institutions with differing setups and levels of technological sophistication, using Internet2 membership and research expenditures as indicators of potential differences across institutions.
Researchers sent the institutional coordinator a description of the topics to be covered during the site visit and asked that the appropriate person(s) to address the various issues be present at the interview. Interviews often included individuals from the offices of facilities, research, budget, and IT, depending on the questions to be addressed and institutional organization. Several days before the site visit, the contact person received a copy of the draft survey to distribute for review; however, participants were asked not to complete the survey ahead of time.
The draft survey was reviewed question by question, although not necessarily in chronological order. Periodic revisions such as adding instructions, definitions, and examples and changing terminology were made between site visits, and the revisions were tested through subsequent site visits or telephone interviews.
Representatives from higher education, government, and biomedical institutions who had experience relevant to the design and purpose of the survey formed the expert panel. Representatives of various interests and agencies were also invited to attend and participate in the panel meeting, including the Council on Governmental Relations, Association of American Medical Colleges, National Center for Education Statistics (NCES), Energy Information Administration, Association of American Universities, OMB, and Federation of American Societies for Experimental Biology. Exhibit 1 shows the membership of the expert panel.
The expert panel convened January 15-16, 2002 at NSF headquarters in Arlington, Virginia. The meeting began with NSF presentations on the Research and Development Statistics program and the Survey of Science and Engineering Research Facilities and included a brief history of the survey along with current issues. NIH representatives also spoke about their role and interest in the survey.
The discussion centered on general design issues, particularly what kinds of data should be collected through the Facilities survey that would be most useful to analysts. The panel addressed the following questions:
Should the survey change how the amount of research space is measured? Cognitive interviews found some inconsistencies in how space was reported; for example, some institutions included corridors whereas others did not. Also, although the data collection might be more burdensome, there is potential for increasing survey usefulness by differentiating between types of space (e.g., wet labs and dry labs) by cost.
Should the survey measure buildings rather than space? The Large Facilities Followup Survey, sent to Facilities survey respondents with projects that exceeded a certain cost threshold, focused on entire buildings rather than net assignable square feet (NASF). One reason for this was OMB's interest in examining construction costs. However, buildings may include space that is not research space.
Should questions pertaining to major instrumentation be included on the survey? Previously, NSF had an instrumentation survey , but it has not been conducted in recent years. The Facilities survey does ask respondents to treat large instruments (i.e., worth more than $1 million) as effectively being part of the facility but otherwise does not discuss instrumentation. NIH has an interest in assessing need for instruments but measuring need is difficult because of the subjective nature of the question.
How might the survey capture data on the new methods of conducting science?
What are the most useful methods to measure the indirect costs of research infrastructure? OMB is interested in measuring whether reimbursements for indirect costs are used to finance new construction. However, the questions that currently ask about indirect costs are seldom answered, with few respondents saying they are able to provide data and even fewer actually providing it.
Should some aspects of the survey design be changed?
What is the most appropriate population to be surveyed? A relatively small number of institutions account for most of the research space. If the survey could be more narrowly targeted, substantial effort might be saved both for the Federal Government and the institutions involved. Also, because of OMB regulations on how to document indirect costs, larger institutions maintain data on facilities differently than do smaller ones (). This affects institutions' ability to respond to the questionnaire.
Can most of the data provided by these institutions be made public? On most of its other major surveys, NSF makes the data for individual institutions publicly available for the majority of items, but it has never released any individual information from the Facilities survey. Releasing this data might increase researchers' use of it and might also increase response rates by giving institutions a product that they also can use. However, there also is a risk that some institutions might not respond, or might respond differently, if they know the data will be made public with institutional identifiers.
Should NSF attempt to collect survey data annually? NSF has used 2-year time periods and collected retrospective data (i.e., for the past 2 years) and prospective data (i.e., for the next 2 years). This is consistent with the fact that the survey is biennial. However, the question remains whether another time period would result in improved response or better data.
Exhibit 1. Expert panel members
Tom Cruse[*]
Associate Provost for Academic Facilities Planning
Office of Senior Vice President and Provost
University of Cincinnati
Don A. Dillman
Deputy Director, Social and Economic Sciences Research Center
Thomas S. Foley Distinguished Professor of Government and Public Policy
Washington State University
Richard Grossi[**]
Chief Financial Officer
Johns Hopkins School of Medicine
Sandra Harris-Hooker
Associate Dean for Research Development
Program Director, Minority Biomedical Research Support
Associate Professor, Medicine and Pathology
Morehouse School of Medicine
John P. Harrod, Jr.
Director, Physical Plant
University of Wisconsin
President, Association of Higher Education Facilities Officers, 2000-2001
Harvey Kaiser
President
Harvey H. Kaiser Associates, Inc.
Leonard K. Peters[*]
Vice Provost for Research, Research Division
Virginia Polytechnic Institute and State University
O. Robert Simha
Massachusetts Institute of Technology Affiliate
Theodore Weidner
Associate Vice Chancellor for Facilities and Campus Services
Administration and Finance
University of Massachusetts
M. Roy Wilson
Dean, School of Medicine
Vice President for Health Sciences
Creighton University
Kaye Wynne[*]
Vice President, Office of Sponsored Programs
Scripps Research Institute
John Yates
Supervisory Program Analyst, Laboratory Infrastructure Division
U.S. Department of Energy
[*]Has served as institutional coordinator for the survey
[**]Unable to attend
A literature review had been prepared and two clusters of site visits had been conducted before the expert panel meeting, so information from these sources, along with data that NSF had compiled from previous Facilities surveys, was available.
The expert panel was not expected to reach a consensus on any of the topics, but rather, to discuss burden issues, relevance of collected data, making data more useful, and improving data quality . Expert panel discussion facilitated the development of new survey questions, which were tested in future cognitive interviews.
An extensive literature review was conducted to gather information about current knowledge and concerns about research and development (R&D) facilities:
The literature review did not attempt to summarize the actual findings or policy recommendations from the various sources identified, but rather focused on the methodological and analytical implications of past research so the redesign would improve the accuracy and usefulness of the data, such as by identifying key variables and ways that they have been measured in other establishment surveys.
As part of the literature review, NSF and Westat identified experts in fields related to research facilities and space management and interviewed several of those experts (exhibit 2). Interviewees received a copy of the 1999 Survey of Scientific and Engineering Research Facilities as a reference. In hour-long interviews, they were asked more about their experiences with facilities data than with the survey itself. For example, they were asked whether they had faced any pitfalls that NSF should take into account or encountered opportunities from which NSF could benefit. They were also asked about measures that NSF could provide that would be particularly useful to facilities planners and if there were ways of asking about these topics that would either minimize institutions' difficulty in responding to the survey or increase the accuracy of their responses.
Exhibit 2. Experts consulted in the literature review
Kenneth Baker
General Manager
Facilities Information Management System
Department of Energy
Dennis Barnes
Director
Higher Education Relations
E-numerate
John Casey
Manager
Engineering
University of Georgia
Physical Plant
James Cayce
Facilities Information Management System
Department of Energy
Katherine Peele
Architect
Boney Architects
O. Robert Simha
Massachusetts Institute of Technology Affiliate
Some of the major findings follow.
Data use. Four primary uses of facilities data are benchmarking (i.e., to evaluate an institution's performance by comparing it to peer institutions), space utilization (e.g., to see how space is divided into various categories or to investigate how efficiently the space is used based on measures such as room utilization rate and station occupancy rate), needs assessment (to project needs and justify capital expenditures), and recovery of indirect costs (as specified in OMB Circular A-21).
Estimating space needs. Many institutions and systems use formulas to analyze their space needs. Although these formulas vary widely in how they are specified, they have common characteristics. They tend to include measures of faculty (or researcher) size and number of graduate students and to assume that there will be differences in space usage by discipline. The formulas are more standardized for analyzing instructional space than for research space. Other variables that are of interest include projected enrollment (especially for instructional space) and faculty recruitment.
Measuring space condition. Space condition is an important issue for several reasons and includes the aging of buildings built as part of a science boom in the 1960s and the changing needs in facilities, such as increased space for mechanical and electrical systems. Condition can be measured through subjective evaluations, building age, date of the last major repair or renovation, and deferred maintenance.
Cyberinfrastructure. Cyberspace has many implications for research facilities, including the need for facility modifications to meet technological advances, the use of distance learning as an alternative to having facilities nearby, the use of large databases as another component of research infrastructure, and the extent to which a campus is wired for Internet use. Some important measures of cyberinfrastructure include bandwidth and its use, storage capacity, and data repository use.
Interdisciplinary research. Interdisciplinary research can prove problematic with respect to space measurement(e.g., departments may share both space and equipment) and the increasing versatile use of space (e.g., less rigidly defined by discipline). However, most institutions still seem capable of dividing their space by disciplines and most researchers are assigned to a home department that can be used for prorating space as necessary. However, there are exceptions in which interdisciplinary space is assigned to organized research units without any tracking of individual disciplines.
Shared resources. Institutions appear to have increased sharing of facilities with other universities or private industry as a way of maximizing the use of funds and meeting needs for sophisticated facilities and equipment. In general, it appears that universities report space as long as they either own or have control of the building. If space is leased, it is counted only by the institution using the space. If space is shared with a private organization, universities tend to report it as their own. Space that is shared but not leased is more difficult to describe; it sometimes is prorated based on funding.
Faculty recruitment. Several respondents mentioned the impact of faculty recruitment on an institution's space needs. Using information about startup packages for prospective appointments might be helpful in distinguishing desires from true needs.
Facilities assessment software. The characteristics of software for assessing facilities were examined to determine the data available to institutions and how these data are used. Institutions report use of several software systems, including the Strategic Assessment Model, Facilities Information Management System, and Facility Information and Maintenance Information System. Data kept in these systems include building number and name, gross square footage, construction date, construction type, functional use, number of floors, and current replacement value. Most systems have the capacity to collect much more extensive data and can perform deficiency analysis, life-cycle data analysis, computations of the facility condition index (FCI) for benchmarking purpose, and also can model FCI funding requirements.
Questionnaire design issues. Rating scales, which are ideally based on five to seven categories, have been found to be helpful in addressing subjective data. Respondents of self-administered questionnaires use categories at the ends of the rating scales more frequently than do respondents in telephone surveys, which suggests greater data validity. Rating scales can also be used for objective data, such as the net age of the building or dividing the sum of deferred maintenance and rehabilitation costs by the facility's replacement plant value.
Some aspects of the questionnaire have the potential to create bias. For example, many institutions maintain their data to support OMB regulations concerning indirect cost reimbursement and this may lead them to be more likely to say that space is used for sponsored research. Also, some institutions may give self-serving responses when reporting on condition or adequacy of space, either to emphasize the institution's strengths or to encourage greater Federal funding. The prospect of publicly releasing some or all data also may create bias, although in some situations, it has resulted in greater accuracy because of the prospect of outside review.
Because technology has significantly changed the way S&E research is conducted, including, in some cases, changing the role of facilities, NSF added a new section to the survey on computing and networking capacity. A methodology workshop was convened to provide NSF with expertise on IT as it relates to S&E research. Specifically, the workshop identified core concepts related to cyberinfrastructure and its contribution to research capacity and potentially appropriate ways to measure these concepts.
Participants were experts in various aspects of IT and were able to discuss the intersection between IT and scientific research. Exhibit 3 identifies the workshop participants.
Exhibit 3. Participants in methodology workshop on cyberinfrastructure
Daniel Atkins
Professor, School of Information and Department of Electrical Engineering and Science
Director, Alliance for Community Technology
University of Michigan
Kenneth C. Green
Director
The Campus Computing Project
Gurcharan Khanna
Associate Director for Research Computing
Dartmouth College
Hanover, NH 03755
Gregory Monaco
Program Director
Division of Advanced Networking Infrastructure and Research
National Science Foundation
Thomas D. Ndousse,
Computer Scientist
U.S. Department of Energy
H. David Lambert
Vice President and Chief Information Officer
Georgetown University
Michael Zastrocky
Vice President, Academic Strategies
Gartner, Inc.
Before the workshop (held August 22, 2002), NSF interviewed their internal experts on cyberinfrastructure and developed draft questions that were tested during the cognitive interview process. The questions had evolved from a set of 3 very limited questions in March 2002 to 11 sometimes complex questions by the time of the workshop. A repeated finding was that institutions varied widely in how their cyberinfrastructure was organized and in the amount of information they were able to provide, so much of the evolution of the questionnaire accommodated this diversity by continuing to expand lists of items and options. For example, institutions described bandwidth as a complex topic that needed to be examined at multiple levels rather than through a single summary question.
The workshop began with NSF presentations on the results of the initial investigation into cyberinfrastructure-related concepts. The goal of the workshop was for participants to provide potential questions and response options for the survey and to suggest salient topic areas not covered by the draft questions. The workshop discussion covered the following themes:
Scope of cyberinfrastructure. Although it was called a survey on cyberinfrastructure, the draft questionnaire focused primarily on a very narrow aspect of that topic: networking capacity. One way of broadening the scope is to add topics such as personnel and research expenditures, although NSF has separate surveys on those topics and declared that its primary interest for this survey was to measure research capacity. Other aspects that the panel determined to be relevant were cybersecurity, networking and connectivity, storage, data structures, middleware, grids, mobility, computation, hardware, networked instruments, finance/funding, bandwidth, network coverage, and software/applications. Of these, the most important area is networking and connectivity. Sometimes it is difficult to craft a precise definition of what is included in infrastructure (e.g., large fundamental databases could be considered either applications or infrastructure).
Relationship between cyberinfrastructure and research capacity. This relationship varies across disciplines but the major underlying themes are the ability to gather, store, and intelligently retrieve massive amounts of information and the use of advanced computational abilities.
Difference between measuring capacity and usage. Capacity can be measured either in theoretical terms or as actual capacity in terms of applications. For example, the practical meaning of network speed or storage capacity depends on the number of users and the types of applications being used. In the Facilities survey, the focus is on capacity (e.g., the amount of space) rather than on how that space is used. NSF declared its intention to focus on theoretical (or maximum) capacity to maintain an analogous approach.
Networking. Bandwidth is key to measuring networking capacity. Connection speeds depend on several different levels of connections and include external connections, the network's backbone, and desktop ports. There is value in determining both each level's speed and the maximum speed obtainable when allowing for the various levels of connections.
Computation speed. Different configurations may be used to achieve high computation speeds and a wide variety of metrics may be needed because the survey is not confined to elite institutions. Panel members suggested differentiating between speeds achieved through external connections and speeds achieved on campus. They expressed an interest in knowing the highest rate that could be achieved both externally and on campus and the configuration used to achieve that speed.
Storage. Panel members were interested in learning the total amount of storage space available, the size of the largest existing data set, and the size of the largest data set that could be stored.
Data repositories. Panel members expressed an interest in learning about data repositories built and maintained by institutions. They noted that no single person may be knowledgeable about every data repository so the answer might depend on who responds to the question. Therefore, responses may not be comprehensive. Still, asking for such a list might help to identify useful databases and might lead to obtaining information about such databases' funding and support.
Expenditures. The panel expressed a strong interest in obtaining data on cyberinfrastructure funding but noted that collecting such data was likely to be difficult. One question was what types of expenditures should be covered (e.g., infrastructure maintenance and operation). Another practical issue concerned institutions' ability to provide the data given the lack of a common reporting standard and wide variations in how data are maintained.
Middleware. Panel members agreed that middleware is an important concept but were uncertain how to design appropriate questions. They suggested talking with a few designated experts on the topic area.
Following the workshop, NSF and Westat determined which topics were highest priority, developed draft questions, and subsequently conducted cognitive interviews to examine institutions' ability to respond to them.
In addition to the expert panel and the methodology workshop, NSF consulted with Dr. Donald Dillman on general questionnaire issues and with Dr. Kenneth Green and some internal staff members concerning the cyberinfrastructure section.
Dr. Dillman is the Thomas S. Foley Distinguished Professor of Government and Public Policy in the Departments of Sociology and Rural Sociology at Washington State University and Deputy Director for Research and Development in the Social and Economic Sciences Research Center. Before holding these positions, Dr. Dillman was the senior survey methodologist in the Office of the Director, U.S. Bureau of the Census, from 1991 to 1995. A recognized expert in the field of survey methodology, Dr. Dillman reviewed draft survey questions, survey question formatting and layout, and introductory survey materials. He also served as a member of the expert panel. His publications include Mail and Internet Surveys: The Tailored Design Method (2002) and How to Conduct Your Own Survey (with Priscilla Salant) (1994).
Dr. Green is the founder and director of the Campus Computing Project, whose annual Campus Computing Survey studies the role of IT in American colleges and universities through the responses of more than 600 2- and 4-year colleges and universities. He is author or coauthor of several books and articles including Who's Going to Run General Motors? What College Students Need to Learn Today to Become Business Leaders Tomorrow (1991) and "Great Expectations: Content, Communications, Productivity, and the Role of Information Technology in Higher Education" (1995) and has been quoted on higher education, IT, and labor market issues in the New York Times, Washington Post, Los Angeles Times, and Chronicle of Higher Education. Dr. Green served as a participant in the methodology workshop and was consulted outside the workshop on reliability and validity issues associated with measuring IT expenditures.
Because the Facilities survey is a repeating survey, the available data from past iterations proved helpful. NSF ran several tabulations on data from previous Facilities survey cycles to help determine the consequences of certain redesign changes and also to help determine topics that warranted further investigation. For example, with respect to survey eligibility, NSF analyzed how raising the minimum amount of R&D expenditures required for survey eligibility or limiting the survey to doctorate institutions and HBCUs would affect the survey population.
Additionally, tabulations were run to check the data for meaningful responses. For example, a question from the 1999 survey asks about various funding sources for repairs, renovations, and new construction of S&E research facilities. If a respondent could identify the amount of indirect costs recovered from Federal grants and contracts, he or she was asked to provide the amounts for repairs and renovations and for new construction. Very few respondents provided an amount, even if they said they were able to provide a figure. This question was then designated for further investigation in the cognitive interviews and by the expert panel. Through these methods, it became clear that institutional funds are extremely difficult to track, methods for so doing are not comparable across institutions, and institutions hesitate to answer this question even if the data are confidential (as they were with previous surveys).
Two separate pretests of the redesigned survey were conducted. The first, which began in September 2002, included all questions except those on computing and networking capacity (or cyberinfrastructure). In December 2002, a second pretest was conducted solely on the computing and networking capacity section.
Institutions were chosen as pretest sites only if they had responded to both the 1999 and 2001 surveys. They also were chosen to show diversity with respect to region, institution type, size, and research expenditures. HBCUs, along with several institutions that have medical schools (to ensure that questions pertaining to medical school research were pretested), were included.
For the base questionnaire pretest, the initial design called for 9 institutions to pretest all 15 questions. However, institutions often were reluctant to complete the full base questionnaire because of the level of effort required, the time involved, and the fact that the data could not be used as the institutions' 2003 survey responses. To aid institutional recruitment, the survey was split into three sections: questions 1-6 pertained to S&E research space; questions 7-10 pertained to repairs, renovation, and new construction of S&E research space in the past 2 years; and questions 11-15 pertained to planned repairs, renovation, and new construction of S&E research space. The majority of institutions were asked to complete only one of the three sections; however, four academic institutions and three biomedical institutions had agreed to complete all 15 questions. Five additional institutions also were asked to complete each of the three sections so that when all of the responses were combined, the total number completing any single question would be nine academic institutions and three biomedical institutions.
Institutional coordinators were recruited for the pretest via telephone. Those who agreed to participate received a Federal Express package that included a full questionnaire with the sections to be completed clearly designated. (The full questionnaire was provided so that institutions could see the context in which the questions were presented.) Institutions received instructions to fax back both the questionnaire and an accompanying commentary guide providing preliminary comments about the questionnaire. Potential problems that might be encountered during completion of the questionnaires were identified through review of the completed questionnaires and commentary guides. Subsequently, pretest respondents were contacted to identify the reasons (and possible solutions) for any problems that occurred and to provide more indepth comments about the questionnaire (including some pre-established questions about known topics of interest).
The commentary guide requested the following information:
After reviewing the completed questionnaires and commentary guides, researchers conducted telephone interviews to probe farther into any issues raised and to ask about several additional topics for study that had been identified. Topics of special interest included the following:
Table 3 provides a summary of the characteristics of the institutions participating in the pretest.
Table 3. Number of pretest institutions, by institution characteristic
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
[*]Includes both academic and biomedical institutions.
[1] Conrad, F., J. Blair, and E. Tracy. 1999. "Verbal Reports Are Data! A Theoretical Approach to Cognitive Interviews," Federal Committee on Statistical Methodology Conference Paper, http://www.fcsm.gov/events/papers1999.html.
[2] Two additional institutions were interviewed about both the project sheet and cyberinfrastructure but not about the remainder of the questionnaire.
|
|
|
|
|