Technical Notes

Reporting categories | Primary data sources | Sampling errors

The data in this report come from many sources, including surveys conducted by the National Science Foundation (NSF), other Federal agencies, and non-Federal organizations. Many methods of data collection are represented. Therefore, users should take great care when comparing data from different sources. These data often will not be strictly comparable because of differences in definitions, survey procedures, and phrasing of questions, among other things.

Survey accuracy is determined by the joint effects of sampling and nonsampling errors.^[1] Sampling errors arise when estimates based on a sample differ from figures that would have been obtained if a complete population had been surveyed. Nonsampling errors can arise from design, reporting, and processing errors as well as from errors due to faulty responses or nonresponses. Nonsampling errors include respondent-based events, such as some respondents interpreting questions differently from other respondents; respondents making estimates rather than giving actual data; and respondents being unable or unwilling to provide complete, correct information. Errors can also arise during the processing of responses, such as during recording and keying.

Reporting categories

This report draws on data collected from many sources, many of which use differing terms to describe the various statistics presented here. Efforts have been made to maintain consistency throughout this text, but in some data reporting, it has been necessary to use distinct terminology that does not match that used in other compilations.

Racial/ethnic information

The collection and reporting of race/ethnicity data pose several additional problems. First, both the naming of population subgroups and their definitions often have changed over time. Second, many of the groups of particular interest are quite small, so it is difficult to measure them accurately without surveys of the entire population of interest. In some instances, sample surveys may not have been of sufficient scope to permit the calculation of reliable racial/ethnic population estimates; consequently, results are not shown for all groups. The U.S. Bureau of the Census's Current Population Survey, for example, cannot provide data on American Indians. Data on this population are available only from the decennial census. Third, data on race/ethnicity are often based on self-identification. These data are less reliable for certain racial/ethnic groups than for others. Data collected at two points in time indicate that self-identification of American Indians is much less reliable than selfidentification of other racial/ethnic groups.^[2] Fourth, it is easy to overlook or minimize heterogeneity within subgroups when only a single statistic is reported for a total racial/ethnic group.

Information about people with disabilities

Data on people with disabilities who study or work in science and engineering (S&E) are seriously limited for several reasons. First, the operational definitions of disability vary, include a wide range of physical and mental conditions, and thus are not totally comparable. The Americans With Disabilities Act of 1990 (ADA) encouraged progress toward standard definitions. Under ADA, an individual is considered to have a disability if he or she has a physical or mental impairment that substantially limits one or more of his or her major life activities, has a record of such impairment, or is regarded as having such an impairment. ADA also contains definitions of specific disabilities. See http://www.usdoj.gov/crt/ada/pubs/ada.txt.

Second, data on disabilities frequently are not included in comprehensive institutional records (e.g., in registrars' records in institutions of higher education). If included at all, such information is likely to be kept only in confidential files at an office responsible for providing special services to students. Institutions of higher education are unlikely to have information regarding any students with disabilities who have not requested special services. In elementary/secondary school programs receiving funds to provide special education, however, statistics on all students identified as having special needs are centrally available.

Third, information about people with disabilities that is gathered from surveys is often obtained from self-reported responses. Typically, respondents are asked whether they have a disability and to specify what kind of disability it is. Resulting data therefore reflect individual perceptions rather than objective measures.

The attempt to provide estimates of the proportion of the undergraduate student population with disabilities is an example of how these factors coalesce. Self-reported data on the undergraduate student population, collected through a survey to ascertain patterns of student financial aid, suggest that about 10 percent of this population have some disability. Estimates from population surveys of higher education institutions, in contrast, place the estimate much lower, between 1 and 5 percent. Whether this discrepancy is the result of self-perception, incomplete reporting, nonevident disabilities, or differing definitions is difficult to ascertain.

In the final analysis, although considerable information is available about individuals with disabilities in the education system and in the S&E workforce, it is often impossible to compare statistics from different sources.

Several sources of data on people with disabilities are cited here. They include four surveys conducted by the Department of Education's National Center for Education Statistics (NCES); the American Council on Education-University of California-Los Angeles Survey of the American Freshman: National Norms; and NSF's Survey of Earned Doctorates (SED), Survey of Doctorate Recipients (SDR), and National Survey of Recent College Graduates (NSRCG). These sources are described in more detail later in this appendix; the following is a brief description of how each survey treats the issue of disability.

NCES surveys. Four NCES surveys collect disability-related information—the National Education Longitudinal Study, the Beginning Postsecondary Students Longitudinal Study, the Baccalaureate and Beyond Study, and the National Postsecondary Student Aid Study (NPSAS). Text table 1 provides a quick comparison of the disability-related information collected by these surveys.

Characteristic	NELS	BPS	B&B	NPSAS
TEXT TABLE 1. Selected characteristics of NCES disability-related surveys


Survey year	1988	1990/94	1993/97	2000
Questionnaire respondent	Parent	Student	Student	Student
Question as asked in the survey	In your opinion, does your eighth-grader have any of the following problems? AND Has your eighth-grader ever received special services for any or all of the following?	Do you have any of the following conditions?	Do you have any of the following disabilities?	Do you have any disabilities, such as a hearing, speech, or mobility impairment, or vision problems that can't be corrected with glasses?
Disability type as categorized by the survey
Visual impairment	Visual handicap (not correctable by glasses)	Visual handicap	Vision impairment that cannot be corrected with glasses, or are you legally blind?	Legally blind or have a vision impairment that cannot be corrected with glasses
Hearing impairment or deaf	Hearing problem OR deafness	Hard-of-hearing OR deafness	Hearing impairment	A hearing impairment
Speech impairment	Speech problem	Speech disability	Speech disability or limitation	A speech disability or limitation
Orthopedic impairment	Orthopedic problem (for example: club foot, absence of arm or leg, cerebral palsy, amputation, polio)	Orthopedic handicap	Orthopedic or mobility limitation	An orthopedic or mobility limitation
Learning disability	Specific learning problem (for example: dyslexia or other reading, writing, or math disability)	Specific learning disability	Specific learning disability	A specific learning disability
Other disability or impairment	Other health problem (includes mental retardation) OR emotional problem OR other physical disability	Other health impairment	Any other type of limitations, disabilities, or handicaps	Other health-related disability or limitation

B&B Baccalaureate and Beyond Study BPS Beginning Postsecondary Students Longitudinal Study NELS National Education Longitudinal Study NPSAS National Postsecondary Student Aid Study SOURCE: U.S. Department of Education, National Center for Education Statistics, Students With Disabilities in Postsecondary Education: A Profile of Preparation, Participation, and Outcomes, NCES 1999-187 (Washington, DC, 1999), p. 6.

Text Table 1 Source Data: Excel File

Survey of the American Freshman. National Norms. The National Norms survey conducted by the American Council on Education and the University of California-Los Angeles asks whether the student has a disability and, if so, whether the student has a disability such as a hearing, speech, orthopedic, learning, health-related, or other disability. The student is asked to mark all that apply.
NSF surveys. The three NSF surveys, SED, SDR, and NSRCG, provide individual respondents' answers. SED asks if the respondent has a disability, then asks the respondent to mark what category applies to the disability. SDR and NSRCG ask the degree of difficulty—none, slight, moderate, severe, or unable to do—an individual with a disability or disabilities may have in performing life activities. Those respondents who answered "moderate," "severe," or "unable to do" for any activity were classified as disabled. Text table 2 compares SED, SDR, and NSRCG treatment of disability.

Disability (mark all that apply)	Difficulty with physical functions (mark one choice for each)
TEXT TABLE 2. SED, SDR, and NSRCG definitions of disability


Visual	SEEING words or letters in ordinary newsprint (with glasses/contact lenses if you usually wear them)
Auditory	HEARING what is normally said in conversation with another person (with a hearing aid, if you usually wear one)
Orthopedic	WALKING without human assistance or using stairs (mobility)
[No corresponding category in SED]	LIFTING or carrying something as heavy as 10 pounds, such as a bag of groceries
Vocal	[No corresponding category in SDR]
Other	[No corresponding category in SDR]

SDR Survey of Doctorate Recipients SED Survey of Earned Doctorates NSRCG National Survey of Recent College Graduates

Text Table 2 Source Data: Excel File

Primary data sources

Data from several sources are presented here. This section provides summary descriptions of major sources and information about the location of more detailed survey descriptions.

Primary NSF sources

The following sources from NSF's Division of Science Resources Statistics (SRS) were used for data tables in this publication. Published data tables from these surveys can be accessed on the SRS website at http://www.nsf.gov/sbe/srs. In addition, researchers may access data directly from the Scientists and Engineers Statistics Data System (SESTAT) or the WebCASPAR database system, which also can be accessed from the SRS website.

Survey of Earned Doctorates

SED has been conducted annually since 1957 for NSF, the U.S. Department of Education, the National Endowment for the Humanities, the National Institutes of Health, the National Aeronautics and Space Administration, and the U.S. Department of Agriculture. This is a survey of all recipients of research doctoral degrees such as the doctor of philosophy (Ph.D.) or doctor of science (D.Sc.); it excludes the recipients of first professional degrees such as the juris doctor (J.D.) or doctor of medicine (M.D.). Therefore, SED data are restricted to research doctorates.

Data for SED are collected directly from individual doctorate recipients contacted through graduate deans at all U.S. universities awarding research doctorates. The recipients are asked to provide information about the field and specialty of their degree as well as their personal educational history, selected demographic data, and information about their postgraduate work and study plans. Over time, approximately 95 percent of the annual cohort of doctorate recipients respond to the questionnaire.

Partial data from public sources, such as field of study, are added to the file for nonrespondents. No imputations are made, however, for nonresponse for data not available elsewhere, such as race/ethnicity information. The data for a given year include all doctorates awarded in the 12-month period ending on June 30 of that year. Information about SED can be found at http://www.nsf.gov/sbe/srs/ssed/start.htm.

Survey of Graduate Students and Postdoctorates in Science and Engineering

The data collected in the Survey of Graduate Students and Postdoctorates in Science and Engineering (GSS) represent national estimates of graduate enrollment and postdoctoral employment at the beginning of the academic year in all academic institutions in the United States that offer doctoral or master's degree programs in any S&E field. Included are data for all branch campuses; affiliated research centers; and separately organized components such as medical or dental schools, schools of nursing, and schools of public health. In fall 2001, the survey population consisted of 721 reporting units at 606 graduate institutions. Data are collected at the academic department level. In fall 2001, the response rate was 98.5 percent for institutions and 99.0 percent for departments.

Available information includes full-time graduate students by source and mechanism of support, including data on women and first-year students enrolled full time, part-time graduate students by sex, and citizenship and racial/ethnic background of all graduate students. In addition, detailed data on postdoctorates are available by source of support, sex, and citizenship, including separate data on those holding first professional doctorates in the health fields; summary information about other doctoral nonfaculty research personnel is also included.

NSF has collected data on graduate S&E enrollment and postdoctoral appointees since 1966. From fall 1966 through fall 1971, data from a limited number of doctorate-granting institutions were collected through the NSF Graduate Traineeship Program, which requested data only on those S&E fields supported by NSF. Beginning with the fall 1972 survey, this data collection effort was assigned to SRS. It was gradually expanded during the period 1972-75 to include additional S&E fields as well as all institutions known to have programs leading to the master's or doctoral degree. Because of this expansion, data for 1974 and earlier years are not strictly comparable with 1975 and later data. Information about GSS can be found at http://www.nsf.gov/sbe/srs/sgss/start.htm.

Survey of Doctorate Recipients

SDR is a longitudinal panel survey of individuals who have received their doctorates in the United States in the sciences or engineering. Since the 1970s, this study has been conducted every 2 years for NSF and other Federal sponsors. It provides information about educational background, occupation, employment, and demographic characteristics of S&E doctorate holders.

The sampling frame for the 2001 SDR included individuals who had earned a doctoral degree from a U.S. college or university in an S&E field; were U.S. citizens or, if non-U.S. citizens, indicated that they had plans to remain in the United States after receiving their degrees; and were under 76 years of age. In 2001, the SDR sample size was 40,000. The overall unweighted response rate for the 2001 SDR was 82.2 percent. The overall weighted response rate was 82.6 percent (weighted responses divided by the number of weighted sample cases). To minimize the effect of nonresponse error, results were adjusted for nonresponse through the use of statistical weighting techniques.

SDR classifies the following broad categories as S&E occupations: computer and mathematical scientists, life and related scientists, physical and related scientists, social and related scientists, and engineers. Postsecondary teachers are included within each of these groups. The following are considered non-S&E occupations: top and midlevel managers; teachers, except S&E postsecondary teachers; technicians/technologists, including computer programmers; people in health and related occupations; social services and related occupations; sales and marketing occupations; and other non-S&E occupations—for example, artists, broadcasters, editors, entertainers, public relations specialists, writers, clerical and administrative support personnel, farmers, foresters, lawyers, judges, librarians, archivists, curators, actuaries, food service personnel, historians (except science and technology), architects, construction tradespeople, mechanics and repairers, and those involved in precision/production occupations, operators (for example, machine setup operators, machine operators and tenders, fabricators, assemblers) and related occupations, transportation/material-moving occupations, and protective and other service occupations. Information about SDR can be found at http://www.nsf.gov/sbe/srs/ssdr/start.htm.

Primary non-NSF sources

The following non-NSF sources were used for data tables in this report.

The Integrated Postsecondary Education Data System Survey: Fall Enrollment, Completions, and Institutional Characteristics

Contact: National Center for Education Statistics
U.S. Department of Education
1990 K Street, NW
Washington, DC 20006
(202) 502-7300
http://nces.ed.gov/ipeds

The Integrated Postsecondary Education Data System (IPEDS) Survey began in 1986 as a supplement to and replacement for the Higher Education General Information Survey (HEGIS), which began in 1966. HEGIS annually surveyed institutions listed in the current NCES Education Directory of Colleges and Universities; IPEDS surveys all postsecondary institutions, including universities and colleges and the institutions that offer technical and vocational education. IPEDS consists of several integrated component surveys that obtain information about types of institutions where postsecondary education is available, student participants, fall enrollments, programs offered and completed, graduation rates, and the human and financial resources involved in the delivery of postsecondary education. Descriptions of these surveys follow.

The IPEDS Institutional Characteristics Survey provides the basis for the list of institutions reported in the Education Directory of Colleges and Universities. The list includes institutions that meet specific accreditation criteria and offer at least a 1-year program of college-level studies leading to a degree. Each fall, institutions listed in the previous year's directory are asked to update information about their school's characteristics.

The IPEDS Completions Survey replaces and extends the HEGIS Degrees and Other Formal Awards Conferred Survey. It is administered to all institutions offering degrees at the bachelor's degree level and above, 2-year institutions, and less-than-2-year institutions.

The IPEDS Fall Enrollment Survey replaces and extends the previous HEGIS surveys of enrollment in institutions of higher education.

The National Postsecondary Student Aid Study

Contact: National Center for Education Statistics
U.S. Department of Education
1990 K Street, NW
Washington, DC 20006
(202) 502-7300
http://nces.ed.gov/npsas

NPSAS was established by NCES to collect information about financial aid allocated to students enrolled in U.S. postsecondary institutions. NPSAS was first administered in the fall of the 1986-87 academic year. NCES conducted subsequent cycles of NPSAS during the 1989-90, 1992-93, 1995-96, and 1999-2000 academic years.

The 1999-2000 survey gathered information from about 62,000 undergraduate and graduate students selected from registrars' lists of enrollees at more than 900 postsecondary institutions. The sample included students who did and did not receive financial aid, as well as students' parents. Student information, such as field of study, educational level, and attendance status (part time or full time), was obtained from registrars' records. Types and amounts of financial aid and family financial characteristics were abstracted from school financial aid records. Data pertaining to family circumstances, background demographic data, educational and work experiences, and expectations were collected from students using a computer-assisted telephone interview. The response rate for obtaining institutional record data for all students was 97.0 percent, and the weighted overall student interview response rate was 65.6 percent.

Survey of Engineering and Technology Enrollments and Survey of Engineering and Technology Degrees

Contact: Matt Doster
Engineering Workforce Commission
American Association of Engineering
Societies
1111 19th Street, NW
Suite 403
Washington, DC 20036
(202) 546-2237
http://www.ewc-online.org/

For 35 years, the Engineering Workforce Commission (EWC) has conducted annual surveys of enrollments and degrees conferred in engineering programs. In 2002, EWC collected data on engineering enrollments and engineering degrees conferred from 351 and 347 institutions, respectively, including all of those with curricula approved by the Accreditation Board for Engineering and Technology (ABET). EWC counts the number of students studying for engineering degrees at all ABET-accredited engineering schools throughout the United States. Historically, EWC has also included schools that are not ABET accredited for a variety of reasons unique to each school. Some schools are in the process of obtaining ABET accreditation; others have simply asked to be included in the survey. The response rate for the 2002 enrollment survey was 87 percent, and the response rate for the 2002 degrees survey was 92 percent. Each year, EWC obtains data from all schools included in the previous year's survey to ensure accurate time-series comparisons.

Sampling errors

Sampling errors occur when estimates are derived from a sample rather than the entire population. The sample used for any particular survey is only one of a large number of possible samples of the same size and design that could have been selected. Even if all other aspects of the survey remained fixed, such as the questionnaire and instructions, the estimates from each sample would differ from other samples. This difference, termed sampling error, occurs by chance, and its variability is measured by the standard error associated with a particular estimate.

The standard error of a sample survey estimate measures the precision with which an estimate from one sample approximates the true population value, and it can be used to construct a confidence interval for a survey parameter to assess the accuracy of the estimate. See the technical notes section of Characteristics of Doctoral Scientists and Engineers in the United States: 2001 for information about calculation of standard errors for data from SDR.

Footnotes
[1] In all surveys cited as sources of data for this report, efforts were made to minimize these errors.

[2] U.S. Bureau of Labor Statistics, A Test of Methods for Collecting Racial and Ethnic Information (Washington, DC: U.S. Department of Labor, 1995).