SESTAT Methodological Research Compendium

This compendium brings together the special SESTAT research which has been done to provide a continuing basis for methodological improvements in the surveys and to give data users a better source of information on the quality of data in the system.

The compendium includes a wealth of findings on survey design, population coverage, data collection methods, response issues, data quality assessment, weighting and estimation. These collected papers include those presented at professional meetings, as well as unpublished works. As the SESTAT system moves forward, additional research papers will be added to this compendium.

Note that the SESTAT system also contains a separate "Methodology" section which describes the methods and procedures used for each of the surveys. In addition, technical and quality information is provided for each variable in the system.


Contents

The Compendium is organized by the research topics shown below, with the research papers associated with each topic listed under each topic heading. Click on the the research paper titles to review the paper abstract and to download the paper.

If you would like additional information about one of the papers in the Compendium, please either contact the author directly or send an email to the individual listed as "alternative contacts" in the paper's citation.

  1. Population Coverage, Definitions, and Sample Frame Issues


  2. Sample Design


  3. Response and Nonresponse Studies


  4. Weighting and Variance Estimation


  5. Research in Survey and Data Quality: Reinterview Reports


  6. Research in Survey and Data Quality: Interviewer Effects


  7. Research in Survey and Data Quality: Mode of Data Collection


  8. Research in Survey and Data Quality: Coding Evaluations


  9. General and Other SESTAT Research

NOTE: The SESTAT research papers are ordered and numbered by section and date of publication/completion, beginning with the most recent articles. As a result, numbering in each section begins at x.31 to allow for additional articles to be added to the compendium over time.

You may download a pdf version of any of the papers included in the compendium. If you are not able to download, you may send a request for a copy to SESTAT@NSF.GOV, giving the number and title of the paper desired and your mailing address.


Alternative Contacts

If you would like additional information about one of the papers in the Compendium, please either contact the author directly or send an email to the individual listed as "alternative contacts" in the paper's citation. E-mail addresses and telephone numbers for these contacts are provided here:

Nirmala Kannankutty
National Science Foundation
E-mail:
nkannank@nsf.gov
Phone: (703) 292-7797

Dave Edson
Mathematica Policy Research, Inc.
E-mail:
Dedson@Mathematica-Mpr.Com
Phone: (202) 484-4683

Mary Collins
Westat
E-mail:
mailto:COLLINM1@WESTAT1.COM
Phone: (301) 251-4273


SESTAT Methodological Research Compendium

Abstracts and Links to Documents

 

1. Population Coverage, Definitions, and Sample Frame Issues

 

1-31.pdf

Brick, J. Michael; Collins, Mary A.; and Gray, Lucinda. (1997). A Comparison of Estimates in the NSRCG and IPEDS. Rockville, MD: Westat, Inc.

Keywords: Survey estimates, estimation methods, sampling errors, measurement errors, non-sampling errors, estimation issues

Alternate Contact Person: Mary Collins

SESTAT Survey: NSRCG

Abstract: The NSRCG and IPEDS both produce estimates of the number of graduates and number of degree awards by race. Comparing the estimates from 1991 through 1994 showed that the total number of graduates from each survey differed by 4 to 5 percent for bachelor's degree recipients and 10 to 15 percent for master's degree recipients. This paper examines the reasons for these differences, including differences in the design or reporting procedures used in the two surveys. It also attempts to quantify the amount of difference that may be attributable to each cause. The main topics of discussion include differences in the survey's target populations, data collection methods, and statistical procedures. The investigation reveals that the NSRCG and IPEDS estimates are comparable when adjustments are made for differences in target populations and for the sampling errors of the NSRCG estimates.

 

1-41.pdf

Burns, Genny. (1996). 1993 National Survey of College Graduates Followup Results from the Uncertain Scientist and Engineer Status Study. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Followup study, scientist and engineer status change, undercoverage, incorrect classifications

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This study is one of a group of special research projects done as part of the 1993 NSCG dealing with sample frame issues. It involved a followup study of 78 respondents who switched from Science and Engineering (S&E) status to non-S&E status and vice versa during a reinterview of NSCG:93 participants. Based on the results of the reinterview, it was estimated that 6.6 percent of the non-S&E respondents in NSCG:93 would switch to S&E if reinterviewed which could lead to undercoverage in NSCG:95. During the followup study, several switchers were contacted for a brief interview to evaluate the accuracy of the undercoverage estimate. The responses of the switchers were also examined to determine if there were any patterns in reporting responses that might suggest problems with questionnaire wording.

 

1-51.pdf

Burns, Genny. (1996). National Survey of College Graduates Follow-up Results from the No Bachelor's Degree Study. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Followup study, no bachelor's degrees, quality of sample, incorrect classification

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey NSCG:93

Abstract: This study is one of a group of special research projects done as part of the 1993 NSCG dealing with sample frame issues. During NSCG:93, about 8.5 percent of the respondents were classified as out-of-scope because they had not earned a bachelor's degree or higher by 1990. This was cause for some concern as the NSCG:93 sample was drawn from a set of persons in the 1990 decennial census who reported having received a bachelor's degree or higher. A sample of 300 out-of-scope cases were contacted for followup interviews in 1995 to determine how many were misclassified in 1993. An examination was also made of the unusually high percentage of cases with no bachelor's degrees that were completed by Field Representatives (FRs) during NSCG:93. Even after excluding the work done by FRs and accounting for 20 percent of the followup respondents saying they had earned bachelor's degrees by 1990, the no bachelor's degree rate of 6.1 percent was still higher than the results of the Census content reinterview survey of the 1990 decennial census.

 

1-61.pdf

Burns, Genny. (1996). National Survey of College Graduates Follow-up Results from the Emigrants Survey. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Followup study, emigrants, quality of sample, undercoverage, incorrect classification, survey eligibility

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This study is one of a group of special research projects done as part of the 1993 NSCG dealing with sample frame issues. During NSCG:93, all respondents that were classified as out-of-scope because they were not living in the United States during the reference week were permanently dropped from the NSCG sample due to the poor prospects and high costs of completing future interviews. This followup study interviewed 100 persons classified as out-of-scope during the NSCG:95 study to determine how many people would have been eligible to participate in NSCG:95. Based on the results of the followup, it was determined that permanently excluding out-of-scope individuals contributed a maximum of .34 percent to NSCG:95 undercoverage.

 

1-71.pdf

Finn, Michael, and Baker, Joe G. (1996). Reconciling Estimates of U.S. Doctorates. Oak Ridge, TN: Oak Ridge Institute for Science and Education.

Keywords: Population coverage, educational and occupational definitions

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: NSCG, SDR

Abstract: This investigation addresses differences in estimates of U.S. doctorates from the NSCG and SDR. The authors address problems associated with definitions of eligible persons, respondent misrepresentation (inflation) of educational credentials, and differences in the treatment of various fields of study and occupations that bear on eligibility. The authors address the question of the degree of discrepancy attributable to these measurement problems, but note the inability to quantify the contribution of each with any degree of confidence.

 

Note: This report led to the understanding that, in the NSCG93, some doctors of medicine (MDs) had incorrectly reported their degrees in the "Doctorate" (i.e. PhD) degree category. The 1993 NSCG data was then re-edited to insure that respondents working as physicians had their degree recorded as a Professional Degree (e.g. MD), rather than a Doctorate. In addition, labeling on later SESTAT surveys was changed to make the Doctorate/Professional degree distinction clearer to respondents.

 

1-81.pdf

Hicks, Lloyd, and Moore, Thomas F. III. (1996). Quality of Initial Names and Addresses in the 1993 National Survey of College Graduates. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Locating graduates, quality of initial contact information, address checks, analysis of bad name cases

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This paper reports on two investigations of the quality of the initial names and addresses used for NSCG:93. The first study examines the quality of locating information that was obtained from various locating services based on the initial address provided. The second study examines several demographic variables to determine if any subdomains experienced a disproportionately large number of unmailable cases due to bad names. The distribution of bad name cases is compared to that of the final NSCG sample for five variables: highest degree, occupation, age, sex, and a special NSF variable that incorporates race, ethnicity, disability status, citizenship status, and place of birth. It was found that the distribution of bad name cases was statistically different from the distribution of the NSCG:93 final sample for all variables except sex.

 


2. Sample Design

 

2-31.pdf

Cox, Brenda G.; Mitchell, Susan B.; and Moonesinghe, Ramal. (1998) Current and Alternative Designs for the Survey of Doctorate Recipients. Report submitted to the National Science Foundation under subcontract to the National Research Council. Washington, DC: Mathematica Policy Research, Inc.

Keywords: Sample design, sampling frames, sampling issues, survey design changes, instrument changes, refreshed panel design, rotating panel design

Contact Person: Dave Edson

SESTAT Survey: SDR

Abstract: The Survey of Doctoral Recipients has undergone many significant changes since its inception. This report presents a historical account of theses changes and highlights strengths and weaknesses of the current design. Two alternative designs are also presented. These alternatives would involve (1) a refreshed panel design or (2) a rotating panel design. Advantages and disadvantages of each design are explored.

 

2-41.pdf

Mitchell, Susan B.; Moonesinghe, Ramal; and Cox, Brenda G. (1998). Using the Survey of Doctorate Recipients in Time-Series Analyses: 1989-1995. Report submitted to the National Science Foundation under subcontract to the National Research Council. Washington, DC: Mathematica Policy Research, Inc.

Keywords: Response rates, sample design, sampling frames, weight adjustment, survey design changes, instrument changes, time-series analysis

Contact Person: Dave Edson

SESTAT Survey: SDR

Abstract: From 1989 to 1995, the Survey of Doctorate Recipients experienced significant changes in instrument and sampling design as well as in response rates. This paper explores the implications of these changes for time series analysis. The authors conclude that caution must be exercised in constructing and interpreting time series that span the 1980s and 1990s.

 

2-51.pdf

Moonesinghe, Ramal. (1998). Sample Design and Precision in the National Survey of Recent College Graduates: 1993, 1995, and 1997. Rockville, MD: Westat.

Keywords: Sample design, precision of estimates

Alternate Contact Person: Dave Edson

SESTAT Survey: NSRCG

Abstract: This paper addresses the effect of sample design changes in the NSRCG on the precision of survey estimates. First, an overview of the sample designs of the 1993, 1995 and 1997 NSRCG surveys is provided, including both the institutional and graduate levels. Changes in the design are noted. The next section of the paper compares sampling rates for each of the three survey years. Estimates from the 1993 and 1995 surveys are compared to examine precision. The results indicate that changes to the NSRCG sample design have resulted in an increase in the precision of estimates of black and Hispanic graduates without any significant loss in precision for white graduates or for all graduates.

 

2-61.pdf

Hardy, Linda. (1992). Balancing multiple demands: Sample redesigns for the NSF system. Paper presented at the annual meeting of the American Statistical Association.

Keywords: Sampling, sample design, sampling frames, sampling problems, sampling goals

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: New Entrants (NSRCG), NSCG, SDR

Abstract: As part of a major survey redesign effort undertaken by the National Science Foundation in the 1990s, it was necessary to develop integrated sampling goals and principles for the survey system and to overcome previously experienced sampling problems. This paper examines the survey sampling frames and coverage as well as changes made to each of the three component surveys on science and engineering graduates and personnel. It also discusses sample design goals and issues and sample allocation guidelines that were used in the 1993 SESTAT surveys. Greater detail on the individual survey samples can be found on the SESTAT website.

 

 


3. Response and Nonresponse Studies

 

3-31.pdf

Briseno, Christina. (1996). The National Survey of College Graduates Locating Test Summary Report. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Tracing, telephone tracing, locating graduates, locating procedures, sources of location information

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG

Abstract: This paper discusses a test that was conducted to determine if sampled persons could be located from telephone centers. The test was also used to determine which resources and procedures were the most useful for locating sample members. The results of the study showed that if the test procedures had been used during NSCG:93, the interviewers could have located and interviewed more sample members. In addition, it was suggested that interviewers needed to receive training, guidelines, and standard documenting procedures for locating sampled persons.

 

3-41.pdf

Mooney, Geraldine; Giesbrecht, Lee; and Shettle, Carolyn. (1993). To Pay or Not to Pay; That is the Question. Paper presented at the 48th Annual Conference of the American Association for Public Opinion Research, St. Charles, Illinois, May 20-23, 1993.

Keywords: Incentives, monetary incentives, cooperation rates, nonresponse bias, data quality

Alternate Contact Person: Dave Edson

Survey population: Three-year old retired CPS sample of college educated people

Abstract: This paper discusses a 1992 pretest for the NSCG. Using four treatment groups, the goal of the study was to determine the effect of questionnaire length and the use of incentives on the cooperation rate, reducing S&E bias, data quality, and "commitment." One of the four groups received a long form version of the questionnaire and a $5 check in the first mailing. A second group also received the long form version of the questionnaire but without the monetary incentive. The other two treatment groups received shorter questionnaires without any financial incentives. The authors found that without an incentive, questionnaire length is inversely proportional to the cooperation rate. They also determined that using an incentive could significantly increase the cooperation rate for a mail survey with one mailing. The results also showed that data quality was not affected by questionnaire length or use of an incentive, S&E bias was only marginally reduced by using incentives, and commitment to being contacted for followups increased with questionnaire length and the use of incentives.

 

3-51.pdf

Moonesinghe, Ramal; Mitchell, Susan, and Pasquini, Daniel. (1995). An Identification Study of Nonrespondents to the 1993 Survey of Doctorate Recipients. Published in Proceedings of the Section of Survey Methods. Alexandria, VA: American Statistical Association.

Keywords: Nonresponse bias, pooled wave estimates, survey estimates, response rates

Alternate Contact Person: Dave Edson

SESTAT Survey: SDR:93

Abstract: This paper identifies differences in response rates to the 1993 Survey of Doctorate Recipients by subgroups, and assesses the effect of these differences on the quality of survey estimates. Subgroups considered include wave of data collection (i.e., first mailing, second mailing, and CATI followup), field, race, and citizenship. The paper also examines how estimates differ based on respondents to different waves. The results showed that while there were few variations in response rates for the SDR subgroups, the variations were important. The results also showed that, in most cases, the interim survey estimates closely resembled the final survey estimates.

 

3-61.pdf

Severynse, Jacqueline. (1995). Determining Significant Differences in New Entrants Flyer Mailing Experiment. Rockville, MD: Westat, Inc.

Keywords: Flyer mailing methods, response rates, incentives

Alternate Contact Person: Mary Collins

SESTAT Survey: 1993 New Entrants (NSRCG:93)

Abstract: This memorandum discusses the results of a flyer mailing experiment aimed at improving survey response rates. Four different types of mailings were used: (1) offer Federal job information/business reply envelope, (2) offer nothing/business reply envelope, (3) offer study results/stamped return envelope, and (4) offer study results/business reply envelope. The results indicate that a higher response rate was achieved using the study results/stamped return envelope than the other flyer mailing methods. Only the stamped return envelope condition had differences that were significant.

 

3-71.pdf

Tremblay, Antoinnette, and Moore, Thomas F. III. (1995). Nonresponse Issues of the National Survey of College Graduates. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Nonresponse, reasons for nonresponse, CART analysis, distinguishing respondents from nonrespondents

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This paper discusses issues of nonresponse from the NSCG:93 study. More specifically, the goal of the nonresponse investigation was to determine if any of fourteen 1990 decennial census demographic variables could reliably distinguish between NSCG:93 respondents and nonrespondents. The authors examine the relationships between each demographic variable and response/nonresponse for the whole dataset, between each variable and the reason for nonresponse reported by the nonrespondents, and between each variable and response/nonresponse for the whole dataset by mode of data collection. The data analysis focuses on response rates, Classification and Regression Trees (CART) statistical software classification analysis, chi-square analysis, and regression.

 

3-81.pdf

Mitchell, Susan; Mooney, Geraldine; and Shettle, Carolyn. (1994). The Impact of Priority Mail in a Mixed Mode Federal Survey. Published in Proceedings of the Section on Government Statistics. Alexandria, VA: American Statistical Association.

Keywords: Mail survey, mailing experiment, response rates, priority mail

Alternate Contact Person: Dave Edson

SESTAT Survey: SDR:93

Abstract: This paper examines the impact priority mailings had on the response rates, cost per completed case, and length of the field period for SDR:93. In order to conduct the study, the SDR:93 sample was divided into three groups (foreign mailing addresses were excluded). Group 1 received a first-class mailing followed by priority mail in the second wave of mailings. Group 2 received two waves of first-class mail followed by a priority mailing. Group 3 received three first-class mailings. All three mailing methodologies yielded comparably high response rates, so the authors dedicate most of their discussion to a comparison of tradeoffs between costs and fielding time. Based on the findings, it was determined that the SDR:95 would receive one first-class mailing followed by a priority mailing and CATI followup.

 

3-91.pdf

Wilkinson, R. Keith, and Hines, Catherine Jucius. (1992). Data Collection Methods in the STPDS Surveys: Improving Response Rates in Mixed Mode Surveys. Paper presented at the annual meeting of the American Statistical Association.

Keywords: Mailing experiments, CATI, questionnaire length, response rates

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: New Entrants (NSRCG), NSCG, SDR

Abstract: This paper discusses factors that have contributed to low response rates in the Scientific and Personnel Data System surveys. Methods for improving response rates that have been tested in these surveys are also detailed. Among the promising changes proposed for the surveys to be conducted in 1993 include the use of the National Change of Address file to update sample member contact information; improving the appearance of mailing and contact materials; including a respondent pre-notification letter; compressing the data collection schedule; and investing in complete telephone follow-up for mail non-respondents.

 

3-101.pdf

Mooney, Geradine and Cheryl De Saw. (1995). 1993 SESTAT: Item Nonresponse

Keywords: Item nonresponse

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: New Entrants (NSRCG), NSCG, SDR

Abstract: This report reviews item nonresponse in the three 1993 SESTAT questionnaires. Using preliminary mail questionnaire data from the SDR and NSCG surveys and the NSRCG Computer Assisted Telephone Interviewing (CATI) data, the report identifies potential problem areas and offers recommendations for the issues raised.


4. Weighting and Variance Estimation

 

4-31.pdf

Jang, Don D.; Cox, Brenda G.; and Edson, David J. (1997). Generalized Variance Functions for Data from Multi-frame Surveys: The SESTAT Experience. Published in Proceedings of the Section on Government Statistics. Alexandria, VA: American Statistical Association.

Keywords: Generalized variance functions, variance approximation, global GVF, integrated GVF, standard errors

Alternate Contact Person: Dave Edson

SESTAT Surveys: NSRCG, NSCG, SDR

Abstract: This paper examines the creation of generalized variance functions (GVFs) for each domain of interest in SESTAT. This global approach produced well-predicted variance estimates but could only provide standard errors for the combined SESTAT data set. The researchers sought to improve on this model by combining three survey-specific GVFs to form an integrated GVF. This integrated GVF was used to predict standard errors for the component surveys, and provided improved standard error prediction over the global model. However, since the global model provided GVFs of acceptable quality and was simpler to use, it was adopted for SESTAT variance approximation.

 

4-41.pdf

Hicks, Lloyd, and Riker, Caroline. (1996). Impact of Weighting on the 1993 National Survey of College Graduates. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Weighting, ratio adjustment, nonresponse adjustment, unit nonresponse, under-coverage

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This report shows the effect of the ratio adjustment weighting procedure on estimates of major NSCG:93 sampling cells. The 863 sampling cells were combined to form 144 ratio adjustment cells. These cells were defined by education, occupation, sex, and NSF group which includes race, ethnicity, disability status, U.S. citizenship, and place of birth. The ratio adjustment procedure was used to inflate the weights of interviewed and out-of-scope persons to account for unit nonresponse and undercoverage.

 

 


5. Research in Survey and Data Quality: Reinterview Reports

 

5-31.pdf

Bushery, John M.; Brick, J. Michael; Severynse, Jacqueline; and McGuinness, Richard A. (1996). Interview Mode Affects Data Reliability.Published in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Keywords: Data reliability, response error, interview mode, reinterview, data quality, response variance

Alternate Contact Person: Mary Collins

SESTAT Surveys: NSRCG:95, NSCG:93

Abstract: This paper compares the simple data response variance obtained for identical questions in the mixed-mode mail/CATI NSCG:93 and the all-CATI NSRCG:95. It includes discussions of the 1993 NSCG and the 1995 NSRCG surveys and their respective reinterviews as well as an examination of the comparability of the NSCG and NSRCG surveys. Their analysis shows that mixed-mode surveys have a slight edge in data reliability over all-CATI surveys; however, other issues, such as timeliness and cost, must also be considered when selecting interview mode.

 

5-41.pdf

Lessard, James M., and Bushery, John M. (1995). Response Variance in the 1993 National Survey of College Graduates. Reinterview Report. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Response variance, reinterview, question reliability

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This report provides measures of response variance for selected questions in the NSCG:93 reinterview. Only cases that were originally completed by mail or CATI were reinterviewed. The report summarizes the reinterview methodology design and major findings. It also provides a detailed report of questions that had a response variance index above 30.

 

5-51.pdf

Lessard, James M., and Bushery, John M. (1995). The 1993 NSCG Reinterview: Investigation of Cases With Uncertain Non-Science/Engineering Status. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Reinterview, response variance, survey undercoverage, education and occupation codes

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This report examines data collected from the NSCG:93 reinterview to investigate the possible undercoverage on the NSCG:95 resulting from respondents being erroneously classified as non-S&E in NSCG:93. It also discusses the specific job and education codes most likely to be misclassified between S&E and non-S&E.

 

 


6. Research in Survey and Data Quality: Interviewer Effects

 

6-31.pdf

Brick, J. Michael; McGuinness, Richard; Lapham, Susan J.; Cahalan, Margaret; Owens, Dedrick; and Gray, Lucinda. (1995). Interviewer Variance in Two Telephone Surveys. Rockville, MD: Westat, Inc.

Keywords: Interviewer effect, variance of estimates

Alternate Contact Person: Mary Collins

SESTAT Surveys: NSRCG:93, NSCG

Abstract: Survey sampling errors do not account for the possibility that the telephone interviewers conducting the study may introduce errors in the responses and that the errors may be correlated between graduates. This report examines the interview effect and its impact on the underestimation of the variance of the estimates. It focuses primarily on responses obtained during the 1993 National Survey of Recent College Graduates, but the results of the interviewer study are compared with results from a similar study of the National Survey of College Graduates data conducted by the Bureau of Census.

 

6-41.pdf

Cahalan, Margaret; Mitchell, Susan; Gray, Lucinda; Chen, Selma; and Tsapogas, John. (1994). Recorded Interview Behavior Coding Study: National Survey of Recent College Graduates. Published in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Keywords: Behavior coding, questionnaire design, cognitive methods, data quality

Alternate Contact Person: Mary Collins

SESTAT Survey: NSRCG:93

Abstract: This study relies on behavior coding as a means to evaluate the reliability and validity of questions appearing on NSRCG:93. Ninety-eight interviews were taped and reviewed by behavior coders who examined interviewer and respondent behavior. The findings examine interviewer variation from the text, probing, affect, whether the respondent provided a "correct" response, and respondent requests for repetition and clarification. The analysis also includes a special look at employment and education questions with less than 85 percent in the "asked and answered only" category. The authors identify the general types of questions with higher than average rates for problem codes. They also acknowledge that the positive results obtained during this study may be due, in part, to the fact that both the interviewers and the respondents knew the interview was being taped. It is recommended that the study be repeated using a larger group of interviewers who are unaware of the reason the interview is being recorded.

 

6-51.pdf

Ringstrom, Danielle; Dedrick Owens; and Richard McGuinness (1994). Interviewer Variance in the 1993 National Survey of College Graduates.

Keywords: Behavior coding, questionnaire design, cognitive methods, data quality

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract:This report measures the extent to which CATI interviewers influenced responses to NSCG questions. The paper measures levels of between- interviewer variance in CATI data for selected questions. Interviewer effects occurred in 86 of the 180 response categories analyzed, but only 15 of these 86 response categories had substantial interviewer effects.

 

 


7. Research in Survey and Data Quality: Mode of Data Collection

 

7-31.pdf

Keathley, Donald; Riker, Caroline; and Hicks, Lloyd. (1995). Results of the 1993 National Survey of College Graduates Modal Study. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Interview mode groups, estimates of differences between mode groups, response rates, item nonresponse rates

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:93

Abstract: This memorandum describes the results of the NSCG:93 modal study. The study compares estimates between the interview mode groups (mail and telephone). The estimates are compared for twelve characteristics for five occupation groups. The results show that the mail group had significantly higher nonresponse rates for the twelve characteristics than the telephone group. The reasons for these higher nonresponse rates are considered on a question-by-question basis. In general, however, most of the questions had nonresponse rates of less than 3 percent for surveys administered by mail and phone.

 

7-41.pdf

Mooney, Geraldine M., and Carlson, Barbara Lepidus. (1996). Reducing Mode Effects in "Mark All That Apply" Questions. Published in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Keywords: Primacy effects, recency effects, mode effects, question format

Alternate Contact Person: Dave Edson

SESTAT Survey: NSCG:93

Abstract: This paper explores the extent to which mode effects in "Mark All That Apply" questions are reduced when such questions are reformatted to require a "yes or no" response after each response category. The authors compare mail and telephone data from six questions that were rewritten using the "yes/no" format to see if these revisions reduced or eliminated primacy effects on mail questionnaires and recency effects on telephone questionnaires. They also examine the positioning of the questions within the surveys and performance measures used to evaluate the success of the "yes/no" format in reducing mode effects. The results show that the "yes/no" format minimized primacy and recency effects but another mode effect emerged-when the questions were administered by telephone, there was an overall increase in the average number of "yes" responses. The results also show that question position in the questionnaire and the number of response categories did not fully explain the modal differences in the total number or proportion of "yes" responses. However, when the authors used cognitive burden and an index combining position, number of response categories, and cognitive burden, they found these measures to be positively correlated with larger differences by mode. This suggests that reformatting may have had less of an effect on questions that were more difficult to answer.

 

 


8. Research in Survey and Data Quality: Coding Evaluations

 

8-31.pdf

Rak, Rebecca; Chen, Selma; and Gray, Lucinda. Westat, Inc. (1997). Occupation Coding: Best Coding and CATI Coding Methods. Rockville, MD: Westat, Inc.

Keywords: Occupation coding, best coding

Alternate Contact Person: Mary Collins

SESTAT Survey: NSRCG:95

Abstract: This report discusses the best coding process used to evaluate the accuracy of respondent-selected occupation codes during the 1995 National Survey of Recent College Graduates. It includes an overview of the different CATI occupation coding methods that were used during survey administration and the occupational worksheet generated to examine the occupation code selected for each occupation reported. The report also includes a comparison of changes made to self-selected codes by coding method as well as a comparison of the time it took to code an occupation by coding method.

 

8-41.pdf

Hardy, Linda P., and Eisenhower, Donna L. (1994). Developing Methods for Collecting and Coding the Occupation of Persons with College Degrees. Published in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Keywords: Occupation coding, occupation data collection

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: NSRCG, NSCG, SDR

Abstract: This paper reviews traditional methods used to collect occupational data and research done on their applicability to the SESTAT population and collection. It analyzes open-ended and structured types of question formats and the types of errors associated with each. It also explains the development of NSF's "Best Coding" process, which attempts to mitigate the errors found in these data collection methods. During best coding, coders use all relevant data captured on the questionnaire to correct respondent errors. NSF also developed occupational coding sheets, coder reference materials, and coder training procedures for use during the best coding process. The end results of the redesign effort were to revise the open-ended questions and occupation list. Finally, the paper discusses initial reports on the functioning of the new coding system and plans for further analysis.

 

8-51.pdf

McGuinness, Richard A. (1997). 1995 NSCG Coding Quality Evaluation. Washington, DC: U.S. Department of Commerce, Bureau of the Census.

Keywords: Occupation and education coding, coding accuracy, error rates, error factors, problematic occupation codes, self-code agreement

Alternate Contact Person: Nirmala Kannankutty

SESTAT Survey: NSCG:95

Abstract: This report provides an overview of the estimates tabulated from the NSCG:95 coding process. Data for the evaluation were obtained from the Coding Quality Control operation. This Quality Control (QC) process involved two production clerks double coding a fraction of the education and occupation responses obtained during the survey process. The coders did not know whether they were assigning best codes or a QC code to an individual case. The authors begin by estimating the index of reliability based on expected and observed agreements between the two coders and the ratio of disagreement to total cases coded. The authors then examine the reliability of coding an occupation S&E or non S&E for major and minor groups. The authors also examine the reliability of three-digit codes and the error rates associated with the coding process. They also identify problematic occupation codes. Error factors providing a measure of how much a published estimate may have over- or underestimated the proportion in a particular occupation category is also considered. Finally, the authors discuss estimates of self-code related measures.

 


9. General and Other SESTAT Research

 

9-31.pdf

Hardy, Linda; Mooney, Geraldine; and Eisenhower, Donna. (1995). Research for the 1993 NSF Surveys of Scientists and Engineers. Published in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Keywords: Quality profile, SESTAT research and evaluation program

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: NSRCG, NSCG, SDR

Abstract: This paper focuses on the on-going SESTAT research and evaluation program that was established in the early 1990's. It discusses the rationale behind having such a program and the main goals and features of the research and evaluation program. The authors also review the factors considered when choosing SESTAT research and evaluation projects. They highlight research activities that were conducted prior to the 1993 data collection and conducted concurrently with the 1993 data collection as well as future and continuing research and evaluation projects. The authors also discuss how the results of the research and evaluation projects were used to improve future survey instruments and explore identified problems. In conclusion, the authors state that the NSF SESTAT program has benefited from the program of research and evaluation, and that this program will continue in the future.

 

9-41.pdf

Shettle, Carolyn. (1992). What Happens When Your Surveys Need a Radical Overhaul. Paper presented at the annual meeting of the American Statistical Association.

Keywords: Survey design, instrument design, CNSTAT recommendations

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: New Entrants (NSRCG), NSCG, SDR

Abstract: This paper presents a brief overview and evaluation of the National Science Foundation Surveys on science and engineering graduates and personnel. Following this review, it discusses the major recommendations made by the Committee on National Statistics (CNSTAT) for improving the surveys. It also provides an item-by-item discussion of the progress that has been made in meeting those goals.

 

9-51.pdf

Tsapogas, John, and Gannon, Joseph P. (1992). Instrument redesign: Trying to make a multitude of users happy while developing a high quality survey instrument. Paper presented at the annual meeting of the American Statistical Association.

Keywords: Questionnaire redesign, instrument design, survey design, questionnaire development

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: New Entrants (NSRCG), NSCG, SDR

Abstract: One area of particular concern during the revisions of the NSF science and engineering personnel surveys in the 1990s was the questionnaires. This paper discusses the research that was conducted on the questionnaires used in the 1980s and determines the impact this research had on the development of new survey instruments for the 1990s. It also examines the measures used to test the new survey instruments. Finally, it discusses the strategies used to develop improvements in the questionnaires and the methods used to implement recommended changes.

 

9-61.pdf

Kasprzyk, Daniel. (1992). Discussion for Section IV. Comment on papers presented to the annual meeting of the American Statistical Association.

Keywords: Survey design/redesign, questionnaire development, sampling

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: New Entrants (NSRCG), NSCG, SDR

Abstract: This paper provides a commentary on four papers discussing NSF activity over a three-year period to address concerns about the science and engineering personnel data system highlighted by CNSTAT. The papers discussed range in topic from an overview of the CNSTAT recommendations and NSF's progress in meeting these goals to data collection problems to questionnaire design changes to sampling issues.

 

9-71.pdf

Mitchell, Susan and Prudy Brown. (1994). Memorandum on Over-Reporting of Postdoctoral Appointments in the Survey of Doctorate Recipients

Keywords: Over-reporting, data quality

Alternate Contact Person: Nirmala Kannankutty

SESTAT Surveys: SDR

Abstract: This report discusses the issue of over-reporting of postdoctoral appointments in the 1993 SDR. In 1991, about two percent of respondents reported having a postdoctoral appointment, while in 1993, four percent of mail respondents and 10 percent of CATI respondents said they had postdoctoral appointments.


Updated: December 12, 2001