Social, Behavioral, and Economic SciencesSocial and  Economic SciencesNational Science Foundation
NSF.GOV SBE HOME BCS HOME SES HOME NUGGETS FASTLANE

Societal Dimensions of Engineering, Science and Technology Program
Doing Survey Research

Many SDEST projects involve examination of perceptions, attitudes, knowledge or behavior in relation to some aspect of science, engineering or technology. Accordingly, some investigators who are untrained in survey research methods will need to develop and administer a survey or questionnaire as part of their SDEST project. This website was created to offer some rudimentary guidance for such investigators.

To those who have responded to interviews, surveys or questionnaires, but have not received formal training in survey research, survey construction, administration and analysis seems beguilingly simple: write down the questions you want answered, send them out to the respondents, and analyze the responses. However, the science of survey research deals with many questions that are easily overlooked by those not trained in survey research, such as:

1. What questions need to be answered by the respondents? (It is not unusual for novice survey researchers to discover, only after they have collected their data, that they have no basis for confirming or refuting some of their hypotheses.)

2. Are the research issues presented from the respondents' perspective? (You can bet respondents' perspective is much different from yours! If you do not write the items from their perspective you will be unlikely to get useful answers.)

3. How will the respondents interpret the questions? Will they understand what is being asked of them?

4. On each multiple choice item, is at least one of the response alternatives a true or appropriate answer to each respondent? ("___Other. Please specify._________" is sometimes the only way to make sure everyone has an appropriate alternative.)

5. Are questions asked in a way that fosters accurate recall?

6. Are some of the questions sensitive? (Many demographic questions are considered personal and sensitive. Also sensitive are questions bearing on whether one is a good citizen, a well-informed and cultured person, a responsible, moral person; also whether one has illnesses or disabilities, whether one engages in illegal or socially undesirable behavior. It is important to learn what is likely to be regarded as sensitive and to ask those questions in a way that will produce valid responses.)

7. What sampling scheme should be employed -- a random sample, a stratified random sample, a convenience sample, a snowball sample, …? (The kind of sample obtained determines the kinds of inferences that can be drawn.)

8. How can an adequate response rate be obtained?

9. How can confidentiality be assured? (There are many techniques for assuring confidentiality; most are not widely known by any but experienced survey researchers.)

10. What questions are the Institutional Review Board likely to ask? (These questions should be anticipated and answered in the IRB protocol.)

11. How will responses be coded and analyzed?

An excellent overview of most of these issues appears in the American Statistical Association's (ASA) website entitled "What is a Survey." It will include the following chapters:

1. What is a Survey?
2. How to Plan a Survey
3. How to Collect Survey Data
4. Judging the Quality of a Survey
5. How to Conduct Pre-testing
6. What are Focus Groups?
7. More about Mail Surveys
8. What is a Margin of Error?
9. Designing a Questionnaire
10. More about Telephone Surveys

This website also includes the following two important, related documents:

"Surveys and Privacy" (ASA Committee on Privacy and Confidentiality)
"Ethical Guidelines for Statistical Practice" (ASA Committee on Professional Ethics)

The following information supplements and extends the material in the ASA website.

How to be sure that all of the necessary questions are asked and answered in the survey.

An effective way to figure out what questions to include in the survey is to write out the hypotheses you want to test. In other words, what do you want to be able to say in your report? This issue and its implications are best elucidated with an example:

Mary McSurvey hypothesizes that executives in large high tech companies feel a greater need for protection of their intellectual property than executives of medium or small high tech companies or of low-tech manufacturing firms of all sizes. To test this hypothesis, Mary will need to decide on the best way to determine WHICH executives in companies are concerned with intellectual property protection and make decisions about ways to protect it. (Having done so she will need to learn how to recruit THESE respondents; she will need to include an item querying respondents about whether they are the persons in the required role.) She will need an objective measure of company size, and a clear definition of "high tech companies." (She must then include questions about the size of the company and the nature of their products to ensure that companies can be classified correctly.) And she will need to know what it means for executives to "worry about intellectual property protection." Does this mean that they hand the problem off to lawyers who write intellectual property agreements? Does this mean they simply don't trust such agreements and will not enter into collaborative arrangements in which they would have to share information about their intellectual property? Does it mean that they have thought much about the problem; if so, what kinds of thoughts are you looking for? If Mary asks "Do you worry about protecting your company's intellectual property?" Executive A may respond "yes" meaning that he hands the problem to a company attorney. Executive B may respond "no" meaning that he hands the problem to a company attorney. Executive C may say "Yes" meaning that he thinks about the problem a lot. Executive D may say "Yes" meaning that it sounds like a serious problem but he doesn't know much about it. (Mary will need to spell out, behaviorally, the various ways one might regard intellectual property issues, and ask the respondent to specify which best describes their behavior.)

How does one decide exactly what one wants to know and what to ask? This leads to question 2 and 3:

How are these matters viewed from the respondents' perspective? A simple survey question won't answer this question, nor will it answer #3:

How does one learn to word questions in language respondents will clearly understand?

To answer these questions, Mary McSurvey may need to conduct some focus groups, to become an insider to the group she hopes to survey, or to bring in consultants who are insiders to the group she hopes to survey. In addition to the ASA website, Krueger (1994) is an excellent source on focus groups. Feldman, Bell and Berger (2003) are excellent sources on becoming an insider or quasi-insider, or working with others who are insiders.

We turn now to question #4: In multiple choice items, do the response alternatives suit the research purposes and are they acceptable to the respondents? The uninitiated may try to solve this problem by leaving off the response alternatives and letting respondents answer in their own words. However, without alternatives to choose among, many respondents will fail to answer at all, or may give responses that are uncodable - that is, useless for research purposes.

It is essential for the researcher to know how people would answer the survey questions in their own words, as that information is needed to guide the creation of usable sets of response alternatives for the actual survey. Where do you find people to answer questions in their own words? Focus groups and individual pilot subjects are one good source of people who can provide this information. However, if your survey is like Mary McSurvey's you will need to be careful how you use up focus group members or pilot subjects, since your subject population is quite limited and specialized. You won't want to use many people in focus groups or pilot studies whom you'll later need as research participants. You may decide it is worth rewarding your few focus group members handsomely in return for the many hours of help you will need from them.

The same goes for question #5: Will respondents consider any of the survey questions as sensitive? Many demographic questions are considered personal. Money is almost always a sensitive matter. It is important to learn what is likely to be regarded as sensitive and to ask those questions in a way that will produce valid responses. One can learn much about how to obtain valid, accurate answers to various kinds of questions from the ASA website and from the classic Asking Questions (Sudman & Bradburn, 1982, especially Chapters 3 & 8). Read both before assembling and consulting a focus group.

How can questions be asked in a way that aids accurate recall? Many questions you might wish to ask about behavior are not sensitive. In fact, they are so mundane that people forget about some of the times they engage in that behavior. "When was the last time you bought a pound of coffee? What brand was it?" "How often do you use an ATM machine?" There are many methods you can use to jog people's memory and obtain accurate answers to such questions. Sudman and Bradburn (1982, especially chapter 2) is the classic on this topic.

What sampling strategy should be used?

In designing a strategy for selecting your respondents, the first question to ask is what you plan to do with the results. To be able to generalize to all populations like the one from which the respondents were drawn, one needs to employ a random (or as nearly random as possible) sample. A random sample is drawn by creating a sample frame, that is, a list of all of the members of the population, then using some sampling scheme so that each member of the population has an equal chance of being chosen. If there are important sub-populations within that larger population, one also needs to stratify and randomly sample the subgroups. Mary McSurvey's research involves both high tech and manufacturing populations of executives, and large, medium and small firms with each kind of industry. Defining her sample frame and deciding how to draw a random sample from it is a major task, calling for expertise both with respect to statistical sampling theory and to the practicalities of actually locating and recruiting the sample. In the case of research designs of this much complexity, it may be a good idea to consult a specialist in survey research to design an appropriate random sampling procedure and plan the data analysis techniques appropriate to that design.

Other kinds of samples are easier to draw. The easiest kind of sample you can draw is a convenience sample -- whoever you can conveniently recruit - from which you cannot generalize, thus largely defeating the purpose of conducting a survey. A convenience sample provides a very rough estimate of what people might respond to the survey, but it is very likely to be a biased sample simply because it consists of those who are easiest to sample.

Sometimes the kinds of people you wish to survey are virtually impossible to locate, except that people of that kind tend to know one another. For example, suppose you wished to survey members of sets of identical triplets. There is no directory that could help you find these people, but many families who have triplets know other families with triplets. Often the only way to identify possible respondents is to do "snowball" sampling. That is, when you identify one individual who meets your criterion (e.g., is one of triplets) you ask that person to identify others whom you might survey. Out of respect for the privacy of these potential respondents, you or the person who identifies additional respondents should contact those persons in such a way that they can readily decline. For example, you or the first respondent may send a letter describing what your study is about, who the sponsor is, what steps will be taken to protect the confidentiality of the data - essentially all the information that will be contained in the informed consent statement. Perhaps you might include a return postcard that can be mailed to you if the person would consider participating. A snowball sample is not a random sample, however, a large enough snow ball sample may roughly approximate an entire sample frame from which to randomly sample.

At times, one wants to survey all of the members of a given group (e.g., all of the high tech firms in a given community, all of the members of a given church). This is not a sample of that group. It is called a census of that group.

How to gain access to the research population?

Whatever method is employed to select those to recruit, the issue of access is crucial. In most cases, both locating and effective recruiting of potential participants require some kind of relationship with the potential participants or members of their community, and a knowledge of their culture. This is true whether one is studying homeless injection drug users, minority populations in a war-torn country, or executives of the World Bank. Success in gaining access is time-consuming, but critical to valid research. The best source on this topic is the book, aptly entitled Gaining Access (Feldman, Bell & Berger, 2003).

How can one assure confidentiality? There are many methods of assuring the confidentiality of survey responses. The appropriate methods depend on the type of questions and the context of the research. A useful source of basic methods of assuring confidentiality of survey responses is the report "Summary of Human Subjects Protection Issues Related to Large Sample Surveys." (Please go to choice of formats, or the Acrobat file.) If the data are later to be shared, they must be prepared and shared in a way that does not violate promises of confidentiality. A summary of methods of restricted access and restricted data is contained in a paper by Virginia deWolf, at http://www.datasciencejournal.org/ (See latest papers link).

What will the Institutional Review Board (IRB) want to know? Federal Regulations of human research specify the questions to be answered in the protocol that is submitted to the IRB. Consult your local IRB for its specific protocol requirements. Many of the specific concerns that the IRB will have are discussed by the American Association of Public Opinion Researchers.

Coding and analysis of responses. Those who do not know, before they administer a survey, how they will code and analyze the responses are likely to find themselves with some unusable data. Methods of coding and analysis are very specific to types of data and complexity of the research design. The researcher who is untrained in methods of survey research is likely to benefit from consultation with a statistician who is expert in survey data coding and analysis.

References

deWolf, Virginia (2002). "Issues in Accessing and Sharing Confidential Survey and Social Science Data".

Feldman, Martha S., Bell, Jeannine & Berger, Michele T. (2003). Gaining Access. Walnut Creek: Altamira Press (A division of Rowman & Littlefield Publishers.

Krueger, Richard. A. (1994). Focus Groups: A practical guide for applied research. Newbury Park: Sage Publications.

Sieber, Joan E. (2001). Summary of human subjects protection issues related to large sample surveys. U.S. Department of Justice, Bureau of Justice Statistics, June 2001, NCJ 1087692.

Sudman, Seymour & Bradburn, Norman. (1982). Asking Questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.

Back to Top



The Division of Social and Economic Sciences
Suite 995, National Science Foundation,
4201 Wilson Boulevard, Arlington, Virginia 22230 USA
Tel: 703-292-8760
Last Updated 11.07.03
Contact SES Webmaster