Doing
Survey Research
Many SDEST projects involve examination of perceptions,
attitudes, knowledge or behavior in relation to some aspect of science,
engineering or technology. Accordingly, some investigators who are
untrained in survey research methods will need to develop and administer
a survey or questionnaire as part of their SDEST project. This website
was created to offer some rudimentary guidance for such investigators.
To those who have responded to interviews, surveys or questionnaires,
but have not received formal training in survey research, survey
construction, administration and analysis seems beguilingly simple:
write down the questions you want answered, send them out to the
respondents, and analyze the responses. However, the science of
survey research deals with many questions that are easily overlooked
by those not trained in survey research, such as:
1. What questions need to be answered by the respondents? (It is
not unusual for novice survey researchers to discover, only after
they have collected their data, that they have no basis for confirming
or refuting some of their hypotheses.)
2. Are the research issues presented from the respondents' perspective?
(You can bet respondents' perspective is much different from yours!
If you do not write the items from their perspective you will be
unlikely to get useful answers.)
3. How will the respondents interpret the questions? Will they
understand what is being asked of them?
4. On each multiple choice item, is at least one of the response
alternatives a true or appropriate answer to each respondent? ("___Other.
Please specify._________" is sometimes the only way to make
sure everyone has an appropriate alternative.)
5. Are questions asked in a way that fosters accurate recall?
6. Are some of the questions sensitive? (Many demographic questions
are considered personal and sensitive. Also sensitive are questions
bearing on whether one is a good citizen, a well-informed and cultured
person, a responsible, moral person; also whether one has illnesses
or disabilities, whether one engages in illegal or socially undesirable
behavior. It is important to learn what is likely to be regarded
as sensitive and to ask those questions in a way that will produce
valid responses.)
7. What sampling scheme should be employed -- a random sample,
a stratified random sample, a convenience sample, a snowball sample,
? (The kind of sample obtained determines the kinds of inferences
that can be drawn.)
8. How can an adequate response rate be obtained?
9. How can confidentiality be assured? (There are many techniques
for assuring confidentiality; most are not widely known by any but
experienced survey researchers.)
10. What questions are the Institutional Review Board likely to
ask? (These questions should be anticipated and answered in the
IRB protocol.)
11. How will responses be coded and analyzed?
An excellent overview of most of these issues appears in the American
Statistical Association's (ASA) website entitled "What
is a Survey." It will include the following chapters:
1. What is a Survey?
2. How to Plan a Survey
3. How to Collect Survey Data
4. Judging the Quality of a Survey
5. How to Conduct Pre-testing
6. What are Focus Groups?
7. More about Mail Surveys
8. What is a Margin of Error?
9. Designing a Questionnaire
10. More about Telephone Surveys
This website also includes the following two important, related
documents:
"Surveys and Privacy" (ASA Committee on Privacy and Confidentiality)
"Ethical Guidelines for Statistical Practice" (ASA Committee
on Professional Ethics)
The following information supplements and extends the material
in the ASA website.
How to be sure that all of the necessary questions
are asked and answered in the survey.
An effective way to figure out what questions to include in the
survey is to write out the hypotheses you want to test. In other
words, what do you want to be able to say in your report? This issue
and its implications are best elucidated with an example:
Mary McSurvey hypothesizes that executives in large high tech companies
feel a greater need for protection of their intellectual property
than executives of medium or small high tech companies or of low-tech
manufacturing firms of all sizes. To test this hypothesis, Mary
will need to decide on the best way to determine WHICH executives
in companies are concerned with intellectual property protection
and make decisions about ways to protect it. (Having done so she
will need to learn how to recruit THESE respondents; she will need
to include an item querying respondents about whether they are the
persons in the required role.) She will need an objective measure
of company size, and a clear definition of "high tech companies."
(She must then include questions about the size of the company and
the nature of their products to ensure that companies can be classified
correctly.) And she will need to know what it means for executives
to "worry about intellectual property protection." Does
this mean that they hand the problem off to lawyers who write intellectual
property agreements? Does this mean they simply don't trust such
agreements and will not enter into collaborative arrangements in
which they would have to share information about their intellectual
property? Does it mean that they have thought much about the problem;
if so, what kinds of thoughts are you looking for? If Mary asks
"Do you worry about protecting your company's intellectual
property?" Executive A may respond "yes" meaning
that he hands the problem to a company attorney. Executive B may
respond "no" meaning that he hands the problem to a company
attorney. Executive C may say "Yes" meaning that he thinks
about the problem a lot. Executive D may say "Yes" meaning
that it sounds like a serious problem but he doesn't know much about
it. (Mary will need to spell out, behaviorally, the various ways
one might regard intellectual property issues, and ask the respondent
to specify which best describes their behavior.)
How does one decide exactly what one wants to know and what to
ask? This leads to question 2 and 3:
How are these matters viewed from the respondents' perspective?
A simple survey question won't answer this question, nor will
it answer #3:
How does one learn to word questions in language respondents
will clearly understand?
To answer these questions, Mary McSurvey may need to conduct some
focus groups, to become an insider to the group she hopes to survey,
or to bring in consultants who are insiders to the group she hopes
to survey. In addition to the ASA website, Krueger (1994) is an
excellent source on focus groups. Feldman, Bell and Berger (2003)
are excellent sources on becoming an insider or quasi-insider, or
working with others who are insiders.
We turn now to question #4: In multiple choice items, do the
response alternatives suit the research purposes and are they acceptable
to the respondents? The uninitiated may try to solve this problem
by leaving off the response alternatives and letting respondents
answer in their own words. However, without alternatives to choose
among, many respondents will fail to answer at all, or may give
responses that are uncodable - that is, useless for research purposes.
It is essential for the researcher to know how people would answer
the survey questions in their own words, as that information is
needed to guide the creation of usable sets of response alternatives
for the actual survey. Where do you find people to answer questions
in their own words? Focus groups and individual pilot subjects are
one good source of people who can provide this information. However,
if your survey is like Mary McSurvey's you will need to be careful
how you use up focus group members or pilot subjects, since your
subject population is quite limited and specialized. You won't want
to use many people in focus groups or pilot studies whom you'll
later need as research participants. You may decide it is worth
rewarding your few focus group members handsomely in return for
the many hours of help you will need from them.
The same goes for question #5: Will respondents consider any
of the survey questions as sensitive? Many demographic questions
are considered personal. Money is almost always a sensitive matter.
It is important to learn what is likely to be regarded as sensitive
and to ask those questions in a way that will produce valid responses.
One can learn much about how to obtain valid, accurate answers to
various kinds of questions from the ASA
website and from the classic Asking Questions (Sudman &
Bradburn, 1982, especially Chapters 3 & 8). Read both before
assembling and consulting a focus group.
How can questions be asked in a way that aids accurate recall?
Many questions you might wish to ask about behavior are not
sensitive. In fact, they are so mundane that people forget about
some of the times they engage in that behavior. "When was the
last time you bought a pound of coffee? What brand was it?"
"How often do you use an ATM machine?" There are many
methods you can use to jog people's memory and obtain accurate answers
to such questions. Sudman and Bradburn (1982, especially chapter
2) is the classic on this topic.
What sampling strategy should be used?
In designing a strategy for selecting your respondents, the first
question to ask is what you plan to do with the results. To be able
to generalize to all populations like the one from which the respondents
were drawn, one needs to employ a random (or as nearly random as
possible) sample. A random sample is drawn by creating a
sample frame, that is, a list of all of the members of the population,
then using some sampling scheme so that each member of the population
has an equal chance of being chosen. If there are important sub-populations
within that larger population, one also needs to stratify and randomly
sample the subgroups. Mary McSurvey's research involves both high
tech and manufacturing populations of executives, and large, medium
and small firms with each kind of industry. Defining her sample
frame and deciding how to draw a random sample from it is a major
task, calling for expertise both with respect to statistical sampling
theory and to the practicalities of actually locating and recruiting
the sample. In the case of research designs of this much complexity,
it may be a good idea to consult a specialist in survey research
to design an appropriate random sampling procedure and plan the
data analysis techniques appropriate to that design.
Other kinds of samples are easier to draw. The easiest kind of
sample you can draw is a convenience sample -- whoever you
can conveniently recruit - from which you cannot generalize, thus
largely defeating the purpose of conducting a survey. A convenience
sample provides a very rough estimate of what people might respond
to the survey, but it is very likely to be a biased sample simply
because it consists of those who are easiest to sample.
Sometimes the kinds of people you wish to survey are virtually
impossible to locate, except that people of that kind tend to know
one another. For example, suppose you wished to survey members of
sets of identical triplets. There is no directory that could help
you find these people, but many families who have triplets know
other families with triplets. Often the only way to identify possible
respondents is to do "snowball" sampling. That
is, when you identify one individual who meets your criterion (e.g.,
is one of triplets) you ask that person to identify others whom
you might survey. Out of respect for the privacy of these potential
respondents, you or the person who identifies additional respondents
should contact those persons in such a way that they can readily
decline. For example, you or the first respondent may send a letter
describing what your study is about, who the sponsor is, what steps
will be taken to protect the confidentiality of the data - essentially
all the information that will be contained in the informed consent
statement. Perhaps you might include a return postcard that can
be mailed to you if the person would consider participating. A snowball
sample is not a random sample, however, a large enough snow ball
sample may roughly approximate an entire sample frame from which
to randomly sample.
At times, one wants to survey all of the members of a given group
(e.g., all of the high tech firms in a given community, all of the
members of a given church). This is not a sample of that group.
It is called a census of that group.
How to gain access to the research population?
Whatever method is employed to select those to recruit, the issue
of access is crucial. In most cases, both locating and effective
recruiting of potential participants require some kind of relationship
with the potential participants or members of their community, and
a knowledge of their culture. This is true whether one is studying
homeless injection drug users, minority populations in a war-torn
country, or executives of the World Bank. Success in gaining access
is time-consuming, but critical to valid research. The best source
on this topic is the book, aptly entitled Gaining Access (Feldman,
Bell & Berger, 2003).
How can one assure confidentiality? There are many methods of assuring
the confidentiality of survey responses. The appropriate methods
depend on the type of questions and the context of the research.
A useful source of basic methods of assuring confidentiality of
survey responses is the report "Summary of Human Subjects Protection
Issues Related to Large Sample Surveys." (Please go to choice
of formats, or the Acrobat
file.) If the data are later to be shared, they must be prepared
and shared in a way that does not violate promises of confidentiality.
A summary of methods of restricted access and restricted data is
contained in a paper by Virginia deWolf, at http://www.datasciencejournal.org/
(See latest papers link).
What will the Institutional Review Board (IRB) want to know? Federal
Regulations of human research specify the questions to be answered
in the protocol that is submitted to the IRB. Consult your local
IRB for its specific protocol requirements. Many of the specific
concerns that the IRB will have are discussed by the American
Association of Public Opinion Researchers.
Coding and analysis of responses. Those who do not know,
before they administer a survey, how they will code and analyze
the responses are likely to find themselves with some unusable data.
Methods of coding and analysis are very specific to types of data
and complexity of the research design. The researcher who is untrained
in methods of survey research is likely to benefit from consultation
with a statistician who is expert in survey data coding and analysis.
References
deWolf, Virginia (2002). "Issues
in Accessing and Sharing Confidential Survey and Social Science
Data".
Feldman, Martha S., Bell, Jeannine & Berger, Michele T. (2003).
Gaining Access. Walnut Creek: Altamira Press (A division of Rowman
& Littlefield Publishers.
Krueger, Richard. A. (1994). Focus Groups: A practical guide for
applied research. Newbury Park: Sage Publications.
Sieber, Joan E. (2001). Summary
of human subjects protection issues related to large sample surveys.
U.S. Department of Justice, Bureau of Justice Statistics, June 2001,
NCJ 1087692.
Sudman, Seymour & Bradburn, Norman. (1982). Asking Questions:
A practical guide to questionnaire design. San Francisco: Jossey-Bass.
|