C H A P T E R 9
Survey Research
CengageNOW for Sociology
Use this online tool to help you make the grade on your next exam. After
reading this chapter, go to “Online Study Resources” at the end of the
chapter for instructions on how to benefit from CengageNOW.
Introduction
Topics Appropriate for Survey
Research
Guidelines for Asking Questions
Choose Appropriate Question
Forms
Make Items Clear
Avoid Double-Barreled Questions
Respondents Must Be Competent
to Answer
Respondents Must Be Willing
to Answer
Questions Should Be Relevant
Short Items Are Best
Avoid Negative Items
Avoid Biased Items and Terms
Questionnaire Construction
General Questionnaire Format
Formats for Respondents
Contingency Questions
Matrix Questions
Ordering Items in a Questionnaire
Questionnaire Instructions
Pretesting the Questionnaire
A Composite Illustration
Self-Administered Questionnaires
Mail Distribution and Return
Monitoring Returns
Follow-up Mailings
Response Rates
A Case Study
Interview Surveys
The Role of the Survey Interviewer
General Guidelines for Survey
Interviewing
Coordination and Control
Telephone Surveys
Computer-Assisted Telephone
Interviewing (CATI)
Response Rates in Interview
Surveys
Online Surveys
Comparison of the Different Survey
Methods
Strengths and Weaknesses of Survey
Research
Secondary Analysis
Ethics and Survey Research
C H A P T E R O V E R V I E W
Researchers have many methods
for collecting data through
surveys—from mailquestionnaires
to personalinterviews toonline
surveys conductedover the Internet.
Socialresearchers shouldknowhow
to selectanappropriate method
and howto implementiteffectively.
254 ■ Chapter 9: Survey Research
Introduction
Surveys are a very old research technique. In the
Old Testament, for example, we find the following:
After the plague the Lord said to Moses and to
Eleazar the son of Aaron, the priest, “Take a
census of all the congregation of the people of
Israel, from twenty years old and upward.”
(Numbers 26: 1–2)
Ancient Egyptian rulers conducted censuses
to help them administer their domains. Jesus was
born away from home because Joseph and Mary
were journeying to Joseph’s ancestral home for a
Roman census.
A little-known survey was attempted among
French workers in 1880. A German political sociologist
mailed some 25,000 questionnaires to workers
to determine the extent of their exploitation
by employers. The rather lengthy questionnaire
included items such as these:
Does your employer or his representative resort
to trickery in order to defraud you of a part of
your earnings?
If you are paid piece rates, is the quality of
the article made a pretext for fraudulent deductions
from your wages?
The survey researcher in this case was not
George Gallup but Karl Marx ([1880] 1956: 208).
Though 25,000 questionnaires were mailed out,
there is no record of any being returned.
Today, survey research is a frequently used
mode of observation in the social sciences. In a
typical survey, the researcher selects a sample of
respondents and administers a standardized questionnaire
to them. Chapter 7 discussed sampling
techniques in detail. This chapter discusses how to
prepare a questionnaire and describes the various
options for administering it so that respondents
answer your questions adequately.
The chapter includes a short discussion of
secondary analysis, the analysis of survey data collected
by someone else. This use of survey results
has become an important aspect of survey research
in recent years, and it’s especially useful for students
and others with scarce research funds.
Let’s begin by looking at the kinds of topics that
researchers can appropriately study by using survey
research.
Topics Appropriate
for Survey Research
Surveys may be used for descriptive, explanatory,
and exploratory purposes. They are chiefly
used in studies that have individual people as the
units of analysis. Although this method can be
used for other units of analysis, such as groups or
interactions, some individual persons must serve
as respondents or informants. Thus, we could
undertake a survey in which divorces were the
unit of analysis, but we would need to administer
the survey questionnaire to the participants in the
divorces (or to some other respondents).
Survey research is probably the best method
available to the social researcher who is interested
in collecting original data for describing a population
too large to observe directly. Careful probability
sampling provides a group of respondents
whose characteristics may be taken to reflect those
of the larger population, and carefully constructed
standardized questionnaires provide data in the
same form from all respondents.
Surveys are also excellent vehicles for measuring
attitudes and orientations in a large population.
Public opinion polls—for example, Gallup,
Harris, Roper, and Yankelovich—are well-known
examples of this use. Indeed, polls have become
so prevalent that at times the public seems unsure
what to think of them. Pollsters are criticized by
those who don’t think (or want to believe) that
polls are accurate (candidates who are “losing” in
polls often tell voters not to trust the polls). But
respondent A person who provides data for analysis
by responding to a survey questionnaire.
Guidelines for Asking Questions ■ 255
polls are also criticized for being too accurate—
for example, when exit polls on election day are
used to predict a winner before the actual voting is
complete.
The general attitude toward public opinion
research is further complicated by scientifically unsound
“surveys” that nonetheless capture people’s
attention because of the topics they cover and/
or their “findings.” A good example is the “Hite
Reports” on human sexuality. While enjoying considerable
attention in the popular press, Shere Hite
was roundly criticized by the research community
for her data-collection methods. For example, a
1987 Hite report was based on questionnaires completed
by women around the country—but which
women? Hite reported that she distributed some
100,000 questionnaires through various organizations,
and around 4,500 were returned.
Now 4,500 and 100,000 are large numbers in
the context of survey sampling. However, given
Hite’s research methods, her 4,500 respondents
didn’t necessarily represent U.S. women any more
than the Literary Digest’s enormous 1936 sample
represented the U.S. electorate when their 2 million
sample ballots indicated that Alf Landon would
bury FDR in a landslide.
Sometimes, people use the pretense of survey
research for quite different purposes. For example,
you may have received a telephone call indicating
you’ve been selected for a survey, only to find
that the first question was “How would you like to
make thousands of dollars a week right there in
your own home?” Or you may have been told you
could win a prize if you could name the president
whose picture is on the penny. (Tell them it’s
Elvis.) Unfortunately, a few unscrupulous telemarketers
try to prey on the general cooperation
people have given to survey researchers.
By the same token, political parties and charitable
organizations have begun conducting phony
“surveys.” Often under the guise of collecting public
opinion about some issue, callers ultimately ask
respondents for a monetary contribution.
Recent political campaigns have produced another
form of bogus survey, the “push poll.” Here’s
what the American Association for Public Opinion
Polling has said in condemning this practice (see
also Figure 3-1):
A “push poll” is a telemarketing technique
in which telephone calls are used to canvass
potential voters, feeding them false or misleading
“information” about a candidate under
the pretense of taking a poll to see how this
“information” affects voter preferences. In fact,
the intent is not to measure public opinion but
to manipulate it—to “push” voters away from
one candidate and toward the opposing candidate.
Such polls defame selected candidates by
spreading false or misleading information about
them. The intent is to disseminate campaign
propaganda under the guise of conducting a
legitimate public opinion poll.
(Bednarz 1996)
In short, the labels “survey” and “poll” are
sometimes misused. Done properly, however, survey
research can be a useful tool of social inquiry.
Designing useful (and trustworthy) survey research
begins with formulating good questions. Let’s turn
to that topic now.
Guidelines
for Asking Questions
In social research, variables are often operationalized
when researchers ask people questions as a
way of getting data for analysis and interpretation.
Sometimes the questions are asked by an
interviewer; sometimes they are written down
and given to respondents for completion. In other
cases, several general guidelines can help researchers
frame and ask questions that serve as excellent
operationalizations of variables while avoiding
pitfalls that can result in useless or even misleading
information.
Surveys include the use of a questionnaire—an
instrument specifically designed to
elicit information that will be useful for analysis.
Although some of the specific points to follow are
more appropriate to structured questionnaires than
256 ■ Chapter 9: Survey Research
Both questions and statements can be used
profitably. Using both in a given questionnaire gives
you more flexibility in the design of items and
can make the questionnaire more interesting
as well.
Open-Ended and Closed-Ended Questions
In asking questions, researchers have two options.
They can ask open-ended questions, in which
case the respondent is asked to provide his or her
own answers to the questions. For example, the
respondent may be asked, ”What do you feel is
the most important issue facing the United States
today?” and be provided with a space to write in
the answer (or be asked to report it verbally to an
interviewer). As we’ll see in Chapter 10, in-depth,
qualitative interviewing relies almost exclusively
on open-ended questions. However, they are also
used in survey research.
In the case of closed-ended questions, the respondent
is asked to select an answer from among
a list provided by the researcher. Closed-ended
questions are very popular in survey research
because they provide a greater uniformity of responses
and are more easily processed than openended
ones.
Open-ended responses must be coded before
they can be processed for computer analysis, as
we’ll see in Chapter 14. This coding process often
requires the researcher to interpret the meaning
of responses, opening the possibility of misunderstanding
and researcher bias. There is also a
danger that some respondents will give answers
that are essentially irrelevant to the researcher’s
intent. Closed-ended responses, on the other hand,
can often be transferred directly into a computer
format.
The chief shortcoming of closed-ended questions
lies in the researcher’s structuring of responses.
When the relevant answers to a given
question are relatively clear, there should be no
problem. In other cases, however, the researcher’s
structuring of responses may overlook some important
responses. In asking about “the most important
issue facing the United States,” for example,
his or her checklist of issues might omit certain
to the more open-ended questionnaires used in
qualitative, in-depth interviewing, the underlying
logic is valuable whenever we ask people questions
in order to gather data.
Choose Appropriate
Question Forms
Let’s begin with some of the options available
to you in creating questionnaires. These options
include using questions or statements and choosing
open-ended or closed-ended questions.
Questions and Statements
Although the term questionnaire suggests a collection
of questions, an examination of a typical questionnaire
will probably reveal as many statements
as questions. This is not without reason. Often, the
researcher is interested in determining the extent
to which respondents hold a particular attitude
or perspective. If you can summarize the attitude
in a fairly brief statement, you can present that
statement and ask respondents whether they agree
or disagree with it. As you may remember, Rensis
Likert greatly formalized this procedure through
the creation of the Likert scale, a format in which
respondents are asked to strongly agree, agree,
disagree, or strongly disagree, or perhaps strongly
approve, approve, and so forth.
questionnaire A document containing questions
and other types of items designed to solicit information
appropriate for analysis. Questionnaires are used
primarily in survey research but also in experiments,
field research, and other modes of observation.
open-ended questions Questions for which the
respondent is asked to provide his or her own answers.
In-depth, qualitative interviewing relies almost
exclusively on open-ended questions.
closed-ended questions Survey questions in
which the respondent is asked to select an answer
from among a list provided by the researcher. Popular
in survey research because they provide a greater
uniformity of responses and are more easily processed
than open-ended questions.
Guidelines for Asking Questions ■ 257
issues that respondents would have said were
important.
The construction of closed-ended questions
should be guided by two structural requirements.
First, the response categories provided should be
exhaustive: They should include all the possible
responses that might be expected. Often, researchers
ensure this by adding a category such as “Other
(Please specify: ________).” Second, the answer
categories must be mutually exclusive: The respondent
should not feel compelled to select more
than one. (In some cases, you may wish to solicit
multiple answers, but these may create difficulties
in data processing and analysis later on.) To ensure
that your categories are mutually exclusive, carefully
consider each combination of categories,
asking yourself whether a person could reasonably
choose more than one answer. In addition, it’s useful
to add an instruction to the question asking the
respondent to select the one best answer, but this
technique is not a satisfactory substitute for a carefully
constructed set of responses.
Make Items Clear
It should go without saying that questionnaire
items need to be clear and unambiguous, but
the broad proliferation of unclear and ambiguous
questions in surveys makes the point worth
emphasizing. We can become so deeply involved
in the topic under examination that opinions
and perspectives are clear to us but not to our
respondents—many of whom have paid little or
no attention to the topic. Or, if we have only a
superficial understanding of the topic, we may fail
to specify the intent of a question sufficiently. The
question “What do you think about the proposed
peace plan?” may evoke in the respondent a
counterquestion: “Which proposed peace plan?”
Questionnaire items should be precise so that the
respondent knows exactly what the researcher is
asking. The possibilities for misunderstanding are
endless, and no researcher is immune (Polivka and
Rothgeb 1993).
One of the most established research projects
in the United States is the Census Bureau’s ongoing
“Current Population Survey” or CPS, which
measures, among other critical data, the nation’s
unemployment rate. A part of the measurement
of employment patterns focuses on a respondent’s
activities during “last week,” by which the Census
Bureau means Sunday through Saturday. Studies
undertaken to determine the accuracy of the
survey found that more than half the respondents
took “last week” to include only Monday through
Friday. By the same token, whereas the Census
Bureau defines “working full-time” as 35 or more
hours a week, the same evaluation studies showed
that some respondents used the more traditional
definition of 40 hours per week. As a consequence,
the wording of these questions in the CPS was
modified in 1994 to specify the Census Bureau’s
definitions.
Similarly, the use of the term Native American to
mean American Indian often produces an overrepresentation
of that ethnic group in surveys. Clearly,
many respondents understand the term to mean
“born in the United States.”
Avoid Double-Barreled Questions
Frequently, researchers ask respondents for a single
answer to a question that actually has multiple
parts. That seems to happen most often when the
researcher has personally identified with a complex
question. For example, you might ask respondents
to agree or disagree with the statement “The United
States should abandon its space program and spend
the money on domestic programs.” Although
many people would unequivocally agree with
the statement and others would unequivocally
disagree, still others would be unable to answer.
Some would want to abandon the space program
and give the money back to the taxpayers. Others
would want to continue the space program but also
put more money into domestic programs. These
latter respondents could neither agree nor disagree
without misleading you.
As a general rule, whenever the word and
appears in a question or questionnaire statement,
check whether you’re asking a double-barreled
question. See “Double-Barreled and Beyond” for
some imaginative variations on this theme.
258 ■ Chapter 9: Survey Research
Respondents Must
Be Competent to Answer
In asking respondents to provide information, you
should continually ask yourself whether they can
do so reliably. In a study of child rearing, you might
ask respondents to report the age at which they
first talked back to their parents. Quite aside from
the problem of defining talking back to parents, it’s
doubtful that most respondents would remember
with any degree of accuracy.
As another example, student government leaders
occasionally ask their constituents to indicate
how students’ fees ought to be spent. Typically,
respondents are asked to indicate the percentage
of available funds that should be devoted to a long
list of activities. Without a fairly good knowledge of
the nature of those activities and the costs involved
in them, the respondents cannot provide meaningful
answers. Administrative costs, for example, will
receive little support although they may be essential
to the program as a whole.
Double-Barreled and Beyond
Even established,professional researchers havesometimescreated
double-barreled questions andworse.Consider this question,asked
of U.S.citizensin April 1986,at a timewhen the country’srelationship
withLibya was at an especially lowpoint.Some observerssuggested
thattheUnitedStatesmight end up ina shootingwarwith the small
North African nation.The HarrisPoll sought tofind outwhat U.S.public
opinionwas.
If Libya now increases itsterrorist acts against the U.S.and wekeep
inflicting moredamage onLibya,theninevitablyit will allend
in the U.S.going towarand finally invading thatcountry which
would be wrong.
Respondents weregiven the opportunityof answering“Agree,”
“Disagree,”or“Notsure.”Noticetheelementscontained in the complex
statement:
1. Will Libyaincrease its terrorist acts againstthe U.S.?
2. Will the U.S.inflict moredamageonLibya?
3. Will the U.S.inevitably or otherwisego to war againstLibya?
4. WouldtheU.S.invade Libya?
5. Wouldthat berightor wrong?
These several elements offer the possibility of numerous points
of view—far morethan the threealternativesofferedtothesurvey
respondents.Even ifwewere to assume hypothetically that Libya
would“increaseitsterroristattacks”and the United States would“keep
inflicting moredamage”in return,youmighthaveany one of atleast
seven distinct expectationsabout theoutcome:
U.S.will War isprobable
not go butnot Waris
towar inevitable inevitable
U.S.willnotinvadeLibya 1 2 3
U.S.willinvadeLibya butit
would be wrong 4 5
U.S.willinvadeLibya andit
would be right 6 7
Theexamination of prognoses abouttheLibyansituation is notthe
only example of double-barreled questions sneakingintopublicopinion
research.Herearesome questionstheHarrisPoll asked inanattempt
to gauge U.S.public opinion aboutthen Soviet General Secretary
Gorbachev:
He looks likethekind of Russianleaderwhowill recognizethat
both the Soviets and the Americans can destroy eachotherwith
nuclearmissiles soit isbetter to cometoverifiable armscontrol
agreements.
Heseemsto be moremodern,enlightened,andattractive,
whichis agood sign for the peaceoftheworld.
Even though he looks much moremodern and attractive,it
would be amistake tothink he will be much different from other
Russianleaders.
Howmanyelementscanyouidentifyin eachof thequestions?
Howmanypossible opinionscould peoplehave in each case?Whatdoes
a simple“agree”or“disagree”really mean in suchcases?
Sources:ReportedinWorld Opinion Update,October 1985 andMay 1986,respectively.
Guidelines for Asking Questions ■ 259
One group of researchers examining the driving
experience of teenagers insisted on asking an
open-ended question concerning the number of
miles driven since receiving a license. Although
consultants argued that few drivers would be able
to estimate such information with any accuracy,
the question was asked nonetheless. In response,
some teenagers reported driving hundreds of thousands
of miles.
Respondents Must Be
Willing to Answer
Often, we would like to learn things from people
that they are unwilling to share with us. For
example, Yanjie Bian indicates that it has often
been difficult to get candid answers from people in
China.
[Here] people are generally careful about what
they say on nonprivate occasions in order to
survive under authoritarianism. During the
Cultural Revolution between 1966 and 1976,
for example, because of the radical political
agenda and political intensity throughout the
country, it was almost impossible to use survey
techniques to collect valid and reliable data
inside China about the Chinese people’s life experiences,
characteristics, and attitudes towards
the Communist regime.
(1994: 19–20)
Sometimes, U.S. respondents say they’re undecided
when, in fact, they have an opinion but think
they’re in a minority. Under that condition, they
may be reluctant to tell a stranger (the interviewer)
what that opinion is. Given this problem, the Gallup
Organization, for example, has used a “secret ballot”
format, which simulates actual election conditions,
in that the “voter” enjoys complete anonymity.
In an analysis of the Gallup Poll election data from
1944 to 1988, Andrew Smith and G. F. Bishop
(1992) have found that this technique substantially
reduced the percentage of respondents who said
they were undecided about how they would vote.
This problem is not limited to survey research,
however. Richard Mitchell (1991: 100) faced a
similar problem in his field research among U.S.
survivalists:
Survivalists, for example, are ambivalent about
concealing their identities and inclinations.
They realize that secrecy protects them from
the ridicule of a disbelieving majority, but enforced
separatism diminishes opportunities for
recruitment and information exchange. . . .
“Secretive” survivalists eschew telephones,
launder their mail through letter exchanges,
use nicknames and aliases, and carefully conceal
their addresses from strangers. Yet once I
was invited to group meetings, I found them
cooperative respondents.
Questions Should Be Relevant
Similarly, questions asked in a questionnaire should
be relevant to most respondents. When attitudes
are requested on a topic that few respondents have
thought about or really care about, the results
are not likely to be useful. Of course, because the
respondents may express attitudes even though
they’ve never given any thought to the issue, you
run the risk of being misled.
This point is illustrated occasionally when
researchers ask for responses relating to fictitious
people and issues. In one political poll I conducted,
I asked respondents whether they were familiar
with each of 15 political figures in the community.
As a methodological exercise, I made up a name:
Tom Sakumoto. In response, 9 percent of the
respondents said they were familiar with him. Of
those respondents familiar with him, about half reported
seeing him on television and reading about
him in the newspapers.
When you obtain responses to fictitious issues,
you can disregard those responses. But when the issue
is real, you may have no wayof tellingwhich responses
genuinely reflect attitudes and which reflect
meaningless answers to an irrelevant question.
Ideally, we would like respondents to simply
report that they don’t know, have no opinion, or
are undecided in those instances where that is the
case. Unfortunately, however, they often make up
answers.
260 ■ Chapter 9: Survey Research
Short Items Are Best
In the interests of being unambiguous and precise
and of pointing to the relevance of an issue,
researchers tend to create long and complicated
items. That should be avoided. Respondents are
often unwilling to study an item in order to understand
it. The respondent should be able to read an
item quickly, understand its intent, and select or
provide an answer without difficulty. In general,
assume that respondents will read items quickly
and give quick answers. Accordingly, provide clear,
short items that will not be misinterpreted under
those conditions.
Avoid Negative Items
The appearance of a negation in a questionnaire
item paves the way for easy misinterpretation.
Asked to agree or disagree with the statement “The
United States should not recognize Cuba,” a sizable
portion of the respondents will read over the word
not and answer on that basis. Thus, some will agree
with the statement when they’re in favor of recognition,
and others will agree when they oppose it.
And you may never know which are which.
Similar considerations apply to other “negative”
words. In a study of support for civil liberties,
for example, respondents were asked whether
they felt “the following kinds of people should be
prohibited from teaching in public schools” and
were presented with a list including such items as
a Communist, a Ku Klux Klansman, and so forth.
The response categories “yes” and “no” were given
beside each entry. A comparison of the responses
to this item with other items reflecting support for
civil liberties strongly suggested that many respondents
gave the answer “yes” to indicate willingness
for such a person to teach, rather than to indicate
that such a person should be prohibited from
teaching. (A later study in the series using the answer
categories “permit” and “prohibit” produced
much clearer results.)
In 1993 a national survey commissioned by the
American Jewish Committee produced shocking
results: One American in five believed that the
Nazi Holocaust—in which six million Jews were
reportedly killed—never happened; further, one
in three Americans expressed some doubt that it
had occurred. This research finding suggested that
the Holocaust Revisionist movement in America
was powerfully influencing public opinion (“1 in 5
Polled Voices Doubt on Holocaust” 1993).
In the aftermath of this shocking news,
researchers reexamined the actual question that
had been asked: “Does it seem possible or does it
seem impossible to you that the Nazi extermination
of the Jews never happened?” On reflection,
it seemed clear that the complex, double-negative
question could have confused some respondents.
A new survey was commissioned and asked,
“Does it seem possible to you that the Nazi extermination
of the Jews never happened, or do you
feel certain that it happened?” In the follow-up
survey, only 1 percent of the respondents believed
the Holocaust never happened, and another 8
percent said they weren’t sure (“Poll on Doubt of
Holocaust Is Corrected” 1994).
Avoid Biased Items and Terms
Recall from our discussion of conceptualization and
operationalization in Chapter 5 that there are no
ultimately true meanings for any of the concepts
we typically study in social science. Prejudice has
no ultimately correct definition; whether a given
person is prejudiced depends on our definition of
that term. The same general principle applies to
the responses we get from people completing a
questionnaire.
The meaning of someone’s response to a question
depends in large part on its wording. This is true
of every question and answer. Some questions seem
to encourage particular responses more than other
questions do. In the context of questionnaires, bias
bias That quality of a measurement device that
tends to result in a misrepresentation of what is being
measured in a particular direction. For example,
the questionnaire item “Don’t you agree that the
president is doing a good job?” would be biased in
that it would generally encourage more favorable
responses.
Guidelines for Asking Questions ■ 261
Democratic primary, many voters who might have
been reluctant to vote for an African American
(Barack Obama) or a woman (Hillary Clinton)
might have also been reluctant to admit their
racial or gender prejudice to a survey interviewer.
(Some, to be sure, were not reluctant to say how
they felt.)
The best way to guard against this problem is to
imagine how you would feel giving each of the
answers you intend to offer to respondents. If you
would feel embarrassed, perverted, inhumane, stupid,
irresponsible, or otherwise socially disadvantaged
by any particular response, give serious
thought to how willing others will be to give those
answers.
The biasing effect of particular wording is
often difficult to anticipate. For example, in both
surveys and experiments, researchers sometimes
ask respondents to consider hypothetical situations
and say how they think they would behave. Those
situations often involve other people, however, and
the names used can affect responses. For instance,
researchers have long known that male names for
the hypothetical people can produce different responses
than female names do. Research by Joseph
Kasof (1993) points to the importance of what the
specific names are: whether they generally evoke
positive or negative images in terms of attractiveness,
age, intelligence, and so forth. Kasof’s review
of past research suggests there has been a tendency
to use more positively valued names for men than
for women.
The Center for Disease Control (Choi and Pak
2005) has provided an excellent analysis of various
ways in which your choice of terms can bias and
otherwise confuse responses to questionnaires.
Among other things, they warn against using
ambiguous, technical, uncommon, or vague words.
Their thorough analysis provides many concrete
illustrations.
As in all other research, carefully examine
the purpose of your inquiry and construct items
that will be most useful to it. You should never be
misled into thinking there are ultimately “right”
and “wrong” ways of asking the questions. When
in doubt about the best question to ask, moreover,
remember that you should ask more than one.
refers to any property of questions that encourages
respondents to answer in a particular way.
Most researchers recognize the likely effect of
a question that begins, “Don’t you agree with the
President of the United States that . . .” No reputable
researcher would use such an item. Unhappily,
the biasing effect of items and terms is far subtler
than this example suggests.
The mere identification of an attitude or position
with a prestigious person or agency can bias
responses. The item “Do you agree or disagree
with the recent Supreme Court decision that . . .”
would have a similar effect. Such wording may not
produce consensus or even a majority in support of
the position identified with the prestigious person
or agency, but it will likely increase the level of support
over what would have been obtained without
such identification.
Sometimes the impact of different forms of
question wording is relatively subtle. For example,
when Kenneth Rasinski (1989) analyzed the results
of several General Social Survey studies of attitudes
toward government spending, he found that the
way programs were identified had an impact on
the amount of public support they received. Here
are some comparisons:
MoreSupport LessSupport
“Assistance to thepoor” “Welfare”
“Haltingrisingcrimerate” “Lawenforcement”
“Dealingwith drug addiction” “Drugrehabilitation”
“Solving problems ofbigcities” “Assistance tobig cities”
“Improving conditions ofblacks” “Assistance toblacks”
“Protecting social security” “Social security”
In 1986, for example, 62.8 percent of the respondents
said too little money was being spent on “assistance
to the poor,” whereas in a matched survey
that year, only 23.1 percent said we were spending
too little on “welfare.”
In this context, be wary of what researchers
call the social desirability of questions and answers.
Whenever we ask people for information, they answer
through a filter of what will make them look
good. This is especially true if they’re interviewed
face-to-face. Thus, for example, during the 2008
262 ■ Chapter 9: Survey Research
have been forced to reread confusing, abbreviated
questions. Nor will they have been forced to write a
long answer in a tiny space.
Similar problems can arise for interviewers
in a face-to-face or telephone interview. Like
respondents to a self-administered questionnaire,
interviewers may miss questions, lose their place,
and generally become frustrated and flustered.
Interview questionnaires need to be laid out in a
way that supports the interviewer’s work, including
special instructions and guidelines that go beyond
what respondents to a self-administered questionnaire
would need.
The desirability of spreading out questions in
the questionnaire cannot be overemphasized.
Squeezed-together questionnaires are disastrous,
whether completed by the respondents themselves
or administered by trained interviewers. The processing
of such questionnaires is another nightmare;
I’ll have more to say about that in Chapter 14.
Formats for Respondents
In one of the most common types of questionnaire
items, the respondent is expected to check
one response from a series. For this purpose my
experience has been that boxes adequately spaced
apart are the best format. Word processing makes
the use of boxes a practical technique these days;
setting boxes in type can be accomplished easily
and neatly. You can approximate boxes by using
brackets: [ ], but if you’re creating a questionnaire
on a computer, you should take the few extra
minutes to use genuine boxes that will give your
questionnaire a more professional look. Here are
some easy examples:
□ ❍ ❑
Rather than providing boxes to be checked,
you might print a code number beside each response
and ask the respondent to circle the appropriate
number (see Figure 9-1). This method has
the added advantage of specifying the code number
to be entered later in the processing stage (see
Chapter 14). If numbers are to be circled, however,
you should provide clear and prominent instructions
to the respondent, because many will be
These, then, are some general guidelines for
writing questions to elicit data for analysis and
interpretation. Next we look at how to construct
questionnaires.
Questionnaire Construction
Questionnaires are used in connection with many
modes of observation in social research. Although
structured questionnaires are essential to and most
directly associated with survey research, they are
also widely used in experiments, field research,
and other data-collection activities. For this reason,
questionnaire construction can be an important
practical skill for researchers. As we discuss the
established techniques for constructing questionnaires,
let’s begin with some issues of questionnaire
format.
General Questionnaire Format
The format of a questionnaire is just as important
as the nature and wording of the questions asked.
An improperly laid out questionnaire can lead
respondents to miss questions, confuse them about
the nature of the data desired, and even lead them
to throw the questionnaire away.
As a general rule, a questionnaire should be
spread out and uncluttered. If a self-administered
questionnaire is being designed, inexperienced
researchers tend to fear that their questionnaire
will look too long; as a result, they squeeze several
questions onto a single line, abbreviate questions,
and try to use as few pages as possible. These efforts
are ill-advised and even dangerous. Putting
more than one question on a line will cause some
respondents to miss the second question altogether.
Some respondents will misinterpret abbreviated
questions. More generally, respondents who find
they have spent considerable time on the first page
of what seemed like a short questionnaire will be
more demoralized than respondents who quickly
complete the first several pages of what initially
seemed like a rather long form. Moreover, the
latter will have made fewer errors and will not
Questionnaire Construction ■ 263
There are several formats for contingency questions.
The one shown in Figure 9-2 is probably the
clearest and most effective. Note two key elements
in this format. First, the contingency question is
isolated from the other questions by being set off to
the side and enclosed in a box. Second, an arrow
connects the contingency question to the answer
on which it is contingent. In the illustration, only
those respondents answering yes are expected to
answer the contingency question. The rest of the
respondents should simply skip it.
Note that the questions shown in Figure 9-2
could have been dealt with in a single question.
The question might have read, “How many times,
if any, have you smoked marijuana?” The response
categories, then, might have read: “Never,”
“Once,” “2 to 5 times,” and so forth. This single
question would apply to all respondents, and each
would find an appropriate answer category. Such
a question, however, might put some pressure on
tempted to cross out the appropriate number,
which makes data processing more difficult. (Note
that the technique can be used more safely when
interviewers administer the questionnaires, because
the interviewers themselves record the responses.)
Contingency Questions
Quite often in questionnaires, certain questions will
be relevant to some of the respondents and irrelevant
to others. In a study of birth control methods,
for instance, you would probably not want to ask
men if they take birth control pills.
This sort of situation often arises when researchers
wish to ask a series of questions about
a certain topic. You may want to ask whether
your respondents belong to a particular organization
and, if so, how often they attend meetings,
whether they have held office in the organization,
and so forth. Or, you might want to ask whether
respondents have heard anything about a certain
political issue and then learn the attitudes of those
who have heard of it.
Each subsequent question in series such as
these is called a contingency question: Whether
it is to be asked and answered is contingent on
responses to the first question in the series. The
proper use of contingency questions can facilitate
the respondents’ task in completing the questionnaire,
because they are not faced with trying to
answer questions irrelevant to them.
FIGURE 9-1
Circling the Answer FIGURE 9-2
Contingency Question Format. Contingency questions offer a
structure for exploring subject areas logically in some depth.
contingency question A survey question intended
for only some respondents, determined by their
responses to some other question. For example, all
respondents might be asked whether they belong to
the Cosa Nostra, and only those who said yes would
be asked how often they go to company meetings
and picnics. The latter would be a contingency
question.
264 ■ Chapter 9: Survey Research
respondents to report having smoked marijuana,
because the main question asks how many times
they have smoked it, even though it allows for
those exceptional cases who have never smoked marijuana
even once. (The emphases used in the previous
sentence give a fair indication of how respondents
might read the question.) The contingency question
format illustrated in Figure 9-2 should reduce
the subtle pressure on respondents to report having
smoked marijuana.
Used properly, even rather complex sets of
contingency questions can be constructed without
confusing the respondent. Figure 9-3 illustrates a
more complicated example.
Sometimes a set of contingency questions is
long enough to extend over several pages. Suppose
you’re studying political activities of college
students, and you wish to ask a large number of
questions of those students who have voted in a
national, state, or local election. You could separate
out the relevant respondents with an initial question
such as “Have you ever voted in a national,
state, or local election?” but it would be confusing
to place the contingency questions in a box stretching
over several pages. It would make more sense
to enter instructions, in parentheses after each
answer, telling respondents to answer or skip the
contingency questions. Figure 9-4 provides an illustration
of this method.
In addition to these instructions, it’s worthwhile
to place an instruction at the top of each
page containing only the contingency questions.
For example, you might say, “This page is only for
respondents who have voted in a national, state,
or local election.” Clear instructions such as these
spare respondents the frustration of reading and
puzzling over questions irrelevant to them and
increase the likelihood of responses from those for
whom the questions are relevant.
Matrix Questions
Quite often, you’ll want to ask several questions
that have the same set of answer categories. This
is typically the case whenever the Likert response
categories are used. In such cases, it is often possible
to construct a matrix of items and answers as
illustrated in Figure 9-5.
This format offers several advantages over other
formats. First, it uses space efficiently. Second,
respondents will probably find it faster to complete
a set of questions presented in this fashion
than in other ways. In addition, this format may
increase the comparability of responses given to
different questions for the respondent as well as for
the researcher. Because respondents can quickly
review their answers to earlier items in the set,
they might choose between, say, “strongly agree”
and “agree” on a given statement by comparing
the strength of their agreement with their earlier
responses in the set.
There are some dangers inherent in using this
format, however. Its advantages may encourage
you to structure an item so that the responses fit
FIGURE 9-3
Contingency Table. Sometimes it will be appropriate for certain
kinds of respondents to skip over inapplicable questions. To
avoid confusion, you should be sure to provide clear instructions
to that end.
FIGURE 9-4
Instructions to Skip
Questionnaire Construction ■ 265
Similarly, if respondents are asked to assess
their overall religiosity (“How important is your
religion to you in general?”), their responses
to later questions concerning specific aspects of
religiosity will be aimed at consistency with the
prior assessment. The converse is true as well. If
respondents are first asked specific questions about
different aspects of their religiosity, their subsequent
overall assessment will reflect the earlier
answers. The order of responses within a question
can also make a difference (Bishop and Smith
2001).
The impact of item order is not uniform. When
J. Edwin Benton and John Daly (1991) conducted
a local government survey, they found that the
less-educated respondents were more influenced
by the order of questionnaire items than those with
more education were.
Some researchers attempt to overcome this
effect by randomizing the order of items. This effort
is usually futile. In the first place, a randomized set
of items will probably strike respondents as chaotic
and worthless. The random order also makes it
more difficult for respondents to answer, because
they must continually switch their attention from
one topic to another. Finally, even a randomized
ordering of items will have the effect discussed
previously—except that you’ll have no control
over the effect.
The safest solution is sensitivity to the problem.
Although you cannot avoid the effect of item order,
into the matrix format when a different, more
idiosyncratic set of responses might be more
appropriate. Also, the matrix question format can
foster a response-set among some respondents:
They may develop a pattern of, say, agreeing with
all the statements. This would be especially likely if
the set of statements began with several that indicated
a particular orientation (for example, a liberal
political perspective) with only a few later ones
representing the opposite orientation. Respondents
might assume that all the statements represented
the same orientation and, reading quickly, misread
some of them, thereby giving the wrong answers.
This problem can be reduced somewhat by alternating
statements representing different orientations
and by making all statements short and clear.
Ordering Items
in a Questionnaire
The order in which questionnaire items are presented
can also affect responses. First, the appearance
of one question can affect the answers given
to later ones. For example, if several questions have
been asked about the dangers of terrorism to the
United States and then a question asks respondents
to volunteer (open-endedly) what they believe to
represent dangers to the United States, terrorism
will receive more citations than would otherwise be
the case. In this situation, it’s preferable to ask the
open-ended question first.
FIGURE 9-5
Matrix Question Format. Matrix questions offer an efficient format for presenting a set of closed-ended questionnaire items that
have the same response categories.
266 ■ Chapter 9: Survey Research
It’s useful to begin every self-administered
questionnaire with basic instructions for completing
it. Although many people these days have
experience with forms and questionnaires, begin
by telling them exactly what you want: that they
are to indicate their answers to certain questions by
placing a check mark or an X in the box beside the
appropriate answer or by writing in their answer
when asked to do so. If many open-ended questions
are used, respondents should be given some
guidelines about whether brief or lengthy answers
are expected. If you wish to encourage your respondents
to elaborate on their responses to closedended
questions, that should be noted.
If a questionnaire has subsections—political
attitudes, religious attitudes, background data—introduce
each with a short statement concerning its
content and purpose. For example, “In this section,
we would like to know what people consider to
be the most important community problems.” Demographic
items at the end of a self-administered
questionnaire might be introduced thus: “Finally,
we would like to know just a little about you so we
can see how different types of people feel about the
issues we have been examining.”
Short introductions such as these help the
respondent make sense of the questionnaire. They
make the questionnaire seem less chaotic, especially
when it taps a variety of data. And they help
put the respondent in the proper frame of mind for
answering the questions.
Some questions may require special instructions
to facilitate proper answering. This is especially
true if a given question varies from the
general instructions pertaining to the whole questionnaire.
Some specific examples will illustrate this
situation.
Despite attempts to provide mutually exclusive
answers in closed-ended questions, often more than
one answer will apply for respondents. If you want
a single answer, you should make this perfectly
clear in the question. An example would be “From
the list below, please check the primary reason for
your decision to attend college.” Often the main
question can be followed by a parenthetical note:
“Please check the one best answer.” If, on the other
hand, you want the respondent to check as many
answers as apply, you should make this clear.
try to estimate what that effect will be so that you
can interpret results meaningfully. If the order of
items seems especially important in a given study,
you might construct more than one version of the
questionnaire with different orderings of the items.
You will then be able to determine the effects by
comparing responses to the various versions. At the
very least, you should pretest your questionnaire
in the different forms. (We’ll discuss pretesting in a
moment.)
The desired ordering of items differs between
interviews and self-administered questionnaires.
In the latter, it’s usually best to begin the questionnaire
with the most interesting set of items. The
potential respondents who glance casually over the
first few items should want to answer them. Perhaps
the items will ask for attitudes they’re aching
to express. At the same time, however, the initial
items should not be threatening. (It might be a
bad idea to begin with items about sexual behavior
or drug use.) Requests for duller, demographic
data (age, gender, and the like) should generally
be placed at the end of a self-administered questionnaire.
Placing these items at the beginning, as
many inexperienced researchers are tempted to do,
gives the questionnaire the initial appearance of a
routine form, and the person receiving it may not
be motivated to complete it.
Just the opposite is generally true for interview
surveys. When the potential respondent’s
door first opens, the interviewer must gain rapport
quickly. After a short introduction to the study,
the interviewer can best begin by enumerating the
members of the household, getting demographic
data about each. Such items are easily answered
and generally nonthreatening. Once the initial rapport
has been established, the interviewer can then
move into the area of attitudes and more sensitive
matters. An interview that began with the question
“Do you believe in witchcraft?” would probably
end rather quickly.
Questionnaire Instructions
Every questionnaire, whether it is to be completed
by respondents or administered by interviewers,
should contain clear instructions and introductory
comments where appropriate.
Self-Administered Questionnaires ■ 267
When the respondent is supposed to rankorder
a set of answer categories, the instructions
should indicate this, and a different type of answer
format should be used (for example, blanks instead
of boxes). These instructions should indicate how
many answers are to be ranked (for example: all;
only the first and second; only the first and last; the
most important and least important). These instructions
should also spell out the order of ranking (for
example: “Place a 1 beside the most important item,
a 2 beside the next most important, and so forth”).
Rank-ordering of responses is often difficult for respondents,
however, because they may have to read
and reread the list several times, so this technique
should be used only in those situations where no
other method will produce the desired result.
In multiple-part matrix questions, giving special
instructions is useful unless the same format is
used throughout the questionnaire. Sometimes respondents
will be expected to check one answer in
each column of the matrix; in other questionnaires
they’ll be expected to check one answer in each
row. Whenever the questionnaire contains both
formats, it’s useful to add an instruction clarifying
which is expected in each case.
Pretesting the Questionnaire
No matter how carefully researchers design a datacollection
instrument such as a questionnaire, there
is always the possibility—indeed the certainty—of
error. They will always make some mistake: an ambiguous
question, one that people cannot answer,
or some other violation of the rules just discussed.
The surest protection against such errors is to
pretest the questionnaire in full or in part. Give
the questionnaire to the ten people in your bowling
league, for example. It’s not usually essential
that the pretest subjects comprise a representative
sample, although you should use people for whom
the questionnaire is at least relevant.
By and large, it’s better to ask people to complete
the questionnaire than to read through it
looking for errors. All too often, a question seems
to make sense on a first reading, but it proves to be
impossible to answer.
Stanley Presser and Johnny Blair (1994)
describe several different pretesting strategies and
report on the effectiveness of each. They also
provide data on the cost of the various methods.
Paul Beatty and Gordon Willis (2007) offer a useful
review of “cognitive interviewing.” In this technique,
the pretest includes gathering respondents’
comments about the questionnaire itself, so that
the researchers can see which questions are communicating
effectively and collecting the information
sought.
There are many more tips and guidelines for
questionnaire construction, but covering them all
would take a book in itself. For now I’ll complete
this discussion with an illustration of a real questionnaire,
showing how some of these comments
find substance in practice.
Before turning to the illustration, however, I
want to mention a critical aspect of questionnaire
design: precoding. Because the information collected
by questionnaires is typically transformed
into some type of computer format, it’s usually appropriate
to include data-processing instructions on
the questionnaire itself. These instructions indicate
where specific pieces of information will be stored
in the machine-readable data files. Notice that the
following illustration has been precoded with the
mysterious numbers that appear near questions
and answer categories.
A Composite Illustration
Figure 9-6 is part of a questionnaire used by the
University of Chicago’s National Opinion Research
Center in its General Social Survey. The questionnaire
dealt with people’s attitudes toward the government
and was designed to be self-administered,
though most of the GSS is conducted in face-toface
interviews.
Self-Administered
Questionnaires
So far we’ve discussed how to formulate questions
and how to design effective questionnaires. As important
as these tasks are, the labor will be wasted
(Text continues on p. 270.)
10. Here are some things the government might do for the economy. Circle one number for
each action to show whether you are in favor of it or against it.
1. Strongly in favor of
2. In favor of
3. Neither in favor of nor against
4. Against
5. Strongly against
PLEASE CIRCLE A NUMBER
a. Control of wages by legislation .......................................... 1 2 3 4 5 28/
b. Control of prices by legislation 1 2 3 4 5 29/
c. Cuts in government spending 1 2 3 4 5 30/
d. Government financing of projects to
create new jobs 1 2 3 4 5 31/
e. Less government regulation of business 1 2 3 4 5 32/
f. Support for industry to develop new
products and technology 1 2 3 4 5 33/
g. Supporting declining industries to
protect jobs 1 2 3 4 5 34/
h. Reducing the work week to create
more jobs 1 2 3 4 5 35/
11. Listed below are various areas of government spending. Please indicate whether you
would like to see more or less government spending in each area. Remember that if
you say “much more,” it might require a tax increase to pay for it.
1. Spend much more
2. Spend more
3. Spend the same as now
4. Spend less
5. Spend much less
8. Can’t choose
PLEASE CIRCLE A NUMBER
a. The environment 1 2 3 4 5 8 36/
b. Health 1 2 3 4 5 8 37/
c. The police and law enforcement 1 2 3 4 5 8 38/
d. Education 1 2 3 4 5 8 39/
e. The military and defense 1 2 3 4 5 8 40/
f. Retirement benefits 1 2 3 4 5 8 41/
g. Unemployment benefits 1 2 3 4 5 8 42/
h. Culture and the arts 1 2 3 4 5 8 43/
12. If the government had to choose between keeping down inflation or keeping down unemployment,
to which do you think it should give highest priority?
Keeping down inflation 1 44/
Keeping down unemployment 2
Can’t choose 8
13. Do you think that labor unions in this country have too much power or too little power?
Far too much power 1 45/
Too much power 2
About the right amount of power 3
Too little power 4
Far too little power 5
Can’t choose 8
...........................................
............................................
..................................................................
...........................
....................................................
........................................................................
...........................................................................
.......................................................
........................................................................
...............................
..................................................................
...........................................
...................................................
.............................................
...................................................
.......................................................................................................................
............................................................................................................
......................................................................................................................................
...........................................................................................................................
................................................................................................................................
.......................................................................................................
...................................................................................................................................
.............................................................................................................................
......................................................................................................................................
FIGURE 9-6
A Sample Questionnaire. This questionnaire excerpt is from the General Social Survey, a major source
of data for analysis by social researchers around the world.
14. How about business and industry, do they have too much power or too little power?
Far too much power 1 46/
Too much power 2
About the right amount of power 3
Too little power 4
Far too little power 5
Can’t choose 8
15. And what about the federal government, does it have to much power or too little
power?
Far too much power 1 47/
Too much power 2
About the right amount of power 3
Too little power 4
Far too little power 5
Can’t choose 8
16. In general, how good would you say labor unions are for the country as a whole?
Excellent 1 48/
Very good 2
Fairly good 3
Not very good 4
Not good at all 5
Can’t choose 8
17. What do you think the government’s role in each of these industries should be?
1. Own it
2. Control prices and profits
but not own it
3. Neither own it nor control its
prices and profits
8. Can’t choose
PLEASE CIRCLE A NUMBER
a. Electric power ..................................................................... 1 2 3 8 49/
b. The steel industry 1 2 3 8 50/
c. Banking and insurance 1 2 3 8 51/
18. On the whole, do you think it should or should not be the government’s responsibility
to . . .
1. Definitely should be
2. Probably should be
3. Probably should not be
4. Definitely should not be
8. Can’t choose
PLEASE CIRCLE A NUMBER
a. Provide a job for everyone who wants one 1 2 3 4 8 52/
b. Keep prices under control ........................................................ 1 2 3 4 8 53/
c. Provide health care for the sick 1 2 3 4 8 54/
d. Provide a decent standard of living for
the old 1 2 3 4 8 55/
.......................................................................................................
..................................................................................................................................
............................................................................................................................
......................................................................................................................................
..........................................................................................................................
................................................................................................................................
.......................................................................................................
..................................................................................................................................
.............................................................................................................................
......................................................................................................................................
............................................................................................................................................
..........................................................................................................................................
.........................................................................................................................................
....................................................................................................................................
...................................................................................................................................
......................................................................................................................................
..........................................................................................................................
................................................................................................................................
...............................................................
.......................................................
..............................
...............................................
......................................................................................
FIGURE 9-6
(Continued)
270 ■ Chapter 9: Survey Research
specifically to the mail survey, which is still the
typical form of self-administered questionnaire.
Mail Distribution and Return
The basic method for collecting data through the
mail has been to send a questionnaire accompanied
by a letter of explanation and a self-addressed,
stamped envelope for returning the questionnaire.
The respondent is expected to complete the
questionnaire, put it in the envelope, and return it.
If, by any chance, you’ve received such a questionnaire
and failed to return it, it would be valuable
to recall the reasons you had for not returning it
and keep them in mind any time you plan to send
questionnaires to others.
A common reason for not returning questionnaires
is that it’s too much trouble. To overcome
this problem, researchers have developed several
ways to make returning them easier. For instance,
a self-mailing questionnaire requires no return
envelope: When the questionnaire is folded a
particular way, the return address appears on the
outside. The respondent therefore doesn’t have to
worry about losing the envelope.
More-elaborate designs are available also. The
university student questionnaire to be described
later in this chapter was bound in a booklet with
a special, two-panel back cover. Once the questionnaire
was completed, the respondent needed
only to fold out the extra panel, wrap it around
the booklet, and seal the whole thing with the
adhesive strip running along the edge of the panel.
The foldout panel contained my return address
and postage. When I repeated the study a couple
of years later, I improved on the design. Both the
front and back covers had foldout panels: one for
sending the questionnaire out and the other for
getting it back—thus avoiding the use of envelopes
altogether.
The point here is that anything you can do
to make the job of completing and returning the
questionnaire easier will improve your study. Imagine
receiving a questionnaire that made no provisions
for its return to the researcher. Suppose you
had to (1) find an envelope, (2) write the address
on it, (3) figure out how much postage it required,
unless the questionnaire produces useful data—
which means that respondents actually complete
the questionnaire. We turn now to the major
methods for getting responses to questionnaires.
I’ve referred several times in this chapter to
interviews and self-administered questionnaires.
Actually, there are three main methods of administering
survey questionnaires to a sample of respondents:
self-administered questionnaires, in which
respondents are asked to complete the questionnaire
themselves; surveys administered by interviewers
in face-to-face encounters; and surveys
conducted by telephone. This section and the next
two discuss each of these methods in turn. A fourth
section addresses online surveys, a new technique
growing in popularity.
The most common form of self-administered
questionnaire is the mail survey. However, there
are several other techniques that are often used as
well. At times, it may be appropriate to administer
a questionnaire to a group of respondents gathered
at the same place at the same time. For example, a
survey of students taking introductory psychology
might be conducted during class. High school students
might be surveyed during homeroom period.
Some recent experimentation has been conducted
with regard to the home delivery of questionnaires.
A research worker delivers the questionnaire
to the home of sample respondents and
explains the study. Then the questionnaire is left
for the respondent to complete, and the researcher
picks it up later.
Home delivery and the mail can also be used in
combination. Questionnaires are mailed to families,
and then research workers visit homes to pick up
the questionnaires and check them for completeness.
Just the opposite technique is to have questionnaires
hand-delivered by research workers with
a request that the respondents mail the completed
questionnaires to the research office.
On the whole, when a research worker either
delivers the questionnaire, picks it up, or both, the
completion rate seems higher than it is for straightforward
mail surveys. Additional experimentation
with this technique is likely to point to other ways
to improve completion rates while reducing costs.
The remainder of this section, however, is devoted
Self-Administered Questionnaires ■ 271
reports the cumulative number or percentage. In
part, this activity provides the researchers with
gratification, as they get to draw a picture of their
successful data collection. More important, however,
it serves as their guide to how the data collection
is going. If follow-up mailings are planned, the
graph provides a clue about when such mailings
should be launched. (The dates of subsequent mailings
should be noted on the graph.)
As completed questionnaires are returned,
each should be opened, scanned, and assigned
an identification (ID) number. These numbers
should be assigned serially as the questionnaires
are returned, even if other identification numbers
have already been assigned. Two examples
should illustrate the important advantages of this
procedure.
Let’s assume you’re studying attitudes toward
a political figure. In the middle of the data collection,
the media break the story that the politician is
having extramarital affairs. By knowing the date of
that public disclosure and the dates when questionnaires
were received, you’ll be in a position to
determine the effects of the disclosure. (Recall from
Chapter 8 the discussion of history in connection
with experiments.)
In a less sensational way, serialized ID numbers
can be valuable in estimating non-response biases
in the survey. Barring more-direct tests of bias,
you may wish to assume that those who failed to
answer the questionnaire will be more like respondents
who delayed answering than like those who
answered right away. An analysis of questionnaires
received at different points in the data collection
might then be used for estimates of sampling bias.
For example, if the grade point averages (GPAs)
reported by student respondents decrease steadily
through the data collection, with those replying
right away having higher GPAs and those replying
later having lower GPAs, you might tentatively conclude
that those who failed to answer at all have
lower GPAs yet. Although it would not be advisable
to make statistical estimates of bias in this fashion,
you could take advantage of approximate estimates
based on the patterns you’ve observed.
If respondents have been identified for purposes
of follow-up mailing, then preparations
and (4) put the stamps on it. How likely is it that
you would return the questionnaire?
A few brief comments on postal options are in
order. You have options for mailing questionnaires
out and for getting them returned. On outgoing
mail, your choices are essentially between first-class
postage and bulk rate. First class is more certain,
but bulk rate is far cheaper. (Check your local
post office for rates and procedures.) On return
mail, your choice is between postage stamps and
business-reply permits. Here, the cost differential is
more complicated. If you use stamps, you pay for
them whether people return their questionnaires
or not. With the business-reply permit, you pay for
only those that are used, but you pay an additional
surcharge of about a nickel. This means that stamps
are cheaper if a lot of questionnaires are returned,
but business-reply permits are cheaper if fewer are
returned (and you won’t know in advance how
many will be returned).
There are many other considerations involved
in choosing among the several postal options. Some
researchers, for example, feel that using postage
stamps communicates more “humanness” and sincerity
than using bulk rate and business-reply permits
does. Others worry that respondents will steam
off the stamps and use them for some purpose other
than returning the questionnaires. Because both
bulk rate and business-reply permits require establishing
accounts at the post office, you’ll probably
find stamps much easier for small surveys.
Monitoring Returns
The mailing of questionnaires sets up a new research
question that may prove valuable to a study.
Researchers shouldn’t sit back idly as questionnaires
are returned; instead, they should undertake
a careful recording of the varying rates of return
among respondents.
An invaluable tool in this activity is a return
rate graph. The day on which questionnaires were
mailed is labeled Day 1 on the graph, and every day
thereafter the number of returned questionnaires
is logged on the graph. It’s usually best to compile
two graphs. One shows the number returned each
day—rising over time, then dropping. The second
272 ■ Chapter 9: Survey Research
mailing time—out and in—is more than two or
three days.)
If the individuals in the survey sample are not
identified on the questionnaires, it may not be possible
to remail only to nonrespondents. In such a
case, send your follow-up mailing to all members of
the sample, thanking those who may have already
participated and encouraging those who have not
to do so. (The case study reported later describes
yet another method you can use in an anonymous
mail survey.)
Response Rates
A question that new survey researchers frequently
ask concerns the percentage return rate, or the
response rate, that should be achieved in a survey.
The body of inferential statistics used in connection
with survey analysis assumes that all members of
the initial sample complete the survey. Because this
almost never happens, non-response bias becomes
a concern, with the researcher testing (and hoping)
for the possibility that the respondents look essentially
like a random sample of the initial sample,
and thus a somewhat smaller random sample of
the total population.
Nevertheless, overall response rate is one
guide to the representativeness of the sample respondents.
If a high response rate is achieved, there
is less chance of significant non-response bias than
with a low rate. Conversely, a low response rate is
a danger signal, because the nonrespondents are
likely to differ from the respondents in ways other
than just their willingness to participate in the survey.
Richard Bolstein (1991), for example, found
that those who did not respond to a preelection
political poll were less likely to vote that those who
did participate. Estimating the turnout rate from
just the survey respondents, then, would have
overestimated the number who would show up
at the polls. Ironically, of course, since the non-respondents
were unlikely to vote, the preferences of
the survey participants might offer a good estimate
of the election results.
As you can imagine, one of the more persistent
discussions among survey researchers concerns
for those mailings should be made as the
questionnaires are returned. The case study later in
this section discusses this process in greater detail.
Follow-up Mailings
Follow-up mailings may be administered in several
ways. In the simplest, nonrespondents are simply
sent a letter of additional encouragement to
participate. A better method, however, is to send
a new copy of the survey questionnaire with the
follow-up letter. If potential respondents have not
returned their questionnaires after two or three
weeks, the questionnaires have probably been lost
or misplaced. Receiving a follow-up letter might
encourage them to look for the original questionnaire,
but if they can’t find it easily, the letter may
go for naught.
The methodological literature strongly suggests
that follow-up mailings provide an effective
method for increasing return rates in mail surveys.
In general, the longer a potential respondent delays
replying, the less likely he or she is to do so at all.
Properly timed follow-up mailings, then, provide
additional stimuli to respond.
The effects of follow-up mailings will be seen
in the response rate curves recorded during data
collection. The initial mailings will be followed by
a rise and subsequent subsiding of returns; the follow-up
mailings will spur a resurgence of returns;
and more follow-ups will do the same. In practice,
three mailings (an original and two follow-ups)
seem the most efficient.
The timing of follow-up mailings is also important.
Here the methodological literature offers
less-precise guides, but I’ve found that two or three
weeks is a reasonable space between mailings.
(This period might be increased by a few days if the
response rate The number of people participating
in a survey divided by the number selected in
the sample, in the form of a percentage. This is also
called the completion rate or, in self-administered surveys,
the return rate: the percentage of questionnaires
sent out that are returned.
Self-Administered Questionnaires ■ 273
study was conducted by the students in my graduate
seminar in survey research methods.
As you may recall, 1,100 students were selected
from the university registration database through a
stratified, systematic sampling procedure. For each
student selected, six self-adhesive mailing labels
were printed by the computer.
By the time we were ready to distribute the
questionnaires, it became apparent that our meager
research funds wouldn’t cover several mailings to
the entire sample of 1,100 students (questionnaire
printing costs were higher than anticipated). As
a result, we chose a systematic two-thirds sample
of the mailing labels, yielding a subsample of 733
students.
Earlier, we had decided to keep the survey
anonymous in the hope of encouraging morecandid
responses to some sensitive questions.
(Later surveys of the same issues among the same
population indicated this anonymity was unnecessary.)
Thus, the questionnaires would carry no
identification of students on them. At the same
time, we hoped to reduce the follow-up mailing
costs by mailing only to nonrespondents.
To achieve both of these aims, a special postcard
method was devised. Each student was mailed
a questionnaire that carried no identifying marks,
plus a postcard addressed to the research office—
with one of the student’s mailing labels affixed to
the reverse side of the card. The introductory letter
asked the student to complete and return the questionnaire—assuring
anonymity—and to return
the postcard simultaneously. Receiving the postcard
would tell us—without indicating which questionnaire
it was—that the student had returned his
or her questionnaire. This procedure would then
facilitate follow-up mailings.
The 32-page questionnaire was printed in
booklet form. The three-panel cover described earlier
in this chapter permitted the questionnaire to
be returned without an additional envelope.
A letter introducing the study and its purposes
was printed on the front cover of the booklet. It
explained why the study was being conducted (to
learn how students feel about a variety of issues),
how students had been selected for the study, the
ways of increasing response rates. You’ll recall that
this was a chief concern in the earlier discussion
of options for mailing out and receiving questionnaires.
Survey researchers have developed many
ingenious techniques addressing this problem.
Some have experimented with novel formats. Others
have tried paying respondents to participate.
The problem with paying, of course, is that it’s
expensive to make meaningfully high payment to
hundreds or thousands of respondents, but some
imaginative alternatives have been used. Some
researchers have said, “We want to get your twocents’
worth on some issues, and we’re willing to
pay”—enclosing two pennies. Another enclosed a
quarter, suggesting that the respondent make some
little child happy. Still others have enclosed paper
money. Similarly, Michael Davern and his colleagues
(2003) found that financial incentives also
increased completion rates in face-to-face interview
surveys (discussed in the next section).
Don Dillman (2007) has spent decades painstakingly
assessing the various techniques that survey
researchers have used to increase return rates
on mail surveys, and he evaluates the impact of
each. More important, Dillman stresses the necessity
of paying attention to all aspects of the study—
what he calls the “Tailored Design Method”—
rather than one or two special gimmicks.
Having said all this, thereis no absolutely acceptablelevel
of response to a mail survey, except for
100 percent. Whileit is possible to achieve response
ratesof 70 percent or more,most mailsurveys probably
fallbelowthat level. Thus,it’s important to test
for non-response bias wherever possible.
A Case Study
The steps involved in the administration of a mail
survey are many and can best be appreciated in a
walk-through of an actual study. Accordingly, this
section concludes with a detailed description of
how the student survey we discussed in Chapter
7 as an illustration of systematic sampling was
administered. This study did not represent the
theoretical ideal for such studies, but in that regard
it serves our present purposes all the better. The
274 ■ Chapter 9: Survey Research
The follow-up mailing stimulated a resurgence
of returns, as expected, and the same logging procedures
continued. The returned postcards told us
which additional mailing labels to destroy. Unfortunately,
time and financial pressures made a third
mailing impossible, despite initial plans to do so,
but the two mailings resulted in an overall return
rate of 62 percent.
This illustration should give you a fairly good
sense of what’s involved in the execution of mailed
self-administered questionnaires. Let’s turn now to
the second principal method of conducting surveys,
in-person interviews.
Interview Surveys
The interview is an alternative method of collecting
survey data. Rather than asking respondents to
read questionnaires and enter their own answers,
researchers send interviewers to ask the questions
orally and record respondents’ answers. Interviewing
is typically done in a face-to-face encounter,
but telephone interviewing, discussed in the next
section, follows most of the same guidelines.
Most interview surveys require more than
one interviewer, although you might undertake a
small-scale interview survey yourself. Portions of
this section will discuss methods for training and
supervising a staff of interviewers assisting you
with a survey.
This section deals specifically with survey interviewing.
Chapter 10 discusses the less-structured,
in-depth interviews often conducted in qualitative
field research.
The Role of the Survey
Interviewer
There are several advantages to having a questionnaire
administered by an interviewer rather than
a respondent. To begin with, interview surveys
typically attain higher response rates than mail
surveys do. A properly designed and executed
interview survey ought to achieve a completion
rate of at least 80 to 85 percent. (Federally funded
importance of each student’s responding, and the
mechanics of returning the questionnaire.
Students were assured that their responses
to the survey were anonymous, and the postcard
method was explained. A statement followed about
the auspices under which the study was being conducted,
and a telephone number was provided for
those who might want more information about the
study. (Five students called for information.)
By printing the introductory letter on the questionnaire,
we avoided the necessity of enclosing a
separate letter in the outgoing envelope, thereby
simplifying the task of assembling mailing pieces.
The materials for the initial mailing were assembled
as follows. (1) One mailing label for each
student was stuck on a postcard. (2) Another label
was stuck on an outgoing manila envelope. (3)
One postcard and one questionnaire were placed
in each envelope—with a glance to ensure that the
name on the postcard and on the envelope were
the same in each case.
The distribution of the survey questionnaires
had been set up for a bulk rate mailing. Once the
questionnaires had been stuffed into envelopes,
they were grouped by zip code, tied in bundles, and
delivered to the post office.
Shortly after the initial mailing, questionnaires
and postcards began arriving at the research office.
Questionnaires were opened, scanned, and assigned
identification numbers as described earlier in
this chapter. For every postcard received, a search
was made for that student’s remaining labels, and
they were destroyed.
After two or three weeks, the remaining mailing
labels were used to organize a follow-up mailing.
This time a special, separate letter of appeal
was included in the mailing piece. The new letter
indicated that many students had returned their
questionnaires already, and it was very important
for all others to do so as well.
interview A data-collection encounter in which
one person (an interviewer) asks questions of another
(a respondent). Interviews may be conducted
face-to-face or by telephone.
Interview Surveys ■ 275
The interviewer must also fit into this ideal
situation. The interviewer’s presence should affect
neither a respondent’s perception of a question nor
the answer given. In other words, the interviewer
should be a neutral medium through which questions
and answers are transmitted.
As such, different interviewers should obtain
exactly the same responses from a given respondent.
(Recall our earlier discussions of reliability.)
This neutrality has a special importance in area
samples. To save time and money, a given interviewer
is typically assigned to complete all the
interviews in a particular geographic area—a city
block or a group of nearby blocks. If the interviewer
does anything to affect the responses
obtained, the bias thus interjected might be interpreted
as a characteristic of that area.
Let’s suppose that a survey is being done to determine
attitudes toward low-cost housing in order
to help in the selection of a site for a new government-sponsored
development. An interviewer
assigned to a given neighborhood might—through
word or gesture—communicate his or her own
distaste for low-cost housing developments. Respondents
might therefore tend to give responses
in general agreement with the interviewer’s own
position. The results of the survey would indicate
that the neighborhood in question strongly resists
construction of the development in its area when
in fact their apparent resistance simply reflects the
interviewer’s attitudes.
General Guidelines for Survey
Interviewing
The manner in which interviews ought to be conducted
will vary somewhat by survey population
and survey content. Nevertheless, some general
guidelines apply to most interviewing situations.
Appearance and Demeanor
As a rule, interviewers should dress in a fashion
similar to that of the people they’ll be interviewing.
A richly dressed interviewer will probably
have difficulty getting good cooperation and responses
from poorer respondents; a poorly dressed
surveys often require one of these response rates.)
Respondents seem more reluctant to turn down
an interviewer standing on their doorstep than to
throw away a mail questionnaire.
The presence of an interviewer also generally
decreases the number of “don’t knows” and “no
answers.” If minimizing such responses is important
to the study, the interviewer can be instructed
to probe for answers (“If you had to pick one of the
answers, which do you think would come closest to
your feelings?”).
Further, if a respondent clearly misunderstands
the intent of a question or indicates that he or she
does not understand, the interviewer can clarify
matters, thereby obtaining relevant responses. (As
we’ll discuss shortly, such clarifications must be
strictly controlled through formal specifications.)
Finally, the interviewer can observe respondents
as well as ask questions. For example, the
interviewer can note the respondent’s race if this
is considered too delicate a question to ask. Similar
observations can be made regarding the quality
of the dwelling, the presence of various possessions,
the respondent’s ability to speak English, the
respondent’s general reactions to the study, and
so forth. In one survey of students, respondents
were given a short, self-administered questionnaire
to complete—concerning sexual attitudes
and behavior—during the course of the interview.
While respondents completed the questionnaire,
the interviewer made detailed notes regarding their
dress and grooming.
This procedure raises an ethical issue. Some
researchers have objected that such practices violate
the spirit of the agreement by which the respondent
has allowed the interview. Although ethical issues
seldom are clear-cut in social research, it’s important
to be sensitive to them, as we saw in Chapter 3.
Survey research is of necessity based on an
unrealistic stimulus-response theory of cognition
and behavior. Researchers must assume that a
questionnaire item will mean the same thing to
every respondent, and every given response must
mean the same when given by different respondents.
Although this is an impossible goal, survey
questions are drafted to approximate the ideal as
closely as possible.
276 ■ Chapter 9: Survey Research
Ultimately, the interviewer must be able to read
the questionnaire items to respondents without
error, without stumbling over words and phrases.
A good model is the actor reading lines in a play or
movie. The lines must be read as though they constituted
a natural conversation, but that conversation
must follow exactly the language set down in
the questionnaire.
By the same token, the interviewer must be
familiar with the specifications prepared in conjunction
with the questionnaire. Inevitably some
questions will not exactly fit a given respondent’s
situation, and the interviewer must determine how
the question should be interpreted in that situation.
The specifications provided to the interviewer
should give adequate guidance in such cases,
but the interviewer must know the organization
and contents of the specifications well enough to
refer to them efficiently. It would be better for the
interviewer to leave a given question unanswered
than to spend five minutes searching through the
specifications for clarification or trying to interpret
the relevant instructions.
Following Question Wording Exactly
The first part of this chapter discussed the
significance of question wording for the responses
obtained. A slight change in the wording of a given
question may lead a respondent to answer “yes”
rather than “no.” It follows that interviewers must
be instructed to follow the wording of questions
exactly. Otherwise all the effort that the developers
have put into carefully phrasing the questionnaire
items to obtain the information they need and to
ensure that respondents interpret items precisely as
intended will be wasted.
While I hope the logic of this injunction is
clear, it is not necessarily a closed discussion. For
example, Giampietro Gobo (2006) argues that we
might consider giving interviewers more latitude,
suggesting that respondents sometimes make errors
that may be apparent to the interviewer on the
spot. Allowing the interviewer to intervene, as he
notes, does increase the possibility that the interviewer
will impact the data collected.
interviewer will have similar difficulties with richer
respondents. To the extent that the interviewer’s
dress and grooming differ from those of the respondents,
it should be in the direction of cleanliness
and neatness in modest apparel. If cleanliness is
not next to godliness, it appears at least to be next
to neutrality. Although middle-class neatness and
cleanliness may not be accepted by all sectors of
U.S. society, they remain the primary norm and
are the most likely to be acceptable to the largest
number of respondents.
Dress and grooming are typically regarded as
signs of a person’s attitudes and orientations. Torn
jeans, green hair, and razor blade earrings may
communicate—correctly or incorrectly—that the
interviewer is politically radical, sexually permissive,
favorable to drug use, and so forth. Any of
these impressions could bias responses or affect the
willingness of people to be interviewed.
In demeanor, interviewers should be pleasant
if nothing else. Because they’ll be prying into a
respondent’s personal life and attitudes, they must
communicate a genuine interest in getting to know
the respondent, without appearing to spy. They
must be relaxed and friendly, without being too
casual or clinging. Good interviewers also have the
ability to determine very quickly the kind of person
the respondent will feel most comfortable with, the
kind of person the respondent would most enjoy
talking to. Clearly, the interview will be more successful
in this case. Further, because respondents
are asked to volunteer a portion of their time and
to divulge personal information, they deserve the
most enjoyable experience the researcher and
interviewer can provide.
Familiarity with the Questionnaire
If an interviewer is unfamiliar with the questionnaire,
the study suffers and the respondent faces an
unfair burden. The interview is likely to take more
time than necessary and be unpleasant. Moreover,
the interviewer cannot acquire familiarity by skimming
through the questionnaire two or three times.
He or she must study it carefully, question by question,
and must practice reading it aloud.
Interview Surveys ■ 277
respondent, however, may reply: “I think that’s
true.” The interviewer should follow this reply with
“Would you say you strongly agree or agree somewhat?”
If necessary, interviewers can explain that
they must check one or the other of the categories
provided. If the respondent adamantly refuses to
choose, the interviewer should write in the exact
response given by the respondent.
Probes are more frequently required in eliciting
responses to open-ended than closed-ended
questions. For example, in response to a question
about traffic conditions, the respondent might
simply reply, “Pretty bad.” The interviewer could
obtain an elaboration on this response through
a variety of probes. Sometimes the best probe is
silence; if the interviewer sits quietly with pencil
poised, the respondent will probably fill the pause
with additional comments. (This technique is used
effectively by newspaper reporters.) Appropriate
verbal probes might be “How is that?” or “In what
ways?” Perhaps the most generally useful probe is
“Anything else?”
Often, interviewers need to probe for answers
that will be sufficiently informative for analytical
purposes. In every case, however, such probes
must be completely neutral; they must not in any
way affect the nature of the subsequent response.
Whenever you anticipate that a given question
may require probing for appropriate responses,
you should provide one or more useful probes
next to the question in the questionnaire. This
practice has two important advantages. First, you’ll
have more time to devise the best, most neutral
probes. Second, all interviewers will use the same
probes whenever they’re needed. Thus, even if the
probe isn’t perfectly neutral, all respondents will
be presented with the same stimulus. This is the
same logical guideline discussed for question wording.
Although a question should not be loaded or
Recording Responses Exactly
Whenever the questionnaire contains open-ended
questions (ones soliciting the respondent’s own answers),
the interviewer must record those answers
exactly as given. No attempt should be made to
summarize, paraphrase, or correct bad grammar.
This exactness is especially important because
the interviewer will not know how the responses
are to be coded. Indeed, the researchers themselves
may not know the coding until they’ve
read a hundred or so responses. For example, the
questionnaire might ask respondents how they
feel about the traffic situation in their community.
One respondent might answer that there are too
many cars on the roads and that something should
be done to limit their numbers. Another might say
that more roads are needed. If the interviewer recorded
these two responses with the same summary—“congested
traffic”—the researchers would
not be able to take advantage of the important
differences in the original responses.
Sometimes, verbal responses are too inarticulate
or ambiguous to permit interpretation. However,
the interviewer may be able to understand
the intent of the response through the respondent’s
gestures or tone. In such a situation, the interviewer
should still record the exact verbal response
but also add marginal comments giving both the
interpretation and the reasons for arriving at it.
More generally, researchers can use any marginal
comments explaining aspects of the response
not conveyed in the verbal recording, such as the
respondent’s apparent anger, embarrassment, uncertainty
in answering, and so forth. In each case,
however, the exact verbal response should also be
recorded.
Probing for Responses
Sometimes respondents in an interview will give
an inappropriate or incomplete answer. In such
cases, a probe, or request for an elaboration, can
be useful. For example, a closed-ended question
may present an attitudinal statement and ask the
respondent to strongly agree, agree somewhat,
disagree somewhat, or strongly disagree. The
probe A technique employed in interviewing to
solicit a more complete answer to a question. It is a
nondirective phrase or question used to encourage
a respondent to elaborate on an answer. Examples
include “Anything more?” and “How is that?”
278 ■ Chapter 9: Survey Research
present problems. Suppose a respondent says he or
she will be 25 next week. The interviewer might
not be sure whether to take the respondent’s current
age or the nearest one. The specifications for
that question should explain what should be done.
(Probably, you would specify that the age as of last
birthday should be recorded in all cases.)
If you’ve prepared a set of specifications,
review them with the interviewers when you go
over the individual questions in the questionnaire.
Make sure your interviewers fully understand the
specifications and the reasons for them as well as
the questions themselves.
This portion of the interviewer training is likely
to generate many troublesome questions from
your interviewers. They’ll ask, “What should I do
if . . . ?” In such cases, avoid giving a quick, offhand
answer. If you have specifications, show how the
solution to the problem could be determined from
the specifications. If you do not have specifications,
show how the preferred handling of the situation
fits within the general logic of the question
and the purpose of the study. Giving unexplained
answers to such questions will only confuse the
interviewers and cause them to take their work less
seriously. If you don’t know the answer to such
a question when it’s asked, admit it and ask for
some time to decide on the best answer. Then
think out the situation carefully and be sure to give
all the interviewers your answer, explaining your
reasons.
Once you’ve gone through the whole questionnaire,
conduct one or two demonstration
interviews in front of everyone. Preferably, you
should interview someone other than one of the
interviewers. Realize that your interview will be a
model for those you’re training, so make it good. It
would be best, moreover, if the demonstration interview
were done as realistically as possible. Don’t
pause during the demonstration to point out how
you’ve handled a complicated situation: Handle it,
and then explain later. It’s irrelevant if the person
you’re interviewing gives real answers or takes on
some hypothetical identity for the purpose, as long
as the answers are consistent.
After the demonstration interviews, pair off
your interviewers and have them practice on each
biased, it’s essential that every respondent be presented
with the same question, even if it’s biased.
Coordination and Control
Most interview surveys require the assistance of
several interviewers. In large-scale surveys, interviewers
are hired and paid for their work. Student
researchers might find themselves recruiting
friends to help them interview. Whenever more
than one interviewer is involved in a survey, their
efforts must be carefully controlled. This control
has two aspects: training interviewers and supervising
them after they begin work.
The interviewers’ training session should begin
with a description of what the study is all about.
Even though the interviewers may be involved
only in the data-collection phase of the project, it
will be useful to them to understand what will be
done with the interviews they conduct and what
purpose will be served. Morale and motivation
are usually lower when interviewers don’t know
what’s going on.
The training on how to interview should
begin with a discussion of general guidelines and
procedures, such as those discussed earlier in this
section. Then the whole group should go through
the questionnaire together—question by question.
Don’t simply ask if anyone has any questions
about the first page of the questionnaire. Read the
first question aloud, explain the purpose of the
question, and then entertain any questions or comments
the interviewers may have. Once all their
questions and comments have been handled, go on
to the next question in the questionnaire.
It’s always a good idea to prepare specifications
to accompany an interview questionnaire.
Specifications are explanatory and clarifying comments
about handling difficult or confusing situations
that may occur with regard to particular
questions in the questionnaire. When drafting
the questionnaire, try to think of all the problem
cases that might arise—the bizarre circumstances
that might make a question difficult to answer.
The survey specifications should provide detailed
guidelines on how to handle such situations. For
example, even as simple a matter as age might
Telephone Surveys ■ 279
emphasis on practice applies equally to the oneperson
project and to the complex funded survey
with a large interviewing staff.
Telephone Surveys
For years telephone surveys had a rather bad
reputation among professional researchers. By
definition, telephone surveys are limited to people
who have telephones. Years ago, this method
produced a substantial social-class bias by excluding
poor people from the surveys. This was vividly
demonstrated by the Literary Digest fiasco of 1936.
Recall that, even though voters were contacted
by mail, the sample was partially selected from
telephone subscribers, who were hardly typical in
a nation just recovering from the Great Depression.
By 2003, however, the U.S. Bureau of the Census
(2006: 737, Table 1117) estimated that 95.5 percent
of all housing units had telephones, so the earlier
form of class bias has substantially diminished.
A related sampling problem involves unlisted
numbers. A survey sample selected from the pages
of a local telephone directory would omit all those
people—typically richer—who requested that
their numbers not be published. This potential bias
has been erased through a technique that has advanced
telephone sampling substantially: randomdigit
dialing (RDD).
Imagine that you were to select a set of sevendigit
telephone numbers at random. Even those
whose numbers were unlisted would have the
same chance of selection as those who were in
the directory would. However, if you were to start
dialing randomly selected numbers, a high proportion
of those would turn out to be “not in service,”
government offices, commercial enterprises, and
so forth. Fortunately, you can obtain ranges of
numbers that are (mostly) active residential numbers.
Selecting a set of those numbers at random
other. When they’ve completed the questionnaire,
have them reverse roles and do it again. Interviewing
is the best training for interviewing. As your interviewers
practice on each other, wander around,
listening in on the practice so you’ll know how
well they’re doing. Once the practice is completed,
the whole group should discuss their experiences
and ask any other questions they may have.
The final stage of the training for interviewers
should involve some “real” interviews. Have
them conduct some interviews under the actual
conditions that will pertain to the final survey. You
may want to assign them people to interview, or
perhaps they may be allowed to pick people themselves.
Don’t have them practice on people you’ve
selected in your sample, however. After each
interviewer has completed three to five interviews,
have him or her check back with you. Look over
the completed questionnaires for any evidence of
misunderstanding. Again, answer any questions
that the interviewers have. Once you’re convinced
that a given interviewer knows what to do, assign
some actual interviews, using the sample you’ve
selected for the study.
It’s essential to continue supervising the work
of interviewers over the course of the study. You
should check in with them after they conduct no
more than 20 or 30 interviews. You might assign
20 interviews, have the interviewer bring back
those questionnaires when they’re completed, look
them over, and assign another 20 or so. Although
this may seem overly cautious, you must continually
protect yourself against misunderstandings that
may not be evident early in the study. Moreover,
Kristen Olson and Andy Peytchev (2007) have
discovered that interviewers’ behavior continues
to change over the course of a survey project. For
example, interviewers speed through the interview
more quickly and are more likely to judge respondents
as uninterested in it.
If you’re the only interviewer in your study,
these comments may not seem relevant. However,
it would be wise, for example, to prepare
specifications for potentially troublesome questions
in your questionnaire. Otherwise, you run the risk
of making ad hoc decisions, during the course of
the study, that you’ll later regret or forget. Also, the
random-digit dialing (RDD) A sampling technique
in which random numbers are selected from
within the ranges of numbers assigned to active
telephones.
280 ■ Chapter 9: Survey Research
the greatest advantages are money and time, in that
order. In a face-to-face, household interview, you
may drive several miles to a respondent’s home,
find no one there, return to the research office,
and drive back the next day—possibly finding no
one there again. It’s cheaper and quicker to let your
fingers make the trips.
Interviewing by telephone, you can dress any
way you please without affecting the answers respondents
give. And sometimes respondents will be
more honest in giving socially disapproved answers
if they don’t have to look you in the eye. Similarly,
it may be possible to probe into more-sensitive
areas, though this isn’t necessarily the case. People
are, to some extent, more suspicious when they
can’t see the person asking them questions—
perhaps a consequence of “surveys” aimed at
selling magazine subscriptions and time-share
condominiums.
Interviewers can communicate a lot about
themselves over the phone, however, even though
they can’t be seen. For example, researchers
worry about the impact of an interviewer’s name
(particularly if ethnicity is relevant to the study)
and debate the ethics of having all interviewers
use bland “stage names” such as Smith or Jones.
(Female interviewers sometimes ask permission to
do this, to avoid subsequent harassment from men
they interview.)
Telephone surveys can allow greater control
over data collection if several interviewers
are engaged in the project. If all the interviewers
are calling from the research office, they can get
clarification from the person in charge whenever
problems occur, as they inevitably do. Alone in
the boondocks, an interviewer may have to wing
it between weekly visits with the interviewing
supervisor.
Finally, another important factor involved in
the growing use of telephone surveys has to do
with personal safety. Don Dillman (1978: 4) describes
the situation this way:
Interviewers must be able to operate comfortably
in a climate in which strangers are viewed
with distrust and must successfully counter
respondents’ objections to being interviewed.
Increasingly, interviewers must be willing to
will provide a representative sample of residential
households. As a consequence, random-digit dialing
has become a standard procedure in telephone
surveys.
The growth in popularity of cell phones has
become a new source of concern for survey researchers,
however, since cell phone numbers are
typically not included in phone surveys. Those who
use cell phones exclusively, moreover, tend to be
younger. This, of course, can affect survey outcomes.
For example, younger voters in 2004 were
more likely to vote for John Kerry than older voters
were. In 2008 they were more likely than the average
voter to support Barack Obama. Further, in a
study of this matter, Scott Keeter and his colleagues
(2008) found a distinct bias by age and the variables
closely related to it (such as marital status)
distinguishing those who were reachable only by
cell phone and those reachable by landline:
One of the most striking differences between
cell-only respondents and people reached on
a landline telephone is their age. Nearly half
of the cell-only respondents (46%) are under
age 30 compared to only 12% in the landline
sample. Related to their younger age, only 26%
of cell-only respondents are married, compared
with 57% percent of those in the landline
sample. Similarly, about half of cell-only
respondents have never been married (51%),
compared with only 16% in the landline
sample.
(Keeter et al. 2008)
At the 2008 meetings of the American Association
for Public Opinion Research (AAPOR), several
research papers examined the implications of cell
phone popularity. Overall, most of the researchers
found that, for most purposes, ignoring those
with only cell phones does not seriously bias
survey results, because these customers represent a
relatively small portion of all telephone customers.
However, virtually all of the researchers concluded
by saying that this situation was likely to change in
the years ahead. The role of cell phones is clearly a
development that social researchers will continue
to examine and deal with.
Telephone surveys offer many advantages that
underlie the popularity of this method. Probably
Telephone Surveys ■ 281
sociodemographic characteristics of owners will
change.” This fact made it likely that “different
behavior patterns associated with the utilization of
the answering machine” could emerge (1991: 216).
More-recent research has shown that several
factors, including answering machines, have
reduced response rates in telephone surveys. Peter
Tuckel and Harry O’Neill (2002) and others have
examined the impact of such factors as Caller ID,
answering machines, telemarketing, and phone
lines being tied up by faxes and Internet access.
All these constitute difficulties modern survey
researchers must deal with.
Computer-Assisted Telephone
Interviewing (CATI)
In Chapter 14, we’ll see some of the ways computers
have influenced the conduct of social research—particularly
data processing and analysis.
Computers are also changing the nature of telephone
interviewing. One innovation is computerassisted
telephone interviewing (CATI). This
method is increasingly used by academic, government,
and commercial survey researchers. Though
there are variations in practice, here’s what CATI
can look like.
Imagine an interviewer wearing a telephone
headset, sitting in front of a computer terminal
and its video screen. The central computer selects
a telephone number at random and dials it. (Recall
that random-digit dialing avoids the problem of unlisted
telephone numbers.) On the video screen is
an introduction (“Hello, my name is . . .”) and the
first question to be asked (“Could you tell me how
many people live at this address?”).
When the respondent answers the phone, the
interviewer says hello, introduces the study, and
asks the first question displayed on the screen.
work at night to contact residents in many
households. In some cases, this necessitates
providing protection for interviewers working
in areas of a city in which a definite threat to
the safety of individuals exists.
Concerns for safety, thus, work in two ways to
hamper face-to-face interviews. Potential respondents
may refuse to be interviewed, fearing
the stranger-interviewer. And the interviewers
themselves may incur some risks. All this is made
even worse by the possibility of the researchers being
sued for huge sums if anything goes wrong.
Telephone interviewing presents its own
problems, however. As I’ve already mentioned,
the method is hampered by the proliferation of
bogus “surveys” that are actually sales campaigns
disguised as research. If you have any questions
about any such call you receive, by the way, ask the
interviewer directly whether you’ve been selected
for a survey only or if a sales “opportunity” is involved.
It’s also a good idea, if you have any doubts,
to get the interviewer’s name, phone number, and
company. Hang up if the caller refuses to provide
any of these.
For the researcher, the ease with which people
can hang up is another shortcoming of telephone
surveys. Once you’ve been let inside someone’s
home for an interview, the respondent is unlikely
to order you out of the house in midinterview. It’s
much easier to terminate a telephone interview
abruptly, saying something like, “Whoops! Someone’s
at the door. I gotta go.” or “Omigod! The pigs
are eating my Volvo!” (That sort of thing is much
harder to fake when the interviewer is sitting in
your living room.)
Another potential problem for telephone interviewing
is the prevalence of answering machines or
voicemail. A study conducted by Walker Research
(1988) found that half of the owners of answering
machines acknowledged using their machines to
“screen” calls at least some of the time. Research by
Peter Tuckel and Barry Feinberg (1991), however,
showed that answering machines had not yet
had a significant effect on the ability of telephone
researchers to contact prospective respondents.
Nevertheless, the researchers concluded that as
answering machines continue to proliferate, “the
computer-assisted telephone interviewing
(CATI) A data-collection technique in which a
telephone-survey questionnaire is stored in a computer,
permitting the interviewer to read the questions
from the monitor and enter the answers on the
computer keyboard.
282 ■ Chapter 9: Survey Research
It’s also possible to set up questionnaires in
personal data assistants (PDAs) for use by an interviewer
or for direct data entry by respondents.
Some of these systems include the possibility of
voice capture, as described in the box.
Response Rates
in Interview Surveys
Earlier in this chapter we looked at the issue of response
rates in mail surveys, and this is an equally
important issue for interview surveys. In Chapter
7, when we discussed formulas for calculating sampling
error to determine the accuracy of survey estimates,
the implicit assumption was that everyone
selected in a sample would participate—which is
When the respondent answers the question, the
interviewer types that answer into the computer
terminal—either the verbatim response to an
open-ended question or the code category for the
appropriate answer to a closed-ended question.
The answer is immediately stored in the computer.
The second question appears on the video screen, is
asked, and the answer is entered into the computer.
Thus, the interview continues.
In addition to the obvious advantages in terms
of data collection, CATI automatically prepares
the data for analysis; in fact, the researcher can
begin analyzing the data before the interviewing
is complete, thereby gaining an advanced view of
how the analysis will turn out. Sill another innovation
that computer technology makes possible is
described in “Voice Capture.”
Voice Capture
JamesE.Dannemiller
SMSResearch,Honolulu
Thedevelopmentof various CATI techniques hasbeen aboon to
survey and marketing research,thoughmostly it has supported
the collection,coding,andanalysis of“dataasusual.”TheVoice Capture
technique developed by Survey Systems,however,offersquiteunusual
possibilities,which weareonly beginningtoexplore.
In the course ofa CATI-based telephone interview,the interviewer
cantriggerthecomputer tobegindigitallyrecordingtheconversation
with the respondent.Having determinedthattherespondent has
recently changed his orherfavoriteTVnews show,forexample,the
interviewercanask,“Why did youchange?” and begin recordingthe
verbatim response.(Early in the interview,theinterviewer has asked
permissionto record parts of the interview.)
Later on, coders can play backtheresponsesandcode them—
much asthey would dowith the interviewer’stypescriptof the
responses. This offers an easierandmoreaccurateway of accomplishing
a conventional task. But that’s a tame useof the newcapability.
It’salsopossible toincorporatesuch oraldataas parts of a crosstabulation
during analysis. Wemay createa tableofgenderby age by
reasonsforswitching TV newsshows. Thus,we can hear, inturn,the
responses of the youngmen, young women, middle-aged men, andso
forth. Inone such study wefoundthe younger and oldermen tending
to watch one TV news show, while the middle-agedmen watched
somethingelse.Listeningto the responses of themiddle-aged men, one
after another,we heard acommon comment: “Well, nowthat I’molder
. . .”This kind of asidemight havebeen lost inthenoteshastilytyped
by interviewers,but such commentsstood out dramaticallyin the oral
data. The middle-agedmen seemed to betellingusthey felt“maturity”
required them to watch aparticular show,whilemore yearsundertheir
belts letthem driftback towhat they liked in thefirstplace.
Thesekinds ofdata are especially compellingtoclients,particularly
in customersatisfaction studies. Rather thansummarizewhat we feel
a client’s customers like and don’tlike, wecan lettherespondents
speak directly to the clientin theirown words. It’s like afocus group
ondemand.Goingonestep further, we havefound thatletting line
employees (bank tellers,for example)listen to the responses hasmore
impactthanhavingtheirsupervisors tellthemwhattheyaredoingright
orwrong.
Asexcitingasthese experiences are,I have the strong feeling that
wehave scarcely begun totap into the possibilities for such unconventional
forms of data.
Online Surveys ■ 283
evidenced by a fax entitled “Should Hand Guns
Be Outlawed?” Two fax numbers were provided
for expressing either a “Yes” or “No” opinion. The
smaller print noted, “Calls to these numbers cost
$2.95 per minute, a small price for greater democracy.
Calls take approx. 1 or 2 minutes.” You can
imagine where the $2.95 went.
Online Surveys
An increasingly popular method of survey research
involves the use of the Internet and the
World Wide Web—two of the most far-reaching
developments of the late twentieth century. Some
researchers feel that the Internet can be used to
conduct meaningful survey research, and this
technique has been getting especially popular in
marketing research, for example. Some online surveys
are conducted completely via e-mail; others
are conducted via websites. Commonly, potential
respondents will receive an e-mail asking them to
go to a web link where the survey resides.
Mick Couper (2001: 464) provides an excellent
overview of the issues concerning the present and
prospective state of online surveys.
The rapid development of surveys on the World
Wide Web (WWW) is leading some to argue
that soon Internet (and, in particular, Web)
surveys will replace traditional methods of survey
data collection. Others are urging caution
or even voicing skepticism about the future
role Web surveys will play. Clearly, we stand at
the threshold of a new era for survey research,
but how this will play out is not yet clear.
As we’ve seen, one immediate objection that
many social researchers make to online surveys
concerns representativeness: Will the people who
can be surveyed online be representative of meaningful
populations, such as all U.S. adults, all voters,
and so on? This is the criticism raised with regard
to surveys via fax or by telephone interviewers.
Camilo Wilson (1999), founder of Cogix (see
the link on this book’s website), points out that
some populations are ideally suited to online
surveys: specifically, those who visit a particular
almost never the case. Lacking perfection, researchers
must maximize participation by those selected.
Although interview surveys tend to produce higher
response rates than mail surveys do, interview success
has recently declined.
By analyzing response-rate trends in the
University of Michigan’s Survey of Consumer
Attitudes, Richard Curtin, Stanley Presser, and
Eleanor Singer (2005) have sketched a pattern of
general decline over recent years. Between 1979
and 1996, the response rate in this telephone survey
dropped from 72 to 60 percent, representing an
average annual decline of three-quarters of a percent.
Since 1996, the rate of decline has doubled.
The increased non-responses reflected both refusals
and those they were unable to contact.
By contrast, the General Social Survey, using
personal interviews, experienced response rates
between 73.5 and 82.4 percent in the years from
1975 to 1998. In the 2000 and 2002 surveys,
however, the GSS completion rate was 70 percent.
Their decline came primarily from refusals rather
than being unable to contact respondents, because
household interviews produce higher rates of contact
than telephone surveys do.
In recent years, both household and telephone
surveys have experienced a decline in response
rates. A special issue of the Public Opinion Quarterly
(2006) was devoted entirely to analyzing the many
dimensions of the decline in response rates in
household surveys. As the analyses show, lower response
rates do not necessarily produce inaccurate
estimates of the population being studied, but the
variations on this issue defy a simple summary.
Many researchers believe that thewidespread
growth of telemarketing has been abig part of the
problems experienced by legitimate telephone surveys,
and there are hopes that the state and national
“do not call” lists may ease that problem. Further,
we saw that other factors suchas answering machines
and voicemail also contribute to these problems
(Tuckel and O’Neill 2002). Response rate is likely to
remain anissue of high concern in survey research.
As a consumer of social research, you should
be wary of “surveys” whose apparent purpose
is to raise money for the sponsor. This practice
has already invaded the realm of “fax surveys,”
284 ■ Chapter 9: Survey Research
Harris Interactive has demonstrated success in
predicting election results.
Many of the cautions urged in relation to
online surveys today are similar to those urged in
relation to telephone surveys in the first edition
of this book, in 1975. Mick Couper (2001: 466)
makes a similar observation:
Several years ago, I predicted that the rapid
spread of electronic data collection methods
such as the Internet would produce a bifurcation
in the survey industry between highquality
surveys based on probability samples
and using traditional data collection methods,
on the one hand, and surveys focused more
on low cost and rapid turnaround than on representativeness
and accuracy on the other. In
hindsight, I was wrong, and I underestimated
the impact of the Web on the survey industry.
It has become much more of a fragmentation
than a bifurcation (in terms of Web surveys at
least), with vendors trying to find or create a
niche for their particular approach or product.
No longer is it just “quick and dirty” in one
corner and “expensive but high quality” in
the other; rather, there is a wide array of approaches
representing varying levels of quality
and cost.
In the meantime, researchers are amassing a
body of experience with this new technique, yielding
lessons for increasing success. For example,
Survey Sampling, Inc., suggests the following dos
and don’ts for conducting online surveys:
Do use consistent wording between the invitation
and the survey. Don’t use terms such as
“unique ID number” in the invitation, then ask
respondents to type their “password” when
they get to the survey. Changing terminology
can be confusing.
Do use plain, simple language.
Don’t force the respondent to scroll down the
screen for the URL for the study location.
Do offer to share selected results from the
study with everyone who completes the survey.
Respondents will often welcome information as
website. For example, Wilson indicates that market
research for online companies should be conducted
online, and his firm has developed software called
ViewsFlash for precisely that purpose. Although
website surveys could easily collect data from all
who visit a particular site, Wilson suggests that
survey-sampling techniques can provide sufficient
consumer data without irritating thousands or millions
of potential customers.
But how about general population surveys?
How about political polling? These are probably
the main issues raised regarding online surveys
today. Not everyone of interest can be reached via
Internet nor feels comfortable using it for participation
in surveys. Moreover, people who are less
available to online surveys do not represent a random
segment of the overall population. The poor
and the elderly, for example, are likely to be underrepresented
in online surveys. At the same time, as
more and more people gain access to the Internet,
this problem will decline. (An early criticism of
telephone surveys was that not everyone had a
phone.)
In one solution to this problem, the National
Opinion Research Center, which conducts the periodic
General Social Survey (GSS), used probability
sampling methods to create a representative sample
of potential respondents (T. Smith 2001). Each
person in the sample was provided with WebTV
access to the Internet, with an agreement that
they would participate in polls from time to time.
While these online respondents were demographically
representative, there were differences in their
responses on survey issues that will require further
study. For example, the online respondents were
more likely to choose extreme responses (such as
“strongly agree”) than those surveyed in face-toface
interviews were.
Commercial research firms, such as Harris
Interactive and Knowledge Networks (see the
links on this book’s website) report they have
developed large-scale panels of online respondents
from whom they are able to select samples that
are representative of whatever populations are of
interest for study. Because their specific methods
are proprietary, assessing their methodological
strengths and weaknesses is difficult. However,
Comparison of the Different Survey Methods ■ 285
The Gallup Organization
SMS Research
The Survey/Marketing Research e-Store
Zogby International
Online surveys appear to have response rates
approximately comparable to mail surveys, according
to a large-scale study of Michigan State
University students (Kaplowitz, Hadlock, and
Levine 2004), especially when the online survey
is accompanied by a postcard reminder encouraging
respondents to participate. While producing a
comparable response rate, the online survey costs
substantially less than a conventional mail survey.
The cost of paper, printing, and postage alone can
constitute a large expense.
In another study of ways to improve response
rates in online surveys, Stephen Porter
and Michael Whitcomb (2003) found that some
of the techniques effective in mail surveys, such
as personalizing the appeal or varying the apparent
status of the researcher, had little or no impact
in the new medium. At the same time, specifying
that the respondents had been specially selected
for the survey and setting a deadline for participation
did increase response rates. The years ahead
will see many experiments aimed at improving the
effectiveness of online surveys. See “How to Do It:
Conducting an Online Survey” for more.
During the 2004 election campaign, the role
and nature of polls—in-person, telephone, and
online—drew considerable public attention. Mark
Blumenthal (2005) reviewed the public discussion
and considered the implications for the future of
polling. Clearly, this discussion will not end any
time soon.
Comparison of the Different
Survey Methods
Now that we’ve seen several ways to collect survey
data, let’s take a moment to compare them directly.
Self-administered questionnaires are generally
cheaper and quicker than face-to-face interview
surveys. These considerations are likely to be
a reward for taking the study, especially when
they are young adults and teens.
Do plan the time of day and day of week to
mail, depending on the subject of the study
and type of respondent. Send the invitation
late afternoon, evening, or weekend, when
respondents are most likely to be reading
mail at home, especially if the study requests
respondents to check an item in the kitchen
or other area in the home. If a parent-child
questionnaire is planned, send the invitation
late afternoon when children are home,
not early in the day, when respondents can’t
complete the study because children are at
school.
Do be aware of technical limitations. For
example, WebTV users currently cannot access
surveys using Java. If respondents’ systems
need to be Java-enabled or require access to
streaming video, alert panelists at the beginning
of the study, not midway through.
Do test incentives, rewards, and prize drawings
to determine the optimal offer for best
response. Longer surveys usually require larger
incentives.
Do limit studies to 15 minutes or less.*
Over the years, members of industrialized nations
have become familiar with the format and
process of self-administered questionnaires, but
the web presents a new challenge for many. Leah
Christian, Don Dillman, and Jolene Smyth (2007)
provide a wealth of guidance on the formatting
of web surveys. Their aim is, as their article title
suggests, “helping respondents get it right the first
time.”
The web is already seeing extensive use as a
marketplace for surveys and other research techniques.
As only a few illustrative examples, see the
following links on this book’s website: http://www
.cengage.com/sociology/babbie:
*Source: http://www.worldopinion.com/the_frame/
frame4.html. Reprinted with permission.
286 ■ Chapter 9: Survey Research
With interviews, you can conduct a survey
based on a sample of addresses or phone numbers
rather than on names. An interviewer can arrive
at an assigned address or call the assigned number,
introduce the survey, and even—following
instructions—choose the appropriate person at
that address to respond to the survey. In contrast,
self-administered questionnaires addressed to “occupant”
receive a notoriously low response.
Finally, as we’ve seen, interviewers questioning
respondents face-to-face can make important observations
aside from responses to questions asked
in the interview. In a household interview, they
may note the characteristics of the neighborhood,
the dwelling unit, and so forth. They can also note
characteristics of the respondents or the quality of
their interaction with the respondents—whether
the respondent had difficulty communicating, was
hostile, seemed to be lying, and so on.
The chief advantages of telephone surveys over
those conducted face-to-face center primarily on
time and money. Telephone interviews are much
cheaper and can be mounted and executed quickly.
Also, interviewers are safer when interviewing
people living in high-crime areas. Moreover, the
impact of the interviewers on responses is somewhat
lessened when the respondents can’t see
them. As only one indicator of the popularity of
telephone interviewing, when Johnny Blair and
his colleagues (1995) compiled a bibliography on
sample designs for telephone interviews, they listed
over 200 items.
Online surveys have many of the strengths
and weaknesses of mail surveys. Once the available
software has been further developed, they will
likely be substantially cheaper. An important weakimportant
for an unfunded student wishing to undertake
a survey for a term paper or thesis. Moreover,
if you use the self-administered mail format,
it costs no more to conduct a national survey than
a local one of the same sample size. In contrast, a
national interview survey (either face-to-face or
by telephone) would cost far more than a local one.
Also, mail surveys typically require a small staff:
You could conduct a reasonable mail survey by
yourself, although you shouldn’t underestimate the
work involved. Further, respondents are sometimes
reluctant to report controversial or deviant attitudes
or behaviors in interviews but are willing to respond
to an anonymous self-administered questionnaire.
Interview surveys also offer many advantages.
For example, they generally produce fewer
incomplete questionnaires. Although respondents
may skip questions in a self-administered questionnaire,
interviewers are trained not to do so. In CATI
surveys, the computer offers a further check on
this. Interview surveys, moreover, have typically
achieved higher completion rates than self-administered
questionnaires have.
Although self-administered questionnaires may
be more effective for sensitive issues, interview surveys
are definitely more effective for complicated
ones. Prime examples include the enumeration
of household members and the determination of
whether a given address corresponds to more than
one housing unit. Although the concept of housing
unit has been refined and standardized by the Census
Bureau and interviewers can be trained to deal
with the concept, it’s extremely difficult to communicate
this idea in a self-administered questionnaire.
This advantage of interview surveys pertains
generally to all complicated contingency questions.
How to Do It: Conducting an Online Survey
If you’reinterestedin conductingan online survey,you can experiment
with a limitedversion of an online program called Survey Monkey,at
nocharge.To get started,gotothe Survey Monkey link onthis book’s
website(http://www.cengage.com/sociology/babbie) and click
“CreateSurvey.”
Theprogram is quite user-friendly with regard todesigning
questionnaire items.Toreach yourintended respondents,youentertheir
e-mail addresses,andtheythen receive an e-mail invitation to visitthe
survey webpage and participate.The freebeginner packagewill also
provideyouwith abasic analysisof the survey results.
Strengths and Weaknesses of Survey Research ■ 287
generally. Earlier chapters have discussed the
ambiguous nature of most concepts: They have no
ultimately real meanings. One person’s religiosity
is quite different from another’s. Although you
must be able to define concepts in those ways most
relevant to your research goals, you may not find
it easy to apply the same definitions uniformly to
all subjects. The survey researcher is bound to this
requirement by having to ask exactly the same
questions of all subjects and having to impute the
same intent to all respondents giving a particular
response.
Survey research also has several weaknesses.
First, the requirement of standardization often
seems to result in the fitting of round pegs into
square holes. Standardized questionnaire items
often represent the least common denominator in
assessing people’s attitudes, orientations, circumstances,
and experiences. By designing questions
that will be at least minimally appropriate to all respondents,
you may miss what is most appropriate
to many respondents. In this sense, surveys often
appear superficial in their coverage of complex
topics. Although this problem can be partly offset
by sophisticated analyses, it is inherent in survey
research.
Similarly, survey research can seldom deal with
the context of social life. Although questionnaires
can provide information in this area, the survey
researcher rarely develops the feel for the total life
situation in which respondents are thinking and
acting that, say, the participant observer can (see
Chapter 10).
In many ways, surveys are inflexible. Studies
involving direct observation can be modified
as field conditions warrant, but surveys typically
require that an initial study design remain unchanged
throughout. As a field researcher, for
example, you can become aware of an important
new variable operating in the phenomenon you’re
studying and begin making careful observations of
it. The survey researcher would probably be unaware
of the new variable’s importance and could
do nothing about it in any event.
Finally, surveys are subject to the artificiality
mentioned earlier in connection with experiments.
Finding out that a person gives conservative
ness, however, lies in the difficulty of assuring that
respondents to an online survey will be representative
of some more general population.
Clearly, each survey method has its place in
social research. Ultimately, you must balance the
advantages and disadvantages of the different
methods in relation to your research needs and
your resources.
Strengths and Weaknesses
of Survey Research
Regardless of the specific method used, surveys—
like other modes of observation in social research—
have special strengths and weaknesses. You should
keep these in mind when determining whether a
survey is appropriate for your research goals.
Surveys are particularly useful in describing
the characteristics of a large population. A carefully
selected probability sample in combination with
a standardized questionnaire offers the possibility
of making refined descriptive assertions about
a student body, a city, a nation, or any other large
population. Surveys determine unemployment
rates, voting intentions, and so forth with uncanny
accuracy. Although the examination of official
documents—such as marriage, birth, or death
records—can provide equal accuracy for a few
topics, no other method of observation can provide
this general capability.
Surveys—especially self-administered ones—
make large samples feasible. Surveys of 2,000 respondents
are not unusual. A large number of cases
is very important for both descriptive and explanatory
analyses, especially wherever several variables
are to be analyzed simultaneously.
In one sense, surveys are flexible. Many questions
can be asked on a given topic, giving you
considerable flexibility in your analyses. Whereas
an experimental design may require you to commit
yourself in advance to a particular operational
definition of a concept, surveys let you develop
operational definitions from actual observations.
Finally, standardized questionnaires have an
important strength in regard to measurement
288 ■ Chapter 9: Survey Research
As with all methods of observation, a full
awareness of the inherent or probable weaknesses
of survey research can partially resolve them in
some cases. Ultimately, though, researchers are on
the safest ground when they can employ several
research methods in studying a given topic.
Secondary Analysis
As a mode of observation, survey research involves
the following steps: (1) questionnaire construction,
(2) sample selection, and (3) data collection,
through either interviewing or self-administered
questionnaires. As you’ve gathered, surveys are
usually major undertakings. It’s not unusual for a
large-scale survey to take several months or even
more than a year to progress from conceptualization
to data in hand. (Smaller-scale surveys can, of
course, be done more quickly.) Through a method
called secondary analysis, however, researchers
can pursue their particular social research interests—analyzing
survey data from, say, a national
sample of 2,000 respondents—while avoiding the
enormous expenditure of time and money such a
survey entails.
Secondary analysis is a form of research in
which the data collected and processed by one
researcher are reanalyzed—often for a different
purpose—by another. Beginning in the 1960s,
survey researchers became aware of the potential
value that lay in archiving survey data for analysis
by scholars who had nothing to do with the
survey design and data collection. Even when one
researcher had conducted a survey and analyzed
the data, those same data could be further analyzed
by others who had slightly different interests. Thus,
if you were interested in the relationship between
political views and attitudes toward gender equality,
you could examine that research question
through the analysis of any data set that happened
to contain questions relating to those two variables.
The initial data archives were very much like
book libraries, with a couple of differences. First,
instead of books, the data archives contained
data sets: first as punched cards, then as magnetic
tapes. Today they’re typically contained on comanswers
in a questionnaire does not necessarily
mean the person is conservative; finding out that a
person gives prejudiced answers in a questionnaire
does not necessarily mean the person is prejudiced.
This shortcoming is especially salient in the realm
of action. Surveys cannot measure social action;
they can only collect self-reports of recalled past action
or of prospective or hypothetical action.
The problem of artificiality has two aspects.
First, the topic of study may not be amenable to
measurement through questionnaires. Second, the
act of studying that topic—an attitude, for example—may
affect it. A survey respondent may have
given no thought to whether the governor should
be impeached until asked for his or her opinion by
an interviewer. He or she may, at that point, form
an opinion on the matter.
Survey research is generally weak on validity
and strong on reliability. In comparison with field
research, for example, the artificiality of the survey
format puts a strain on validity. As an illustration,
people’s opinions on issues seldom take the form
of strongly agreeing, agreeing, disagreeing, or
strongly disagreeing with a specific statement. Their
survey responses in such cases must be regarded
as approximate indicators of what the researchers
had in mind when they framed the questions. This
comment, however, needs to be held in the context
of earlier discussions of the ambiguity of validity
itself. To say something is a valid or an invalid measure
assumes the existence of a “real” definition of
what’s being measured, and many scholars now
reject that assumption.
Reliability is a clearer matter. By presenting
all subjects with a standardized stimulus, survey
research goes a long way toward eliminating unreliability
in observations made by the researcher.
Moreover, careful wording of the questions can also
significantly reduce the subject’s own unreliability.
secondary analysis A form of research in which
the data collected and processed by one researcher
are reanalyzed—often for a different purpose—by
another. This is especially appropriate in the case of
survey data. Data archives are repositories or libraries
for the storage and distribution of data for secondary
analysis.
Secondary Analysis ■ 289
might get away with an offhand, unsubstantiated
assertion, imagine how much more powerful your
paper would be with the following addition.
1. Access the General Social Survey data set at the
National Opinion Research Center, at the link
on this book’s website.
2. Follow the instructions by clicking successively,
in the left column,
a. The plus sign beside “GSS”
b. The plus sign beside “General Social Surveys,
1972–2006”
c. The plus sign beside “Variable Description”
d. The plus sign beside “Subject Index”
Your screen should now contain the excerpt shown
in Figure 9-7.
In the header to the right of this excerpt on
your screen, you’ll see three tabs: DESCRIPTION,
TABULATION, AND ANALYSIS. You’ll see that
DESCRIPTION has been selected. To begin creating
puter disks, CD-ROMs, or online servers. Second,
whereas you’re expected to return books to a conventional
library, you can keep the data obtained
from a data archive.
The best-known current example of a resource
for secondary analysis is the General Social Survey
(GSS). The National Opinion Research Center
(NORC) at the University of Chicago conducts this
major national survey, currently every other year,
to collect data on a large number of social science
variables. These surveys are conducted precisely for
the purpose of making data available to scholars at
little or no cost and are supported by a combination
of private and government funding. Recall that
the GSS was created by James A. Davis in 1972;
it is currently directed by Davis, Tom W. Smith,
and Peter V. Marsden. Their considerable ongoing
efforts make an unusual contribution to social
science research and to education in social science.
You can learn more about the GSS at the link on
this book’s website: http://www.cengage.com/
sociology/babbie.
Numerous other resources are available for
identifying and acquiring survey data for secondary
analysis. The Roper Center for Public Opinion
Research at the University of Connecticut is one
excellent resource. The center also publishes the
journal Public Perspective, on public opinion polling.
Polling the Nations is an online repository for
thousands of polls conducted in the United States
and 70 other nations. A paid subscription allows
users to obtain specific data results from studies
they specify, rather than obtaining whole studies.
Although the cost of the subscription may be too
steep for the average student, you might check to
see if your school’s library has subscribed. (See the
links for the Roper Center and Polling the Nations
on this book’s website.)
Because secondary analysis has typically
involved obtaining a data set and undertaking an
extensive analysis, I would like you to consider
another approach as well. Often you can do limited
analyses by investing just a little time. Let’s say
you’re writing a term paper about the impact of
religion in contemporary American life. You want
to comment on the role of the Roman Catholic
church in the debate over abortion. Although you
FIGURE 9-7
Subject Index for the General Social Survey
290 ■ Chapter 9: Survey Research
the analysis for your term paper, click the TABULATION
tab. A portion of your screen will now look
like Figure 9-8.
Let’s select “Religious Preference” as the column
variable in our table. You’ll have to explore a
bit, beginning with the plus sign beside the letter
R in the Subject Index. Once you’ve opened that
letter, open the entry for “Religion.” Then open
“R’s Religious Preference.” Finally, open “Respondent
(current).” I know these are a lot of steps,
but it demonstrates the wealth of data available for
analysis. Once you get more familiar with the data
set and the program, you’ll move around deftly.
At this point, you’ll find several options under
“Respondent (current).” We’re interested in the
first of these: RS RELIGIOUS PREFERENCE. Notice
that instead of a plus sign, this notation is preceded
by a right-pointing arrow. Click on the arrow, and
you should see the display shown in Figure 9-9.
Click on the phrase “Add to column.” Your
table is halfway complete. To add attitudes toward
abortion, repeat the process just outlined but select
the following:
A
Abortion
In case of . . .
any reason
This will show the GSS questionnaire items
that asked respondents whether a woman should
have the right to a legal abortion for any reason.
Opening the arrow next to ABORTION IF
WOMAN WANTS FOR ANY REASON should give
you the display shown in Figure 9-10.
Click on the phrase “add to row” and you’ll be
presented with a two-variable table, a portion of
which is shown in Figure 9-11. (I’ve limited the
display to those religious preferences with more
than a few responses.) In reviewing these analytic
results, you should realize that they represent the
cumulation of surveys done between 1972 and
2006. If you want to limit your analysis to a single
year, instructions on the website will tell you how
to do that.
The results of this analysis may surprise you.
Whereas Catholics are less supportive of abortion
FIGURE 9-8
Instructions for Creating a Table
Secondary Analysis ■ 291
(35.4 percent) than Jews (77.8 percent) and those
with no religion (66.3 percent) are, they do not
differ substantially from Protestants (36.0 percent).
Imagine a term paper that says, “Whereas
the Roman Catholic Church has taken a strong,
official position on abortion, many Catholics do
not necessarily agree, as shown in Table . . .“ Moreover,
this might be just the beginning of an analysis
that looks a bit more deeply into the matter, as
described in Chapter 14, on quantitative analysis.
The key advantage of secondary analysis is that
it’s cheaper and faster than doing original surveys,
and, depending on who did the original survey,
you may benefit from the work of topflight professionals.
The ease of secondary analysis has also
enhanced the possibility of meta-analysis, in which
a researcher brings together a body of past research
on a particular topic. To gain confidence in your
understanding of the relationship between religion
and abortion, for example, you could go beyond
the GSS to analyze similar data collected in dozens
or even hundreds of other studies.
There are disadvantages inherent in secondary
analysis, however. The key problem involves the
recurrent question of validity. When one researcher
collects data for one particular purpose, you have
no assurance that those data will be appropriate
for your research interests. Typically, you’ll find
that the original researcher asked a question that
“comes close” to measuring what you’re interested
in, but you’ll wish the question had been asked
just a little differently—or that another, related
question had also been asked. For example, you
may want to study how religious various people
are and the survey data available to you only asked
about attendance at worship services. Your quandary,
then, is whether the question that was asked
provides a valid measure of the variable you want
to analyze. Nevertheless, secondary analysis can
be immensely useful. Moreover, it illustrates once
again the range of possibilities available in finding
the answers to questions about social life. Although
no single method unlocks all puzzles, there is no
limit to the ways you can find out about things.
And when you zero in on an issue from several
independent directions, you gain that much more
expertise.
I’ve discussed secondary analysis in this chapter
on survey research because it’s the type of analysis
most associated with the technique. However,
there is no reason that the reanalysis of social
FIGURE 9-9
Selecting RS RELIGIOUS PREFERENCE as the Column Variable
FIGURE 9-10
Selecting ABORTION IF WOMAN WANTS FOR ANY REASON
as the Row Variable
FIGURE 9-11
The Relationship between Religion and Attitudes toward
Abortion
292 ■ Chapter 9: Survey Research
• Questionnaires provide a method of collecting
data by (1) asking people questions or (2) asking
them to agree or disagree with statements representing
different points of view. Questions may
be open-ended (respondents supply their own
answers) or closed-ended (they select from a list
of provided answers).
Guidelines for Asking Questions
• Items in a questionnaire should follow several
guidelines: (1) The form of the items should be
appropriate to the project; (2) the items must be
clear and precise; (3) the items should ask only
about one thing (that is, double-barreled questions
should be avoided); (4) respondents must be
competent to answer the item; (5) respondents
must be willing to answer the item; (6) questions
should be relevant to the respondent; (7) items
should ordinarily be short; (8) negative terms
should be avoided so as not to confuse respondents;
(9) the items should be worded to avoid
biasing responses.
Questionnaire Construction
• The format of a questionnaire can influence the
quality of data collected.
• A clear format for contingency questions is necessary
to ensure that the respondents answer all the
questions intended for them.
• The matrix question is an efficient format for presenting
several items sharing the same response
categories.
• The order of items in a questionnaire can
influence the responses given.
• Clear instructions are important for getting appropriate
responses in a questionnaire.
• Questionnaires should be pretested before being
administered to the study sample.
• Questionnaires are usually administered in one of
three main ways: through self-administered questionnaires,
face-to-face interviews, or telephone
surveys. Researchers are exploring online surveys
as well.
Self-Administered Questionnaires
• It’s generally advisable to plan follow-up mailings
in the case of self-administered questionnaires,
sending new questionnaires to those respondents
who fail to respond to the initial appeal. Properly
monitoring questionnaire returns will provide
a good guide to when a follow-up mailing is
appropriate.
research data needs to be limited to those collected
in surveys. Nigel Fielding (2004), for example, has
examined the possibilities for the archiving and
reanalysis of qualitative data as well.
Ethics and Survey Research
Survey research almost always involves a request
that peopleprovide us with information about the
mselves that is not readily available. Sometimes, we
ask for information (about attitudes and behaviors,
for example) that would be embarrassing to the
respondents if that information became publicly
known. Insome cases, such revelations could result
in the loss of a job or a marriage. Hence, maintaining
the norm of confidentiality, mentioned earlier
in the book, is particularly important in survey
research.
Another ethical concern relates to the possibility
of psychological injury to respondents. Even if
the information they provide is kept confidential,
simply forcing them to think about some matters
can be upsetting. Imagine asking people for their
attitudes toward suicide when one of them has recently
experienced the suicide of a family member
or close friend. Or asking people to report on their
attitudes about different racial groups, which may
cause them to reflect on whether they might be
racists or at least appear as such to the interviewers.
The possibilities for harming survey respondents
are endless. While this fact should not prevent you
from doing surveys, it should increase your considered
efforts to avoid the problem wherever possible.
MAIN POINTS
Introduction
• Survey research, a popular social research
method, is the administration of questionnaires
to a sample of respondents selected from some
population.
Topics Appropriate for Survey Research
• Survey research is especially appropriate for making
descriptive studies of large populations; survey
data may be used for explanatory purposes as well.