See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/267938670
Question Appraisal System QAS-99 By
Article · January 1999
CITATIONS
76
READS
5,709
2 authors, including:
Some of the authors of this publication are also working on these related projects:
Tobacco Surveillance View project
Tobacco Surveillance and Research View project
Gordon B Willis
National Cancer Institute (USA)
143 PUBLICATIONS   8,221 CITATIONS   
SEE PROFILE
All content following this page was uploaded by Gordon B Willis on 15 January 2015.
The user has requested enhancement of the downloaded file.
Question Appraisal System
QAS-99
By:
Gordon B.Willis and Judith T. Lessler
Research Triangle Institute
Suite 420
6110 Executive Blvd.
Rockville, MD 20852
August, 1999
Questionnaire Appraisal System
QAS-99:
Research Triangle Institute-ii-
Acknowledgments
This version of the RTI Questionnaire Appraisal System (QAS-99) is based on a system
that we developed for Behavioral Surveillance Branch of the Centers for Disease Control and
Prevention (Contract No.: 200-98-0103 TO#3) for use in evaluating questions for the Behavioral
Risk Factor Surveillance System (BRFSS). The BRFSS-QAS can be obtained by contacting:
Centers for Disease Control and Prevention
National Center for Chronic Disease Prevention and Health Promotion
Division of Adult and Community Health
Behavioral Surveillance Branch
4770 Buford Highway, NE, MS K47
Atlanta GA 30341-3717
770.488.2455
http://www.cdc.gov/nccdphp/brfss
Questionnaire Appraisal System
QAS-99:
Research Triangle Institute-iiiTable
of Contents
Section Page
Section 1. Background and Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
Overview of the QAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-1
Learning to Use the QAS-99 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-2
Applying the QAS-99 in Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1-3
Section 2. QAS-99 Coding Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2-1
Section 3. QAS-99 Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Instructions on Using the Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Purpose of the QAS-99 Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
How to Read the Manual, the First Time Through . . . . . . . . . . . . . . . . . . . . . . 3-1
Two Issues to Consider, Up Front. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-1
Finally, a Note About Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-3
STEP 1 - READING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-5
Problem Type 1a - What to Read . . . . . . . . . . . . . . . . . . . . . . . . 3-5
Problem Type 1b - Missing Information . . . . . . . . . . . . . . . . . . 3-6
Problem Type 1c - How to Read . . . . . . . . . . . . . . . . . . . . . . . . 3-7
STEP 2 - INSTRUCTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-8
Problem Type 2a - Conflicting or Inaccurate Instructions . . . . 3-8
Problem Type 2b - Complicated Instructions . . . . . . . . . . . . . . 3-9
STEP 3 - CLARITY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10
Problem Type 3a - Wording . . . . . . . . . . . . . . . . . . . . . . . . . . 3-10
Problem Type 3b - Technical Terms . . . . . . . . . . . . . . . . . . . . 3-11
Problem Type 3c - Vague . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-12
Problem Type 3d - Reference Periods . . . . . . . . . . . . . . . . . . . 3-16
STEP 4 - ASSUMPTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-17
Problem Type 4a - Inappropriate Assumptions . . . . . . . . . . . . 3-17
Problem Type 4b - Assumes Constant Behavior . . . . . . . . . . . 3-19
Problem Type 4c - Double-Barreled . . . . . . . . . . . . . . . . . . . . 3-20
STEP 5 - KNOWLEDGE/MEMORY . . . . . . . . . . . . . . . . . . . . . . . . . 3-21
Problem Type 5a - Knowledge . . . . . . . . . . . . . . . . . . . . . . . . 3-21
Problem Category 5b - An Attitude . . . . . . . . . . . . . . . . . . . . . 3-23
Problem Type 5c - Recall . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-24
Problem Type 5d - Computation . . . . . . . . . . . . . . . . . . . . . . . 3-25
Questionnaire Appraisal System
QAS-99:
Research Triangle Institute-ivTable
of Contents (continued)
Section Page
STEP 6 - SENSITIVITY/BIAS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-26
Problem Type 6a - Sensitive Content . . . . . . . . . . . . . . . . . . . 3-26
Problem Type 6b - Sensitive Wording . . . . . . . . . . . . . . . . . . 3-26
Problem Type 6c - Socially Acceptable . . . . . . . . . . . . . . . . . 3-27
STEP 7 - RESPONSE CATEGORIES . . . . . . . . . . . . . . . . . . . . . . . . 3-28
Problem Type 7a - Open-Ended Question . . . . . . . . . . . . . . . . 3-28
Problem Type 7b - Mismatch . . . . . . . . . . . . . . . . . . . . . . . . . 3-30
Problem Type 7c - Technical Terms . . . . . . . . . . . . . . . . . . . . 3-32
Problem Type 7d - Vague . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-33
Problem Type 7e - Overlapping . . . . . . . . . . . . . . . . . . . . . . . . 3-34
Problem Type 7f - Missing . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-34
Problem Type 7g - Illogical Order . . . . . . . . . . . . . . . . . . . . . . 3-35
STEP 8 - OTHER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3-36
Section 4. Example Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-1
How to Code Problem Types: Two Detailed Example Questions . . . . . . . . . . . . . . . . . 4-1
Coding Form for Example Question 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-2
Coding Form for Example Question 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4-10
Section 5. Practice Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-1
Four Practice Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-2
Answers to the Four Practice Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-3
1
Lessler, Judith T., and Forsyth, Barbara H. (1996) “A Coding System for Appraising
Questionnaires.” In Schwarz, Norbert, and Seymour Sudman (Eds.), Answering Questions: Methodology
for Determining Cognitive and Communicative Processes in Survey Research. San Francisco: JosseyBass
Publishers.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute1-1
Section 1. Background and Instructions
The questionnaire appraisal system is designed to assist questionnaire designers in
evaluating survey questions, and in finding and fixing problems, before the questions “go into
the field.” We strongly believe that many improvements to questions can be made through the
process of appraisal—the systematic review of a questionnaire, using a system like the QAS-99
as a guide. Our goal is to improve the efficiency of the questionnaire review process by
providing developers with an easy-to-use method for identifying and correcting potential
problems with draft questionnaire items. QAS-99 was based on several previous question
appraisal systems and, in part, from a method developed to examine and classify the cognitive
processes inherent in the question-answering process (Lessler and Forsyth, 1996).1
This version is
different from ones that we have used in other studies. In this version, we have eliminated codes
that characterize the nature of the response task, that focus on mnemonic and judgment
processes, and have, instead, focused on question characteristics that are likely to present
problems when administered in the field.
Overview of the QAS
The QAS guides users through a systematic appraisal of survey questions and helps them
to spot potential problems in the wording or structure of the questions that may lead to
difficulties in question administration, miscommunication, or other failings. The user examines
proposed questions by considering specific categories of question characteristics in a step-wise
fashion and, at each step, decides whether the question exhibits features that are likely to cause
problems. In completing the appraisal, the user indicates whether the problem is present by
circling YES or NO on an accompanying coding form (see Section 2) and, for each YES circled,
notes the reason a YES code was assigned. In brief, the steps are as follows, with a full
discussion of each step given in the QAS-99 Manual (Section 3):
STEP 1: READING: Determine if it is difficult for the interviewers to read the
question uniformly to all respondents.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute1-2
STEP 2: INSTRUCTIONS: Look for problems with any introductions, instructions,
or explanations from the respondent’s point of view.
STEP 3: CLARITY: Identify problems related to communicating the intent or
meaning of the question to the respondent.
STEP 4: ASSUMPTIONS: Determine if there are problems with assumptions made
or the underlying logic.
STEP 5: KNOWLEDGE/MEMORY: Check whether respondents are likely to not
know or have trouble remembering information.
STEP 6: SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and
for bias.
STEP 7: RESPONSE CATEGORIES: Assess the adequacy of the range of responses
to be recorded.
STEP 8: OTHER: Look for problems not identified in Steps 1 - 7.
Learning to Use the QAS-99
You can learn to use the QAS-99 on your own by following these steps:
1. Review the QAS-99 Coding Form (Section 2). Although the way in which to
use this form will not be immediately apparent, it provides a good overview of the
system.
2. Read the QAS Manual (Section 3). The manual contains detailed discussion and
examples of the various problems encountered in survey questions, instructions
on how to “code” these problems on the coding form, and suggestions for
correcting problems. As you read the QAS-99 manual, look at the corresponding
part of the coding form, so you can associate the detailed discussion with the
form. As you begin using the coding form to appraise questions, it will be helpful
to return to the manual until you are throughly familiar with how the codes are
applied.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute1-3
3. Examine the Example Questions (Section 4). Following the manual are two
example questions that have been appraised using the QAS-99, with detailed
explanations of why particular codes were assigned. Review these two question
appraisals in detail before beginning to work with the system.
4. Practice using the QAS-99 by coding the four practice questions in Section 5.
5. Check your work by comparing your codes to those assigned by expert
appraisers. The results from expert coders are given for all of the questions
contained in Section 5. If you find substantial differences between the codes that
you assigned and those of the expert coders, we recommend that you re-read the
QAS-99 manual and attempt to understand why the coders assigned each
particular code.
Applying the QAS-99 in Practice
QAS-99 is a tool for you to use to evaluate and revise draft questions. If one diligently
applies the QAS-99, few questions may escape being assigned a problem code because no
question is perfect for every respondent that will be encountered in a survey. Your goal is to
understand the potential problems and to reduce them as much as possible. Thus, we
recommend that you be strict in assigning codes and err on the side of finding a problem rather
than not finding any. Otherwise, you will miss opportunities to consider more effective
wordings and structures for the questions that you are drafting.
You can use the QAS-99 in several ways:
1. Improving questions: You can use the QAS-99 to guide the revision of your own
questions after you write them. Keep in mind that even after learning the system, and
considering how to avoid various pitfalls, it is still important to apply the QAS-99, and to
do the question coding, rather than just concluding that “now that I know what to avoid,
I’ll write good questions.” Even very expert questionnaire designers benefit from
considering a number of separate aspects of a question—systematically, and one at a
time—and the QAS-99 forces one to do this. So, after you have drafted questions, use
the QAS-99 to identify potential problems with them, then revise the questionnaire to
remove these potential problems.
2. “Flagging” questions for further testing: Because many questions may be
subsequently evaluated through cognitive testing or field pretests, you can use your
Questionnaire Appraisal System
QAS-99
Research Triangle Institute1-4
findings to highlight issues that you believe should be investigated in this subsequent
testing.
3. Stimulating collaborative review: One particularly helpful way to use the system is to
have one or more of your colleagues also evaluate the questions, and then discuss with
you the findings and ways to improve the questions.
Remember to focus on the respondent’s question-answering task when appraising the
questions. In particular, think about the different types of respondents who may be administered
the question:
Think about how people of varying ages and different levels of education,
income, and life experience may react to the questions. For example, if the
question focuses on a particular health condition, remember that it will be
asked of both people with and without the condition. Also, it is important to
keep in mind the general conditions of the survey. If respondents are
answering questions over the telephone, they may not have been expecting a
call, and that there may be distractions in their environment that make it
difficult for them to concentrate on their answers. Respondents may be
reluctant to provide truthful answers to sensitive questions because they are
concerned that the interviewer will judge them or that members of their
household may overhear their responses. They are often in a hurry and may not
have thought about the topics in the way that is demanded by the questionnaire.
Your goal is to develop questions that are robust within these general survey
conditions.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute2-1
Section 2. QAS-99 Coding Form
This section contains a sample QAS-99 coding form. Although the details of how to use
this form are contained in the section that follows (Section 3), take a quick look at this form first
to get an idea of the basic steps and problem types that are involved.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute2-2
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION.
Question number or question here:
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read. YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view.
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations.
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute2-3
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict. YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary. YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute2-4
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question.
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for. YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute2-5
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult. YES NO
7b. MISMATCH between question and response categories. YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations. YES NO
7e. OVERLAPPING response categories. YES NO
7f. MISSING eligible responses in response categories. YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
2
Because this version is derived from the project that was done for the Behavioral Surveillance Branch of
CDC, most of the example questions are health questions that commonly occur in the BRFSS.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-1
Section 3. QAS-99 Manual
Instructions on Using the Manual
Purpose of the QAS-99 Manual. The manual provides detailed directions on how to fill
out the coding form (in Section 2) by following the eight basic steps of the QAS-99. These steps
are arranged in an order that loosely mirrors the question asking-and-answering process. These
steps are global in nature; under each one are a number of specific problems types (labeled 1a,
1b, 1c…). The manual provides detailed directions on how to determine whether each problem
type exists and, therefore, when to code these by circling YES on the QAS-99 coding form.2
How to Read the Manual, the First Time Through. Look at a copy of the coding form
when reading the manual. Pay special attention to the examples that illustrate each problem type.
The manual also contains check marks (U ) that provide additional clarifying tips on
questionnaire design and problem type coding issues. So, by reading through the manual and
making frequent reference to the coding form, you should be able to develop a good working
knowledge of the way the system works. In a later section, you will get to practice applying the
QAS-99.
Two Issues to Consider, Up Front. Once you begin to apply the QAS-99, you may
have two basic reactions:
1. “I can’t tell which problem type to select. It seems like more than one may apply, and I
don’t know which one to choose.”
Although the different problem types under each step are intended to address separate
aspects of a question that may be problematic, there clearly is overlap between these, and this
can make it difficult to identify (to “code”) a unique problem type category in a particular
situation. This reaction, however, should not be a reason for getting bogged down in making
coding decisions. Remember that the ultimate purpose of the QAS-99 is to find problems so that
these may be rectified. Therefore, identification is more important than exact classification.
Once you have identified a problem, do not dwell excessively on figuring out which category fits
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-2
best (several may fit). Think of the QAS-99 as a series of fishing nets—if one net misses,
another one may “make the catch.”
2. “There are problems with every question.”
This reaction is common because there is no such thing as a perfect survey question, and
the QAS-99 is designed to find a lot of different types of potential flaws. So, instead of just
concluding that it is futile to script good questions, it may help to consider a few points that put
the appraisal process in the appropriate context:
! Using a system such as this to critique the question in effect forces the
user to consider many aspects of the question and to search extensively for
problems. Under such a microscope, problems will likely be found
frequently. It may not be possible to fix everything. If the most basic
problems can be identified, and addressed, however, the exercise has been
worthwhile.
! Problems differ in their seriousness. Whereas some may simply make the
question less than ideal for purposes of analysis, others could render the
information obtained completely useless. The appraiser should consider
the ramifications of the problems found, given the intended purposes of
the question. Sometimes we have to “live with” questions that have some
degree of vagueness because we cannot specify terms to the extent that
legal text does. Or, the topic may be sensitive in nature because there is
no way to escape the fact that some risk behaviors involve such behaviors
as sex and drug use. The fact that a question may contain one or more
potential problems does not mean we should absolutely not ask it. Rather,
we should recognize that there is no such thing as a perfect question, and
attempt to develop the best form possible, consider the limitations, and
make a decision regarding the worthiness of administering that question.
! Reasonable people can disagree about which problem type applies, or
whether a problem even exists. A degree of subjectivity is inherent in all
question appraisals. We have no evidence other than our own experience
and judgment to use in deciding whether respondents will know or
remember certain information, whether a question is too long, or whether
the response categories used are meaningful to the majority of the
population surveyed. Such decisions may be further confirmed or refuted
by pretesting. Through question appraisal, we simply do our best to make
a number of these subjective judgments, and to create the best questions
that are possible at this point. The QAS-99 is a tool that guides us through
this process.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-3
In summary, remember that the overall purpose of the QAS-99 is not to find ways to
criticize every survey question (or questionnaire designer!), but rather, to help the designer to
find basic potential problems and to fashion the version that will require the least additional
work, in order to contribute to a measurement system that fulfills the research objectives.
Finally, a Note About Terminology. The QAS-99 divides the “survey question” into
two basic parts:
! the question
! the response categories
When we discuss survey questions in the manual, the term question has a specific
meaning and refers to the part that is read to the respondent over the phone. For example,
consider the following:
QUESTION: How useful is the QAS-99? Would you say very useful, somewhat
useful, or not at all useful?
RESPONSE __Very useful
CATEGORIES: __Somewhat useful
__Not at all useful
__I don’t know yet
For purposes of discussion below, we refer to the part “How useful is the QAS-99?
Would you say very useful, somewhat useful, or not at all useful?” as the question. The eligible
answers (__ Very useful ... __ I don’t know yet) are referred to as response categories. Note that
because the response categories are communicated to the respondent in this case, we do consider
them to be part of the question. If the interviewer did not read these to the respondent (i.e., just
“How useful is the QAS-99?”), but instead, they are only listed for the interviewer’s use, then
they would not be considered to be a part of the question and would only be termed as response
categories. This practice allows us to distinguish between problems that pertain to response
categories that are read and those that are not.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-4
The next pages detail all of the QAS-99 steps and problem types.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-5
STEP 1 - READING: Determine if it is difficult for the interviewers to
read the question uniformly to all respondents.
Problem Type 1a - WHAT TO READ: Interviewers may have difficulty
determining what parts of the question are to be read.
! Circle YES on the QAS-99 coding form if …
a) The interviewer must make a decision as to whether or not to read certain parts of the
question to a particular respondent.
OR
b) The question contains phrases in parentheses or instructions labeled READ IF
NECESSARY.
Example In the past month, have you talked to a doctor or other health professional
about your own health? (READ IF NECESSARY - include doctors, nurses,
dentists, psychologists, osteopaths, and chiropractors. Do not include
telephone calls only for the purpose of making appointments).
Comment How will the interviewer know whether it is necessary to read the question’s
additional phrases? If you want all respondents to include or exclude certain
things, then the explanatory information must be read to each respondent.
U Remember that when questions are administered over the telephone, respondents can
respond only to what is read to them.
STEP 1 - READING (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-6
Problem Type 1b - MISSING INFORMATION: Information the
interviewer needs to administer the question is not contained in the question.
! Circle YES for 1b if …
1) The interviewer must remember something about the respondent, consult other
materials, or return to another question to determine if the question should be asked
for that respondent.
OR
2) The question depends on information that may not have been collected previously in
the interview.
Example (Ask if age >60): Would you say that your health in general is excellent, very
good, good, fair, or poor?
Comment The interviewer may not know the respondent’s age at this point in the survey.
Or, given computer-based administration, even if the computer is taking care
of this, we need to at least determine if the appropriate information (here, age)
will have been collected previously.
STEP 1 - READING (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-7
Problem Type 1c - HOW TO READ: Question is not fully scripted and
therefore difficult to read.
! Circle YES for 1c if the wording necessary to read the question is not provided. This
frequently occurs when questions are converted from a self-administered method to
another method, such as telephone administration, in which the questions are to be
read by the interviewer.
Example How many glasses (8 oz) of milk (whole, 2%, or skim milk) did you drink
yesterday?
Comment It is clear what to read (the parenthetical parts are important), but it is unclear
how to read it. The parts in parentheses can easily be read silently, and would
be fine for a self-administered questionnaire. However, if read by an
interviewer, the question needs to be rephrased as “How many 8 ounce
glasses of whole, two-percent, or skim milk did you drink yesterday?”
U Again, it cannot be stressed enough that questions that LOOK perfectly fine when read
silently may produce severe problems when read out loud over the telephone. So a good
check on reading problems is to actually read the question out loud.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-8
STEP 2 - INSTRUCTIONS: Look for problems with any introductions,
instructions, or explanations from the respondent’s point of view.
U Note that a survey question often includes information other than a series of words with a
question mark at the end. Here, we are focusing on such instructions or introductory
statements that are intended to guide the respondent in knowing how to answer.
Problem Type 2a - CONFLICTING OR INACCURATE
INSTRUCTIONS, introductions, or explanations.
! Circle YES for 2a if an instruction, introduction, or explanation contains conflicting
information, or conflicts with the question.
Example 1 The next questions are about the claims process involved with [Health Plan
Name]. How would you rate the range of services covered by [Health Plan
Name]?
Comment Here, “claims process” conflicts with “range of services.” It would be better
to simply open with “The next questions are about [Health Plan Name]”
Example 2 The next questions are about arthritis. Do you have some form of arthritis,
gout, bursitis, tendinitis, or lupus?
Comment The instruction concerns only “arthritis,” whereas the question asks about a
number of items that will not be recognized by most respondents as types of
arthritis. The question would be clearer without the introduction.
Problem Type 2b - COMPLICATED INSTRUCTIONS, introductions, or
explanations.
STEP 2 - INSTRUCTIONS (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-9
! Circle YES for 2b if the question has extensive instructions that may be difficult for
respondents to follow.
Example Behaviors associated with having the AIDS virus include male homosexual
contact, injecting illegal drugs, and sex with multiple partners or a person with
any of these risks. In terms of your own risk, what would you say your
chances are of having the AIDS virus?
Comment Respondents generally find it difficult to remember a set of complicated
instructions. Moreover, attempting to teach respondents about risk within the
question is most likely misguided. If people do not know these things, one
sentence in a questionnaire will be unlikely to inform them. It would
probably be more effective to assess knowledge by asking several questions
about factors that influence HIV/AIDS risk before asking about degree of
AIDS risk.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-10
STEP 3 - CLARITY: Identify problems related to communicating the
intent or meaning of the question to the respondent.
Problem Type 3a - WORDING: The question is lengthy, awkward,
ungrammatical, or contains complicated syntax.
! Circle YES for 3a if the question contains a large amount of text or uses complex syntax,
or the wording could be made more “natural.”
Example 1 Which of the following best describes whether you have a smoke alarm in
your home? You own a smoke alarm, and it is installed and working; you
own a smoke alarm, but it is broken or not installed; you own a smoke alarm
but the battery is missing; you don’t own a smoke alarm because it is too
expensive; you don’t own a smoke alarm because you don’t think it is
necessary; or you don’t own a smoke alarm for some other reason?
Comment The problem is self-evident. A good way to identify questions that are
excessive in length is to read them out loud, especially to another person.
Again, remember especially that telephone administration puts a severe limit
on question length.
Example 2 What are the disabilities or health problems of this person?
Comment It would be better to ask “What disabilities or health problems does this
person have?”
U This problem type is similar to 2b (Complicated Instructions), but is intended to be more
general. Long, convoluted questions should get “picked up” by either of 2b or 3a.
STEP 3 - CLARITY (continued)
3
Here we are concerned with terms that are part of the question, as defined earlier, and are read
to respondents. Technical terms contained in response categories seen only by the interviewer are
addressed in Step 7.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-11
Problem Type 3b - TECHNICAL TERMS are undefined, unclear, or
complex.3
! Circle YES for 3b if the question contains specific technical terms that are undefined or
for which simpler alternatives exist.
Example What kind of doctor treats your diabetes: A general or family practitioner, an
internal medicine doctor, a diabetologist, or someone else?
Comment “Diabetologist” is better conveyed as “a specialist who deals with
diabetes”; “practitioner” may be better communicated as “doctor.”
Examples Technical terms that could be either replaced or explained:
Dilated eye exam
Digital rectal exam
Retinopathy
Extremity
Pneumococcal vaccination
Neurological problems
COPD
DUI
U Here is another situation in which we need to consider the mode of interview. If it is a
telephone interview, remember that complex, multisyllabic words that are passable when
read in print often are difficult to make out, or are misheard, when read over the telephone.
STEP 3 - CLARITY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-12
Problem Type 3c - VAGUE: The question is vague because there are multiple
ways in which to interpret it, or to determine what is to be included and excluded.
U This involes a huge category of problems. Typically, the problem is not with particularly
complex or hard-to-understand terms, but with the entire question; the words each make
sense, but strung together, they ask a question that is vague, as in the following ways:
! Circle YES for 3c if there are two (or more) possible alternative interpretations and it is
unclear which is meant.
Example 1 Have you had your blood tested for the AIDS virus?
Comment It is unclear whether this means “Did I take the initiative in deciding to have my
blood tested?” versus “Was it tested as part of any type of blood test?”
Sometimes respondents will say that “I needed it tested for my job—but I didn’t
go out of my way to have it done.” If the issue of interest is the act of testing,
simply ask “As far as you know, has your blood ever been tested for the AIDS
virus?”
Example 2 Do you think that diabetes can cause problems with your kidneys?
Comment When asked of diabetics, it may not be clear whether this means (a) in general,
or (b) does it cause problems for me. If the former is intended, rephrase: “Do
you think that diabetes can cause kidney problems?”
STEP 3 - CLARITY (continued)
4
Again, we are concerned here with terms that are read to respondents. Terms contained in
response categories seen only by the interviewer are addressed in Step 7.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-13
! Circle YES for 3c if the question contains undefined or unclear common terms4
Many common terms are exceedingly vague when used in survey questions. Even a
“normal” question, in plain English, with no complicated words, can cause serious
interpretation problems.
Example 1 Do you have a car?
Comment You: Who is “you” — me, or anyone in the household?
Have: What does this mean — own? lease?
Car: Does this include pickup truck? Van?
It is because of these problems that survey questions must be constructed more
carefully. An equivalent might be: “Does anyone in your household now own
or lease a car, truck, or other type of motor vehicle?”
Example 2 Are there any guns located in your house?
Comment What type of gun? Does a BB gun count? Where in the house—does the
garage count? What if someone is out hunting, as we speak?
Example 3 Does your local health department provide any public health services?
Comment What is a “public health service”? Will members of the public have any idea
what we mean? The question needs to either provide a definition, or else ask
separately about a range of specific services.
STEP 3 - CLARITY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-14
! Circle YES for 3c if it is unclear what is to be included and excluded from the question.
Example 1 How often do you take part in community organizations, meetings, or other
activities involving Hispanic people or culture?
Comment What is to be included here? If I have a meeting at work with two people who
happen to be Hispanic, does that count? For this question to work as it is
(probably) intended, it would need to ask specifically about “activities that
focus on Hispanic culture or people of Hispanic origin.”
Example 2 In the past 12 months, have you talked to a doctor or other health professional?
Comment Do phone calls to make appointments count? Do visits for children count?
What counts as a “health professional”? Does a visit to an acupuncturist
count? To be thorough, the question might ask about a series of health
professionals. At the least, it can be improved by indicating what to include or
exclude (by adding, for example “Include phone conversations, except if these
were only to make appointments”).
! Circle YES for 3c if it is unclear what “frame of reference” the respondent is supposed to
adopt. This problem is often in evidence if a typical reaction might be “compared to
what?” or “compared to whom?”
Example How much would you say you know about Lyme disease? A lot, some, a little,
or nothing?
Comment This can be difficult to answer because the respondent does not really know to
whom they should compare themselves. Although I may have heard of Lyme
disease, know that it comes from deer ticks, and know some of the symptoms,
that is not a lot compared to, for example, a medical expert. Also, it is
premature to ask a detailed question before establishing that the question is
relevant. It may be better to first ask: “Have you ever heard of Lyme disease?”
STEP 3 - CLARITY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-15
! Circle YES for 3c if there is simply not enough information given to prompt a meaningful
answer.
Example Would you support an increase in cigarette taxes if the additional revenue was
spent on community cancer prevention and control programs?
Comment Respondents who smoke, and who are making a serious attempt to answer the
question, may object that it does not contain sufficient information. If the
increase were a nickel a pack, that may be different than if it were a dollar a
pack. The amount needs to be specified. Often, a question having this type of
problem naturally elicits a response of “it depends...” (on additional information
that is not supplied).
U Distinguishing Problem Types 3b (TECHNICAL TERMS) and 3c (VAGUE): 3b
involves particular technical terms that can be picked out as too complex for the target
population.. If there are no such technical terms, but clarity is difficult because of
general vagueness of relatively common terms, then choose 3c instead of 3b.
STEP 3 - CLARITY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-16
Problem Type 3d - REFERENCE PERIODS are missing, not well specified,
or are in conflict.
For questions that refer to some time in the past, responses are usually more reliable and
precise when they are tied to a well-defined time period.
! Circle YES for 3d when the time period asked about is not clear.
Example 1 Before your last pregnancy, had you stopped using all methods of birth
control?
Comment How long before pregnancy? Do we mean days, months, years, or something
else? Depending on the reference period of interest, the question could instead
start with: “During the 3 months before that pregnancy...”
Example 2 How many glasses of low-fat or skim milk per day do you usually drink?
Comment No reference period is indicated. Is this in the past week, past month, past year,
or some other period of time?
Example 3 How many times have you visited a dentist during the past year?
Comment The term “past year” is commonly misunderstood. This may mean “past
calendar year,” “since January 1,” or “since 365 days ago.” If the latter is
intended, it makes more sense to say “in the past 12 months.” If we are very
concerned about this being exactly 12 months, it also helps to indicate the date
(...that is, “since May 20, 1999”).
Example 4 Have you ever been examined within the past 12 months by a doctor for skin
cancer?
Comment In this case, the term “ever” implies a lifetime reference period. If the desired
reference period is the past 12 months, then “ever” should be removed.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-17
STEP 4 - ASSUMPTIONS: Determine if there are problems with
assumptions made or the underlying logic.
Problem Type 4a - INAPPROPRIATE ASSUMPTIONS are made about the
respondent or his/her living situation.
U These are not primarily problems related to the respondent’s understanding the
question but, rather, relate to the appropriateness of the question for that respondent.
! Circle YES for 4a if the question contains (often subtle) logical problems, such as those in
these examples:
Example 1 How often do you usually select low-salt foods at the grocery store—Most of
the time, sometimes, rarely, or never?
Comment We do not know if the person even goes to the grocery store, so it is hard to
interpret a “never” response. To provide the information desired, it is necessary
to first ask how often the person shops for food, and if it is never, to skip him or
her from the follow-up question.
Example 2 Have you had (medical test X) done?
IF YES:
When you got your most recent test result, was it positive or negative?
Comment We have not established that the person received the results of the test. This
needs to be asked as a separate question (“Did you receive the results of that
test?”), or at least to include a “did not receive the results” response category.
STEP 4 - ASSUMPTIONS (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-18
Example 3 How confident are you in your doctor’s ability to help you with your health
problems?
Comment This sounds simple enough, but even assuming we have established already that
the person has one main doctor who he/she has visited recently, it ignores the
fact that some people may go to that doctor only for screening and general
checkups and may not have any health problems. Again, this would serve best
as a follow-up to an initial question, such as: “In the past 12 months, have you
visited that doctor to get treatment for any health problem, or only for general
care and routine checkups?”
Example 4 Have you ever had an amputation?
IF YES: What extremity was amputated?
__ Toe
__ Foot
__ Leg
__ Arm
__ Finger
Comment It is possible that may have been multiple amputations. If a questionnaire data
coding system is set up to allow only one response, this will cause problems.
STEP 4 - ASSUMPTIONS (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-19
Problem Type 4b - ASSUMES CONSTANT BEHAVIOR: The question
inappropriately assumes a constant pattern of behavior or experience for situations that in
fact vary.
! Circle YES for 4b if the behavior or experience is typically irregular rather than constant
in nature, and the question forces an average or typical value that may be difficult or not
meaningful. If the answer is likely to be “it varies...,” then this problem exists.
Example 1 How often does your arthritis prevent you from doing your usual work or taking
part in social activities? Would you say every day, almost every day, once a
week, occasionally, or never?
Comment Note that flare-ups are a common characteristic of many chronic medical
conditions. Respondents tend to say that there are weeks when they cannot do
these things at all, and then it gets better again. Although this problem has not
been solved by questionnaire designers, it might help to first ask a question
about the regularity of the experience.
Example 2 When you go out in the sun for an hour or more, how often do you wear
sunscreen or protective clothing?
Comment In the middle of summer I may be very likely to, but in February, this is
extremely unlikely. How should I give an overall judgment? Specifying the
season would help somewhat.
U Distinguishing Between Problem Types 4a (Inappropriate Assumptions) and 4b
(Assumes Constant Behavior): Type 4b, like 4a, involves an inappropriate assumption.
The difference is that in 4b the assumption relates not to whether something is true for
the respondent (it may well be, to some extent), but rather, whether it is true in the
“static” sense that is implied. If the problem relates to variation in some attribute or
behavior that the question does not take account of, then select 4b.
STEP 4 - ASSUMPTIONS (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-20
Problem Type 4c - DOUBLE-BARRELED question that contains multiple
implicit questions.
! Circle YES for 4c if there are two or more questions in one.
Example 1 Are you covered by Medicaid or any type of dental benefits plan?
Comment Does this mean a Medicaid dental plan? Or any type of Medicaid? This needs
to be divided into two separate questions.
Example 2 Do you think that ministers, politicians, and other community leaders should
speak out against cigarettes and tobacco?
Comment This one is actually triple-barreled. One could think that politicians and other
community leaders should, but that ministers should not. These could instead
be asked about separately (as in a “stem-based” question):
Do you think each of the following should speak out against cigarettes and
tobacco…
Ministers?
Politicians?
Other community leaders?
U Note that some problems may be fixed, yet others remain. Here, we have solved the
“triple-barreled” problem, but this does nothing to address a basic vagueness that
seems to exist with this question (see Problem Type 3c). Remember to think through
different aspects of the question and to consider that improving them may involve
multiple types of alterations.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-21
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are
likely to not know or have trouble remembering information.
Problem Type 5a - KNOWLEDGE: The respondent is unlikely to know the
answer.
Circle YES for 5a in several situations. First:
! The question is factual in nature, and it is likely that the respondent simply does not have
the knowledge necessary to answer the question, and may never have.
Note that this problem is sometimes caused by asking for an excessive level of detail:
Example 1 What kind of calcium pills do you take? (Mark those mentioned)
calcium carbonate (generic)
Citracal
caltrate
calcium gluconate
Biocal
calcium lactate
Tums
Tums E-X
Oscal 500
vitamin/mineral supplement
other
Comment People who regularly take pills often have no idea of precisely what type of
pills these are. Solutions may involve reducing the level of specificity, or even
re-thinking whether the question can be posed. Logically, if people do not
know the answer, we should not be asking the question.
STEP 5 - KNOWLEDGE/MEMORY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-22
Example 2 Thinking about your most recent mammogram, how much did it cost, regardless
of who paid for it? Include just the cost of the x-ray itself and not any fee
charged by the doctor at the x-ray facility, or the cost for an office visit where
the test was ordered.
Comment Respondents in a health maintenance organization (HMO) who simply make a
co-payment may have no idea how much the mammogram cost. Also, even
those who pay out-of-pocket, or according to a fee-for-service model, may have
no way of disentangling the cost of the test from the cost of the office visit.
Example 3 Have you been examined within the past year by a doctor, nurse, or other health
professional for oral or mouth cancer?
Comment People often are not aware of what a check for oral or mouth cancer consists of,
and especially, whether their dentist has done one.
! Finally, circle YES for 5a if there may be a problem that involve a projection or
hypothetical situation.
Example How would you rate your own chances of getting Lyme disease in the coming
year—high, medium, low, or none?
Comment Respondents are often unable to make such judgments. It may be better to
simply ask about behaviors which we know increase the chances of getting
Lyme disease, or about knowledge concerning how Lyme disease is contracted.
STEP 5 - KNOWLEDGE/MEMORY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-23
Problem Category 5b - An ATTITUDE that is asked about may not exist.
! Circle YES for problem category 5b for attitudinal questions that ask about issues the
respondent may have no opinion on, or given any thought to.
Example 1 Do you think that laws restricting the sale of tobacco products to minors have
been adequately enforced?
Comment This may be very difficult for 65-year-old respondent who does not know any
minors, let alone any who smokes. At the least, a question like this needs to
explicitly allow respondents to indicate that they have no opinion on the issue
(here, by adding: “...or are you not aware of how well these laws have been
enforced?”).
Example 2 Do you think that current regulations for mammography equipment and
personnel are adequate to insure the safety of women?
Comment This is clearly well beyond the range of knowledge that exists in most
population respondent groups.
Example 3 In general, would you say that your State’s Congressmen are doing a good job,
or a bad job?
Comment Respondents will happily answer this question, even in the absence of knowing
who their congressmen are, or anything in particular that they have done. At
the least, if there is the likelihood that people may reasonably be expected not to
have a true opinion, then they should be allowed to express this, as in “... or do
you have no opinion?”
STEP 5 - KNOWLEDGE/MEMORY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-24
Problem Type 5c - RECALL failure.
! Circle YES for 5c when there is a good chance that a significant segment of the
population will no longer remember what we may be interested in asking them.
Even if a respondent had learned the relevant information at some point, he/she may no
longer remember it because the time interval (the reference period) is too long, or the events
are routine or non-memorable. Clearly, this may be a difficult judgment to make because
we cannot always tell what respondents will know. However, these are the types of
questions that illustrate this problem:
Example 1 How many mammograms have you had in the past 10 years?
Comment An exact count for something like this can be difficult. It would be better to
present a series of response category ranges (“Would you say none, one, one to
five, or more than five?”), to make the task easier.
Example 2 During the past 12 months, how many days did illnesses or injury keep you
from working?
Comment It is very difficult to both (a) remember all occurrences in the past 12 months,
and (b) remember to exclude occurrences previous to that. Again, providing
response category ranges would facilitate this cognitive task.
U How will we be able to tell what respondents will recall? In assessing whether
respondents will likely know the answer to the question, a good test is to ask yourself,
colleagues, and family members the question. If they cannot remember an answer,
other respondents are also likely to have problems.
U Distinguishing Between Problem Types 5c (RECALL) and 5a (KNOWLEDGE): How
can we distinguish between the case in which the respondent never knew the answer
(5a) from when he/she did at one time, but simply cannot remember (5c)? Often, this
must be a judgment call, depending on what assumptions we are willing to make about
what people are likely to know or now know. One way to think about this issue is to
STEP 5 - KNOWLEDGE/MEMORY (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-25
consider whether putting a few “cues” or reminders in the question will help or not—
logically, it cannot for 5a problems, but may for those covered by 5c.
Problem Type 5d - COMPUTATION or calculation problem.
! Circle YES for 5d if the question simply presents a task that is complex in terms of
computational difficulty, as this overloads the person’s working memory.
Example In a typical workweek, about what proportion of your time is spent on each of
the following activities?
__ Walking?
__ Sitting?
__ Standing?
Comment Answering this question can be very difficult: The respondent must consider
the total number of hours worked, the total number of hours devoted to each
activity, and then express these as a ratio. Further, without knowing up front
how many items in the list, the respondent will have no idea how to allocate
these estimates so that they will sum to a meaningful total. It would be better to
instead ask about the number of hours per day the person spends on each
activity, and then to compute proportions in the analysis.
U This is similar to 5c (Recall Failure), except that the fundamental problem is not that
the person cannot remember how often he or she engages in each activity. Rather, he
or she is unable to add up the various bits and pieces of information in the way that the
question demands.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-26
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or
wording, and for bias.
Problem Type 6a - SENSITIVE CONTENT: The question is on a topic that
people will generally be uncomfortable talking about.
U Clearly, questions on sexual behavior, drug use, or other illegal activities may produce
bad data, or may simply threaten the rapport developed in the interview between the
interviewer and the respondent. Often, it is necessary to provide an introduction that
explains why a series of sensitive questions needs to be asked. Consider also whether it
makes sense to ask sensitive questions over the telephone, when the interviewer cannot
even be seen.
! Circle YES for 6a if the question asks about sensitive issues or behaviors:
Example How many different people have you had sex with in the past 12 months?
Comment Would a person whose spouse might possibly overhear on another extension be
willing to give a response other than “one”? Such a question may not be
appropriate for a telephone survey, unless special procedures (such as use of
touch-tone keying to enter the response) are used.
Problem Type 6b - SENSITIVE WORDING.
! Given that the question may be generally sensitive (as defined in 6a), Circle YES for 6b
for questions that contain terms that are emotionally loaded, offensive, or that can be
stated more “gently.’”
Example In the past 12 months, have you driven when you were drunk?
Comment The term drunk has negative connotations. It may be better to soften this, as in
“...when you may have had too much to drink.” It might be objected that the
latter is too vague, but drunk is not much better because respondents will not
know whether they were legally intoxicated, or have any other way of better
interpreting the intended concept.
STEP 6 - SENSITIVITY/BIAS (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-27
Problem Type 6c - A SOCIALLY ACCEPTABLE response is implied.
! Circle YES for 6c if the question contains phrasing or response categories that imply that
one answer should be chosen because it is preferable to other alternatives.
Example 1 Did you use condoms as protection to prevent disease?
Comment This suggests only one use (disease prevention), and it is very easy to say “yes,”
even if the real reason was something else (e.g., avoiding pregnancy). If there
is a small list of likely candidate reasons, it is best to read them all: “Did you
use condoms to avoid pregnancy, to prevent disease, or both?”
Example 2 Do you agree that cigarette smoking can cause cancer?
Comment It would be much less biased to rephrase, as in : “For the following statements,
tell me whether you agree, disagree, or have no opinion. (1) Cigarette smoking
can cause cancer.”
Example 3 On an average day, for about how many hours do you watch television? Would
you say none, between 1 and 2 hours, 2 to 4 hours, or more than 4 hours?
Comment A great deal of research has shown that the category ranges provided by
behavior frequency questions such as this tend to guide the respondent’s
choices—changing the nature of these ranges markedly alters response
distributions, when certain responses may be seen as more desirable than others.
So, especially where there may be some social desirability to the behavior, it is
better to ask this simply as an open-ended question and code the response
directly.
U Note that for quantitative questions like this one, open-ended formats tend to work. For
questions with qualitative response categories, open-ended formats can be problematic
(see Step 7, Problem Type 7a).
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-28
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range
of responses to be recorded.
U Problems with response categories will sometimes overlap with problems involving
question intent/clarity covered in Step 3. One way of distinguishing these is to note that
the problems covered in Step 3 involve what is read to respondents, whereas the
problems in Step 7 often involve what is not read.
Problem Type 7a - OPEN-ENDED QUESTION that is inappropriate or
difficult.
! For open-ended questions (those for which the response categories are not read to
respondent) circle YES for 7a when:
Respondents will have trouble providing an unguided response.
OR
The answers given by respondents are likely to be difficult for the interviewer to code.
U There are two general types of “open-ended” questions that may cause difficulty:
Open-ended response with verbatim coding: One case is in which the question asks for
a verbal response that must be recorded in words by the interviewer (e.g., “Why are you
not covered by health insurance?” Here, the problem may be either for the respondent
(who simply does not know what the range of appropriate responses are), or the
interviewer (who may have difficulty entering a long open-ended response). In this
case, it may be better to “close” the question and to provide the respondent with a
number of clear alternative to choose from.
Open-ended response with precoded categories: The second type of open-ended
question that is problematic is the “precoded” question, which asks an open-ended
question of the respondent (again, “Why are you not covered by health insurance?”),
but for which a number of preprinted categories are provided for the interviewer to use,
by selecting the one that is closest to the respondent’s answer (e.g., “became ineligible
because of age or leaving school”). Again, the respondent may have problems
providing an acceptable response. Further, the interviewer’s task is complicated
because of the need to match what the respondent has said with the list of categories,
which may be very long, or may simply not provide an easy match. So, look especially
for cases in which the answers likely to be given will not match “precodes,” where
precodes are numerous.
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-29
Example 1 What prompted you to have your most recent mammogram?
(Mark those mentioned)
__ Doctor recommended
__ Other health professional recommended
__ Part of regular checkup
__ Friend encouraged
__ Awareness of guidelines/need
__ Breast problems
__ Advertisement
__ Other
Comment Respondents will have no way of knowing what the intended range of
acceptable answers are. Interviewers will also have problems coding the types
of answers that respondents tend to give (“Well, it was time to get one done...”;
“I was told to”; “I wanted to see if there was a problem”). It is better to let
respondents know the acceptable responses by reading these, if the list is not too
long. If the list is long, and covers more than one dimension of an issue (such
as who indicated the test should be done versus why it should be done), then this
can be “decomposed” into multiple questions that are more specific. For
example, it might be better to divide this example into two questions.
A) Who recommended that you have your most recent mammogram—a doctor,
another type of health professional, someone else you know, or did you decide
on your own?
B) Did you receive this mammogram as part of a regular checkup, because of a
breast problem, or for some other reason?
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-30
Example 2 People go to the dentist for many reasons. What was the main reason for your
last visit?
__ Emergency
__ Periodic checkup
__ Filling
__ Cleaning
__ Extraction
__ Orthodontics
__ Gum treatment
__ Denture work
__ Crown or bridge
Comment Again, the respondent does not see the available choices. Therefore, he or she
may simply say “checkup.” The interviewer has no way of knowing whether
this is periodic (the respondent may not have been to the dentist in years, so we
cannot assume the regularity of the behavior. At the least, the term periodic
should be deleted.
Problem Type 7b - MISMATCH between question and answer categories.
! Circle YES for 7b if the answers presented or implied by the question itself conflict with
the response categories:
Example 1 Since you’ve lived in your current residence, have any special locks been
installed on the doors of your home?
__ Yes
__ No
__ No opinion
Comment Note that the response categories implied do not match those that are given to
the interviewer to use. This is not, in fact, an opinion question (it is either true
or it is not, or the respondent does not know). So, this question should use
Don’t know instead of No opinion.
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-31
Example 2 Thinking about your last pregnancy—Did you become pregnant sooner that
you wanted, later than you wanted, or at the time you wanted to?
__ Sooner
__ Later
__ At the time
__ Didn’t want pregnancy
Comment The last category is provided to the interviewer in case it is volunteered, but the
respondent does not know this and may get confused and not know to say this.
The question should include the phrase “...or didn’t you want to get pregnant at
all?”
Example 3 When was your diabetes diagnosed?
__ Within the past 12 months
__ Within the past 5 years
__ More than 5 years ago
Comment The respondent does not know that the interviewer will be marking only a wide
range and so may struggle to remember an exact date, when this is totally
unnecessary. It is important to provide, in the question, information about the
level of specificity required in the answer: “When was your diabetes
diagnosed—Was it within the past 12 months, between 1 and 5 years ago, or
more than 5 years ago?”
U It is tempting to class the previous example as a type of memory problem (see Step 5).
However, the root of the problem is not difficulty in the recall process (remembering
whether it was the past 12 month, 5 years, or earlier may actually be very easy), but
rather the failure of the question to make clear what the relevant recall task is, with
respect to the requirements of the response categories.
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-32
Example 4 Do you plan to have a mammogram in the future?
__ Yes - next 12 months
__ Yes - next 24 months
__ No
Comment This is asked as a “yes/no” question, whereas the interviewer is intended to
determine when the respondent next intends to have a mammogram. Clearly,
the latter concept needs to be asked as a follow-up question: IF YES: “When
do you plan to have your next mammogram—in the next 12 months, between 1
and 2 years from now, or more than 2 years from now?”
Problem Type 7c - TECHNICAL TERMS are undefined, unclear, or complex.
! For examples, see Step 3, Problem 3b, “Question contains undefined, unclear, or complex
technical terms.” Note that this problem may have already been “picked up” there if the
response categories are read to the respondent. If not, and the question contains response
categories with undefined or vague technical terms, circle YES for 7c.
Again, here are examples of terms that appear in response categories that are not
typically well-understood by survey respondents:
Dilated eye exam
Digital rectal exam
Retinopathy
Extremity
Pneumococcal vaccination
Neurological problems
COPD
DUI
U Remember that it does not really make a big difference whether you “pick up” this problem
when looking at “the question” (3c) or when looking at the “response categories” (7c), as
long as you note it.
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-33
Problem Type 7d - VAGUE response categories.
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-34
! See Step 3, Problem 3c, Vague, “There are multiple ways to interpret the question.”
Again, note that this problem may have already been detected there, if the response
categories are read by the interviewer and are therefore “part of the question.” For cases
where the response categories are separate from the question, but display that same type
of vagueness, circle YES for 7d.
Example In the past 30 days, how often have you had a headache? [ENTER
RESPONSE GIVEN BY RESPONDENT]
__Never
__Seldom
__Occasionally
__Frequently
Comment Besides not working very well as an open-ended question (see Problem Type
7a), there is another problem: People tend to vary with respect to their
interpretation of vague terms, such as “seldom” and “occasionally.” One
person may consider 3 or 4 headaches as “seldom,” another “occasionally.”
Therefore, it would be better to ask for the quantitative frequency with which
headaches have occurred:
In the past 30 days, how often have you had a headache? Would you say
every day, between 3 and 6 times a week, once or twice a week, or less than
once a week?
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-35
Problem Type 7e - OVERLAPPING response categories.
! Circle YES for 7e when the response categories are NOT mutually exclusive.
Example __ Strongly agree
__ Agree
__ Disagree
__ Strongly disagree
Comment If someone agrees strongly, they also agree. It would be better to add
somewhat to the second and third categories here.
Problem Type 7f - MISSING response categories.
! Circle YES for 7f when there are “gaps” between response categories:
Example 1 __ All of the time
__ Most of the time
__ Seldom
__ Never
Comment Someone who engages in the behavior “sometimes” will have trouble finding
a response category that fits. So, “some of the time” should be added. Again,
it would be better not to use words like “seldom.” “Always, More than half
the time, Half the time, Less than half the time, Never” are good substitutes.
Example 2 Do you consider yourself to be overweight, underweight, or about average?
Comment Some people may not fit any of the given categories. A National Football
League lineman (or a horse race jockey) might consider himself to be none of
these things; he is at a desirable weight, but it is not anywhere near average.
Instead, use “overweight, underweight, or about the right weight?”
STEP 7 - RESPONSE CATEGORIES (continued)
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-36
Problem Type 7g - ILLOGICAL ORDER of response categories.
! Circle YES for 7g when the categories that are read to respondents, or presented for the
interviewer to select, are out of order.
Example Over the past 12 months, about how many sex partners did you have? Would
you say only one, two or three, between four and ten, more than ten, or none?
Comment Because this is an ordered list that increases in magnitude, it sounds strange to
place “none” at the end.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute3-37
STEP 8 - OTHER: Look for problems not identified in Steps 1-7.
QAS-99 Steps 1 to 7 will likely identify the majority of common problems that exist in
survey questions. A careful reviewer, however, may be able to detect additional problems that
sometimes crop up. Although the QAS-99 has no specific problem type codes for these, here are
a couple of things to look for:
U Does the question fulfill the objectives? In particular, consider whether the question makes
sense to ask in the first place. Although a complete discussion of the ways in which survey
questions serve to fulfill data objectives is beyond the scope of this manual, it behooves the
designer to always keep in mind the purpose of asking the question, with respect to the type
of data that will be of value. A few such checks would include the following:
! If a question is likely to be answered the same way by almost all respondents, it will
have no diagnostic value.
! If a question is intended to be part of a ratio measure (such as a rate), make sure that
questions that provide numerator and denominator data are asked, and are asked for the
same reference period and of the appropriate respondents.
! If a question involves a level of measurement (e.g., binary YES-NO) that does not
match the analysis objective (e.g., measurement of a continuous distribution of some
attribute), it will not function as intended.
U Does the question ordering work? The focus of the QAS-99 is the individual question as it
is placed under our “microscope.” As such, we do not explicitly consider the relationships
that may exist between questions. However, a questionnaire is in a real sense greater than
the sum of its parts; questions that are asked earlier may effect those that come later.
In general, if you do uncover problems not otherwise described by the previous QAS-99
steps, circle YES for 8, and enter a note indicating the nature of the problem found.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-1
Section 4. Example Questions
How to Code Problem Types: Two Detailed Example Questions
The two example questions below demonstrate exactly how one goes through the coding
procedure. We will use one coding form for each question. So, to do the coding, we first type
the full question text (with response categories) at the top of the coding form. We then select (in
this document, by highlighting) YES or NO for each problem type (1a... 8) on the coding form
and also enter more detailed comments directly onto the form.
Q1) Do you own a swimming pool, or is a swimming pool available to you at your
residence? [APARTMENT, CONDO ARE INCLUDED - BEACH CLUB,
SWIM CLUB ARE NOT INCLUDED.]
_Yes
_No
Q2) When swimming, how often do you swim within 2 hours of consuming an
alcoholic beverage? (Check response volunteered).
_Always
_Nearly always
_Sometimes
_Rarely
_Never
_I do not drink
_I do not swim
_Don’t know/Refused
Here is the coding form for the first example question. Following this is a detailed
explanation of why each code was circled YES or NO.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-2
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION.
Question number or question here:
Q1) Do you own a swimming pool, or is a swimming pool available to you at your residence?
[APARTMENT, CONDO ARE INCLUDED - BEACH CLUB, SWIM CLUB ARE NOT
INCLUDED.]
_Yes
_No
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
Not clear when to read part in brackets.
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read.
Not clear how to read part in brackets.
YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view,
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations.
Conflict between “swim club” and “residence.”
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-3
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
Assuming the parenthetical part is to be read, it is awkward sounding.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex.
“Residence” too technical; also not clear what type of swimming pool is intended.
YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
May be confusing – is this asking a “yes/no” or “select which one is true” question?
It is also unclear what type of pool to include.
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict. YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
Is this first clause even necessary? If the person doesn’t own the pool, but it’s still
there, it seems like we still want to know about this.
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary. YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-4
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question.
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for. YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-5
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult. YES NO
7b. MISMATCH between question and response categories. YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations. YES NO
7e. OVERLAPPING response categories. YES NO
7f. MISSING eligible responses in response categories. YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-6
STEP 1: READING
1a. WHAT TO READ: It appears that the question itself can be read without much
difficulty. But the part in brackets is less clear. Is this to be read? Is it optional? How does
the interviewer know? Because this is not clear, we circle YES for 1a. So we will
remember later why we circled this, write “1a: Not clear when to read part in brackets”
under the question.
1b. MISSING INFORMATION: Circle NO. There is no missing information that would be
needed to read the question.
1c. HOW TO READ: If one assumes that the optional parts are to be read in some cases, one
should also circle 1c, because “apartment, condo are included” is not really fully scripted,
the way we normally speak English. Write “Not clear how to read part in brackets” next to
1c on the coding form.
STEP 2: INSTRUCTIONS
2a. CONFLICTING INSTRUCTIONS: Again, assuming that there are cases in which the
part in brackets is to be read to the respondent, this does serve as an instruction. It could be
argued that the instructions conflict with the question, because the latter mentions “swim
club,” which people do not live at, whereas the former mentions “residence.” So, on this
basis, one might also circle YES for 2a. Write “Conflict between swim club and residence
under 2a on the coding form.
STEP 3: CLARITY
3a. WORDING: It is not overly long. But the wording of the parenthetical section is
awkward. So, circle YES.
3b. TECHNICAL TERMS: “Residence” may be too technical—something like “the place
where you live” would probably be better. So, circle YES for this, and write “Residence
may be too technical” on the coding form.
3c. VAGUE: Looking at the question again, it could be interpreted as an “Either-or”
question (“Which of these is true: You own a pool, OR there’s one at your residence?”). So,
one might choose to circle YES on this basis, and again, write a note on the coding form
describing the possible problem. Also, it is not clear whether to include children’s pools,
etc.
3d. REFERENCE PERIODS: Is the reference period specified? The implication is “now,”
which seems clear enough. Circle NO.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-7
STEP 4: ASSUMPTIONS
4a. INAPPROPRIATE ASSUMPTIONS: At this point, consider what we are assuming:
First, that they have a home, which is probably OK for a household telephone survey. But
when we ask “do you own a swimming pool,” this begs the question of whether ownership
of the pool is important, or just that there is a pool at the residence. Assuming that the
objective is to be able to measure access to swimming, then it may be irrelevant whether it is
“owned or not.” So, circle YES, and add a note like “Is this first clause even necessary?” on
the coding form.
4b. ASSUMES CONSTANT BEHAVIOR: This is not really relevant, so circle NO.
4c. DOUBLE-BARRELED: It is arguable whether owning a swimming pool is a separate
issue from whether one is at the residence, but that has already been picked up in 4a. If not,
it could be picked up here.
STEP 5: KNOWLEDGE/MEMORY
5a/b/c/d. Hopefully, the respondent knows this, can recall it, and does not need to do any
mental computation. We’re OK here (circle NO).
STEP 6: SENSITIVITY/BIAS
6a/b/c. This does not appear to be a very sensitive issue, or one that presents any real social
desirability bias. It also does not suggest that saying YES or NO is necessarily better.
These are all circled NO.
STEP 7: RESPONSE CATEGORIES
7a. No. It is not open-ended.
7b. MISMATCH. If interpreted correctly, it is a yes/no question with yes/no response
categories. There could be a mismatch if the respondent misinterprets this as an either/or
question and says “the first one” (own a pool), but this has already been picked up and
noted. So, it is safe to circle this NO.
7c/d/e/f/g. (other problems with response categories). These are generally not a problem
with yes/no questions.
STEP 8: OTHER
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-8
We have already found quite a bit. There is no need to note anything else.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-9
EXAMPLE QUESTION #2
Now we are ready to tackle the second question. On a second coding form, we enter the text for
Question 2 at the top, and proceed through each step with the selection of YES and NO for each
problem type (see the detailed coding form). Here is a reprinting of the question, a filled-out
coding form, and an explanation of how the coding was done.
Q2) When swimming, how often do you swim within 2 hours of consuming an
alcoholic beverage? (Check response volunteered).
_Always
_Nearly always
_Sometimes
_Rarely
_Never
_I do not drink
_I do not swim
_Don’t know/Refused
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-10
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION.
Question number or question here:
Q2) When swimming, how often do you swim within 2 hours of consuming an alcoholic beverage? (Check response
volunteered).
_Always
_Nearly always
_Sometimes
_Rarely
_Never
_I do not drink
_I do not swim
_Don’t know/Refused
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read. YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view.
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-11
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
The phrasing “consuming an alcoholic beverage” is very unnatural.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex.
The term “consuming” is too technical; give examples of alcoholic beverages, like
beer, wine, or liquor.
YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
What exactly is “swimming,” as far as this question is concerned?
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict.
The reference period for the behavior is unclear.
YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
Assumes that person swims at all.
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary.
Doesn’t work if person has only swum once or twice.
YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question.
This implicitly consists of (a) do you swim, and (b) how often do you drink when
swimming.
YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-12
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for. YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
This may be a sensitive issue.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-13
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult.
Respondent won’t know what acceptable response categories are.
YES NO
7b. MISMATCH between question and response categories.
Same problem as 7a.
YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations. YES NO
7e. OVERLAPPING response categories.
The categories “always” and “nearly always” are very close. Replace the latter with
“more than half the time.”
YES NO
7f. MISSING eligible responses in response categories. YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-14
STEP 1: READING. No problems reading. It is completely scripted. Circle NO.
STEP 2: INSTRUCTIONS. There are no instructions. Circle NO.
STEP 3: CLARITY
3a. WORDING: The question is not especially long or complicated, but the wording
“consuming an alcoholic beverage” is somewhat unnatural. So, we circle YES for this.
3b. TECHNICAL TERMS: “Consuming” is a technical term that does not reflect normal
English expression. “Drinking” would be an improvement. So, circle YES, and write
“consuming is too technical.” Also, “Alcoholic beverage” is not really well defined. It
might be better to include examples. So, write “give examples like beer, wine, liquor.”
3c. VAGUE: The term “swimming” may be somewhat ill-defined (does this mean
swimming laps, or just sitting in a pool?). However, this will be difficult to rectify unless
we script a detailed series of questions on swimming. The only way to determine whether
this is a problem is to make a decision of whether this level of ambiguity is acceptable or
not. Thus, the issue of the intent—what we hope to learn from the question—is vital in
making these decisions. These issues can only be decided by the questionnaire designer.
For now, it makes sense to circle YES, if only to force a consideration of the “vagueness”
issue.
3d. REFERENCE PERIOD: The question asks for a frequency of behavior, but there is no
reference period given. Do we mean generally? Or over a certain time interval? So, circle
YES, and write “reference period unclear.”
STEP 4: LOGIC
4a. ASSUMPTIONS: By asking “When swimming...” the question assumes that the
respondent swims. He or she may not, though. The question attempts to get at this by
including “I do not swim” in the response categories. But, if the respondent just says
“never,” meaning “I never swim,” we will not be able to distinguish this from never drinking
alcohol and swimming. It would be better to ask first if the person has gone swimming over
a certain time period, and then if he or she has drunk alcohol beforehand. So circle 4a, and
write “Assumes that person swims at all.”
4b. ASSUMES CONSTANT BEHAVIOR: The question assumes that one swims regularly
enough that it makes sense to assess the frequency with which one engages in drinking
before doing so. It will not work well when someone has only gone swimming a few times
(if I swam once in the last year, and had been drinking, is that “always”?). Circle 4b and
write “Doesn’t work if person has only swum once or twice.”
4c. DOUBLE-BARRELED: Yes, the question implicitly includes the questions (a) do you
swim, and (b) do you drink alcohol before swimming. If we did not notice this before, we
can circle YES and note it here.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-15
STEP 5: KNOWLEDGE/MEMORY
5a. KNOWLEDGE: Is it likely the person will not know the answer? It is possible that
respondents will not know if 2 hours has passed, so we have to make a judgment about
whether they will generally know this or not. This is a judgment call, but assuming that they
will probably have a general idea, we can let this go (and circle NO).
5c. RECALL: This is an even more difficult judgment call. Will people remember how
often they have had alcohol before swimming? One thing in our favor is that we are not
asking for exact recall, but instead “vague quantifiers” like always, sometimes, etc. (so we
are not requiring an exacting type of recall). However, given that there is no reference
period, and that this perhaps should be reassessed, it also must be considered whether the
new reference period successfully avoids the problem (note that many problems with
questions are interrelated, and that solving one may create another).
5d. COMPUTATION: Again, because of the nonquantitative nature of the response
categories, this should not be a problem.
STEP 6: SENSITIVITY/BIAS
6a. SENSITIVE CONTENT: Drunken swimming is not as bad as drunken driving, but it
may be perceived as sensitive. So, one might choose to circle YES here, and write “May be
sensitive issue.”
6b. SENSITIVE WORDING: The wording does not ask about being drunk, or anything
pejorative, so this is OK (circle NO).
6c. SOCIALLY ACCEPTABLE response: The question phrasing does not appear to
suggest a particular answer that is most acceptable (beyond the simple fact that it makes one
look best to say “Never.” So, circle NO.
STEP 7: RESPONSE CATEGORIES
7a. OPEN-ENDED QUESTION: This is circled YES because the respondent is never told
what the acceptable response categories are and will be likely to give answers other than
“always” or “sometimes.”
7b. MISMATCH: The question and answers are not specifically mismatched. On the other
hand, the response categories implied by the question do not match the given precodes. So,
one could choose to circle YES. Note that the problem was already picked up in 7a.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute4-16
7c/d. TECHNICAL/VAGUE response categories: The terms in the response categories are
vague quantifiers, which are sometimes the source of ambiguity, but are not especially
vague by survey standards.
7e. OVERLAP: The categories “always” and “nearly always” are very close. It might be
better to replace the latter with “more than half the time.” So, circle YES and write a note
in.
7f/g. Circle NO.
STEP 8: OTHER
Again, we have found a number of problems already. There is no need to note anything
else.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-1
Section 5. Practice Questions
The best way to learn how to use the QAS-99 is to practice on real survey questions. So,
the next task is to code a number of representative questions. In this section, we first provide a
“basic” set containing four questions, and then a more extensive set for users who would like
more practice. For each of these questions, follow these guidelines:
a. Assume that the questions will be administered in a telephone survey.
b. The questions vary in the number and type of problems they are likely to have. So, do
not feel like you have to find every type of problem with each question.
c. Use one coding form for each question. Enter written comments for each YES circled on
the coding form.
d. The questions are not intended to be part of an organized questionnaire. Therefore, do
not worry if they seem to be placed haphazardly (they are).
e. When doing the coding, it is helpful to imagine different types of respondents and
different living situations. Ask your self things like:
Could Bill who lives in an assisted living situation answer this question?
Does this question work for people with little education?
Will young people without much life experience be able to answer this question?
For the first four practice questions, the “answers” from expert coders appear on coding
sheets starting on page 5-4. We recommend that you completely code each question before
examining how your findings compare with those of the expert coders.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-2
Four Practice Questions
1) In the past month, have you visited a dentist (include dental hygienist) for any type
of dental care?
[ ] Yes
[ ] No
[ ] Don’t know
2) Sometimes health care professionals try to prevent injuries by counseling people
about safety. Have you had a routine checkup in the past year?
[ ] Yes - one time
[ ] Yes - more than once
[ ] No
[ ] Don’t know
3) When you receive health care, would you prefer a provider who shares your ethnic
background?
[ ]Yes
[ ] No
[ ] Doesn’t matter
4) How often do you use a restraining system (INCLUDE ALL TYPES) when you
drive or ride in a car? (MARK ONE)
[ ] Always
[ ] Nearly always
[ ] Rarely
[ ] Never
[ ] Don’t know
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-3
Answers to the Four Practice Questions
The following pages provide the coding sheets that serve as “answers” for the four
practice questions. We include both the codes and the notes that the expert coders wrote.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-4
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION
Question number or question here: Q1) In the past month, have you visited a dentist (include dental hygienist) for
any type of dental care?
[ ] Yes
[ ] No
[ ] Don’t know
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
Should the parenthetical part (include dental hygienists) be read?
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read.
See 1a - if the interviewer attempts to read the parenthetical part, the question does not
flow well.
YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view.
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations.
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-5
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
Parenthetical part is awkward.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex.
What does dental care include? What about dental care received some other place
(i.e., emergency room, clinic, etc.)
YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
People may vary in what they include.
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict.
It is unclear what “the past month” means.
YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary. YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-6
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question.
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for.
Respondent may not remember information, or whether the visit occurred within the
past month.
YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-7
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult. YES NO
7b. MISMATCH between question and response categories. YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations. YES NO
7e. OVERLAPPING response categories. YES NO
7f. MISSING eligible responses in response categories. YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-8
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION
Question number or question here:
Q2) Sometimes health care professionals try to prevent injuries by counseling people about safety. Have you
had a routine checkup in the past year?
[ ] Yes - one time
[ ] Yes - more than once
[ ] No
[ ] Don’t know
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read. YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view.
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations.
First sentence does not match the question.
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-9
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex.
“Health care professionals” and “routine checkup” may need to be defined.
YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
It’s not necessarily clear what a routine checkup is for all respondents.
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict.
It’s not clear what “the past year” means. This could be either “since January 1,” or
since this date a year ago.”
YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary. YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-10
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question.
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for.
Information may be difficult to recall over a 1-year period.
YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-11
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult. YES NO
7b. MISMATCH between question and response categories.
The question implies a YES/NO response, but the YES response categories are further
subdivided, without any question that will serve to prompt an appropriate response.
YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations. YES NO
7e. OVERLAPPING response categories. YES NO
7f. MISSING eligible responses in response categories. YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-12
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION
Question number or question here:
Q3) When you receive health care, would you prefer a provider who shares your ethnic background?
[ ]Yes
[ ] No
[ ] Doesn’t matter
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read. YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view.
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations.
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-13
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex.
“Ethnic background” and “provider” may need to be defined.
YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
Respondents may vary in what they include in “ethnic background.” Is race to be
included as well?
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict.
Because this is an attitude question, it may not be a serious problem, but it really isn’t
clear whether we’re talking about lifetime experiences, current ones, or something
else.
YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
Assumes respondent receives health care.
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary. YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-14
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question.
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
I wonder if people have even thought of this very much.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for. YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
Questions about ethnicity may be sensitive.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question.
Respondents may feel that the “safe” answer is NO, even if they do prefer someone of
their own ethnic background.
YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-15
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult. YES NO
7b. MISMATCH between question and response categories.
The question implies only YES and NO answers, but the response categories include
“it doesn’t matter” as well.
YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations. YES NO
7e. OVERLAPPING response categories.
It’s difficult to differentiate between answers of “no” and “it doesn’t matter.”
YES NO
7f. MISSING eligible responses in response categories. YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-16
QUESTION APPRAISAL SYSTEM (QAS-99):
CODING FORM
INSTRUCTIONS. Use one form for EACH question to be reviewed. In reviewing each question:
1) WRITE OR TYPE IN QUESTION NUMBER. ATTACH QUESTION
Question number or question here:
Q4) How often do you use a restraining system (INCLUDE ALL TYPES) when you drive or ride in a car?
(MARK ONE)
[ ] Always
[ ] Nearly always
[ ] Rarely
[ ] Never
[ ] Don’t know
2) Proceed through the form - Circle or highlight YES or NO for each Problem Type (1a... 8).
3) Whenever a YES is circled, write detailed notes on this form that describe the problem.
STEP 1 - READING: Determine if it is difficult for the interviewers to read the question
uniformly to all respondents.
1a. WHAT TO READ: Interviewer may have difficulty determining what parts of the
question should be read.
Should “include all types” be read?
YES NO
1b. MISSING INFORMATION: Information the interviewer needs to administer the
question is not contained in the question.
YES NO
1c. HOW TO READ: Question is not fully scripted and therefore difficult to read.
You can’t literally read the parenthetical part without being very awkward.
YES NO
STEP 2 - INSTRUCTIONS: Look for problems with any introductions, instructions, or
explanations from the respondent’s point of view.
2a. CONFLICTING OR INACCURATE INSTRUCTIONS, introductions, or
explanations.
YES NO
2b. COMPLICATED INSTRUCTIONS, introductions, or explanations. YES NO
STEP 3 - CLARITY: Identify problems related to communicating the intent or meaning
of the question to the respondent.
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-17
3a. WORDING: Question is lengthy, awkward, ungrammatical, or contains complicated
syntax.
As noted above, reading the parenthetical part makes this very awkward.
YES NO
3b. TECHNICAL TERM(S) are undefined, unclear, or complex.
“Restraining system” is not a common term.
YES NO
3c. VAGUE: There are multiple ways to interpret the question or to decide what is to be
included or excluded.
It’s not clear what types of “cars” are to be included. Does this mean “car” in the
generic sense, or is it to exclude things like mini-vans, trucks, and so on?
YES NO
3d. REFERENCE PERIODS are missing, not well specified, or in conflict.
The reference period is not specified.
YES NO
STEP 4 - ASSUMPTIONS: Determine if there are problems with assumptions made or
the underlying logic.
4a. INAPPROPRIATE ASSUMPTIONS are made about the respondent or about his/her
living situation.
This assumes that the person drives or rides in a car. Further, it assumes that the
behavior is similar whether they drive or ride (it may differ).
YES NO
4b. ASSUMES CONSTANT BEHAVIOR or experience for situations that vary.
Respondents tend to say things like “I always do on the highway, but not so much
around town.”
YES NO
4c. DOUBLE-BARRELED: Contains more than one implicit question.
Driving and riding are two completely different questions. This needs to be broken up.
YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-18
STEP 5 - KNOWLEDGE/MEMORY: Check whether respondents are likely to not know
or have trouble remembering information.
5a. KNOWLEDGE may not exist: Respondent is unlikely to know the answer to a factual
question.
YES NO
5b. ATTITUDE may not exist: Respondent is unlikely to have formed the attitude being
asked about.
YES NO
5c. RECALL failure: Respondent may not remember the information asked for. YES NO
5d. COMPUTATION problem: The question requires a difficult mental calculation. YES NO
STEP 6 - SENSITIVITY/BIAS: Assess questions for sensitive nature or wording, and for
bias.
6a. SENSITIVE CONTENT (general): The question asks about a topic that is
embarrassing, very private, or that involves illegal behavior.
This is not extremely sensitive, but does involve behavior (not wearing a seat bealt) that
is illegal in some States.
YES NO
6b. SENSITIVE WORDING (specific): Given that the general topic is sensitive, the
wording should be improved to minimize sensitivity.
YES NO
6c. SOCIALLY ACCEPTABLE response is implied by the question.
The “how often” phrasing implies that one does this. If we were to first ask “do you
ever” and then ask “how often” when the initial answer is “yes,” the answers may look
very different.
YES NO
Questionnaire Appraisal System
QAS-99
Research Triangle Institute5-19
STEP 7 - RESPONSE CATEGORIES: Assess the adequacy of the range of responses to
be recorded.
7a. OPEN-ENDED QUESTION that is inappropriate or difficult.
It’s not clear that respondents will use the response categories that are pre-coded.
These should be read out loud.
YES NO
7b. MISMATCH between question and response categories.
Same problem as 7a. The response categories may not be those that respondents will
use.
YES NO
7c. TECHNICAL TERM(S) are undefined, unclear, or complex. YES NO
7d. VAGUE response categories are subject to multiple interpretations.
These are vague response quantifiers that may have very different meanings to
different respondents.
YES NO
7e. OVERLAPPING response categories. YES NO
7f. MISSING eligible responses in response categories.
There’s a big gap between “nearly always” and “”rarely.” (“Sometimes” is missing.)
YES NO
7g. ILLOGICAL ORDER of response categories. YES NO
STEP 8 - OTHER PROBLEMS: Look for problems not identified in Steps 1 - 7.
8. Other problems not previously identified. YES NO
View publication statsView publication stats