Also available from Continuum
Real World Research series
Case Study Research Methods, Bill Gillham
The Research Interview, Bill Gillham
Observation Techniques, Bill Gillham
Developing a Questionnaire 2nd Edition, Bill Gillham
Also available
Questionnaire Design, Interviewing and Attitude Measurement, A. N. Oppenheim
Small-Scale Social Survey Methods
Bill Gillham
Real World Research
continuum
Questions and Answers
6
The Relationship between Questions and Answers
Forming specific questions, the fine grain of survey research, is not the start of such a project but about half-way along the development phase. More than that: like the writer of a murder mystery who sets out the plot knowing how it is to be resolved, so in writing questions you need to be clear in what way they are to be answered.
Open and closed questions
The issue of what questions you want to ask and how you want to ask them comes down to how the responses are to be analysed. To recap, there are basically two kinds of questions: those where the answer is left 'open' and those where the answer is 'closed' - in the sense of offering a limited range of specified answers. An open question might be:
• Which daily newspaper do you read most often? A closed equivalent might be:
• Which of the following daily newspapers do you read most often? (Followed by a list with tick boxes and probably an
The closed version, with its simple tick response, is a more efficient way of posing the question and less trouble to answer. It also prompts people, reminding them of elements they might overlook. With specific 'factual' questions about behaviour (what people do) there is not much of a problem: the closed question works well.
However, when you are dealing with opinions the choice is not so clear-cut. For example, if you wanted to survey attitudes to the war in Iraq an open question could be:
• What were your views, at the time, on the Allied invasion of Iraq in 2003?
That is a complicated question which, as it is posed, may require a complicated and extended answer. If you have 100 - or a 1,000 — such answers in a postal questionnaire you have an enormous task of analysing (or categorizing) the responses. And if they are part of a questionnaire similarly constructed then it becomes an almost impossible task particularly for a lone researcher. Indeed, much simpler (and less emotive) questions than this will pose problems of analysis. It is not that the question is not worth asking but that it is inappropriate to the medium of a postal questionnaire; there being other reasons such as the need to explore the answers given. This is where in-depth face-to-face interviews would be more appropriate.
A closed version of the above question might be:
SAME QUESTION (Please tick the most appropriate box
below):
□ Broadly in favour
□ Not sure/no opinion
□ Mainly against
Note that it is the answer that is closed, not the question. This is called a pre-coded format. It is then a simple matter to analyse the answers - a basic count formula. The weakness of such a pre-
rnHpd anal™? is that it rlnpsn't tfll -vnn mnrlr Knt tn pynprt it tn
Small-Scale Social Survey Methods
Questions and Answers
do so is to misunderstand not just the practicalities of forming questions for a questionnaire but also what you can expect of such a survey. Questionnaire (or other) survey instruments are best at:
• providing a straightforward 'descriptive' account of the wider framework;
• indicating those areas where further in-depth research is necessary (perhaps where the balance of responses is unexpected or shows up subgroup differences).
negative responses, or simply the full range of answers, you often have to resort to a forced choice. For example:
• time allocated		In the consultation with your doctor, what was the most satisfactory part (TICK V) and what was the least satisfactory (CROSS X)?
• feeling that you were listened to		
• diagnostic advice		
9 explanation of treatment		
• follow-up appointments		
Constructing survey questions
In constructing a questionnaire you may find that the range of possible answers alters the way in which you frame the question. This is because the answers almost always follow some kind of multiple-choice format. There are not so many of these, but using a variety is a way of keeping your respondent interested. Here are some examples where we give the answers first, for emphasis:
• YES/NO Have you seen your doctor during the past year?
• up to 1 week		If YES how long did you have to wait for an appointment? (TICK ONE BOX) JVB This choice of direction linked to a previous question is called routing a response.
• 1 — 2 weeks		
• 2 — 3 weeks		
• more than 3 weeks		
• very satisfied		How satisfied were you with the
• satisfied		consultation? (TICK ONE BOX)
• not satisfied		
• very dissatisfied		
This last kind of scale, widely used and often with more choice (5 to 7 options) including a neutral one in the centre ('not sure' or similar) tends to produce a positive response bias - perhaps the reason why it is so popular with commercial enterprises! To get
Another way to get people to express a preferential judgement is to weight a set of scaled responses so as to emphasize that you are interested in critical judgements set against a positive statement ('Well, I don't agree with thatf.') For instance:
• The Health Centre provides a very good appointments service (TICK ONE BOX).
□ agree
□ not sure
□ disagree
□ strongly disagree
An alternative is to put different elements in rank order, as a way of expressing judgements. For example, if the Health Centre runs a weight-control programme, you might ask the following:
• Which aspect of the programme did you find most useful? Put 1 against what you found most useful, 2 against the next most useful, and so on down to 5 for the least useful.
□ scientific information
□ fallacies about dieting
□ advice on healthy eating
□ information on unhealthy eating (what to avoid)
□ changing exercise patterns
Small-Scale Social Survey Methods
Questions and Answers
The strength of this technique is that it gives the researcher a better idea of the relative merits of the different components of the programme. Note that because numbers are used it is tempting to construct an average 'score' and this is not legitimate (see p. 90-1).
Specified response questions
Where possible you should provide the probable answers so that the respondent simply has to check a given choice. But sometimes you know what kind of answer you want but the range of exact responses cannot be identified or there would be too many to list. For example:
• Where did you do your teacher training?
Please write in: _
• In what year did you complete your training? Please write in: _____
• In what year did you obtain your first teaching post?
Please write in:__
Here the required responses are more simply obtained, and can then be classified as appropriate.
Slightly open questions
The trouble with questionnaire judgements of the multiple-choice variety is that you don't know why a particular answer was chosen. If your aim is to improve the quality of service it is here that adverse judgements need amplification. If, for example, in relation to the Health Centre appointments service some respondents have ticked either strongly disagree or disagree, then in order to take action you need to know what lies behind the judgement. To get more insight you can phrase what is known as an indicated response where you are indicating the required answer but not savins1' what it should be:
•  If you have ticked either disagree or strongly disagree, please say why:
The responses to this will require a simple content-analysis approach (see Chapter 14). But do not be deceived by that word 'simple': categorizing open responses is a time-consuming business. Questions of this type should be of the essential variety and few in number - probably no more than two. Problems of analysis apart, they are more trouble to answer and may impair your response rate. If there is one index of those who lack experience in constructing questionnaires it is that they include a lot of loosely constructed open questions.
Subject descriptor questions
This category has been left till last because it is the most straightforward yet often the most carelessly constructed for that reason.
The answers to subject descriptor questions are important because they allow you to carve up the questions and answers that follow (the focus of the survey research) in terms of differences on the subject descriptors (gender, age, income, occupational status, educational level, marital status). Their very 'factuality' may suggest that there is no problem in getting this information. But there are two issues: that the questions should be entirely unambiguous; and that they should be sensitive to people's feelings about the information that is sought.
Gender may be an easy one. But asking people about their age is another matter. Consider the following which shows a common error:
Small-Scale Social Survey Methods
Questions and Answers
• Please tick the box against your age range:
□ 20-30
□ 30-40
□ 40-50
□ 50-60
□ 60-70
□ 70 +
So which box do you tick if you are 30 or 40 or 50, etc.? The format needs to be:
□ 20-29
□ 30-39
— and so on.
The 10-year range is adequate for most purposes; a 5-year range is a possible alternative if it is really necessary. People are remarkably vain about their age presumably because they like to think they don't look it. So you don't ask for an exact age, except perhaps in the case of the under-20s.
Income is another sensitive area. Depending upon your survey group you may opt for £10,000 or £5,000 intervals. Here a single-digit overlap in the categories is unimportant because people don't know their income precisely. So:
• Please tick the box against your gross annual income range:
□ under £10,000
□ £10,000-20,000
□ £20,000-30,000
□ £30,000-40,000
□ £40,000-50,000
□ £50,000 +
The ceiling income indicated, as well as the size of the steps, should take into account the probable range in the group being
surveyed. The above example would be too high and too coarsely graded for undergraduate students, or pensioners for example.
Occupational status is, again, not entirely straightforward. In the standardized world we inhabit, financial institutions are thrown by those who do not fit the single categories: employed, retired, unemployed or self-employed. So we might offer the following choice:
□ Full-time employed
□ Part-time employed
□ Unemployed
□ Retired
□ Self-employed
□ Unwaged carer (of children or adults)
You are usually asked to tick one box. But in my case, for example, I am both retired (no job') and self-employed (writer, consultant) and until last year I was also part-time employed. So which box should I tick? An appropriate instruction might be: Tick all relevant boxes.
Occupatio7ial category is commonly used as an index of social class or socio-economic status but that is not without its critics -educational level is another (but correlated) dimension - see below. You can't expect people to classify themselves socially and there is much disagreement among social scientists; for those interested, look at the debates in Twentieth Century British Social Trends edited by A. H. Halsey and Josephine Webb (2000).
Classifications by occupation are traditionally divided between manual and non-manual and further sub-divided (professional, semi-professional, routine white-collar workers, small-business workers, skilled manual, semi-skilled and unskilled). At one time these correlated quite closely with income and educational level but that is no longer the case particularly in relation to income; many 'manual' workers are now educationally well qualified while many 'white-collar' workers are not. In any case the subjective perceptions of social class are subtler than such 'objective'   descriptive  categories:   social  behaviour, accent,
Small-Scale Social Survey Methods
Questions and Answers
clothes, styles of grooming, tastes and habits of mind loom large here.
Although occupational category is one variable of importance, in small-scale surveys you are not usually seeking to mimic main population-level parameters. For guidance on classifying occupations in the UK the reader is referred to the General Household Survey (ONS 2006). So, however you use the information you can ask:
• What is your current job (or your last job if caring for others, unemployed or retired)? _
Educational level is a less ambiguous way of classifying people than their occupation with its bewildering diversity. In the UK you could specify it as follows:
• Please tick the highest level of your educational qualifications:
□ None
□ GCSE (Grades D to G)
□ GCSE (Grade G or above)
□ A level or equivalent (e.g. Scottish Highers, NVQ Level 3)
□ First degree or graduate level professional qualification
□ Postgraduate qualification (diploma or certificate)
□ Master's degree
□ Doctorate
Marital status is, if anything, even trickier. The once standard choice: Married/Divorced/Separated/Single no longer applies. Many people co-habit (and the partnership may not be heterosexual). How do you classify that? Arid we now have the category of 'civil partnership' between same sex couples. A more adequate choice would be:
□ married
□ separated
□ divorced
□ widowed
□ single
□ civil partnership
□ in a stable relationship
As with the occupational category, people should be given the choice of ticking more than one box, e.g. 'divorced' and 'in a stable relationship', if they so wish.
In the next chapter we deal with question development. The present chapter (in illogical order) means that when you come to draft a possible question you will at the same time be thinking: How shall I set out the answer choices? Which answer format would work best?
This process is part of what is involved in producing clear and well-focused questions. And that stage is fundamental to producing a questionnaire or structured interview that works: one that the respondent can interpret unambiguously and where the researcher gets the kind of information that is being sought.
Refining the Questions
7
Refining the Questions
The last chapter emphasized the need to consider how questions might be answered because this has a bearing on how they are framed. But questions might be presented slighdy differently according to the data-collection method used:
• a printed questionnaire, which respondents have to complete by themselves;
• a structured face-to-face interview (essentially a personally administered questionnaire, which we refer to as a recording schedule);
• a structured telephone interview, which falls somewhere between the two in that we suggest you send the respondents a questionnaire and talk them through it.
But whichever method you choose, the process of question development is the same; it is only later that certain adjustments might be made depending on how the questions are to be presented.
The unstructured phase
How much time you devote to this stage depends on the degree of familiarity with the topic area. The mistake is to think you know it so well that this open 'finding out' procedure can be bypassed
It involves talking to people who are members of (or similar to) the potential survey group. As indicated in Chapter 5 this can be done systematically with short unstructured interviews that you tape-record and content-analyse, in a focus group (a slightly later stage where you're fairly clear what you want to focus on), or by informal conversations, which may be all that is feasible.
Where possible audio-recording should be used, as written note-taking interrupts the flow, distracts your attention and involves on-the-hoof selection that may be ill-judged. And if you just rely on recollected impressions you will lose a lot of material. There is much to be said for listening carefully to a tape-recorded conversation (perhaps more than once) where attention is focused on the content and not on maintaining the interaction. Writing down the substantive topics that come up also clarifies your thinking, even if you are not conscious of the process. In each case you make it clear to the people involved what your area of interest is, that you need guidance on the detail and will be using what they tell you to develop questions for a questionnaire or similar. What are the things they think are important?
Brainstorming
You can do this sooner or later: usually when you feel the need to put possible questions down on paper.
This is not the point to concern yourself with exact forms of words, whether the questions fall into groups, are in the wrong order, are different ways of saying the same thing (useful in itself) or - as will certainly be the case - are too many.
You will find that after you've disgorged all this material you will add to it gradually, rather like a shopping list. At this stage don't try to categorize or edit — at least on paper - although that process will be going on in your mind because you can't help it. This element of unconscious work is highly productive and you need to allow time for it to bear fruit.
And for the moment keep the list to yourself while you pick
Small-Scale Social Survey Methods_ _
The Delphi technique
This technique (with its classical reference) can be used in various ways: we use it here as a way of getting further items for your questionnaire/interview. For example, if your research area is people's attitudes to alternative medicine, you ask those with some knowledge of the area for suggested questions (perhaps no more than three or four) that could be included in the questionnaire. You don't show them or tell them what you've drafted so that their ideas are not 'contaminated' by yours.
There is usually some overlap with what you already have but perhaps expressed in a better form of words and, quite often, something is suggested that had not occurred to you.
Sorting questions under topic headings
Logically 'topics' - the groupings of a set of questions — come before the specific questions themselves. For some, whose mental processes are highly organized, that may be the case, but for most of us questions come first - rather out of order - and, by inspection, we can then see how they fall into groups or topics. When we do that, gaps (questions we need to ask) and redundancies (two or more questions that say essentially the same thing) will soon become apparent.
For example, in the fictional instance of a study of alternative medicine we might infer a grouping of questions under the topic heading Doubts about alternative medicine, which could be set out as below (with answer formats). (Note how the answer choices arc designed to fit the questions or statements.)
a. How do you feel about 'self-prescription'?
□ No problem
□ Not sure
□ Doubtful
*________ Refining the Questions
j        b. The advertising is too 'commercial'. : □ Agree
□ Not sure
□ Disagree
;        c. Do you feel able to choose the right medicine'?
□ Yes
□ Not always
□ No
d. Do you feel you understand what is said about using the medicines?
□ Yes
□ Not always
□ Sometimes
e. Are you confident about taking the right amount of medicine?
□ Yes
□ Not always
□ No
f. The advertising tries to 'blind you with science'.
* □ Agree
□ Not sure
□ Disagree
!        g. Are you confident about making the right choice of medicine?
□ Yes
, □ Not always
' □ No
i
-48 .    .. .
; When you read through these you might feel that (a), (c) and (g) are
j essentially the same; as are (b) and (f); and then possibly (d) and (f); in
= addition (e) could be seen as overlapping with (a). So the exercise has
« set you thinking, but you may decide to stay with them all for the
, moment. The revision process will be helped if you set out similar
{ questions in parallel, i.e. side-by-side rather than having to dot
j around in a vertically arranged list. You can do this in spreadsheet
1 format on a computer using Microsoft Excel or similar; or hand-write
j them on to blank A3 sheets following a similar format. It is curious
* how this simple visual rearrangement assists vour thin kin p.
Small-Scale Social Survey Methods_
For example:
The advertising is too 'commercial'./ The advertising tries to 'blindyou with science'.
Are they equivalent? Is one better than the other?
Writing questions is as much art as science: some people have a definite knack for it. And conversely you can find yourself trapped by the first form of words that occurred to you. You can have doubts about a word or a phrase without being clear as to an alternative. If you highlight these you can come back to them. For example, the phrase 'blinding with science' might seem too strong so that respondents will, inevitably, react against it.
Overlap is one issue, clarity is another, redundancy is yet another. All writing, after the initial formulation, is about revision, particularly eliminating unnecessary words. Repetition and piling on the adjectives (or adverbs) are the main culprits here. For example:
• What was the very first thing that made you decide to give alternative medicine a try?
Please write in:__
— could be replaced as:
• What made you try alternative medicine?
Please write in:_____
There is no loss of meaning in the revision and its impact is greater. 'Simple' and direct questions always work best because they are better focused, which means the response is better focused. Interestingly, in free-flowing face-to-face interviews there is a greater temptation for the interviewer to be more 'wordy', paradoxically when it is harder for the person being interviewed to attend to all that is being asked. A particular tendency is for the interviewer to ask a compound question such as: Why did you try
rmnblcmeninm) merlirine nn/t nthnf 7PI/7C vmir health  brnhlem nf thp bmp?
|_______________Refining the Questions
I
That s not one question but two and, in written format, one can
see immediately that it won't do. Writing is inherently more
economical than speech; although it is worth noting that an 1 expert interview is characterized by the economy of the questions.
Perhaps just one word: Why?
We need to distinguish trialling from piloting: these topics are
covered in Chapters 8-10. Trialling is where you take your (too I        long) list of questions - and answer formats - and try them out on
a small number of people similar to those in your survey group.
You will already have your questions sorted under topic headings i        with alternatives about which you can't make up your mind.
These will be what you present to your trialling group, but one f        further revision is needed before you do that.
i
[
Putting questions in developmental order
Questions in a questionnaire or interview schedule should follow
•j a 'logical' order: where one leads on to the next which, nonetheless, presents something different. If the person being
t interviewed feels 'I've already answered that question' then the process starts to lose momentum - with an associated irritation
: factor. One question should also be a kind of orientation or preparation for the next. Hence the emphasis on establishing a logical or developmental sequence. However, the rather piecemeal way that questions are generated means that they won't be in quite that order (if at all). We'll take a practical example.
i if you were researching the motivation for, and experience of,
'mature' students (defined as 30+) taking a postgraduate course,
[       you might have a topic heading Expectations of the course under
|       which the following questions have been accumulated:
\
I       a. Have you found the course as you expected?
\ □ Yes
I □ More or less
Small-Scale Social Survey Methods
b. How accurate did you find the description of the course in the prospectus?
□ Accurate
□ Fairly accurate
□ Inaccurate
c. How have you found the academic level of the course?
□ Higher than expected
□ As expected
□ Lower than expected
d. How have you found the workload on the course?
□ Very heavy
□ About right
□ Easily managed
e. Are your fellow students of the type you had expected?
□ Yes
□ More or less
□ No
f. How well does the course fit what you feel you need?
□ Very well
□ Quite well
□ Not very well
g. Does the course provide the intellectual stimulation you require?
□ Yes
□ Sometimes
□ Not really
How would you revise the order of these questions? Are there any that you feel are essentially the same, or redundant? Take jive minutes to make your own revisions before reading the next paragraph.
My judgement is that (e) and (f) should come just after (b) - and in reverse order (f) and then (e). Questions (g) and (c) seem pretty well equivalent but it might be worth trying out both to see which works better; (a) is a clear lead in to the questions so should stay first, while (d) could also stay if (f) and (e) were transposed. Your judgement may well differ: all you need to do is review your