Key points
•• Traditional reviews allow you to be flexible and to explore ideas
•• They can be insightful and original
•• They can be undertaken by one person at undergraduate or postgraduate level
•• Subjectivity is implicit; there is no protocol but it is good practice to tell the reader
on what basis the material was selected
•• It is only relatively recently that the literature review has been deemed a research
methodology in its own right
Be aware
•• The result depends on the skills of the writer
•• Traditional reviews can be dismissed as an ‘opinion piece’
•• Traditional review is less helpful for policy development because it is not a systematic
review (see Chapter 7)
Overview of the debate
So far in Part I of this book we have covered the early stages of doing a literature
review. We have:
•• identified a topic, funnelled down and decided on a specific research area and
research question
•• identified keywords and used the library resources to search for resources and
information
•• read the material, and
•• made notes after reading.
Part II of the book concentrates on two different approaches to literature
review. The language – or labels – used to describe literature reviews tends to
be confused. So each time you read the words ‘literature review’ you should
the traditional review
5
06-Jesson et al-4192-Ch-05.indd 73 27/01/2011 10:08:28 AM
doing your literature review
74
check, read and categorise what type of review it is. In this book we set out to
differentiate between a ‘traditional literature review’ and a ‘systematic literature
review’, which has a very specific meaning. The first part of this chapter
examines a selection of published reviews showing how each one has a different
focus. The latter part of the chapter begins working through the review
process. To make the most of this chapter you need to refer to the original
article because sometimes we quote from the original work and other times
we only provide a commentary.
A literature review is a desk-based research method by which the researcher
critically describes and appraises what is already known about a topic using
secondary sources. In some instances, a literature review is described as a traditional
narrative review (Torgerson, 2003) because it consists of a narrative
style of presentation.
•• The traditional review, as undertaken by undergraduate and Masters-level students,
aims to be comprehensive, which means it aims to present a summary review of the
current state of knowledge about a particular subject.
•• The traditional review also seeks to add new insights on the topic.
Critique of traditional review
Some writers who promote a systematic review methodology take a pejorative
approach to the traditional review on the grounds that it does not produce
reliable evidence (see Petticrew and Roberts, 2006, on health and social care).
Which approach to review you take depends on the context of review, the
academic discipline of study, and the purpose. It also depends on what level of
study you are at. Traditional review is the norm at undergraduate level. At
postgraduate level traditional review may be less helpful for guiding policy or,
as Torgerson (2003) asserts, for contributing to an informed debate of the
issues in education. Critics dismiss the traditional review as of little value
because it is ‘non-scientific’, or it is merely a discussion paper, or an opinion
piece.This is because in traditional reviews the author’s subjectivity is implicit;
there is no protocol and quite often no description of how the review was carried
out. In some journal articles, for example, there might not be a methods
section to help the reader understand the choices made on selection or on the
review process. The critique claims that the absence of a systematic protocol
means that an uninformed reader is unable to judge the completeness of the
arguments put forward in such a review (the systematic protocol is described
in Chapter 7).
But the argument presented in this book is that before being able to move
on to doing a systematic review you have to be confident in your searching
06-Jesson et al-4192-Ch-05.indd 74 27/01/2011 10:08:29 AM
the traditional review
75
skills and basic reviewing ability. To start with a traditional review helps you to
develop those reviewing skills and gives you the insights to enable you to work
your way up to doing the systematic review. It can be argued that this form of
review is a ‘scoping review’, that is, a review which sets the scene for a future
research agenda. It is unlikely that at undergraduate level you will have the
time to do more than a traditional review, but that does not mean you do not
need to know what a systematic review methodology involves. Postgraduates
are most likely to attempt a systematic review.
Critics of traditional narrative review argue:
•• that there is no formal methodology, so there is a lack of transparency and no academic
rigour
•• that the reasons for including some material and excluding others is not discussed
•• on selection grounds, because the review is only a small, potentially biased selection
of the whole range of literature on the subject
•• since there is no methodological audit trail, the review cannot be replicated by others
•• that there is no quality assessment of the material included; incorrect interpretations
may therefore result
•• that contrary or conflicting views may not be identified or included in the review.
When will you need to review the literature?
In Chapter 1 we suggested that there are six different scenarios when literature
reviews are undertaken:
1	 When writing a research proposal, usually for postgraduate dissertations of approximately
3,000 words in length. The review would take up approximately one-third
of this word count. So this review would be a preliminary taster of the more in-depth
review that you write in your dissertation. On the other hand, a proposal seeking
funding for further research would be a review which summarises key findings before
highlighting the knowledge gap, thereby justifying the rationale for further research.
2	 For an undergraduate or postgraduate Masters research project, where the dissertation
is between 10,000 and 20,000 words in length. The review might be one or
two chapters covering policy or theory and empirical applied studies. This would
be a more comprehensive review of the topic, still identifying the research gap and
explaining or justifying the project.
3	 For a doctoral dissertation.
4	 For a journal article publishing research findings, which often begins with a summary
or a section that ‘strings together’ the literature without providing an in-depth
analysis.
5	 When writing a literature review in its own right to provide a stand-alone review of
a topic.
6	 For evidence-based policy development.
06-Jesson et al-4192-Ch-05.indd 75 27/01/2011 10:08:29 AM
doing your literature review
76
Types of review: critical, conceptual, state of the art,
expert and scoping
To recap, traditional reviews are exploring issues, developing ideas, identifying
research gaps, whereas systematic reviews are compiling evidence to
answer a specific research or policy problem or question, using a protocol. It
can be argued that both approaches are used to answer a research question
or problem. However, the main difference is in the design and the methodological
approach. Within the traditional review model, there are different
types or reasons for reviewing. The type or reason is often indicated in the
article title. The types are listed next and then followed by an example of
each type.
•• A traditional review usually adopts a critical approach which might assess theories
or hypotheses by critically examining the methods and results of single primary
studies, with an emphasis on background and contextual material. The material is
selected in order to present an argument. Example 5.1 is a paper on marketing
recycling and is representative of a typical academic paper – setting up the story
so far.
•• A conceptual review aims to synthesise areas of conceptual knowledge that contribute
to a better understanding of the issues. Example 5.2 is a discussion about the
two concepts: public health and population health.
•• A state-of-the-art review brings readers up to date on the most recent research on
the subject. This might be a seminal work, so it could be a useful beginning to your
research project. Example 5.3 is a state-of-the-art review of green supply-chain
management.
•• Similarly, an expert review is just that, written by an acknowledged expert. This may
be heavily influenced by the writer’s own ideology and paradigm. Example 5.4 is
an expert review on organisational and managerial change.
•• A scoping review sets the scene for a future research agenda. This is comparable
to what you have to do for your student project. The review documents what is
already known, and then, using a critical analysis of the gaps in knowledge, it
helps to refine the questions, concepts and theories to point the way to future
research. It is also used as the first step in refining the questions for a subsequent
systematic review. The output is a document which maps out the general topic area
and makes recommendations for future research. Example 5.5 explores the
research agenda for research on firm acquisition.
To summarise, these types of traditional review are often based on a personal
selection of materials because the author has some important contribution to
make to the knowledge base and the point is to help develop an argument or
tell a story.This approach offers greater scope to be reflective, but may provide
a one-sided or even biased argument, as discussed in Chapter 4.
06-Jesson et al-4192-Ch-05.indd 76 27/01/2011 10:08:29 AM
the traditional review
77
TASK
Take a look at the contents pages of recent editions of the journal The International
Journal of Management Reviews, which publishes only review articles (note: some
articles contain no methodology section at all). You will see that the article titles carry
a pointer, indicating what types of a review they are. Sometimes they declare the
method, as in: ‘content analysis’, ‘towards a conceptual model’, ’a review of theories’,
‘towards a research agenda’, ‘a narrative review’.
The key point about traditional reviews is that it is not necessary to conduct a
comprehensive systematic search, but you will help the reader more if the
method and selection rationale is described because they will be able to judge
the completeness of your argument. So, we are talking here about a search
description (building on Chapter 2), then a selection rationale, prior to analysis
and synthesis. The reader can then understand better the relevance and importance
of the review and its findings with respect to their own information needs.
Some examples of published traditional reviews
To benefit from the following examples you will need to read the original
papers.
Critical reviews
Example 5.1 is taken from Smallbone (2005), an article entitled ‘How can
domestic households become part of the solution to England’s recycling problems?’
In this article, Smallbone is reading the recycling literature from a marketing
perspective. So the critical reading skill here is to apply one academic
perspective,‘marketing’, on to another academic perspective, in this case ‘environmental
studies’. Example 5.1 shows you how this review is assembled. You
can see how many articles were used in the review and that having an alternative
focus helps you to identify a gap.
TASK
It would help if you can look at Smallbone’s article for yourself to see how the review
is constructed (you can find it online at WileyInterscience, Business Strategy and the
Environment, 4(2): 110–122.
The first paragraph of the review (Example 5.1, column 1) is laid out to show
how it has been constructed and the behind-the-scenes reading and analysis
06-Jesson et al-4192-Ch-05.indd 77 27/01/2011 10:08:29 AM
doing your literature review
78
that underpins the statements (column 2). This opening section presents the
reader with a summary overview of the current state of knowledge on the topic.
Example 5.1
A critical review
Taken from a paper on recycling by Smallbone (2005), this is an example of a good critical
literature review put together with survey results to make policy recommendations.
The original text Deconstructing notes on
the text
The academic literature on pro-environmental
behaviour and why people do and do not participate
in recycling is extensive, but it is scattered in journals
that range from those concerned with psychology
through business, marketing and environmental
science and sociology. It goes back to the 1970s,
and by the mid 1980s a meta-analysis aiming to
formulate a model of environmental behaviour could
include 128 studies (Hines et al., 1986).
Despite this effort, the results are frequently
contradictory and are bedevilled by differences in
local rubbish collection systems, variations in cultural
expectations, and reliance on self-reported behaviour
and small or biased samples, leaving a recent study
to conclude that current knowledge on recycling
behaviour is ‘fragmented and inconclusive’ (Davies
et al., 2002: 54).
Nevertheless it is possible to tease out a number of
strands of thinking which together help to shed light
on the validity of the three marketing assumptions
described above.
First sentence – tells me there is a lot
of published material, and that
contributions have been made from
a variety of academic disciplines
and perspectives.
Second sentence – tells me how old
the topic is and that there is already
a meta-analysis of 128 studies,
published in 1986.
This is the summary analysis (an
overall statement of ‘what does
this material tell us’) and notes
differences – reaffirming this opinion
by citing a recent paper (at the time
of writing her work).
Then she brings the paragraph back
to her own research question.
Examine the paper again. You will see that Smallbone organises the material
under three main headings (the framework):
•• Targeting the green consumer using marketing communications.
•• Green intentions and recycling behaviour.
•• Could recycling become a social norm?
06-Jesson et al-4192-Ch-05.indd 78 27/01/2011 10:08:29 AM
the traditional review
79
If we examine this structure further, we can see that each of these headings
contains a varying number of paragraphs and references. This gives you an
indication of how much material there was to review in total, and then how
many authors had covered each theme. This information helps you, as a critical
reader, to make judgements about the quantity of work reviewed and the relative
importance of each topic.
•• Targeting the green consumer using marketing communications – five paragraphs,
covers 18 articles.
•• Green intentions and recycling behaviour – three paragraphs, covers three articles.
•• Could recycling become a social norm – three paragraphs, covers three articles.
Altogether, Smallbone has reviewed and analysed 24 papers.
So the critical dimension is achieved by reading the available material from
a different perspective, looking at the topic through an alternative lens – the
eyes of a marketing discipline. This is providing an alternative focus and that
is one way that knowledge is advanced. Smallbone wants to make a point
about the gap she found in the recycling knowledge and is using the literature
to show that gap. In a policy-related context she wants to show that current
policy may be focused on the wrong trajectory. Of course, she also wants to set
the scene for future marketing campaigns.
Tip
See if you can apply an alternative discipline lens to the topic you are reviewing.
Example 5.2
Conceptual reviews
An example of conceptual review in an emerging field is Kindig’s article, ‘Understanding
population health terminology’ (2007). We suggest that you look up this article to get the
most out of the discussion that follows.
We have to be sure that we are in agreement that the words we use have a shared meaning
and understanding. Conceptual reviews are able to compare and contrast the different
ways in which authors have used a specific word or concept. The following paragraph is
my observation, not a reproduction of the original text. At first glance you might not think
it is a review at all, but on closer reading of the original paper you will notice that Kindig
(2007) is reviewing the state of knowledge on two concepts, which are often used in a
confused way in public health discourse: public health and population health. So what
makes this a conceptual review?
06-Jesson et al-4192-Ch-05.indd 79 27/01/2011 10:08:29 AM
doing your literature review
80
Kindig (2007) does not contain a methods section; neither does it state that it is a
literature review, although it is. The context for the paper is the realisation that the
different disciplines now contributing to the discourse on public health have different
understandings and use of the core concepts. Now it is not just medicine that has an
input, but also epidemiology, economics, sociology and psychology. Kindig defines
and discusses many of the terms and concepts characterising this emerging field. This
paper is written by an expert in the field, drawing on a wealth of personal and professional
experience. The concepts are public health, an old term whose definition
was once clear but is now becoming problematic, and population health, a relatively
new term. In the final paragraph, Kindig gives his rationale for the study. This is a
policy agenda and an important research question is proposed: ‘What is the optimal
balance of investments in the multiple determinants of health over the life course that
will maximise overall health outcomes and minimise health inequalities at the population
level?’ Without an agreed definition of the core concepts the subsequent research
will be of little use.
Kindig faced a similar problem that students have to face. There is a research
question, but before you can begin to specify the research question and design
the research methodology you have to be clear how the core concepts are
understood and used (operationalised). Using Kindig (2007) as an example,
this paper shows the complexity that might arise when examining current
literature from a different paradigm or academic discipline.
Tip
Note down from each article you read how the core concepts have been used or
operationalised. Are there different definitions in other disciplines and how has the
definition changed over time?
Example 5.3
State-of-the-art reviews
We have selected a paper by Srivastava (2007) as an example of a state-of-the-art review.
The paper is a comprehensive review of supply chain literature on green supply. (Srivastava,
2007). This is an expert review because the authors have developed an original framework
through which to analyse the known body of work, as shown in the following quotation.
The paragraph below is based solely on the journal article abstract, using the original
text. Note that the published review is described as ‘comprehensive and all inclusive’. It
must therefore have taken a long time to do. This paper does have a detailed methodological
description. So what makes this a state-of-the-art review?
06-Jesson et al-4192-Ch-05.indd 80 27/01/2011 10:08:29 AM
the traditional review
81
A perusal of the literature showed that a broad frame of reference for this subject is
not adequately developed. A succinct classification is needed. The literature on green
supply chain management is covered exhaustively from its conceptualisation, primarily
taking a ‘reverse logic angle’. This review classifies the problem context, on the
methodological approach, maps mathematical tools and provides a timeline.
(Srivastava, 2007: 53)
Example 5.4
Expert reviews
Ferlie is a well known expert on public policy. His review paper presents an overview of
the organisational changes and management literature on recent large scale change within
health care organisations. Ferlie notes that seven critics may argue that the approach
adopted here is too subjective in orientation (Ferlie, 1997: 181). This example of a literature
review is a self-declared subjective selection written by an expert on the topic of
organisational and managerial change. It is therefore a traditional review, not a systematic
review. Ferlie (1997), as an acknowledged expert, gives his opinion on the literature as it
applies to health care. This following paragraph is my observation, not the original text. So
what makes this an expert review?
Ferlie (1997) explains his search and selection rationale. He observes that the subject
matter is a diffuse field where the unit of analysis – organisational and managerial reform
– is multifaceted. So, there is no conceptual consistency then. He observes that studies
are based on diverse methodological and theoretical orientations, and randomised trials
are rarely employed. Ferlie’s search design consists of a two-stage model. The first
stage involved a manual search of eight key journals known to the author. Ferlie traced
earlier work by examining citations, and then selected those studies which appeared to
be of interest. The second stage was the selection of a group of nine key texts which
appeared to be of particular interest. The selected studies were chosen on the basis that:
•• they discuss organisational and managerial changes in health care at a macro
rather than a micro level
•• they present primarily empirical data as well as interpretation
•• in the judgement of the author, they reflect the work of established scholars and
research groups working in this field as indicated by professional reputation,
citation and the winning of grants.
It is the description of the methodology that tells you that this is not a systematic
review and that the author has used his expert knowledge to limit the
search. An insight here is that by using citations and scanning key journals it is
possible to identify a useful range of information. But it is the expertise that
drives selection of material.
06-Jesson et al-4192-Ch-05.indd 81 27/01/2011 10:08:29 AM
doing your literature review
82
Tip
By scanning the citation list for each article you will quickly spot the key journals
publishing material on your topic. It does not take long to identify them in the search
stage as the same ones keep popping up. This will help to concentrate your search.
Example 5.5
Scoping reviews
Our example of a scoping review is a paper on the subject of firm acquisition by Barkema and
Schijven (2008). As a reminder, a scoping review sets the scene for a future research agenda.
This is my observation, not the original published text, on what makes this a scoping review.
Barkema and Schijven (2008) provide an example of a critical literature review
which summarises past research and presents a research agenda for the future.
Barkema and Schijven set their review of firm acquisition in the contemporary management
context at the time of writing. The consensus is that most acquisitions fail.
Yet despite many academic insights into what needs to be done when acquiring a
new company, the same mistakes appear to be repeated. Therefore, the authors note
that there is value in taking stock of past research and in outlining what remains to
be explored and in drawing an agenda for future work (2008: 595). There is a short
methods section explaining what is covered in the review – studies published since
1980 in leading management journals plus some as yet unpublished work and work
from other settings. The structure of the paper is balanced, consisting of 11 pages on
the past review, six pages on the future agenda, together with summary tables of the
selected papers. The summary tables are the evidence, presented to the reader.
The review documents what is already known. Then, using a critical analysis
of the gaps in knowledge, it helps to refine the questions, concepts and theories
and thus points the way to future research. It is also used as the first step in
refining the questions for a subsequent systematic review. The pages of summary
present the evidence to the reader, so that you can make your own assessment
of the validity of the authors’ claims.
So now you have looked at some published examples of different types of
traditional critical reviews, this should help when you are reading for your own
literature review and give you some ideas on how to get going.
Drawing up an analytical framework – how to sort the material
Although you might set out with a limited plan for the scope of your research,
it can actually be a fluid and flexible process. The library search will show you
06-Jesson et al-4192-Ch-05.indd 82 27/01/2011 10:08:29 AM
the traditional review
83
the size of the body of work that exists. If you can’t find anything, you may
need to revise your research question or try alternative keywords. The advantage
of a traditional review, which is less formally prescribed than a systematic
review, is that you can add new thoughts and new themes to your plan
throughout the process.
Figure 5.1 is a stylised presentation showing an overview of the typical process
as straightforward and linear. Step 1 begins when you have obtained some
papers, some information on theories and on the empirical applications of the
theory. Step 2 is to read and begin to think what approach your critique will
take. Make an analytical assessment of what you have in front of you. Step 3
is the point at which you can spot a knowledge gap.
Theory
Step 1 Step 2
Step 3
Examples
applied in
research
Read
and assess
Knowledge
gap
Figure 5.1  The analytical process
The review process
Figure 5.1 is an abstract representation. If we take a real-world example, you
can see how this might begin to work out. Remember the Smallbone (2005)
example covered 24 articles, so start this process as soon as you can. Begin to
make notes, categorise your papers and start to write.
Mind map or table display of material
The assumption at this point is that you have printed off several articles and
are ready to begin the next phase of analysis. One useful way to start is to lay
out the material on your desk or on a big table, in bundles or piles. Figure 5.2
is based on an evaluation project where we needed to find out quickly how to
evaluate a local Sure Start Partnership. Sure Start was a UK-government intervention
in disadvantaged communities. The aim of the programme was to
support parents of children under 5 years by bringing together all the statutory
06-Jesson et al-4192-Ch-05.indd 83 27/01/2011 10:08:29 AM
doing your literature review
84
public and voluntary services working for families – thus forming a partnership.
Over six months we had gathered together huge piles of paper, on a research
question ‘What is to be learned from Sure Start programme evaluations?’
Figure 5.2 represents the desk display of the different materials that were
obtained. Displaying the materials in this way helped us to conceptualise
the topic and create a mind map. The skills you need to perform this task
are ‘differentiating’ and ‘categorising’ your data.
Sure Start
Public sector
partnerships
Key writers
(1) Glendinning, Powell
and Rummery – book
definitions
(2) Tom Ling – typology
(3) Skelcher
(4) Murray Stewart
Empirical
studies
Few published
Good practice guides
Toolkits for education and
measurement tools
Conceptual
Political and Policy Studies
Government systems
Hierarchy Markets Third Way
partnerships
Policy statements
National Evaluation of Sure Start
Website guidance papers
Local evaluations
Definitions
No consensus,
vague
Cross-organisational
working, cross-sectional
working, collaboration
Figure 5.2  Example of a mind map/table sort of materials on Sure Start public partnership evaluation
This project was about public sector partnerships, so partnerships sit at the
centre of the map. Both conceptual and empirical studies were retrieved –
with some overlap; there were also government practice guidelines and public
partnership evaluation toolkits.After skim reading all of this material we could
confidently state that there was no clear definition of public partnership.There
were several abstract models and public partnership evaluation toolkits, but
few empirical examples of applying these models and none applying evaluation
toolkits to local Sure Start interventions. This does not mean that there
were none, just that we had not yet found them because they were not in the
public domain.
06-Jesson et al-4192-Ch-05.indd 84 27/01/2011 10:08:30 AM
the traditional review
85
Begin to categorise
So to recap. Step 1 begins when you have obtained some articles, possibly
covering theories and some empirical applications of the theory. Step 2 is to
read and begin to think what approach your critique will take, making an analytical
assessment of what you have in front of you. Step 3 is to begin to categorise
your material. The next section shows how you can take a single text
and summarise the key points, authors, and concepts into a table.
Change management is a topical feature of modern management. The example
on change management is summarised in Table 5.1. Shacklady-Smith (2006)
advises her readers to be mindful of the historical, social, economic and political
contexts when reviewing the change management literature. This is a useful
example because it has several approaches, lots of empirical studies testing out
the theory and a succinct critical summary and overview of the literature.
This is an example from a textbook, where the literature has already been
summarised for you – remember, wherever possible you should attempt to
consult the original publications to make up your own mind and develop your
own critique. Nevertheless this type of review does serve as an introduction to
help you get started and charts your ‘enquiry chain’ (O’Neill, 2005).
Table 5.1  Change management theory as a topic
Theoretical approaches Relevant authors How this differs
Mechanistic and planned Lewin, 1958 Historical beginnings
Emergent process Burnes, 1996 Challenges Lewin
Typologies of change:
Developmental
Transitional
Transformational
Ackerman, 1997 Modern applications
How to change organisations Empirical case studies Applied to real situations
Source: Shacklady-Smith (2006: )
Moving to analysis and synthesis
So far this chapter has described different types of review and illustrated the
types with published examples from a range of topics. This was followed by
some advice on how to begin making sense of the literature you are reading.
In his classic text Doing a Literature Review, Hart (1998) has given us a
checklist of the types of question you might use to interrogate an article and
which will also provide a framework for your write-up:
06-Jesson et al-4192-Ch-05.indd 85 27/01/2011 10:08:30 AM
doing your literature review
86
•• What are the key sources?
•• What are the key theories, concepts and ideas?
•• What are the key epistemological and ontological grounds for the discipline?
•• What are the main questions and problems that have been addressed to date?
•• How has knowledge in the topic been structured and organised?
•• How have approaches to these questions increased our understanding and knowledge?
•• What are the origins and definitions of the topic?
•• What are the major issues and debates about the topic?
You will find your own preferred way of beginning to put information together.
The process of searching, reading, then refining the scope and re-searching is
a cyclical process. It is unlikely that you will do each of these activities once
only. By now you will have explored several sources and articles, maybe even
different sorts of literature, you will know the key issues and have found a
focus for your review. Next, you have to concentrate on documenting the
themes, similarities and differences in the literature you are reviewing.
Synthesise on a thematic basis, so the evidence from single studies is pooled.
This takes us to the next three steps towards analysis.
Stage 1
Write a summary of the important parts of each paper; take three papers to
begin with. You will start, with three sets of information and two or three
themes (which might include theory, results or data analysis). Look again at
Example 5.1 where Smallbone (2005) stated the three key topic areas the
literature review covered. Evaluate the evidence that is presented, question for
yourself how valid and reliable the evidence is. This is easier to do if you
already have knowledge of one subject area and can apply it to a different
discipline or a different focus, as Smallbone did with marketing and recycling.
At this stage you are writing descriptively – it is just a summary. Later you will
expand on the number of papers you review and build up the evidence, and
maybe revise your themes as you go along.
Stage 2
Now compare and contrast the three papers and themes: what is the same,
what is similar, what is different? Write short summary paragraphs of the key
points you want to make, drawing from each of the three individual extracts
you produced in step 1. Now you should have one comparative set of information
but it is no longer in a bibliographic format but combined. Be careful with
time sequences – analyse the work in a chronological order. If it is a new subject
area for you, question the plausibility. Does what you read make sense from
06-Jesson et al-4192-Ch-05.indd 86 27/01/2011 10:08:30 AM
the traditional review
87
your own experience? You may be the person who makes a critical challenge
to longstanding ideas or theory. At this stage you are writing critically.
Stage 3
This is a cyclical process. You will read more, make more notes, and discard
some notes, read and review, write, read, review. Build up your case now. It is
important to document the good and negative or weak features, the strengths
and limitations of the method as well as the explanatory arguments. Keep
returning to your original research question, aim and objectives to make sure
what you are writing is still relevant. Some students get to this point and
change direction because an in-depth understanding of the problem helps
them to refocus on the research problem.
Tip
When writing, check the words at the beginning of each paragraph. If each paragraph
begins with an author’s name then maybe you still have a format that is based
on description, where each source of information is presented independently one
after the other. Now repeat this test with each sentence. Try not to begin every
sentence or paragraph with the author’s name.
The presentation of your review
The aim of a traditional narrative literature review is to provide a critical
review, not a description, a catalogue or shopping list. It is a new picture or
story you are presenting, with your judgements made from a sound basis of
evidence, reflection and sometimes experience.
Make sure that you summarise current knowledge in a clear and consistent
logical way. Think of your review as a story, with an introduction, middle and
end. The format could be like this:
•• Introduction – guide the reader through. Give an overview of what is known and
how you will present your critique – the trailer. Make the purpose clear from the
start, explain the structure.
•• Method – this is optional but it does help the reader to see how and from where
you have obtained your information. You can also state here possible themes/issues
that you have decided not to include.
•• Theme 1.
•• Theme 2.
06-Jesson et al-4192-Ch-05.indd 87 27/01/2011 10:08:30 AM
doing your literature review
88
•• Theme 3.
•• Discussion and critical summary paragraphs.
•• Conclusion – states what your contribution to the debate is. Show the gap. Do not
introduce new material. If relevant, how does it link into your research project?
Summarising the gap – dare to have an opinion
You need to convince the reader that you are fully conversant with the current
debate on your topic, that you know who the key writers are and the ideological
agendas or perspectives from which they work.You want to be original
and show how your analysis and synthesis brings insights or a new dimension
to the topic. Finally, show that you have identified the knowledge gap, especially
if the purpose of the review is scoping to set up a research project.
Highlight any consensus, any exceptions to that consensus, and note the
methodological or theoretical limitations in the work you have reviewed.
Summary
This chapter has tried to tease out the different types of traditional literature review
that you may encounter in your reading. Tell the reader what the purpose of your
review is and what type of review it is. The traditional review is flexible. It allows us
to use different types of evidence, draw on quantitative and qualitative work, on
research and non-research materials. A final point is to remind you to be self-aware,
reflective and critical. It remains the key research method that all students have to
undertake at some point in their academic career. It does not have to be labour
intensive, and can be done within the budget and time constraints of a student
project. It is the first step you need to undertake before you can begin a systematic
review.
In Chapter 7 we cover systematic review. To aspire to the systematic review as the
gold standard is good, but first you have to know how to do an effective traditional
review. Systematic reviews, if done properly, are time-consuming, expensive and are
usually undertaken by a team of research assistants and researchers. Researchers
who produce systematic reviews have to have an overview of the current literature.
In effect, they have insights from a scoping review to formulate or modify the research
question, before they can apply the specific research question and designate the
inclusion and exclusion criteria.
06-Jesson et al-4192-Ch-05.indd 88 27/01/2011 10:08:30 AM
Key points
•• To undertake a systematic review you need some working knowledge and understanding
of the field
•• Systematic review uses a standardised, structured, protocol-driven methodology
•• The methodology is focused, explicit and must be transparent
•• Systematic review requires a rigorous, systematic, comprehensive and exhaustive
search for all the relevant literature
•• Systematic review claims to be objective, balanced and unbiased
Be aware
•• Systematic review methodology may not be appropriate for undergraduate-level
reviews
•• Systematic review is time-consuming and can be expensive
•• Systematic reviews are usually undertaken by more than one person; they are
usually a team effort, to do the scanning, screening and quality assessment to
reduce bias
•• Systematic review is dependent on access to electronic databases and a range of
available databases and can therefore be limited by the effectiveness of the databases
•• Systematic review is typically restricted to published, peer-reviewed, academic work
Overview
Literature reviews can be envisaged as a continuum, ranging from traditional
review to systematic review.
Traditional review ——————————— Systematic review
No defined method ——————————— Rigorous method
Exploratory/creative ——————————— Transparent/replicable
This chapter begins by describing the context in which systematic review has
developed as a methodology in evidence-based practice. It then takes you step
the systematic review
7
08-Jesson et al-4192-Ch-07.indd 103 27/01/2011 10:09:48 AM
doing your literature review
104
by step through the explicit stages of the methodology, as shown in Figure 7.1.
You will learn how to develop your review protocol and the importance of
documenting every stage of the process.The chapter ends with some examples
of more sophisticated and complex reviews.
Scope and map
Plan and protocol
Document
Inclusion and exclusion criteria
Search and screen
Quality appraisal
Data extraction
Synthesis
Report
Figure 7.1  Key stages in a systematic review
Definitions
Systematic review
The term, or concept, systematic review is used in two ways: it can refer to
the prescribed methodology (a means – method) or to the output report itself
(a report). This first definition, by Sweet and Moynihan (2007: 1), written
with a healthcare context in mind, encapsulates all the key buzzwords which
define a systematic review:
Systematic reviews provide a systematic, transparent means for gathering, synthesising
and appraising the findings of studies on a particular topic or question. The aim is to
minimise the bias associated with single studies and non systematic reviews. (our italics)
Or it describes the output:
A systematic review is a research article that identifies relevant studies, appraises
their quality and summarises their results using scientific methodology. (Kahn et al.,
2003: 1, our italics)
08-Jesson et al-4192-Ch-07.indd 104 27/01/2011 10:09:49 AM
the systematic review
105
Identifying and sifting through all the relevant studies and evaluating each
according to predefined criteria is what distinguishes a systematic review from
a traditional review. Table 7.1 sets out the key differences.
Table 7.1  Comparative table of scoping review and systematic review
Traditional (scoping) review Systematic review
Aim To gain a broad understanding,
and description of the field
Tightly specified aim and objectives with
a specific review question
Scope Big picture Narrow focus
Planning the review No defined path, allows for
creativity and exploration
Transparent process and documented
audit trail
Identifying studies Searching is probing, moving from
one study to another, following up
leads
Rigorous and comprehensive search for
ALL studies
Selection of studies Purposive selection made by the
reviewer
Predetermined criteria for including and
excluding studies
Quality assessment Based on the reviewer’s opinion Checklists to assess the methodological
quality of studies
Analysis and
synthesis
Discursive In tabular format and short summary
answers
Methodological
report
Not necessarily given Must be presented for transparency
Source: Adapted from Pilbean and Denyer (2008)
Task
Go to the internet and enter the keywords: ‘traditional literature review’ and then
‘systematic literature review’ in the search box. Note the difference in the number of
hits your search engine throws up.
The sheer volume of new research studies published these days makes it hard
for researchers, practitioners and policy makers to know what is currently useful.
This is a knowledge management problem and the aim of systematic
reviews is to help bring this problem under control. For practitioners, the
limitations of acting on single case study articles is that they may misrepresent
the balance of available research evidence. In addition, studies can be of
variable quality in terms of their design, execution, analysis and reporting.
Moreover, research reports can be biased.
Systematic review for clinical biomedical research was formalised over
20 years ago in the UK through the Cochrane Collaborative, but since then it
has been adopted worldwide and to some extent by most other fields of research.
The purpose is to combine information from various sources to provide more
08-Jesson et al-4192-Ch-07.indd 105 27/01/2011 10:09:49 AM
doing your literature review
106
data to answer a specified research question without the need to set up a new
study. The Cochrane papers provide a valuable resource and guidance to practitioners
and students undertaking a systematic review. These reviews adopt a
strict scientific methodology, as shown in Table 7.2, which may not be appropriate
for your student subject or topic. It is best to check with your supervisor
which type of review you should undertake.
Evidence-based practice
Evidence-based policy or practice (EBP) began in healthcare research, in
medicine and nursing, but it is now to be found in other areas, such as education,
the probation service, regeneration policy and practice, housing, social
care and criminal justice. You will notice from this list that in the UK these are
all public services, where a large body of research is commissioned regularly, so
there is ample material to review.
Several research centres, based in academic institutions, have been set up to
collate the evidence and to help researchers adapt the Cochrane guidelines
by producing specialised guidelines and toolkits. Some of these are listed in
Appendix 3. For example, in social care, the Social Care Institute for Excellence
(SCIE) sets its own framework for systematic reviews which covers five domains:
policy knowledge, organisational knowledge, practitioner knowledge, user
knowledge and research knowledge. The SCIE quality agenda covers empirical
evidence of policy and practice, including the use of testimony from users and
carers, a field which would not be highly rated in other subject contexts.
As we noted earlier, some authors argue that systematic review is a ‘better’
research method than traditional review because it is a more rigorous and therefore
‘scientific’ approach to the practice of literature reviewing (see Torgerson
(2003) on education research and Petticrew and Roberts (2006) on the social
sciences). Rigorous and scientific, in this context, refers to the systematic protocol
methodology, compared to the looser and more flexible approach implicit
in the traditional review. However, there is rather less consensus in the social
science disciplines as to what constitutes evidence and appropriate methodological
paradigms, each academic field having its own preferences.
Business and management research
The development of evidence-based management and the practice of undertaking
systematic reviews in the business and management field has been slow. The
problems for business and management subjects has been well articulated by
Tranfield et al. (2003: 208). However, it should be acknowledged that the business
and management field covers a diverse range of subjects, including marketing,
finance and accounting, work organisation and psychology, economics and
08-Jesson et al-4192-Ch-07.indd 106 27/01/2011 10:09:49 AM
the systematic review
107
international business,each of which has its own research paradigm and acceptable
approaches to research. Thus it can be difficult to impose the systematic review
methodology on to their work. To meet this challenge the Advanced Institute of
Management Research (AIM) researchers at Cranfield University have developed
a training package to encourage the use of systematic review in business schools
(Pilbean and Denyer, 2007/8). Tranfield et al. (2003) argue convincingly that,
compared with the medical and biological sciences, there are significant ontological,
epistemological and methodological differences in the many disciplines
which go under the ‘management’ umbrella, and this makes it even more
problematic to adopt wholesale the prescribed approach to systematic review.
Social science and cross-disciplinary systematic reviews
Other authors have noted that the systematic review methodology has not always
been successful in social science and multidisciplinary research.A really interesting
discussion on the limitations of a protocol-driven methodology is to be found in a
systematic review which set out to answer this question: What empirical evidence
is available on the relationships between mental health problems and social exclusion?
(Curran et al., 2007). This study had to combine the literature covering two topics:
health (mental health problems) and social policy (social exclusion). Both of
these concepts are subject to varying usage over time and place. The authors of
this review noted other challenges arose because much of the work was located in
grey literature, such as in policy documents and government reports, or research
from major charities or think tanks. The conclusion, in the context of trying to
apply the systematic protocol to social science and cross-disciplinary research, was:
There are a number of challenges to be overcome: poorly defined topics; inconsistent
use of key words and controlled vocabulary; abstracts that do not effectively communicate
the content of the paper or are not accessible in bibliographic databases
and resource and technology problems. (Curran et al., 2007: 305)
The observations made about business and management research and social
science and cross-disciplinary systematic reviews are here to illustrate some of
the hazards implicit in taking a methodology from its original discipline –
medicine and healthcare – and adapting it to other disciplines.
A half-way review – rapid appraisal
So what are we saying here? We argue in this book that a systematic review
actually builds on a traditional, critical or scoping review, as described in
Chapters 5 and 6, but there are two main differences: the reason for undertaking
a review and the manner of performing a review.
08-Jesson et al-4192-Ch-07.indd 107 27/01/2011 10:09:49 AM
doing your literature review
108
•• A systematic review is a comprehensive review of all published articles selected to
address a specific question using a systematic method of identifying relevant studies
in order to minimise biases and error.
•• The details of the approach used in a systematic review must be documented in the
methods section of the review report.
•• Systematic reviews have a structured methodology which must be transparent to its
readers.
•• The starting point is having a key specific question to answer.
•• The whole process is summarised in Table 7.2.
Table 7.2  The key phases of a systematic review
Phase 1: Mapping the field through a scoping review.   What do we know and what are
the knowledge gaps (as described in Chapter 5). How much relevant material is available?
Prepare the review plan. This includes the method and the protocol for the systematic review. Define
the question or questions, compile key words. Set up the inclusion and exclusion criteria. Design the
data extraction pro-forma or data sheet.
Phase 2: Comprehensive search.   Access the electronic databases and search using your key
words. Search and document the search results.
Check whether the hits are relevant or are you coming up with too many hits. If so, do you need to
refine the search and revise the key words? Do you need to revise the inclusion and exclusion
criteria? Do you need to change the research question being addressed? Document the results/
numbers in a table. Screen the title, the abstract and, if relevant, print or obtain the paper.
Phase 3: Quality assessment.   Read the full paper and apply the quality assessment, using the
‘hierarchy of research’. Decide whether papers are IN or OUT of your review. Document the reasons
for excluding papers and compile a numerical table of the process.
Phase 4: Data extraction.   Write down the relevant data on to your pre-designed extraction
sheet. This can be handwritten or in an electronic format.
Phase 5: Synthesis.   Synthesise the data from each individual article into one. Shows what we
know now and what we still need to know. Is a meta-analysis or a mathematical synthesis feasible?
Phase 6: Write up.   Write up a balanced, impartial and comprehensive report, using a
systematic review format, presenting the process reports which will enable another researcher to
replicate your review. Disseminate to inform practice.
One of the limitations of the systematic methodology is that to do a good
systematic review takes time, resources and ideally more than one researcher.
You will note that most published systematic reviews are multi-authored.That
should not automatically deter you from doing your review using the systematic
methodology, but you do need to recognise and note the limitations. This
methodology is more appropriate for Masters-level and doctoral work. Such
a review might be called a rapid review. In the professional sphere, ‘rapid
appraisals’ are reviews of existing evidence which are not fully developed
systematic reviews. They are descriptive and can be completed in 8–12 weeks,
as advised by the Government Social Research Unit (GSRU):
08-Jesson et al-4192-Ch-07.indd 108 27/01/2011 10:09:49 AM
the systematic review
109
Rapid evidence appraisals collate descriptive outlines of the available evidence on a
topic, critically appraise them (including an economic appraisal), sift out studies of
poor quality and provide an overview of what that evidence is telling us and what is
missing from it. (GSR, 2008: 12)
The GSR process is based on a fairly comprehensive electronic search, which
can include some print materials, but it does not involve an exhaustive search.
Development of the review protocol
In Chapter 5 we described the key features of traditional review and in
Chapter 6 we gave more guidance on writing up your review. So at this stage
we assume that you have undertaken a traditional review. Use the checklist by
Hart (1998) and map your findings:
•• What are the key sources?
•• What are the key theories, concepts and ideas?
•• What are the key epistemological and ontological grounds for the discipline?
•• What are the main questions and problems that have been addressed to date?
•• How has knowledge in the topic been structured and organised?
•• How have approaches to these questions increased our understanding and knowledge?
•• What are the origins and definitions of the topic?
•• What are the major issues and debates about the topic?
Figure 7.2 shows the simple mapping (visual) of a scoping literature review for
an intervention through a community pharmacy to improve services to men,
with the workplace playing a strategic role. The interconnecting circles show
where the knowledge is; the detached circles show where there are no crossovers
or linkages – this is where the gaps are. This is an example of a scoping
Workplace health
Public health
Community
pharmacy
Government/
Pharmacy policy
Figure 7.2  Literature scoping map for a project on men’s health in community pharmacy
08-Jesson et al-4192-Ch-07.indd 109 27/01/2011 10:09:49 AM
doing your literature review
110
study. Alternatively, it could be labelled as a mapping study. A mapping study
helps you assess the size of the work, the terminology used, and the methodological,
epistemological and ontological basis of the field. In this example, there
was no literature linking workplace health and community pharmacy. By doing
this, you will also assimilate knowledge of the range of theoretical approaches.
The plan or protocol
Next, you have to draw up a plan or protocol of the proposed research to
establish the theoretical, empirical and conceptual background to your review.
You specify the research with a clear aim and objectives, state the research
question or questions, and clarify the purpose of the review. The plan helps to
establish a degree of objectivity because it is an explicit statement and
description of the steps that are to be taken. To complement it, there will be
a documented audit of your progress.
To recap, then. Before you can make an explicit plan you have to be aware
of the type of information available, the quantity and the quality.This should
have been identified during your scoping/mapping review and culminate in
the development of your systematic review question. In a limited period of
time you will be unlikely to address a complex question that requires the
detailed evaluation of thousands of published articles. Conversely, where
the topic is relatively new there will be minimal published information
available and your review will be easier to complete and relatively short.
Formulating the review question
The review question is critical to the systematic review.The question addressed
by the systematic review needs to be defined very precisely because you will
have to make a dichotomous (yes or no) decision as to whether each potentially
relevant paper will be included or rejected from your review.
Example 7.1 is a policy review problem concerning mental health and social
exclusion by Curran et al. (2007). It shows how a mapping/scoping review
question was set out.
Example 7.1
A mapping/scoping review question
What empirical evidence is available on the relationship between mental health problems
and social exclusion?
08-Jesson et al-4192-Ch-07.indd 110 27/01/2011 10:09:49 AM
the systematic review
111
Sub-questions
What is the nature of this evidence? Is it qualitative or quantitative?
Which mental health and social exclusion topics are well researched and which are not?
Which countries are the studies set in?
What research designs are used to generate the evidence? (Curran et al., 2007: 293)
More concise guidance on how to frame the question is taken from the
Magenta Book (2005). There are four components to designing a systematic
policy review question, and these are outlined in Example 7.2.
Example 7.2
Components of a policy review question
1	 Give a clear specification of the intervention, factors or processes in question.
2	 Give a clear specification of the population or subgroups in question.
3	 Give a clear specification of the outcomes that are of interest to the user.
4	 Give a clear specification of the contexts in which the questions are set.
By applying these four criteria, we can draw up a systematic review research question
about a policy intervention:
What is the effect of a personal adviser service (1. intervention) in terms of retaining
(3. outcome [a]) and advancing (3. outcome [b]) lone parents (2. population) in the UK
workforce (4. context)?
And an example about implementation:
What are the barriers (1. factors/processes [a]) and facilitating factors (1.factors/processes
[b]) to getting lone parents (2. population) to participate (3. outcome [a]) and advance
(3. outcome [b]) in the UK workforce (4. context ). (Magenta Book, 2005)
Documenting your progress
Throughout the process you have to document your decisions so that the
process is transparent to the reader and can therefore be replicated by other
researchers. The information you need is usually:
•• the title of the database
•• date searches conducted
•• years covered
•• search terms (keywords)
•• language restrictions
•• number of hits.
08-Jesson et al-4192-Ch-07.indd 111 27/01/2011 10:09:49 AM
doing your literature review
112
Documenting as you go along is crucial because you will obtain a different
set of results with every electronic search. You may want to refresh your
ideas with a review of the contents of Chapter 2 on library search. The first
table you compile will describe the search process. A summary narrative
illustration is shown in Example 7.3, which is a systematic review of barriers
to recycling in the UK. The documentation can be provided for the reader
either in summative narrative form in the report (as in Example 7.3), or in a
technical table (Table 7.3).
Example 7.3
The search report from a systematic review ‘What are the barriers to
recycling’ (Jesson and Stone, 2008)
The search began in October 2007 and continued until the final GoogleScholar search
in May 2008. A cut-off date of 2001 was deliberately chosen to reflect changes in
knowledge about recycling. A systematic search was undertaken of the computerised
databases Metalib®, ABI/INFORMS EBSCO, and SWETSWISE. The individual journal
databases searched were: Sage, Wiley Interscience online, Oxford, Taylor and Francis
Informaworld. Links within these databases to similar journal pages were followed up,
as were references at the end of each relevant paper. Once it was noted that one
journal was registering frequently, every issue of Resources, Conservation and
Recycling was scanned for the years 2001–2008 (vols 32–52), which covered our
stated timeframe.
The initial electronic search identified 522 papers which contained the words ‘barriers to
recycling UK’. Each title and abstract was screened using the inclusion/exclusion criteria.
Then paper copies were obtained and read more closely for information about barriers to
recycling. The final number of papers which met the inclusion criteria was 14. They all had
something interesting to say about current barriers to recycling.
The final second GoogleScholar search in May listed 7,090 items using the word string
‘barriers to recycling household waste in the UK’. From the first 120 scanned, five new
sources, including two conference papers, were identified.
Table 7.3  Search report table (Jesson and Stone, 2008)
GoogleScholar 7,090
e-library electronic databases 522
Potential in scope and interesting 27
In scope after reading 14
Data on current barriers 8
08-Jesson et al-4192-Ch-07.indd 112 27/01/2011 10:09:49 AM
the systematic review
113
Tip
Documenting your literature search as you go along is good practice and essential in
systematic review.
Locating studies and sources of information
Keeping to your plan and research question you can now undertake a comprehensive
search for all potentially relevant articles or studies through electronic
databases. Technical guidance and instructions on searching electronic databases
is to be found in Chapter 2. If you are getting too many hits, refine the
keywords, synonyms or related terms and do the search again. This stage may
be an iterative procedure until you have covered all keyword options.
You will have decided at the planning stage whether the study is to be based
solely on electronic sources, as some would advise, or whether you will have
to include other material. The range and types of material potentially available
were discussed in Chapter 2, but are shown in Figure 7.3.
Academic peer-reviewed articles are said to be the best source of data for
systematic review, but an electronic database search can only pick up on the
Electronic sources
Databases
Electronic libraries
Electronic Journals
Print sources
Journals
Textbooks
Hand searching
Databases e.g.
Conference proceedings
Research funders
Follow-up referenceGoogleScholar
Media items
Media items
Grey literature
Figure 7.3  Systematic searching: potential resources table
08-Jesson et al-4192-Ch-07.indd 113 27/01/2011 10:09:50 AM
doing your literature review
114
title, subject heading and abstract of the article, as the authors have written it.
The authors may not have given sufficient information for the abstract to be
picked up and some abstracts give a misleading picture of the contents of the
article. Depending on your review question, you may want to widen your
search outside the narrow confines of the electronic search to include other
methods of searching, such as manual examination of printed journals, and look
at other forms of information, such as conference proceedings or commissioned
research reports. Check with your supervisor whether this option will be
acceptable.There are specific databases in many fields where non-peer-reviewed
material is collated, such as www.wastenet.defra.gov.uk, which, in response to
the needs of the research community, is beginning to centralise relevant and
up-to-date information about waste and resources research.
The search will only be as good as the indexing of the databases you use. But
what appears in indexed databases can be just the tip of an iceberg.A Cochrane
review of 22 specialist, indexed, UK healthcare journals found that 35% of
trials identified by a hand search were not indexed by Medline.
Tip
Following up references and hand searching individual journal contents pages can
link you up with supplements, news items, and sometimes letters to the editor, which
may have additional information about other research.
The additional steps suggested in the above tip can help you to avoid selection
bias or publication bias. Sometimes it is easy to take only the more readily
accessible material, which is in the major indexed databases, but this could
defeat the aim of scientific rigour that is associated with systematic review
methodology. Remember, publication bias occurs where journals have a tendency
to promote a given approach and reject papers which have a negative
stance or produce inconclusive findings. Therefore, it can be the case that one
view predominates in the literature. A public discussion of publication bias
arose when leaked emails from climate change researchers at the University of
East Anglia was placed on the internet (Pearce, 2010).
So, to summarise the search process so far, remember that the search is one
of the standard features which distinguishes a systematic review from a traditional
literature review. The search process is more rule-driven and rigorous
than in a traditional basic literature review. There has to be an explicit statement
of the criteria that are being applied, an attempt, if possible, to cover all
08-Jesson et al-4192-Ch-07.indd 114 27/01/2011 10:09:50 AM
the systematic review
115
published material and to state whether any evidence in non-academic forms
(such as those in columns 2 and 3 of Figure 7.3) will be used.
Selecting studies: inclusion and exclusion criteria
You only want the articles that help you to answer your research question so
the inclusion and exclusion criteria you will apply must be explicit. And you
have to set these criteria at the outset, which is why you need a working
knowledge of the topic. This is illustrated in Example 7.4. The first stage of
your decision-making process is to read the title, the abstract and maybe the
introduction and conclusion of the article. You will occasionally find that articles
which appear relevant (because of words picked up by the search engine
in the title or abstract) are in fact misleading. The second stage is to screen the
papers in their entirety, scanning for the key information that you will need for
your data extraction (Phase 4 in Table 7.2). This is where the quality criteria
are applied and you sort out which papers to include and which to exclude.
Assessment involves a degree of subjectivity in the judgements made (Phase 3
in Table 7.2). That is why many systematic review guidelines stress the importance
of more than two people independently evaluating each study.
Example: 7.4
The inclusion and exclusion criteria from a systematic review ‘What are the
barriers to recycling’ (Jesson and Stone, 2008)
Keywords
The strings and combinations of keywords included:
‘household waste recycling’
‘barriers/constraints and recycling’
‘marketing and recycling’
‘recycling and attitudes/motivation/behaviour and kerbside’
Inclusion and exclusion criteria
Inclusion: English language, UK, domestic waste, household and on street/kerbside studies,
empirical evidence of barriers. The time scale was 2001–2008. Grey literature, such as reports
and non-academic research, which were identified from reference lists, and GoogleScholar,
were considered where available.
Exclusion: outside UK, Civic Amenity (CA) and bring sites only, other aspects of the waste
hierarchy – re-use and reduce, measuring participation and set-out rates, volumes of waste,
and papers published pre-2001.
08-Jesson et al-4192-Ch-07.indd 115 27/01/2011 10:09:50 AM
doing your literature review
116
You have to document the decisions you make, noting any articles that have
been rejected or that failed to meet your criteria and stating why.You may find
you will revisit these observations later. Some writers provide a flow chart
showing the numbers involved in this process.There is an example in Chapter 8,
Example 8.1. The documentation tables are to show the transparency of every
decision so that readers can see what you have done.
Appraisal – assessing the quality of research
The next stage in systematic review is the appraisal of the material that you
have selected. But how can you do that if you are a novice researcher? If your
experience is predominantly in medical, quantitative studies, how can you
assess social science or management research based on qualitative studies, or
policy-based research? In this section we discuss some of the ready-made tools
freely available from internet websites that have been designed by the systematic
review centres mentioned earlier.
A key dimension of the appraisal is to examine the methodology of primary
studies.There is often a ‘quality’ threshold applied before a study is included in the
review. A ‘hierarchy of research study designs’ is the model which is used in biomedical
research, but it also sets the standards in other fields of applied research.
This hierarchy is contestable in management and social science fields. The
quality of what is accepted as evidence varies by discipline. Medical science has
adopted the normal scientific approach, where double-blinded controlled
trails are widely accepted as the most rigorous method for testing interventions.
Thus random controlled methodology and double-blind, cross-over randomised
control trials (RCTs) are known as the gold standard, while qualitative
interviews and narrative studies have least credibility and are ranked as anecdotal.
The direction of the arrow in Figure 7.4 shows that the higher in the
table the method, the better the design quality.
But when we try to apply this judgement across other academic and policy
fields of research, such as organisational and business studies, or multidisciplinary
social science studies, the limitations of the standard hierarchy of evidence
model becomes obvious.
Task
Where in the hierarchy measure do the publications in your area of interest lie? What
does it tell you about the nature of research methodology typical in the field?
The gold standard blind randomised controlled trial is rarely used outside
clinical research. Trying to impose the randomised controlled method across
08-Jesson et al-4192-Ch-07.indd 116 27/01/2011 10:09:50 AM
the systematic review
117
organisational or business research is of dubious practical merit and ethical
acceptability, and is rarely attempted.
You will spot what constitutes evidence in your field of inquiry as you scan
the articles in the search stage. In the waste context, the Department for
Environment, Food and Rural Affairs (DEFRA) recognised the limitations of too
narrow an evidence base and widened the scope for their Waste and Resources
Evidence Strategy to include ‘more than hard data, facts, trends, survey information
but also … judgements and opinions, informal and tacit knowledge and
analytical reasoning that sets the data research in context’ (DEFRA, 2007).
You have to be pragmatic about using the quality standards. In the ‘barriers
to recycling’ study used as an example in this chapter, it soon became clear
that nearly all the studies retrieved were based on a survey method. Thus the
quality assessment had to be made on the basis of the details given by the
authors of each article about the survey design.
Task
Look at Example 7.6, Tables 7.1 and 2. This is the standard way of presenting summary
results of a systematic review. Columns 4 and 5 give methodology details.
Now, follow up the original papers yourselves and you will find there is insufficient
methodological detail to assess differences in quality for your review.
Note that if the quality criteria are applied too stringently, then you may not
have much to review. Ogilvie et al. (2005) reported in their systematic review
of public health and health promotion interventions that filtering out studies
for exclusion without examining them in any detail would have deprived the
reviewers of useful insights and evidence.
Systematic review and meta-analysis
Randomised controlled trials (RCT)
Cohort studies
Case control studies
Cross-sectional surveys
Case reports
Expert opinion
Anecdotal
Figure 7.4  The hierarchy of research study design
08-Jesson et al-4192-Ch-07.indd 117 27/01/2011 10:09:50 AM
doing your literature review
118
Example 7.6
Summary tables (Tables 1 and 2) showing key evidence from a systematic
review ‘What are the barriers to recycling?’ (Jesson and Stone, 2008)
Table 1  Articles included in the systematic review on current barriers to recycling domestic
waste
Reference Aim of research
Focus and
location Method
Non-recyclers
Numbers, reasons
given
Perrin, D. &
Barton, J. (2001)
Resources,
Conservation and
Recycling, 33:
61–74
To assess issues
associated with
transforming
household
attitudes and
opinions into
material recovery.
Comparison of
two different
kerbside
schemes.
Leeds
Bradford
Comparative case study.
Door-to-door delivery preintervention
and follow-up
postal self-completion
survey.
Total sample n = 763
Leeds n = 79
Bradford n = 14
Barriers listed
Tucker, P. &
Spiers, D. (2003)
Journal of
Environmental
Planning and
Management,
46(2): 289–307
Attitudes and
behaviour
change in
household waste
management.
Home
composting.
Scotland
Longitudinal case study.
Postal survey and
deliver/collect.
Two samples: those
taking up a bin, those not
taking a bin.
Total sample
n = 412/755
No response rate given
Non-composters not
counted.
Barriers based on
the literature and
this study.
Barriers listed
McDonald, S. &
Oates, C. (2003)
Resources,
Conservation and
Recycling, 39:
369–385
To understand
the non-recycler
better.
Reasons for
not opting-in
to take a
kerbside
container.
Sheffield
Case study.
Postal survey nonparticipants
only.
Total sample
n = 714/1690
Response rate 43%
Content analysis.
Coded barriers into
12 categories.
Barriers listed
Thomas, C.,
Yoxon, M.,
Slater, R. &
Leaman, J.
ISWA World
Congress (2003)
To explore
reasons why
people recycle
linked to a public
communications
and education
campaign.
Kerbside
provision.
London
Boroughs
Western
Riverside
Longitudinal case study.
Part one.
MORI face-to-face
interview survey of
n = 2023 and 13 focus
groups.
Segments: medium,
high, low and non-
recyclers.
Barriers listed
Williams, I.D. &
Kelly, J. (2003)
Resources,
Conservation and
Recycling, 38:
139–159
To identify
reasons for nonparticipation
in
green waste
collection.
Green waste
opt-in or optout
of a taking
a container.
Wyre,
Lancashire 	
Case study.
Two stages and two
samples.
Opt-in participants
response rate 72.5%
Opt-out response rate 49%
Non-participators
n = 611
Barriers listed
08-Jesson et al-4192-Ch-07.indd 118 27/01/2011 10:09:50 AM
the systematic review
119
Reference Aim of research
Focus and
location Method
Non-recyclers
Numbers, reasons
given
Robinson, G.M. &
Read, A.D. (2005)
Resources,
Conservation and
Recycling, 45:
70–83
To assess
kerbside and
bring site
behaviour and
promotional
activity.
Measuring
changes over
time, 2000–
2004
Royal Borough
of Kensington
and Chelsea
Longitudinal case study.
One in four household
sample, face-to-face
interviews
Samples:
2000: n = 8,066
2004: n = 3,367
Percentage nonrecyclers
drops
from 51% to 27%
Barriers listed
Smallbone, T.
(2005) Business
Strategy and the
Environment, 14:
110–122
To measure
consumer views
on household
waste and test
assumptions
underlying policy
approach.
Recycling
behaviour
England,
Scotland,
Wales
Includes a NOP national
telephone survey
Sample n = 1000
Non-recyclers 21%
Barriers listed
Table 2  Articles excluded from the systematic review and reasons
Reference Aim of research Focus and location Method
Reason for
exclusion
Davis, G. Phillips,
P.S., Read, A.D. &
Lida, Y. (2006)
Resources,
Conservation and
Recycling, 46:
115–127
Understanding
recycling
participation using
the Theory of
Planned Behaviour.
Testing theory to
create effective
targeting material.
West Oxfordshire
Survey hand
delivered to 334
houses.
Sampling: ACORN
A, DEF, part of B.
Response rate 22%
Excluded
non-recyclers
from analysis
(n = 2).
Not about
barriers
Shaw, P.J.,
Lyas, J.K.,
Maynard, S.J. &
van Vugt, M.
(2007)
Journal of
Environmental
Management, 83:
34–43
To assess kerbside
schemes using a
mathematical model
based on SOR and
PR.
To prioritise
campaigning.
London Borough of
Havering
Street observation
and Survey sample
n = 4,085.
Literature
review only
Oates, C.J. &
McDonald, S.
(2006)
Sociology, 40(3):
417–433
To investigate
recycling as
domestic labour.
Gendered division
of labour.
Sheffield
Postal selfcompletion
survey.
Sample n =
469/1,532
Response rate 31%
Not about
barriers
Table 1  (Continued)
(Continued)
08-Jesson et al-4192-Ch-07.indd 119 27/01/2011 10:09:51 AM
doing your literature review
120
Reference Aim of research Focus and location Method
Reason for
exclusion
Karousakis, K. &
Birol, E. (2008)
Journal of
Environmental
Management
(internet version)
To examine the
determinants of
household recycling
behaviour, measure
willingness to pay.
London Boroughs of
Kensington and
Chelsea, Richmond
upon Thames and
Westminster
On street interviews
Sample n = 188.
Not about
barriers
Barr, S. (2007)
Environment and
Behaviour, 39(4):
435–473
To develop and test
a conceptual
framework.
Exeter Self-completion
survey. Contact and
collect method.
Sample
n = 673/981
Response rate 69%
Literature review
and theory.
Not about
barriers
There are a range of checklists that you can draw on to appraise the quality of
non-clinical research. For example, we have provided the COREQ 32 checklist
(Tong et al., 2007) in Appendix 2 (and see below). This checklist was written
as formal reporting guidelines to help authors and journal reviewers improve
the quality of work that is submitted to and published in medical journals, but
the ideas are transferable to other disciplines.
Some researchers have found that the hierarchy of evidence model limits
their choice and have devised a personalised quality hierarchy relevant to the
topic or field. For example, in a meta-analysis of a study on stigma and mental
health, quality was assessed along four dimensions: theory, publication bias,
research design and sources of heterogeneity (Mak et al., 2007). Unlike the
hierarchy of evidence which measures research design quality, Table 7.4
shows how to measure the quality of publication. Publication bias assesses
external validity, or the extent to which the results can be generalised to the
population.
Assessing quantitative studies
The previous section described the appraisal of study design using the hierarchy
of evidence model and a customised four-dimensional quality appraisal
model. The usual way to assess the quality of an individual study is to examine
key features of the article or report. The following list is just some of the key
dimensions that you could use as a starting point:
Table 2  (Continued)
08-Jesson et al-4192-Ch-07.indd 120 27/01/2011 10:09:51 AM
the systematic review
121
The introduction
Are the aim and objectives of the study clear?
Why was the study undertaken? (Known as the rationale)
Why now, in this context?
Is there a link to theory?
Method
What is the research design?
Is there detail about the sampling frame, how and why the sample was
selected?
Data
What types of data are there?
How and where, and by whom was the data produced?
How trustworthy, reliable and valid is the data?
Analysis
How was the data analysed?
How rigorous and trustworthy is the analysis?
Table 7.4  Variables in assessing study quality, examining external validity
Dimension of quality assessment Components and operational
definitions
Measure
Type of review process Has the paper undergone peer
review?
·   Peer reviewed
·  Not peer reviewed
(grey)
Publication type Where is the article published?
This list may vary according to
the protocol
·   Academic journal
·   Professional journal
·   Book or book chapter
·   Doctoral dissertation
Publication date When was the article
published?
·   Before 2000
·   2000 and later
Journal impact factor, for prestige,
where 5 is highest
What is the current impact
factor of the journal in which
the paper is published?
·   0–1.0
·   1.1–2.0
·   2.1–3.0
·   3.1–4.0
·   4.1–5.0
Source: Adapted from Mak et al. (2007)
08-Jesson et al-4192-Ch-07.indd 121 27/01/2011 10:09:51 AM
doing your literature review
122
Results
Are the results a true representation of the data?
Do the results relate back to the research question?
Do the authors discuss the methodological limitations of their study?
Assessing qualitative studies
When it comes to assessing or appraising qualitative research there are several
published guidelines to help you. Tong et al. (2007) undertook a systematic
review of the many guidelines that have evolved to assess the quality of qualitative
studies, covering in-depth interviews and focus group techniques, to
produce The Consolidated Criteria for Reporting Qualitative Studies
(COREQ). This is a 32-item checklist based on 76 items from 22 checklists
which you might find useful (see Appendix 2). The comprehensive 32-item
list can be adapted to suit your research field.
Assessing management or organisational studies
Systematic reviews have traditionally been applied in fields of research where
positivist and quantitative approaches are dominant. Consequently, far less has
been written about how to do a systematic review in some other fields of
research, such as management. As mentioned earlier, Tranfield et al. (2003)
explain that management research is a comparatively young field of inquiry,
which is far less well developed than medical science. Nevertheless, there are
a number of original reviews being undertaken by groups of researchers in
several fields of inquiry that are being published in the International Journal of
Management Reviews. There is not, as yet, a specific or appropriate published
checklist to assess management research, so for your own review read the
existing criteria list and then draw up your own criteria for assessing your
specific field of study.
Data extraction
Now you have retrieved your articles, there will be two piles of papers in front
of you (if you have printed them off), or two bibliography lists if you prefer
that approach. One pile IN and one pile OUT.You have documented the process
so far, and accounted and explained why the articles are in each pile. The
next stage is to extract the relevant data from the articles in the IN pile.
Every researcher has his or her own favourite way of highlighting key
aspects from articles; some insights were given in Chapters 5 and 6. One way
to start, probably the old-fashioned way, is to highlight sections of the paper
08-Jesson et al-4192-Ch-07.indd 122 27/01/2011 10:09:51 AM
the systematic review
123
with highlighter pens, and then enter the relevant information on to your data
extraction form. This might be another dynamic phase where you revise the
format of what you want to record. You may want to revisit the scoping and
mapping phase, or revise your inclusion and exclusion criteria.
Example 7.5
The data extraction form from a systematic review ‘What are the barriers to
recycling’ (Jesson and Stone, 2008)
The data extraction form has to reflect the question and planned assessment. It is another
visual record of the decisions you have made. It will include the following details:
(a)	 Author and publication details (bibliographic details)
(b)	 Paradigm (academic discipline)
(c)	 Aim and focus of the paper
(d)	 Method details (sample selection, size, method design, response rate, location of
the study)
(e)	 Theory or models
(f)	 Data about barriers to recycling (either as a literature review/summary or numbers
of non-recyclers or a list of new reasons or barriers)
(g)	 Segmentation
(h)	 Other relevant or useful information
You will soon have several completed data extraction forms. You can then
move on to the next phase and undertake your analysis and synthesis.
Synthesis, drawing conclusions, what the review shows
The analysis and synthesis is probably the most intellectually taxing phase of
the systematic review process. Hart (1998: 110) defined synthesis as ‘the act of
making connections between the parts. It is not simply a matter of re-assembling
them back into the original order but of finding a new order.’ In the data extraction
stage you unpacked each article. In the synthesis stage you have to put them
all together again, but this time telling a new story or making new connections.
That is your contribution to knowledge, or filling the knowledge gap.
There is no single agreed way of synthesising the evidence; it will depend on
the type of review and subject matter. The aim is to collate and present the
extracted data from primary studies so that the characteristics and results of
the study are summarised in a meaningful way.
08-Jesson et al-4192-Ch-07.indd 123 27/01/2011 10:09:51 AM
doing your literature review
124
There are two components to synthesis: (1) the story, and (2) the table
(the evidence). Descriptive or non-quantified studies involve narrative text
and tabulation to present the study characteristics and results – in essence a
short summary of findings. You will not need to tabulate all the data. Select
the important points which help to answer your question, and structure them
to highlight the similarities and differences between the included studies.
Tabulation allows the reader to scan down the columns and see where the
similarities and differences are. The second component, where it may be possible
in some studies to quantify synthesis by using statistics, is described in
Chapter 8.
Evolving formats of systematic review
So far this chapter has described the process used to produce a systematic
review. This process is ‘protocol’-driven and follows a systematic methodology.
We have also observed that this methodology was introduced to deal with
biomedical and healthcare research studies. Business, organisational and social
science researchers have been slower to adapt it for their requirements. Some
researchers argue that systematic review is not appropriate for policy and management
decisions, and you may have come to that conclusion for yourself by
this point. For those who are still curious, carry on reading.
The controversial big question for reviewers has been ‘Is it feasible to combine
the findings of research studies that use different methods? The following
section briefly introduces some of the more recent innovations and adaptations
to the systematic methodology. These are more advanced, complex and specialised
review methodologies, which may be useful for doctoral students.
Systematic reviews of qualitative and quantitative research
To date, there is no single agreed framework for synthesising both qualitative
and quantitative data. A challenging contribution by Mays et al. (2005)
explores this issue in one of several articles in the special edition on synthesising
evidence in the Journal of Health Services Research and Policy, supplement
to issue 3 (2005). Mays et al. (2005) proposed four basic approaches to synthesising
both qualitative and quantitative types of data to inform policy making
in the field of health. The four approaches are:
•• Narrative, which includes traditional literature reviews, thematic analysis, narrative
synthesis, realist synthesis and meta-narrative mapping.
•• Qualitative, which converts all available evidence into a qualitative form using
techniques such as meta-ethnography and qualitative cross-case analysis.
08-Jesson et al-4192-Ch-07.indd 124 27/01/2011 10:09:51 AM
the systematic review
125
•• Quantitative, which converts all evidence into a quantitative form using techniques
such as quantitative case survey or content analysis.
•• Bayesian meta-analysis and decision analysis.
In 2005, when the journal supplement was published, non-biomedical systematic
reviews were in their early stages of development. Two examples of
more recently published articles are presented in Example 7.7 to illustrate the
methodological approaches.
Example 7.7
A meta-narrative mapping analysis (Collins and Hayes, 2010)
Met-analysis mapping is defined as the process of plotting how a particular research tradition
has unfolded over time. It is an approach which combines the analytical dimensions of
traditional narrative research with the comprehensiveness and rigour pursued in systematic
literature reviews (Collins and Hages, 2010: 10). This is said to be a useful technique in
the synthesis of vast and complex evidence bases to inform policy processes.
The aim of this review was to monitor thematic trends in the health inequalities knowledge
base over time and to map local government intervention on local health inequalities. The
reviewers searched for four bodies of knowledge: health promotion, Healthy Cities, population
health and urban health, covering a 20 year timeframe 1986–2006, using only abstracts
(therefore text as written, qualitative data). The timeframe represents the evolution of health
determinants research. 1004 abstracts were reviewed. The lens applied was Canadian.
Three electronic databases were searched. The result is a bibliographic report, which
describes the detail of each article, showing the quantitative changes over the timeframe and
the qualitative change in emphasis of topics and prescriptions for government intervention.
There are some points to note about this review and to show how you can use
articles to generate research questions. First, it is based entirely on abstracts.
Hopefully you will now be aware of some of the limitations associated with
abstracts, so apply your critical lens to the paper. Second, note that the search
ended in 2006 and the article was published in 2010, reflecting the time to
carry out the search, review and synthesis, and then add on more time to write
up the article and finally get it published. So, third, it would be perfectly possible
to take this review as a starting point for a new review and see what has
been published on the topic since that 2006 cut-off date. Or it could be possible
to widen the databases search, maybe applying a European lens or possibly
including grey literature.
Further specialist advice on thematic analysis is freely available online at the
ESRC National Centre for Research Methods.
08-Jesson et al-4192-Ch-07.indd 125 27/01/2011 10:09:51 AM
doing your literature review
126
Grey literature in systematic reviews
One of the obstacles that policy researchers have identified with the systematic
review methodology is that adopting a method used successfully in the
clinical biomedical sciences is not automatically applicable in other fields of
knowledge.A key issue is that in social sciences and policy research much of the
useful knowledge is contained in non-academic, non-peer-reviewed journals, in
so-called grey literature (Curran et al., 2007; Collins and Hayes, 2010).
The reason this is a problem is that grey literature is at the bottom of the
academically acceptable resources table (Table 7.3) and does not register at all
in the hierarchy of research study design (Table 7.4). Another limitation is in
actually identifying this type of resource because academic electronic databases
do not pick them up. This is where a working knowledge of the topic is essential,
or maybe your information specialist in the library can help. If you do not
access grey literature you are excluding that valuable information from service
users, charitable organisations, think tanks, and so on.This is essentially an argument
about what counts as knowledge in your field. The final obstacle is how
to analyse, synthesise and incorporate the material you are allowed to use.
Umbrella reviews
Systematic reviews have been completed on many subjects and we are
approaching the stage when it is possible to analyse the systematic reviews
themselves as a body of knowledge. These are known as umbrella reviews. An
umbrella review is a way of identifying and appraising and synthesising existing
systematic review evidence. As such they are able to present the overarching
findings of existing systematic reviews (see Example 7.8).
Example 7.8
Tackling the wider social determinants of health and health inequalities:
evidence from systematic reviews (Bambra et al., 2009)
A systematic review methodology was used to locate and evaluate published and unpublished
[therefore grey literature] systematic reviews of interventions around the ‘wider social
determinants of health’ (Bambra et al., 2009). The authors set the context of an increasing
policy emphasis on tackling the ‘wider social determinants of health’ through the implementation
of appropriate interventions. So the aim of the review (or to put it another way, the
research question) was to identify what is already known and to highlight areas for further
development. The review identified 30 relevant systematic reviews and concluded that the
effects of interventions on health inequalities were unclear.
08-Jesson et al-4192-Ch-07.indd 126 27/01/2011 10:09:51 AM
the systematic review
127
So, what might the limitations be of a review of reviews? Without reading each
review, and then reading each article or report within each review, the reader
is relying on the skills, knowledge and expertise of Bambra and colleagues to
be reliable interpreters of qualitative, quantitative and grey literature. So is the
review still as reliable as a protocol-driven methodology would suggest it
should be?
Summary
This chapter has given guidance on how to undertake a systematic literature review
using a defined protocol, where the data is fairly homogeneous. We concluded by
considering whether it is feasible to undertake a systematic review with data that are
not homogeneous or with grey material that may not be acceptable to your academic
institution. The answer is left open because this is an ongoing debate. Finally, some
observations about objectivity and bias are necessary. It would be irresponsible to
pretend that this methodology is entirely free from bias. As researchers, we make
judgements at every stage of the review process. Occasionally, some bias may be
unconsciously made as a result of the ideological lens through which we read each
article. Then we need to make a decision on its quality and relevance.
Remember, systematic review is a question-driven methodology. If you do not have a
specific question, you should probably be doing a traditional review. If you are
reviewing research with quantitative data, then meta-analysis could be the way to
synthesise the data, and that is the subject of Chapter 8.
08-Jesson et al-4192-Ch-07.indd 127 27/01/2011 10:09:51 AM