M A S A R Y K O V A U N I V E R Z I T A
F A K U L T A I N F O R M A T I K Y
Factual Knowledge Adaptive
Learning System for
Geography
D I P L O M A T H E S I S
Vít Stanislav
B r n o , s p r i n g 2015
Declaration
Hereby I declare, that this paper is m y original authorial w o r k , w h i c h I
have w o r k e d out by m y o w n . A l l sources, references and literature used or
excerpted during elaboration of this w o r k are properly cited and listed i n
complete reference to the due source.
Advisor: doc. M g r . Radek Pelánek, P h . D .
ii
Acknowledgement
I w o u l d like to thank m y supervisor, doc. M g r . Radek Pelanek, Ph.D., for his
guidance, enthusiasm a n d valuable advices. I thank to m y colleagues from
Adaptive Learning G r o u p (Jan Papousek i n particular) for their enjoyable
cooperation o n development a n d insightful discussions. I thank to all w h o
tested the developed system a n d p r o v i d e d some feedback.
iii
Abstract
This thesis deals w i t h design, implementation, deployment and usage
analysis of a factual knowledge adaptive learning system for geography s
l e p e m a p y . c z . The thesis puts the system into the context of systems for
factual knowledge learning (flashcard software) and systems for learning
geography. It also describes m a i n system parts and explains their function.
The system is implemented as a web application w i t h client-server architecture.
Django framework is used o n server and AngularJS framework on
client. The implemented web system is w i d e l y used and is a source of a b i g
data set used for research.
iv
Keywords
adaptive practice, online education, geography, outline maps, flashcard
software, Python, JavaScript
v
Contents
Contents 1
1 Introduction 2
2 Similar Systems 4
2.1 Flashcard Software 4
2.2 Geography Games 8
3 System Functionality 11
3.1 Server-side Functionality 11
3.2 Client-side Functionality 15
4 Implementation 21
4.1 Project Timeline 21
4.2 Used Technologies 22
4.3 Map-related Functionality 26
4.4 Other Functionality 28
5 System Usage Analysis 31
5.1 Used Technologies 31
5.2 Users of the System 32
5.3 User Registration A/B Experiments 36
5.4 User Feedback 38
6 Subsequent Related Work 41
6.1 Derived Systems 41
6.2 Research on the Collected Data 43
7 Conclusion 45
Bibliography 46
Appendices 49
A Source code 49
B Visualizations 50
1
Chapter 1
Introduction
In recent decades computers started to occupy more and more areas of
h u m a n activities and education is no exception. There are many different
forms of e-learning, such as learning management systems, (intelligent)
tutoring systems, massive open online course platforms or educational
games. Students can acquire knowledge i n various forms and formats,
ranging from text and images through audio and video to highly i n teractive
applications or games.
In some cases students can also practice the knowledge and get feedback
o n their performance. Computers can easily give feedback o n some
tasks, such as learning elementary mathematics operations or learning foreign
language vocabulary, whereas i n the case of more creative tasks h u m a n
feedback is still superior to feedback generated by computer (e.g. evaluation
of an essay or an advanced mathematics proof). However, i n the areas
where computer can evaluate students' knowledge, computer outperforms
h u m a n i n being more patient and available at any time. Computer can also
recommend material for further study and provide learning that is customized
to an individual student's needs, as opposed to one-pace-fits-all
learning style i n traditional classroom.
A d a p t i n g to individual students is an important feature of the learning
system developed i n this thesis. The system allows students to practice association
between a feature on a map (e.g. a country) and its name. The system
is developed i n collaboration w i t h another student, Jan Papoušek, w h o
is responsible for developing server-side features (i.e. student model and
question selection algorithm), while I a m responsible for user-facing features
and preparing the maps.
Te.g. imary goal of this thesis is to develop part of the system as a web
application. Other goals of the thesis are: to provide an overview of relevant
similar systems and their underlying principles, to analyze the data col-
2
1. I N T R O D U C T I O N
lected by the implemented system, and to outline modifications of the i m plemented
system for use i n other areas.
The thesis consists of 7 chapters. Chapter 2 describes various features of
systems similar to the developed system. Chapter 3 describes functionality
of the whole system, that is regardless of w h o implemented it. Chapter 4
describes particular issues of author's part of implementation. In Chapter
5 are described some experiments w i t h the data collected by the system.
Chapter 6 mentions other research and development that was done thanks
to the implemented system.
3
Chapter 2
Similar Systems
The proposed system is essentially a combination of two c o m m o n types of
applications: flashcard software and geography games. This chapter contains
a detailed description of the important features of both types of appli-
cations.
2.1 Flashcard Software
Flashcard software is a computerized version of physical (paper) flashcards.
The basic idea of paper flashcards is that students write a small chunk of
information that they want to learn o n one side of a flashcard and a cue
to recall the information o n the other side (e.g. w h e n learning Spanish vocabulary:
"una mujer" / "a woman"). Then they learn by trying to recall
one side while looking at the other and then checking if they were right by
flipping the flashcard.
There are many different flashcard applications available; however they
all share the following basic use case:
1. User chooses a topic to learn and creates or selects a set of flashcards
covering the topic.
2. User practices the flashcards, i.e. user is asked one side of a flashcard
and is supposed to recall the content of the other side.
3. After answering, the flashcard is scheduled for next practice. The i n terval
between the current and next practice is based of the firmness
of user's knowledge of the flashcard content.
In the rest of this section are described c o m m o n features of flashcard
software illustrated o n sample systems listed i n Table 2.1.
4
2. S I M I L A R S Y S T E M S
N a m e U R L
A n k i h t t p : / / a n k i s r s . n e t /
IS M U N I D r i l h t t p s : / / i s . m u n i . c z / a u t h / d r i l /
Memrise h t t p : / / w w w . m e m r i s e . c o m /
Cerego h t t p : / / c e r e g o . c o m /
Table 2.1: Sample flashcard software
Practicing Flashcards
The general w o r k f l o w of practicing flashcards is shared among all the flashcard
systems; however, the systems w a r y i n some details of the practice. In
some systems it is u p to the users to evaluate their knowledge of the other
side ( A n k i [8] and IS M U N I D r i l [4]). Then they choose to see the other side
of the flashcard and select one of the options (4 i n A n k i , 6 i n Dril) describing
their level of knowledge (ranging from "Very p o o r " to "Very good").
The system schedules next practice of the flashcard based on the selected
option. This is a w o r k f l o w that can also be used w i t h paper flashcards,
w h i c h is not the case of some other systems.
In other systems (Memrise and Cerego), the user is required to type
the answer (or choose it from several options) and the system evaluates its
correctness. The biggest disadvantage of this approach is that typing the answer
takes more time than saying it (especially w h e n typing o n a smartphone).
O n the other h a n d , it is likely to measure knowledge more accurately
as users tend to overestimate their knowledge w h e n assessing themselves
[6].
A n important part of flashcard software is a scheduling algorithm.
The algorithm determines w h e n it is the best time to practice a flashcard
again. It is usually based o n w e l l studied phenomena such as spacing and
forgetting effects [27] and spaced repetition [14]. Considering these phenomena
leads to increasing the interval between two consequent practices
of a flashcard. H o w large is the increase depends o n h o w firm is the user's
knowledge thought to be.
For example, learning algorithm of Cerego consists of learning m o d ule
and review module [29]. The learning module asks students repeatedly
questions about a small group of items for as many times as it is necessary
to make the correct responses automatic (response time around 500 ms).
5
2. S I M I L A R S Y S T E M S
Then the review module schedules the spacing of reviews of the items to
maximize spacing effect.
Feedback for Users
There are various approaches to provide users w i t h information about their
learning progress. Users might want to k n o w " h o w many items do they
k n o w " , " h o w w e l l do they k n o w the items" or " h o w m u c h time d i d they
spend o n studying". This information can help users choose what to learn
next and motivate them to continue learning.
A n k i is capable of displaying various graphs of h o w many cards have
been learned. Most of them are bar charts w i t h bar per day over some period
of time, where bars represent number of different states of cards or
time spent o n learning (e.g. see Figure 2.1). There also is a pie chart depicting
the division of cards among the possible states (learn, young, mature or
relearn, suspended, unseen).
Figure 2.1: A n k i Review Count: The number of answered questions
Memrise visualizes users' knowledge as plants (see Figure 2.2). A t
the begining of learning a plant is planted; then the knowledge/plant has
to be revised/watered i n certain time intervals i n order to bloom, otherwise
the knowledge/plant withers. This metaphor allows Memrise to visualize
the states of the plants; the same visualizations are used for groups of cards
called levels (levels are parts of a course). Memrise also displays progress
bars of learned words i n a level or a course.
Cerego visualizes user's knowledge by plotting the items on a 2 D scatter
plot (see Figure 2.3), where vertical axis roughly represents short-term
memory and horizontal axis long-term memory. After practicing an item
6
2. S I M I L A R S Y S T E M S
Figure 2.2: Memrise knowledge visualization
it moves to the top and slightly to the right. Over time (without practice)
the item moves d o w n . Horizontally every item begins as "unstarted" on
the left; after each practice it moves to the right (through level 1 to 4);
The goal is to reach the "Mastery zone" o n the right.
Level 1 Level 2 Level 3 Level 4
e
o
Z
• QG>
o
u_ o °
o
$
1• • Vrr
^ ^ ^ ^ ^ ^ ^Ui
z
Figure 2.3: Cerego knowledge visualization
Memory Encoding
In order to improve memory encoding of the learned items, various additional
information can be provided w i t h the learned items. It might
be examples, interesting facts, amusing associations or mnemonics. Some
systems provide very good support for memory encoding enhancement
(e.g. Memrise), some systems allow this to be done w i t h more general features
(e.g. Anki) and other do not provide any tools for for this at all (e.g. IS
M U N I Dril).
For example, Memrise calls its memory encoding enhancement feature
'mems" [19] and provides tools to c r o w d source them. More specifically,
7
2. S I M I L A R S Y S T E M S
every user can create mems and vote o n the mems created by others. W h e n
practicing an item for the first time, user is assigned the highest voted m e m
(or can choose any other mem). This approach (as any other use of c r o w d
sourcing) requires many users to be studying the same course, since only
a small fraction of users is w i l l i n g to spend time on creating mems. H o w ever,
i n m y experience, high quality mems can be a great help for example
w h e n learning Chinese characters.
Even if a system does not support mems explicitly, there are often some
ways around it. For example, A n k i allows creation of more than two-sided
cards. Three-sided cards can be defined such that the third side is never
a part of the cue but always a part of the answer, while first and second side
can be cue and answer i n either direction. That is exactly what is needed for
a mnemonic.
Flashcards for Learning Maps
Most flashcard systems support flashcards w i t h images, w h i c h makes
the systems capable of being used for learning locations of countries i n
maps. A Flashcard has to contain an image of an outline map w i t h one item
highlighted i n a different color. For example, for A n k i there is a shared deck
"Countries of the W o r l d " 1
and many others; Memrise also offers m a n y map
courses 2
; last but not least, Cerego offers "Countries of E u r o p e "3
and many
others.
Static images are a reasonable w a y of practicing countries i n a general
purpose learning system; however, a system w i t h native map support
can offer better capabilities. In particular, the system can ask questions
like " F r o m all the countries o n the map choose M o n g o l i a " (as opposed to
" W h i c h of these four pictures represents Mongolia?") and evaluate the an-
swer.
2.2 Geography Games
There are m a n y applications for practicing maps; however, they usually
pay no attention to question selection and ask the questions either i n fixed
order or at random. These games also do not save answers and thus users'
1. h t t p s : / / a n k i w e b . n e t / s h a r e d / i n f o / 2 9 1 5 3 3 2 3 9 2
2. http ://www.memrise.com/courses/english/maps/
3. h t t p s : / / c e r e g o . c o m / s e t s / 7 2 1 5 5 9
8
2. S I M I L A R S Y S T E M S
knowledge cannot be tracked. This section describes three sample geography
games listed i n Table 2.2.
N a m e U R L
Sheppard Software h t t p : / / w w w . s h e p p a r d s o f t w a r e . c o m /
geography games G e o g r a p h y . h t m
World Geography h t t p : / / w o r l d - g e o g r a p h y - g a m e s . c o m /
Games c a p i t a l s _ e u r o p e . s w f
Digital Dialects h t t p : / / w w w . d i g i t a l d i a l e c t s . c o m /
geography games g e o g r a p h y _ g a m e s . h t m
Table 2.2: Sample geography games
A typical example is geography games on Sheppard Software. For most
maps they offer three types of exercises:
• Level 1 - click o n any item to learn its name,
• Level 2 - click o n the item called X,
• Level 3 - drag the outlines from a stack and drop them to their locations
o n the map (easy w h e n the coastline matches part of the shape).
Another example of geography game is W o r l d Geography Games. The
questions are of "click o n the item called X " type. After an incorrect answer
the user has to try again as there is no option to skip the question, w h i c h
can be very frustrating. The answered items are labeled o n the map and
the labels persist till the end of the practice. Users can practice countries,
capitals, flags and 16 other map types (e.g. lakes, rivers or mountains).
A less c o m m o n approach to practice is used i n geography games at D i g ital
Dialects. The type of questions is also "click o n the item called X " , but
the questions are asked i n triplets. In each triplet the first question has three
options (highlighted among all items); the other two options remain for
the second question and the third question has only one option left (see
Figure 2.4). After answering all tree questions the answers are evaluated
and user can proceed to the next triplet of question. Another special feature
of this system are buttons next to the items to click o n instead of clicking on
the items (e.g. countries) themselves. It is a reasonable solution for too small
items, but it is used for all items, w h i c h seems as if it was chosen because it
was easier to implement.
9
2. S I M I L A R S Y S T E M S
Figure 2.4: Digital Dialects geography games - Countries of Asia.
1 0
Chapter 3
System Functionality
This chapter describes system capabilities; it focuses o n introducing
the m a i n ideas a n d avoids implementation details. The system developed
in this thesis differs from the aforementioned flashcard systems mainly i n
two aspects:
• The system estimates global difficulty of each item across all users
and uses it to estimate users' prior knowledge of items.
• One side of a flashcard is an element i n an S V G image (as a better
alternative to pre-rendered raster images uploaded to a general purpose
flashcard system).
The system is divided into several modules a n d uses client-server architecture
(see Figure 3.1). The server-side module "Skills/difficulty estimation"
estimates difficulty of items a n d skills of users based o n previous
answers. The estimates (together w i t h other information about previous answers)
are used b y the "Question construction" module to choose appropriate
questions for "Practice". The estimates of users' skills are also presented
to users i n " K n o w l e d g e m a p " a n d " M a p s overview".
3.1 Server-side Functionality
The description of the server-side modules, i.e. knowledge estimation a n d
question selection, (see Figure 3.1) is given i n E D M paper " A d a p t i v e Practice
of Facts i n Domains w i t h Varied Prior K n o w l e d g e " [25]; therefore, this
thesis contains only m a i n points a n d is based o n the paper.
Skills/Difficulties Estimation
Difficulties a n d skills are estimated b y a variation of Elo rating system [9],
w h i c h w a s originally created for rating chess players. In Elo has each player
11
3. S Y S T E M F U N C T I O N A L I T Y
Figure 3.1: System Architecture.
i associated a number of rating points Bi w h i c h describe their skill (higher Bi
means higher skill). After a player i plays a match, Bi is updated according
to the result R of the match (R = 1/R = 0) according to the following
formula:
Bi :=9i + K(R-P(R=l))
where P(R = 1) = 1/(1 + e~(6i
~6
^) is probability that the user wins
the match and K is a constant w h i c h determines h o w big influence has
the latest match. K can also be a function that decreases w i t h increasing
number of matches played by the player i - the more data there is about
the player the smaller adjustments player's skill needs.
In the developed system is the Elo system used i n the following way.
W h e n a student answers a question about an item it is considered to be
a match between the student and the item and is evaluated as described i n
Elo. Skill/difficulty estimation module computes the following real values:
• Difficulty of an item - computed from the first answer of each student;
decreased by correct answers and increased by incorrect.
• Global skill of a student - computed from the first answer o n each
item; increased by correct answers and decreased by incorrect.
12
3 . S Y S T E M F U N C T I O N A L I T Y
• Local skill of a given student on a given item. It is initialized w i t h
the difference between student's global skill and item difficulty and
modified by subsequent answers.
Local skill can be transformed v i a a sigmoid function to probability of
correct answer, w h i c h is used i n the questions construction module.
Question Construction
Questions construction module has to decide the following aspects of constructing
a question.
• W h i c h item is the most useful to be practiced at the moment.
• Whether to make the question multiple choice or open answer (i.e. all
the items o n the current map are possible answers)
• The direction of the question, i.e. "Choose item o n the m a p . " or " W h a t
is the name of the highlighted item?".
• H o w m a n y choices does the question have (2 to 6) i n the case of m u l tiple
choice question.
• W h i c h items are the alternatives i n the case of multiple-choice ques-
tion.
The item to be practiced is selected among the available items by a scoring
function. The scoring function has three parts:
• predicted probability of correct answers - the system has a target
probability (e.g. 75%) and items far away from the target probability
are penalized the most,
• time since last question about the same item - to avoid practicing
the item w h e n still i n short-term memory,
• number of answers for the given user and item - to ensure that each
item w i l l be practiced eventually.
The choice of target probability has been subject to some research based
on the developed system [24]. Too l o w target probability is suboptimal as
it demotivates users. O n the other hand, too high target probability leaves
1 3
3. S Y S T E M F U N C T I O N A L I T Y
fewer space for learning. The probability of correct answer can be increased
by decreasing number of choices, w h i c h is used w h e n there is no item w i t h
target probability.
Alternative Options Selection
Another important part of the question construction is the selection of
the choices of a multiple-choice question. The naive w a y is to select
the choices at random. That is suboptimal as such questions have highly
variable difficulty. It might result i n questions that can be answered correctly
without the knowledge of the asked item, e.g. a question about A n gola
w i t h alternatives Peru, Kazakhstan and M o l d o v a .
The choices that do not require specific information about an item to be
retrieved are called non-competitive (as opposed to competitive). Impact of
these two types of choices o n retention have been studied i n controlled experiments
[18], w h i c h conclude that competitive choices result i n same retention
of the asked item as non-competitive; however, they result i n higher
retention of the item i n a competitive alternative choice.
To create more competitive choices the system can prefer closer countries
as choices. A possible w a y to determine closeness of two countries
is to use a list countries and their bordering countries. U s i n g only direct
neighbors results i n very easy to guess questions as the correct answers is
usually i n the middle. A method w i t h decent results is to also use neighbors
of neighbors. U s i n g neighbors, however, cannot be applied to cities or
mountains.
A more general approach (currently used i n the system) is to prefer
items that users used as incorrect answers to the item i n open questions.
For each pair of items a confusion factor is defined as the number of times
the first item has been confused w i t h the other one. The options are chosen
randomly among 8 items w i t h highest confusion factor; the randomness is
weighted by the confusion factor (items w i t h higher confusion factor are
more likely to be chosen). For example, Cameroon is most often confused
w i t h Niger (38%), Nigeria (27%), Central African Republic (10%), Republic
of the Congo (9%), G a b o n (6%), Ivory Coast (5%), U g a n d a (3%), and Guinea
(2%) [25].
A disadvantage of the currently used approach is that b y seeing
a multiple-choice question w i t h the same alternatives several times stu-
14
3. S Y S T E M F U N C T I O N A L I T Y
dents can create a n association between the alternatives and later be more
likely to confuse the items [20]. This disadvantage could be prevented b y
choosing non-competitive alternatives w h e n the skill of the asked item is
low. I w o u l d also suggest to avoid asking questions w i t h two (maybe even
three) alternatives as they have high probability of guess and thus are less
useful for skill estimation. The decreased probability of correct answer can
be compensated by choosing alternatives w i t h lower difficulty.
3.2 Client-side Functionality
The essential parts of the client are the practice page, knowledge map and
maps overview (see Figure 3.1). Other features described i n this section i n clude
mnemonics, personal goals and difficulty rating poll.
Practice Page
What is the name of
state highlighted on
the map?
• Belarus
Lithuania
_ Poland
O Doni know
•+ Continue
C Highlight again
Figure 3.2: Practice screenshot.
Practice is the m a i n use case of the whole system. User selects a map
and an item type to practice (e.g. countries of Europe - Figure 3.2). The system
asks user questions about position of the selected type of items. After
answering a question user gets immediate feedback o n the correctness of
the answer. If the answer is correct, the next question follows automatically
in about half a second; whereas after incorrect answer users are given time
15
3. S Y S T E M F U N C T I O N A L I T Y
to reflect o n their mistake and have to explicitly click a button to continue
(when they are ready for the next question).
After a set of ten questions a summary page is displayed. It contains correctly
answered items highlighted i n green and incorrectly answered items
in red. Summary page also presents success rate progress bar and buttons
for further action (view knowledge map or practice again).
Knowledge Map
Knowledge map page provides users w i t h information about their k n o w l edge
of a given map. A n essential part of the page is a map (see Figure 3.3)
w i t h the items colored i n different colors from green-to-red color scale (intuitively
green stands for high level of knowledge, red for poor). The scale
is defined by five colors (red and green o n the far ends and yellow and
two shades of orange i n the middle). In order to express the change of estimated
knowledge, the color can be changed either continuously or i n discrete
steps (e.g. i n 5 or 10 steps).
It is not clear w h i c h of the two possibilities provides users w i t h better
feedback o n their knowledge. In m y opinion, i n the continuous case there
is too m u c h information and it is impossible for user to distinguish slight
changes i n colors after practicing a map, thus it is better to use discrete
Figure 3.3: K n o w l e d g e map of African countries.
16
3. S Y S T E M F U N C T I O N A L I T Y
colors, w h i c h show only bigger changes i n estimated knowledge. Discrete
colors also have a " f i n a l " green color, w h i c h can express achieving mastery
(as i n mastery learning [3]).
Another aspect of the knowledge map to be determined is its initial state
(prior to any practice). O n one h a n d , each item can be initially colored according
to user's prior knowledge of the item. This way, it is likely that user
begins w i t h most items i n red, w h i c h is, i n m y opinion, demotivating. O n
the other hand, items can begin as unpracticed (white) and only after being
practiced for the first time obtain a color according to user's local skill on
the given item. A disadvantage of the second approach is that items w i t h
high prior skill are unlikely to be practiced and therefore can remain white
even w h e n the most of the items are already practiced or even learned,
w h i c h again can be frustrating to the user w h o wants to make the whole
map green.
One w a y to overcome the aforementioned disadvantages is to color
the high prior skill items from the beginning and the rest of the items only
after they have been practiced. This can also be confusing to users, but it is,
in m y opinion, the best options w i t h the given question selection algorithm.
Another option is to change the question selection algorithm so that it selects
the high prior skill items to be practiced at least once early i n the practice
(e.g. one third of questions can be about unpracticed high prior skill
items for as long as there are any such items).
O n the knowledge map page, user can switch between three m a i n feature
categories (Political map, Water and Surface). In each of these categories
there are several feature types (e.g. i n Political map there are States
and Cities), each of w h i c h can be hidden. The three m a i n categories have
been chosen so as to equally distribute feature types among them. A more
equal category to "Political m a p " w o u l d probably be "Physical m a p " , but
that w o u l d result i n too many feature types i n "Physical m a p " , w h i c h
w o u l d be too cluttered.
Maps Overview
With a higher number of maps i n the system it was necessary to create
a page s u m m a r i z i n g all the available content. It has been created as a list
of all maps w i t h links to view user's knowledge maps and buttons to practice
them (see Figure 3.4). Below each button there is a progress bar show-
1 7
3 . S Y S T E M F U N C T I O N A L I T Y
ing h o w many of the items have been practiced and h o w many have been
learned by the user.
O u t l i n e m a p s World Continents T states - Maps overview
World
C o n t i n e n t s
M a p s o v e r v i e w
R i v e r s L a k e s 1 M o u n t a i n s Islands
1 1 ^1 ^^^^^^^^H
Learned 23/84
Practiced Ä 44 184
Africa
Asia
Cities R i v e r s
Cities R i v e r s I M o u n t a i n
Australia and
Oceania
Cities
Europe States Cit
1 R i v e r s M o u n t a i n s Islands
Figure 3.4: M a p s overview.
Mnemonics for Places
Rate of learning of facts can be increased w i t h mnemonics [23]. A relevant
mnemonic can be s h o w n to user after a n incorrectly answered question. It
is relatively easy to implement the feature; however, it is rather difficult to
create the content of the mnemonics. It requires either an expert i n Geography
w i t h interest i n mnemonics, or some k i n d of c r o w d sourcing.
It is useful to have some data about w h i c h items were most often answered
incorrectly. Those items have the biggest room for improvement by
mnemonics; therefore, choosing them is most likely to result i n highest students'
performance improvement at lowest cost. We attempted to crowdsource
mnemonics for the system and got them for 2 0 most often confused
items (e.g. Estonia, Latvia and Romania).
1 8
3. S Y S T E M F U N C T I O N A L I T Y
Personal Goals
The purpose of personal goals is to incentive users to practice regularly i n
order to get the items into their long-term memory. The feature works as
follows: User chooses a map and a completion date of the goal. In order to
complete the goal, user has to achieve certain m i n i m u m probability of correct
answer (0.9 i n this case) of all items o n the map. The system visually
informs user about the goal completion progress and compares the actual
progress to an ideal progress (linear increase of average probability of correct
answer i n time).
Measuring Users' Perceived Difficulty of Questions
Different versions of the question construction algorithm have been compared
i n live experiments (outside of the scope of this thesis). It turned out
to be rather difficult to find a solid measure for comparing w h i c h of the algorithms
is "better".
Essentially we want to measure h o w m u c h has a user learned, but that
is difficult to measure objectively as users' skills are affected by questions
they have been asked, w h i c h are selected adaptively. To avoid this feedback
loop it w o u l d require to introduce pre-test and post-test questions (either
fixed or selected at random). Pre-test/post-test is problematic as the average
amount of answers by user is around 30 and we do not k n o w this
number for a given user i n advance so it is possible that the user leaves
the system before the post-test.
G i v e n the difficulty of measuring learning effect, w e instead measure
the number of questions answered b y a user assuming that a user w h o
spends more time i n the system is likely to learn more (which does not
have to be always true). A l t h o u g h it is easy to count number of answers it
also has its limitations as it is affected not only b y h o w well the questions fit
user's individual knowledge but also by other (external) factors (e.g. h o w
m u c h time is a student i n a school classroom instructed to spend o n learn-
ing)A
s a particular thing w e want to k n o w is if the algorithm selects questions
of appropriate difficulty, a difficulty rating poll has been implemented.
A m o d a l dialog asking " H o w hard are the questions?" w i t h options
"Too easy", "Just right" and "Too h a r d " (see Figure 3.5) is displayed to
19
3. S Y S T E M F U N C T I O N A L I T Y
users after finishing 3rd, 7th, 12th and 20th set of questions. A l t h o u g h perceived
difficulty might depend o n personal preferences and expectations
of a particular user, these level out over a large data set. It also gives us
a valuable insight into users perception of the system.
Figure 3.5: Difficulty rating poll.
20
Chapter 4
Implementation
In the previous chapter the system functionality was described i n a w a y
w h i c h is independent of any particular programming language, framework
or library. Since one of the goals of this thesis is to implement the system
described i n the previous chapter, specific technologies have to be chosen
and used.
This chapter describes w h i c h technologies were used a n d some of
the interesting issues that have been encountered during the development.
The source code of the implemented w e b system is publicly available
i n its Github repository (see A p p e n d i x A ) . Implementation of some of
the features described i n the previous chapter was a straightforward use of
the technologies and thus such features are omitted i n this chapter.
4.1 Project Timeline
The system has been developed over almost t w o years. That influenced
many aspects of the system; its features went through several versions i n
order to experiment w i t h various approaches. The functionality has been
added incrementally a n d iteratively as w e l l as the content. To provide
the reader w i t h an overview of the project timeline a list of major milestones
follows:
• Project Start (June 2013) - an initial vision of a n adaptive practice
system for map outlines.
• Alpha version (July 2013) - practice w i t h naive question selection
(skills and difficulties computed as correct to incorrect answers ratio)
and knowledge map; testing o n A d a p t i v e Learning group members;
only one map map available - states of the U.S.
21
4. I M P L E M E N T A T I O N
• Beta version (Fall 2013) - content extended w i t h countries of
the w o r l d and continents; testing o n dozens of beta testers; polishing
user interactions.
• FI M U N I students' research and development project (December
2013 to M a y 2014) - implemented support for other types of items
(e.g. cities, rivers or mountains); maps overview implemented; S E O
to reach broader audience; Elo for skill/difficulty estimation and comparing
it w i t h a naive algorithm i n A / B experiment.
• C T T project (August to December 2014) - Implemented internationalization,
personal goals, mnemonics, difficulty rating poll, derived
driving license system; user registration A / B experiments launched.
4.2 Used Technologies
Various technologies were used to implement the system. The server-side
part of the web application is written i n P y t h o n web framework Django extended
by several Django packages. The client is build i n AngularJS framew
o r k and uses some other JavaScript libraries, particularly Kartograph.js.
The client source code deployment is managed w i t h Grunt.
Django
The m a i n server building block is a w e l l - k n o w n P y t h o n web framework
Django [10]. Django has been chosen as it is the most popular web framew
o r k written i n Python. P y t h o n has been chosen because it is used for data
analysis and experiments by members of the Adaptive Learning G r o u p and
thus it is easier to migrate an algorithm evaluated i n an off-line experiment
to become a part of the web system.
The most important part of Django used by the system is Django O R M
(object-relational mapping). A l l the system data is stored i n a relational
database ( M y S Q L ) and the O R M handles creating the database schema
and converting database rows to P y t h o n objects and vice versa. Very convenient
is also the Django a d m i n - it provides a web interface to manage
the database content.
Since Django is very popular, there are many packages (2,719 as of
M a r c h 31, 2015 [5]) that extend Django w i t h some functionality and can
be included i n a Django project. The developed geography system uses
22
4. I M P L E M E N T A T I O N
four Django packages: south, django-social-auth, django-lazysignup and
django-modeltranslation.
South is a package for database schema migration. It is such an essential
part of developing a web application that it has been incorporated into
the core Django package since Django 1.7.
django-social-auth is a wrapper around many different social networks
Authentication APIs. It supports dozens of authentication back-ends out of
w h i c h are two used i n the system (Google and Facebook).
django-lazysignup automatically creates user profiles w h e n users enter
a part of the system that needs to save some data about their behavior
(e.g. answer a question). It also allows users to register and thus migrate
their lazy user profile to a regular user profile.
django-modeltranslation is for keeping multiple language versions of
model object fields i n the database. For each translated Django model field,
django-modeltranslation creates multiple database fields (with language
code suffixes). Then it decides w h i c h of the database fields is returned as
Django m o d e l field according to the current language.
Django also offers a template language, w h i c h extends H T M L w i t h tools
like cycles and conditions. The template language is, however, used i n
the system only for the m a i n m e n u and footer as other UIs are handled
by AngularJS templates.
The system uses Django version 1.5 as it was the latest version at
the time the project started. However, version 1.5 is no longer supported
so the system has to be migrated to a newer version. A s it is planned to
also migrate the system to a c o m m o n back-end w i t h other applications described
i n Section 6.1, these two migrations can happen at the same time.
Internationalization
The first implementation was i n C z e c h only. In order to reach broader audience,
the application has been internationalized and translated to E n glish.
The internationalization later allowed a M e x i c a n volunteer to translate
the system also into Spanish and other volunteers are currently translating
the system to German.
2 3
4. I M P L E M E N T A T I O N
Internationalization of the U I is done by standard Django translation
functions (such as u g e t t e x t or u n g e t t e x t ) , w h i c h rely o n G N U
g e t t e x t toolset [11]. Translation of the practiced items is managed by
d j a n g o - m o d e l t r a n s l a t i o n package, w h i c h creates additional database
fields for each Django model T e x t F i e l d registered for translation (e.g. for
field name are created additional fields name_cs and name_en).
AngularJS
The client of the developed system is a single page application b u i l d i n
JavaScript framework AngularJS [15]. AngularJS has been chosen because it
is a M o d e l V i e w Controller ( M V C ) framework, w h i c h makes it significantly
easier to develop an interactive application, such as this one, compared to
traditional JavaScript frameworks like jQuery. W h e n comparing AngularJS
to other JavaScript M V C frameworks (Backbone.js, Amber.js), AngularJS
has been chosen because it is backed by Google and has the largest c o m m u nity
whether it is measured by Stars o n Github, StackOverflow questions
or YouTube results [30].
AngularJS uses templates that are an extension of H T M L by ng- prefixed
attributes (such as n g - i f or ng-repeat) and Angular-specific elements.
Templates can be further extended by custom elements - directives.
The template system can also evaluate expressions i n double curly
brackets notation ({{ e x p r e s s i o n } } ) . These expressions are evaluated
on a scope, w h i c h is the " M o d e l " part of the M V C .
The " C o n t r o l l e r " part of M V C is a function w i t h its dependencies ( A n gularJS
uses Dependency Injection) as arguments. The dependencies of
a controller are services. Services are supposed to be reusable pieces of code
w i t h a given purpose (e.g. to communicate w i t h the server).
Kartograph.js
Kartograph.js [1] has been chosen for displaying the S V G maps generated
by its P y t h o n counterpart, Kartographpy. It is rather a convenience library
because internally it uses Raphael for the D O M manipulation.
Raphael is a well-documented [2] vector graphics library for the web. It
does all the necessary D O M manipulation inside an H T M L 5 S V G element.
24
4. I M P L E M E N T A T I O N
Raphael also uses V M L as a fallback i n Microsoft Internet Explorer (MSIE)
version 8 and older, w h i c h does not support S V G .
Kartograph.js parses an S V G file to JavaScript objects, where it can be
resized and sanitized. Then it converts the objects back to S V G path strings
and feeds them into Raphael, w h i c h displays it i n an S V G element i n side
the H T M L page. This process of converting the image happens to be
the m a i n bottleneck of the client-side performance.
The slowest part is parsing the d attribute of S V G path, w h i c h is basically
a string representation of an array of coordinates. This string is parsed
to a JavaScript Array, w h i c h is more suitable for resizing the image, although
i n order to pass it to Raphael Paper . p a t h () [2], it has to be converted
to a string again.
O n desktop, this bottleneck causes a delay i n order of hundreds of m i l liseconds
w h e n loading a map, w h i c h is not a serious problem as it is preceded
by fetching the map from the server, w h i c h often takes longer. The
problem is worse o n mobile devices w i t h lower performance, but since majority
of traffic comes from desktop, this issue has not been addressed fur-
ther.
Grunt
Homepage initialization performance was negatively influenced by
the large amount of H T T P requests made to the server to collect all
JavaScript files. Grunt (The JavaScript Task Runner) is used to tackle this
issue. Firstly, it is used to minify and concatenate JavaScript files and CSS
styles o n file modification while development and on deploy. There are
about 15 JavaScript libraries files reduced to only one. The amount of
project specific JavaScript files is reduced i n a similar way.
Furthermore, Grunt is used to compile AngularJS templates to
JavaScript files, w h i c h are also included i n the aforementioned concatenation,
thus again reducing the amount of H T T P requests. Last but not least,
Grunt is used to r u n JSHint (a JavaScript code quality tool) i n a Git precommit
hook to prevent any potentially problematic code from being added
to the project. JSHint i n Grunt was at first also used to check files i n real time
while editing, but this was later replaced by use of a V i m plugin, w h i c h is
better integrated to the editor.
25
4. I M P L E M E N T A T I O N
4.3 Map-related Functionality
There is a lot of source code that deals w i t h maps. Firstly S V G images
are generated from geospatial data files; then important information from
the images is loaded to the database; finally parts of the S V G images are
displayed i n a browser.
Generating Maps
M a p s are generated b y a Python script w h i c h is stored i n its o w n Git repository
(see A p p e n d i x A ) . The m a i n responsibility of the script is to pass right
parameters to Kartograph.py g e n e r a t e function. There are different sets
of layers for different maps. In some cases the script does some ad hoc
modifications to the generated maps, such as replacing state codes or fixing
UTF-8 encoding.
Kartograph.py provides a c o m m a n d line interface for generating maps
based on a map file (shapefile format) and a configuration file (JSON);
however, after trying it out I decided to use the Kartograph.py programm
i n g A P I . The m a i n reason was that a Python script is more versatile, thus
the script can generate only a specified map, a group of maps or all the maps
at once. U s i n g a Python script allows avoiding code duplication by use of
inheritance, and text replacements i n the output.
Adding Generated Maps to the Production System
The output of the script described i n the previous subsection is a folder w i t h
S V G images (maps). These files also contain names and types of the items
for practice, w h i c h have to be extracted and loaded to the production system
database.
A t first, Django migrations were used to a d d new Places to the database.
Eventually, I realized that adding new maps w i l l probably never stop and it
should be automated, since all the information is already i n the S V G maps
and it only needs to be parsed and read.
A custom Django c o m m a n d (called update_maps) has been implemented
to automate adding maps to the system. O n input it takes a folder
w i t h S V G files; each file is parsed i n order to find S V G groups w h i c h contain
S V G paths or c i r c l e s . These S V G elements contain all attributes
2 6
4. I M P L E M E N T A T I O N
necessary to create a P l a c e object w i t h its name, code a n d p l a c e _ t y p e .
A l l the elements being part of a specific S V G file also allows creation of
P l a c e R e l a t i o n objects, w h i c h carry the information about w h i c h P l a c e s
are o n w h i c h map (map is a P l a c e , too).
M a p s a n d Layers
Most maps contain multiple layers (e.g. background, countries, cities a n d
rivers). A l t h o u g h all of the layers could be displayed at the same time (since
an S V G file contains all of the layers), for the sake of clarity it is better
to display only some of the layers (as discussed i n Section 3.2). The layers
are managed b y a d d L a y e r () a n d g e t L a y e r ( ) methods of Kartograph.js
map object [1]. The hiding of layers is done b y calling Raphael
Element. h i d e () method o n each element of the given layer.
Each layer consists of S V G elements, w h i c h differ based o n what they
represent (see Figure 4.1). The basic element, w h i c h is used for states, islands
a n d mountains, is an S V G p a t h w i t h a white f i l l a n d a black
s t r o k e . The element slightly changes color o n hover (by decreasing
o p a c i t y ) a n d a callback function can be b o u n d to the element click event.
Other element types differ i n some aspects.
(a) States and cities (b) Rivers (c) Mountains
Figure 4.1: Different layers
A city is an S V G c i r c l e . Its radius depends o n city population a n d
z o o m level. M o r e specifically, there are five population thresholds that i n crease
the size of the circle (e.g. 100 thousand, 5 million). Some cities are
very close to each other (e.g. Bratislava a n d Vienna), w h i c h makes it very
difficult to select the one underneath the other. In order to overcome this
27
4. I M P L E M E N T A T I O N
issue, circles decrease their relative size to other elements w h e n z o o m i n g
the map (but they still increase their absolute size).
A river is a blue s t r o k e , w h i c h increases its s t r o k e - w i d t h o n hover.
Since there is no f i l l , it is the s t r o k e that changes the color to signal correctness
of an answers or student knowledge. A special property of a lake
is its bright blue f i l l and a dark blue s t r o k e . Background has gray f i l l
and does not react to hover.
Boundary Box Cache
A boundary box (bbox) of a path is the smallest rectangle such that all
the points of the path are inside it. Bbox is a useful tool to summarize h o w
large a path is and where it is located, w h i c h is rather difficult to see from
a list of points. Bbox is used i n the system w h e n highlighting an item (its
path). The highlight is an animation w h i c h scales up the size of the path.
Bbox is needed to find the center of the scaling and to determine the scale
ratio (smaller paths are scaled more).
The complexity of computing the bbox is linear to the length of the path
points list because each of the points has to be checked as it can enlarge
the bbox. In the worst case (which is the Canadian territory N u n a v u t w i t h
its many islands w i t h complex coastlines), the bbox computation i n browser
takes more than a second.
To prevent these delays bboxes are computed o n the server o n deploy.
They are saved i n J S O N format and loaded by the client to a global object
called bboxCache. The implementation of getBBox function i n Raphael
library, has been altered to look for a bbox into the bboxCache before computing
the bbox. O n l y if there is no bbox for the given path i n bboxCache,
it is computed and saved to the cache for next time. bboxCache also takes
care of resizing the bboxes to fit the resized map.
4.4 Other Functionality
Apart from the map-related functionality, most of the implementation was
rather a straightforward use of the used technologies. One interesting detail
is requesting questions from the server. Another one is the implementation
of logging data, w h i c h is mentioned i n order give an overview of what data
2 8
4. I M P L E M E N T A T I O N
is logged as there have been done some analyses of the data (see Section 5.2
and Section 6.2).
Requesting Questions from the Server
In the implementation of the practice it h a d to be determined h o w often
to request questions from the server to the client. Questions can be either
requested all at once at the beginning of a set or one at a time. A downside
of the former option is that it reduces adaptivity of the system as the answers
earlier i n the set cannot be used to select the questions later i n the set.
The latter option, o n the other hand, creates bad user experience o n realw
o r l d network as user often has to wait for the next question.
A synthesis of these approaches has been used. The whole set is fetched
at the beginning; however, after each answer the list of questions to be
asked is updated. Since a correct answer is followed b y the next question
automatically i n 0.6 seconds, the next question is often selected from the already
stored list as the request for new questions has not finished yet. H o w ever,
w h e n the user has already w e l l estimated skill, the stored question and
the n e w l y fetched one are likely to be the same.
This point proved to be a source of serious problems. Firstly, the n e w l y
fetched question has been set as the next question, w h i c h results i n the same
question being asked twice i n a row. Another issue was that sometimes
the request after n t h answer can finish earlier than the request after n + l t h
question. If this situation is not handled, it results i n newer questions being
replaced by older questions. A n d as the older questions might contain
a question that has been already asked at any position this can again lead
to the same question being asked twice i n a row. A l l these issues have been
solved.
Logging Data
Users generate a lot of data while using the system. Majority of the data is
users' answers a n d smaller part is the difficulty rating poll results. The rating
poll answer consists of user, inserted date time, value (one of "Too
easy", "Appropriate", "Too difficult").
The answer record consists of the following fields:
• user ID,
29
4. I M P L E M E N T A T I O N
• asked item (e.g. Estonia),
• answered item (if same as item asked, then the answer is correct),
• map (e.g. Europe - to distinguish from Estonia o n the map of
the World),
• date and time w h e n the answer was answered,
• response time (in milliseconds),
• question type ("Choose o n the m a p " or " W h a t is the name of highlighted
item"),
• number of options (2 to 7 i n the case of multiple-choice question or
none i n the case of open question).
• language (Czech, English or Spanish)
• IP address (Can be used to identify group of students from the same
IP address and label them as a school class.)
The total number of answers gathered since launching the system is
in order of millions (8,728,802 as of A p r i l 14, 2015) and g r o w i n g b y about
1 million per month. This b i g data set has been used i n many off-line experiments
conducted by members of A d a p t i v e Learning G r o u p at Faculty of
Informatics M a s a r y k University (see Section 6.2). The data is also examined
in the following chapter of this thesis.
30
Chapter 5
System Usage Analysis
The system has been deployed to h t t p : //slepemapy. cz i n late 2013.
Ever since then it attracts increasing numbers of users. This chapter examines
w h o those users are, what they have i n c o m m o n and i n what aspects
they differ.
Some analyses of users were done i n bachelor thesis of D i o n y z
Lazar [17], but there are still many other aspect of user behavior to be examined.
Also, they were done a year ago and the data set consisted of only
about one eighth of the current amount of answers (1 vs. 8 million).
5.1 Used Technologies
System traffic has been continuously monitored by Google Analytics [12],
the most widely used website traffic analysis tool [33]. F r o m its w i d e variety
of reports, has been especially useful the number of users i n time and where
do the users come from.
Google Analytics also offers capability to conduct A / B experiments.
Usage of this feature is simple - the system has to fetch A / B value from
an A P I , change the system according to the A / B value, and send the A / B
value w i t h all Google Analytics requests. To make it even simpler, there is
an AngularJS module a n g u l a r - g o o g l e - e x p e r i m e n t s [22], w h i c h provides
an AngularJS directive to display the appropriate variation (after setting
an experiment ID).
Off-line analyses have been done i n Python data analysis library
Pandas [31], w h i c h internally uses matplotlib [13] for plotting the data.
The source code of off-line analyses is available i n a separate Github repository
(see A p p e n d i x A ) .
31
5. S Y S T E M U S A G E A N A L Y S I S
5.2 Users of the System
N u m b e r of users is gradually rising, as can be seen i n Figure 5.1. Similar
trends have also numbers of sessions and page views. Different trend show
the last three plots i n Figure 5.1. The changes at the beginning of those plots
are caused by some changes of implementation details and do not carry any
important information, so the important parts of these plots are relatively
constant. The bounce rate keeps o n being rather l o w (11%), w h i c h is caused
by h o w is the system implemented - most users arrive to the homepage,
where they are immediately faced w i t h several buttons to start a practice;
therefore, they make a least this interaction w i t h the system. The average
time i n the system is about 11 minutes during w h i c h the average user visits
11 pages, which, i n m y opinion, signifies that the practice offered by
the system is useful to the users.
• Sessions
30,000
January 2014 April 2014 July 2014 October 2014 January 2015
• New Visitor • Returning Visitor
Sessions Users
Avg. Session Duration Bounce Rate
00:11:05 10.81%
Figure 5.1: N u m b e r s of sessions per month.
A l t h o u g h there are 23 maps i n the system, most of the traffic is concentrated
to just a few of them (World, Europe, A s i a and Czech Republic - see
Figure B . l i n A p p e n d i x B). The figure also shows that after reaching certain
amount of traffic and adding most of the maps into the system (around
week 11 of 2014) users' preference for different maps does not change m u c h
over time.
32
5. S Y S T E M U S A G E A N A L Y S I S
Since vast majority of the users are from the same time zone (Central
European Timezone), fluctuations i n traffic during a day can be observed
(see Figure 5.2). O b v i o u s l y there is l o w traffic at night, then significant i n crease
i n the m o r n i n g and the highest traffic i n the evening around 7 P M .
A n interesting phenomenon are six peaks between 8 A M and 2 P M . A t first
they might look like random noise, but after a more careful examination
they match timings of C z e c h school lessons and breaks, thus confirm that
the system is used i n schools. However, most of the users use the system i n
the evening i n their free time, w h i c h is, i n m y opinion, a good thing, since
users can use the system at their convenience and spend i n the system different
amount of time according to their individual needs.
Figure 5.2: N u m b e r of answers during a day
Another indicator of the school usage of the system is the decrease i n
number of sessions during weekends (see Figure B.2 i n A p p e n d i x B) and
holidays (see number of sessions i n July and A u g u s t i n Figure 5.1). The usage
i n schools is also confirmed by written feedback from users via the feedback
form i n the system. Some of the feedback contains explicit reference to
school and i n many other cases it contains signs of being written by pupils
(e.g. serious grammar issues or expressed annoyance of using the system
unwillingly).
3 3
5. S Y S T E M U S A G E A N A L Y S I S
Dividing Users
If there are significant differences between some groups of users (e.g. school
users differ from other users), w e can classify them into those groups a n d
adjust system parameters for each group to enhance adaptivity of the system.
I examined five possible division of users into two groups:
• m o r n i n g vs. evening users,
• non-returning vs. returning users,
• school vs. out-of-school users,
• l o w vs. high skill users,
• l o w vs. high response time users (i.e. fast vs. slow).
In the following paragraphs are the aforementioned division methods
specified more precisely, e.g. where is the boundary between morning a n d
evening.
Evening users are defined as those whose average time of an answer is
greater than 4 P M . The exact time has been chosen a d hoc, based o n B.3(a)
in A p p e n d i x B, so as to make the two groups of similar size.
School users are detected b y their IP address. M o r e specifically, if there
are more than or equal to 20 users whose first answer comes from a given
IP address, they are all considered to come from a school classroom. A g a i n ,
the threshold of 20 users has been chosen ad hoc. In B.3(b) i n A p p e n d i x B
can be seen that this threshold results i n about 10% of users being detected
as classroom users.
Returning users are determined b y difference between the date a n d
time of their first a n d last answer. Returning users are considered those
whose first a n d last answer are not i n the same day. A b o u t half of users
are being considered returning b y this threshold (see B.3(c) i n A p p e n d i x B).
The same figure also shows comparison w i t h other possible division thresholds
(e.g. hour or month).
High skill users are considered those w i t h skill 0 a n d higher. Skill is
a real number 9i described i n Section 3.1; however, most of users' skills are
between -4 and 4 (as can be seen i n B.3(d) i n A p p e n d i x B). 0 has been chosen
to be the threshold as it is the default skill of a new user.
34
5. S Y S T E M U S A G E A N A L Y S I S
Low response time users are those w i t h the median of response time
lower than 3.5 seconds. The threshold has been chosen to divide the users
into two groups of similar size (see B.3(e) i n A p p e n d i x B). M e d i a n has been
chosen instead of average because average response time is w a y higher
because about half of the users have at least one very high response time
(above 30 seconds). These very high response times are likely to be caused
by users doing other things i n the m i d d l e of practice. Users d o not care
about the response time as the system does not suggest i n any w a y that it
is measured a n d users should pay attention to it.
Divisions Correlation
It has been examined h o w the divisions of users correlate w i t h each other
(see Table 5.1). A vector of booleans (one value for each user) is computed
for each division. For each pair of divisions are their vectors compared b y
Pearson correlation coefficient.
The first observation is that none of the correlations is very strong. The
strongest correlation is between school a n d evening users (-0.27), w h i c h is
not surprising since schools operate mostly i n the morning. In-school users
also have lower prior skill, w h i c h is, again, not very surprising.
evening lowRT high skill school returning
evening 1 -0.009335 0.018608 -0.269656 -0.016726
l o w R T -0.009335 1 0.07515 0.014363 0.113427
high skill 0.018608 0.07515 1 -0.116768 0.072943
school -0.269656 0.014363 -0.116768 1 -0.068281
returning -0.016726 0.113427 0.072943 -0.068281 1
Table 5.1: Pearson correlation coefficient between divisions
The third strongest correlation shows that returning users have lower response
times. I can see t w o reasons for this. Firstly, returning users are more
familiar w i t h the system thus respond faster. Secondly, returning users are
likely to have higher skill a n d thus d o not need to think for so long to recall
an answer. I think that both have an effect to some extent a n d it should be
examined more closely a n d taken into account w h e n conducting research
on response times.
35
5. S Y S T E M U S A G E A N A L Y S I S
Differences of Question Difficulty between the Divisions
I have examined the differences between the two groups of each division.
More specifically were examined results of difficulty rating poll (see Figure
B.4 i n A p p e n d i x B) and success rate (see Figure B.5 i n A p p e n d i x B).
Several next paragraphs describe some observations from the figures.
School users have lower success rate (73% vs. 80%) and more often rate
the questions as "Too difficult" (27% vs. 10%) compared to out-of-school
users. The difference is, i n m y opinion, caused partly b y lower skills of
school users but mainly b y their preference of easier tasks as they use the
system only because they were told to do so. The differences between morning
and evening users are very slight and probably can be explained by the
correlation of m o r n i n g users w i t h school users.
L o w response time users and high prior skill users have very similar
trends. They both have higher success rate (83% vs. 77%, resp. 84% vs. 74%)
and ratings shifted towards "Too easy", w h i c h is, especially i n case of the
high prior skill, rather by definition.
Returning users have slightly higher success rate (81% vs. 78%), but
unlike any other division they have slightly less of both "Too easy" and
"Too difficult" ratings (25% vs. 29%, resp. 10% vs. 11%) compared to nonreturning
users. This can be explained i n various ways. In m y opinion, users
w h o are content w i t h the difficulty are more likely to return. However, the
reason might be also that the system estimates the difficulty more precisely
for returning users as there is more data about them.
5.3 User Registration A/B Experiments
Google Analytics Experiments feature has been used to conduct two online
experiments w i t h the a i m of increasing the portion of registered users
by altering some parts of the U I . W h i l e users do not have to register to
use the system, the m a i n reason to increase registration rate is to decrease
the probability that a single person uses the system as multiple users. That
can happen either w h e n a person uses the system from multiple computers
or w h e n a person returns after more than two weeks and no longer has
the lazysignup session active.
36
5. S Y S T E M U S A G E A N A L Y S I S
The First Experiment
The first experiment has been conducted because of the assumption that
users are not aware of the possibility to register. The assumption stemmed
from the fact that the option for registration i n the m a i n toolbar was labeled
" L o g i n " . It w a s labeled that w a y because the only t w o authentication options
were v i a Facebook a n d v i a Google, w h i c h from the implementation
point of v i e w d o not make any difference between registration a n d login.
A n alternative version w i t h the option labeled " L o g in/Sign i n " has
been created a n d deployed. After 20 days of running the experiment,
the data d i d not show any significant difference between the t w o versions
(see Table 5.2), so the experiment w a s replaced b y the second experiment.
Variation Sessions Convs C o n v Rate C t O P o O O
L o g i n 3,794 91 2.40% 0% 0.0%
L o g i n / 3,486 83 2.38% - 1 % 49.3%
Sign u p
Table 5.2: Results of the first experiment (Abbreviations: C o n v = Conversion;
C t O = Compare to Original; P o O O = Probability of Outperforming
Original
The Second Experiment
The second experiment tested h o w promoting the possibility to register affects
the ratio of registered users. M o r e specifically, i n addition to the original
site w i t h no promotion, there were deployed four alternative variations
w i t h a p o p - u p box (similar to the one for difficulty rating Figure 3.5)
displayed to user after completing 2nd, 4th, 6th, or 8th set of questions.
The p o p - u p box informed users about features for registered users a n d
asked them to register.
I assumed that promoting registration w o u l d have positive effect the o n
ratio of registered users; however, none of the alternatives outperformed
the original, as can be seen i n Table 5.3. The results neither prove that
the original outperforms any of the alternatives. The experiment r u n for
3 months, w h i c h is the m a x i m u m length that Google Experiments allow.
The results of neither experiment showed any w a y to increase ratio of
registered users. In theory, there are t w o possible explanations for this. E i -
37
5. S Y S T E M U S A G E A N A L Y S I S
Variation Sessions Convs C o n v Rate C t O P o O O
Original
After 8th
After 4th
After 6th
After 2nd
8,888
6,127
7,173
5,251
1,603
135
92
102
66
19
1.52%
1.50%
1.42%
1.26%
1.19%
0%
- 1 %
-6%
-17%
-22%
0.0%
44.8%
40.2%
15.9%
19.2%
Table 5.3: Results of the second experiment (Abbreviations: C o n v = C o n version;
C t O = Compare to Original; P o O O = Probability of Outperforming
Original
ther there was not enough traffic to show the difference or the variations
used i n the experiments do not have significant impact o n registration rate.
In case of the first experiment the variation was, i n m y opinion, too small
to have an impact. However, i n the second experiment I think that more
data w o u l d show statistically significant difference, but it w o u l d still be at
most a few percent increase of registration rate. M y conclusion of both experiments
is that users have their o w n motivations and goals and therefore
cannot be easily forced to do something they do not want to do.
With this experience i n m i n d I suggest another A / B experiment w i t h
the a i m to increase registration rate. There is some functionality for registered
users only (e.g. personal goals) but all the links to this functionality
are h i d d e n for unauthenticated users. I suggest not to hide the links but
instead let the users click the links and then tell them that they need to be
registered for this functionality.
5.4 User Feedback
A great amount of user feedback has been gathered through the feedback
form o n the site . E v e n though most users do not send us any feedback (less
than 1% does), these are often very valuable insights.
To gain some general insight about the feedback all received feedback
(162 messages as of M a r c h 29, 2015) has been manually categorized. Exactly
one type has been assigned to each message to simplify the situation,
although i n some cases a message could be of multiple types. Messages
shorter than 50 characters are filtered by the system and have to be ap-
38
5 . S Y S T E M U S A G E A N A L Y S I S
proved to be i n the data set as they usually contain no relevant information.
Figure 5.3 shows h o w many messages of each type are there. M o s t of
the feedback is one of the following types (sorted from most often to least):
• Content request - users w o u l d like like to practice another map
(Oceans and seas are often requested).
• Functionality request - e.g. countries capitals practice, flags practice
or exclude some items from practice.
• Error in functionality - e.g. some item cannot be clicked.
• Praise - users only express their liking of the application.
• Report of error in content - incorrectly named or located item.
• No information value - usually send by children w h o were forced to
use the system at school.
Figure 5.3: Feedback messages by type.
Content request are the most c o m m o n type of feedback probably because
w e lack a person whose m a i n responsibility w o u l d be to create new
content and thus the requests are often not met and reoccurring. To fix this
we should be looking for someone w i t h knowledge of geography and experience
w i t h using a G I S sowtware (e.g. Q G I S ) to edit maps.
3 9
5. S Y S T E M U S A G E A N A L Y S I S
We are also getting many functionality requests that are not implemented
as they do not fit our research directions (e.g. practice of flags). O n
the other hand, w e do our best to fix all reported errors, w h i c h is probably
the reason w h y error reports are fewer than requests for something new.
40
Chapter 6
Subsequent Related Work
Since the project started almost two years ago there has already been a significant
amount of w o r k done based o n the system developed i n this thesis.
Most of the consequent w o r k fits into one of two categories: development
of similar systems (with potential code reuse) and research o n the data collected
by the system.
6.1 Derived Systems
After implementing the geography practice application I have participated
on development of similar applications that reuse some source code and
also b u i l d o n the experience gained w i t h designing and developing the geography
application.
Driving License Tests
A system for practicing C z e c h driving license multiple-choice test questions
has been implemented and deployed to h t t p : / / a u t o s k o l a c h y t r e . cz.
It is similar to the geography application as both applications ask users
questions and estimate users knowledge based o n the answers.
One difference is that questions i n a u t o s k o l a c h y t r e . cz are created
by the C z e c h Ministry of Traffic and are all multiple-choice w i t h two or
three static choices, thus there is no need for adaptive choices selection.
There also is no need for a map, w h i c h saves a lot of space and makes
the application more suitable for use o n mobile devices. I a m w o r k i n g on
converting the web application client into an A n d r o i d mobile application
w i t h Phonegap; however, there are some performance issues w i t h A n g u larJS
to be solved.
41
6. S U B S E Q U E N T R E L A T E D W O R K
The driving license application has been created to be general enough to
allow use i n another d o m a i n w i t h available multiple-choice test questions.
The configuration of the system consits of a J S O N file w i t h the questions
and another file w i t h U I texts. A l l the d o m a i n specific texts i n the U I are
managed by d j a n g o - f l a t b l o c k s [7] and thus saved i n the database as
opposed to be hard-coded.
One particular d o m a i n it could be used i n is TSP (Learning Potential
Test - i n C z e c h "Testy Studijních Přepokladů"), w h i c h are used as admission
tests at M a s a r y k University. Previous years questions of TSP are publicly
available; however, the university distributes them only i n PDF, w h i c h
is the m a i n reason it has not been used yet.
A n a t o m y for H i g h School Students
Our system was a starting point for development of an adaptive learning
system for h u m a n anatomy, w h i c h was created as d i p l o m a thesis by Jan
Kučera [16]. The anatomy system reuses some of the client-side code of
our system, whereas its server-side reuses only the ideas as it is written i n
a different programming language (Java). The system has been deployed to
h t t p : / /www. s l e p a a n a t o m i e . cz / and its content covers the extent of
anatomy i n high school biology.
A n a t o m y for M e d i c a l Students
Currently w e collaborate w i t h M e m o r i x team (authors of an anatomy textbook
for medical students called " M e m o r i x of A n a t o m y " [32]) o n an adaptive
practice of anatomy for medical students. The practice is very similar to
geography (students want to learn names of areas o n a picture); however, it
differs i n other challenging aspects.
A n a t o m y encompasses significantly larger amount of items and pictures
to be learned. The textbook contains about 2000 items and about 600
images and our target group users have to eventually k n o w all of them
(and even more) for their anatomy exam. We need to find a w a y to display
them an overview of all their knowledge. We plan to use the topics
hierarchy from the textbook and aggregate the knowledge i n certain categories/
chapters. We also want to motivate students to use the system continuously
during their three semesters of anatomy and better incorporate
forgetting effects into the student model.
42
6. S U B S E Q U E N T R E L A T E D W O R K
Another challenge is to label all the parts of the images from the textbook
i n a format supported by the practice application. The book is created
in a software that allows the images to be exported to S V G . The S V G paths
in the images do not contain names of the items (body parts) they represent.
Therefore, I developed a web application w h i c h is used b y medical
students w i t h knowledge of anatomy for labeling the items.
6.2 Research on the Collected Data
The implemented system has produced over 8 million answers and currently
produces about 1 million answers per month. This data set is subject
to research conducted i n Adaptive Learning Group.
There is a paper that describes the adaptivity of the system and evaluates
it o n the collected data [25]. The paper divides the adaptivity of the
system into three m a i n parts: estimation of prior knowledge, estimation of
current knowledge and questions selection.
Some experiments focused on comparing different variations of the Elo
system to estimate student prior knowledge and item difficulty [21]. These
experiments used only correctness of first answer of each student-item pair
as the other answers are affected by student's learning. Results of these experiments
show that some are slightly more accurate than others to predict
answer correctness; however, the differences are not very high and their
impact o n user's learning is to be examined.
Other experiments examined h o w to improve student current skill estimation.
One w a y to do this is to use time spacing and response time, i.e.
h o w m u c h time elapsed since last answer the user was asked the same
item and h o w long the user spent on answering it. The assumptions are
that users forget what they have learned over time and that faster answer
indicates better knowledge. The goal of these experiments is to describe
the assumed effect more accurately and use it to improve predictions of
the current skill. Various models of student's memory have been evaluated
on the data [28].
Research o n response times [26] showed (among others) the following
findings. W h e n response times are longer, success rate of the answers de-
43
6. S U B S E Q U E N T R E L A T E D W O R K
creases (see Figure 6.1). The same relation is between response times and
success rate of the next answer about the same item. However, the relation
of response time of a n incorrect answer and success rate of the following
answer is reverse. These findings can be incorporated to the production
system to increase the accuracy of the estimations.
Success vs. Response Time
0.60 ' 1
0-10% 2 l - 3
° ° / o
31-40% 4 l-50% 5 l-60% 6 l-70% 71-80% 8 l - 9
° ° / o
9l-lO°°/ o
Response Time Group
(according to the percentile computed per item)
Figure 6.1: Response times and probability that the (next) answer is correct.
[26]
There has also been some research o n question selection algorithms. It
took form of A / B on-line experiments that compared multiple versions of
the adaptive algorithms [24]. Firstly, the adaptive question selection has
been compared to r a n d o m selection. It has been proved that the adaptive
version causes users to spend more time i n the system. Secondly, the question
selection w i t h given target success rate has been compared to question
selection w i t h target success rate adjusted according to the observed success
rate of the past ten answers. The adjustment was done to choose more
difficult questions w h e n observed success rate was too high and vice versa.
The target success rate has been also manipulated for different users (ranging
from 55% to 95%).
44
Chapter 7
Conclusion
The proposed system has been implemented and deployed to h t t p : //
slepemapy.cz. There are many indicators that suggest the quality and
usefulness of the system. Firstly there is the high amount of users (hundreds
of users per day and about 20 thousand sessions per month) and the
time they have spent i n the system (in total over 3,000 hours per month as
the average is 10 minutes per session). Furthermore, there is the positive
verbal feedback send by the users. Last but not least, the system took 8th
place i n the competition called "Společně otevíráme data 2014" (a competition
of applications that use open data).
Because of its large user base, the system is not only useful to the users,
but also provides opportunities for research. The answers and other collected
data have been subject to numerous off-line experiments. Moreover,
on-line A / B experiments have been conducted to explore different versions
of question selection algorithms and their impact o n users' usage of the sys­
tem.
There are m a n y opportunities to enhance the functionality of the system.
For example, the system can be extended by a teacher's mode. That
is to allow teachers to create classes of students and to display statistics
about individuals or the class as a whole. Another useful extension might
be allowing users to exclude certain items from practice (e.g. to practice
only cities above a given population threshold). According to the feedback,
many users w o u l d also appreciate more content to practice (e.g. seas, lowlands
or deserts).
Another w a y of building o n top of this project is to reuse the developed
functionality i n a similar system focused o n different area than geography.
A system for practicing driving license exams questions has been developed
i n this w a y and currently w e develop a system for practicing h u m a n
anatomy for medical students.
45
Bibliography
[1] Gregor Aisch. Kartograph.js docs, h t t p : / / k a r t o g r a p h . o r g /
d o c s / k a r t o g r a p h . j s / , 2014. [Online; accessed 2015-03-23].
[2] D m i t r y Baranovskiy. Raphael reference, h t t p : / / r a p h a e l j s . c o m /
r e f e r e n c e . html, 2013. [Online; accessed 2013-03-09].
[3] James H Block, Peter W Airasian, Benjamin S Bloom, and John B Carroll.
Mastery learning: Theory and practice. Holt, Rinehart and W i n ston
N e w York, 1971.
[4] M i c h a l Brandejs, Jitka Brandejsová, Miroslava Misáková, Jan
Kasprzak, Daniel Keder, Ľ u b o š Lunter, et al. Spaced-repetition
algorithm i n university context: drilling less but remembering more.
A M Y , 2009.
[5] A u d r e y R o y & Two Scoops Press Daniel Greenfeld. Django packages
: Reusable apps, sites and tools directory. https://www.
d jangopackages . com/, 2015. [Online; accessed 2015-03-31].
[6] D a v i d D u n n i n g , Kerri Johnson, Joyce Ehrlinger, and Justin Kruger.
W h y people fail to recognize their o w n incompetence. Current Directions
in Psychological Science, 12(3):83-87, 2003.
[7] Clint Ecker, K e v i n Fricovsky, Peter Baumgartner, Alexander
Artemenko, and Horst G u t m a n n . django-flatblocks 0.9 :
Python package index. h t t p s : / / p y p i . p y t h o n . o r g / p y p i /
d j a n g o - f l a t b l o c k s , 2009. [Online; accessed 2015-03-23].
[8] D a m i e n Elmes. A n k i - powerful, intelligent flashcards. h t t p : / /
a n k i s r s . net/, 2015. [Online; accessed 2015-03-31].
[9] Arpád E Elo. The rating of chessplayers, past and present, volume 3.
Batsford L o n d o n , 1978.
[10] Django Software Foundation. Django (version 1.5). h t t p : / /
d j a n g o p r o j e c t . com, 2013. [Online; accessed 2015-03-23].
46
7. C O N C L U S I O N
[11] Inc. Free Software Foundation, gettext - g n u project, h t t p s : / /www.
g n u . o r g / s o f t w a r e / g e t t e x t / , 2013. [Online; accessed 2015-03-
23].
[12] Inc. Google. Google analytics. http://www.google.com/
a n a l y t i c s / , 2005. [Online; accessed 2015-03-23].
[13] John D Hunter. Matplotlib: A 2d graphics environment. Computing
In Science & Engineering, 9(3):90-95, 2007.
[14] Jeffrey D Karpicke and H e n r y L Roediger. Repeated retrieval during
learning is the key to long-term retention. Journal of Memory and
Language, 57(2):151-162, 2007.
[15] Pawel K o z l o w s k i and Peter B D a r w i n . Mastering Web Application
Development with AngularJS. Packt Publishing Ltd, 2013.
[16] Jan Kučera. Adaptabilní webový systém pro výuku anatomie.
D i p l o m a thesis, Faculty of Informatics M a s a r y k University, 2014.
[17] Dionýz Lazar. Analýza dat ze systému pro výuku zeměpisu. Bachelor
thesis, Faculty of Informatics M a s a r y k University, 2014.
[18] Jeri L Little and Elizabeth L Bjork. O p t i m i z i n g multiple-choice tests as
tools for learning. Memory & cognition, 43(l):14-26, 2014.
[19] Memrise L t d . A b o u t - memrise. http://www.memrise.com/
about/, 2015. [Online; accessed 2015-03-23].
[20] Elizabeth J Marsh, H e n r y L Roediger, Robert A Bjork, and Elizabeth L
Bjork. The memorial consequences of multiple-choice testing. Psychonomic
Bulletin & Review, 14(2):194-199, 2007.
[21] Juraj Nižnan, Radek Pelánek, and Jiří Řihák. Student models for prior
knowledge estimation, 2015. To appear i n Educational Data M i n i n g
( E D M ) , 2015.
[22] Rudolf Olah. Angular-google-experiments. h t t p : / / g o a u g u s t .
g i t h u b . i o / a n g u l a r - g o o g l e - e x p e r i m e n t s / , 2014. [Online; accessed
2015-03-23].
[23] Robert M Olton. The effect of mnemonic u p o n the retention of pairedassociate
verbal material. Journal of Verbal Learning and Verbal Behavior,
8(l):43-48,1969.
47
7. C O N C L U S I O N
[24] Jan Papoušek and Radek Pelánek. Impact of adaptive educational system
behaviour o n student motivation, 2015. To appear i n Artificial
Intelligence i n Education (AIED), 2015.
[25] Jan Papoušek, Radek Pelánek, and Vít Stanislav. A d a p t i v e practice of
facts i n domains w i t h varied prior knowledge. In Educational Data
Mining (EDM), pages 6-13, 2014.
[26] Jan Papoušek, Radek Pelánek, Jiří Řihák, and Vít Stanislav. A n analysis
of response times i n adaptive practice of geography facts. To appear
in Educational Data M i n i n g ( E D M ) , 2015.
[27] Philip I Pavlik and John R Anderson. Practice and forgetting effects on
vocabulary memory: A n activation-based model of the spacing effect.
Cognitive Science, 29(4):559-586, 2005.
[28] Radek Pelánek. M o d e l i n g students' memory for application i n adaptive
educational systems, 2015. To appear i n Educational Data M i n i n g
( E D M ) , 2015.
[29] A n d y V a n Schaack, Nicolas Schwighofer, and A n d r e w S Lewis. A
new method to maximize the effectiveness and efficiency of learning,
memory, and performance. h t t p s : / / c e r e g o . c o m / p d f / C e r e g o _
W h i t e p a p e r . p d f . [Online; accessed 2015-03-31].
[30] U r i Shaked. Angularjs vs. backbone.js vs. ember.js. h t t p s ://www.
a i r p a i r . c o m / j s / j a v a s c r i p t - f r a m e w o r k - c o m p a r i s o n , 2014.
[Online; accessed 2015-03-23].
[31] PyData Development Team, pandas: Python Data Analysis Library.
Online, 2012.
[32] Ondřej Volný, Radovan H u d a k , D a v i d Kachlik, and Miroslava
Dvořákova. M e m o r i x of anatomy, a modern, didactic book created
by medical students and residents for medical students, trends i n the
teaching of anatomy i n 21st century. INTED2014 Proceedings, pages
3621-3629,2014.
[33] W3Techs. Usage statistics and market share of traffic analysis tools
for websites, march 2015. h t t p : / / w 3 t e c h s . c o m / t e c h n o l o g i e s /
o v e r v i e w / t r a f f i c _ a n a l y s i s / a l l , 2015. [Online; accessed 2015-
03-23].
48
Appendix A
Source code
Source code implemented i n this thesis is i n one of the following Git repositories
a n d clones of the repositories are i n the Archive of Thesis i n Information
System of M a s a r y k University:
The geography w e b application
h t t p s : / / g i t h u b . c o m / a d a p t i v e - l e a r n i n g / g e o g r a p h y
The P y t h o n script for generating the maps
h t t p s : / / g i t h u b . c o m / a d a p t i v e - l e a r n i n g /
geography-map-generator
Data analyses
h t t p s : / / g i t h u b . c o m / s l a w e e t / t h e s i s - a n a l y s i s
49
Appendix B
Visualizations
World
— Europe
— Africa
Asia
— North America
South America
— Czech Rep.
— United States
i mi 1 1 1 1 1
date
Figure B . l : Portion of users o n most practiced maps i n time
50
B. V I S U A L I Z A T I O N S
B. V I S U A L I Z A T I O N S
1.0
0.8
0.6
0.4
0.2
0.0
1.0
0.8
0.6
0.4
0.2
0.0
(a) Morning vs. evening
III,T - H r \ i n n - * t L r i i c i ^ c o a i o < - < < " M m * t L r n D r v
~ c o a i
(c) non-returning vs. returning
1.0
0.8
0.6
0.4
0.2
0.0
I...
1.0
0.8
0.6
0.4
0.2
^ • ^ I-H r o i n r-- c i r H m i n r~ a i i H m i n r~ a i r H m m r-~ a i
i—11—11—ii—11—irsi o j f N f N rsi c o r n c o r n r n
(b) In-school usage
1.0
0.8
0.6
0.4
0.2
0.0 1i n ^ - m f N t - H O ' - H f N n n T j - i n
(d) low vs. high skill
(e) Response time (seconds)
Figure B.3: Different thresholds for d i v i d i n g users
52
B. V I S U A L I Z A T I O N S
Figure B.4: Difficulty ratings b y divisions
Is school Is evening Has low Has high Is returning
u s e r
user response time prior skill user
Figure B.5: Success rate b y divisions
53