MASARYK
U N I V E R S I T Y
FACULTY OF INFORMATICS
Recognition of Markers and Letters
in Images of Cards
Bachelor's Thesis
TOMAS JANOUSEK
Brno, Spring 2022
MASARYK
U N I V E R S I T Y
FACULTY OF INFORMATICS
Recognition of Markers and Letters
in Images of Cards
Bachelor's Thesis
TOMÁŠ JANOUŠEK
Advisor: doc. RNDr. Pavel Matula, Ph.D.
Department of Visual Informatics
Brno, Spring 2022
Declaration
Hereby I declare that this paper is my original authorial work, which
I have worked out on my own. A l l sources, references, and literature
used or excerpted during elaboration of this work are properly cited
and listed in complete reference to the due source.
Tomáš Janoušek
Advisor: doc. RNDr. Pavel Matula, Ph.D.
iii
Acknowledgements
I would like to thank my advisor, doc. RNDr. Pavel Matula, PhD, for
continuous support from the beginning to the end. Moreover, I thank
Be. Jakub Doucek, from CGE, for technical support and helpful advice
that improved the marker scanning algorithms. Lastly, I appreciate
my friends who took their time to help with a dataset for the letter
recognition algorithms.
iv
Abstract
This thesis solves the problem of scanning images containing cards
with markers and letters. It researches both marker scanning, and letter
recognition approaches. Furthermore, it develops several methods
using both approaches. It defines metrics to assess their capabilities,
mainly accuracy and speed and evaluates methods on three custom
datasets. The transfer learned convolutional neural network EfficientNetV2
is the most accurate method, and the fastest is the altered
ArUco scanning algorithm. The thesis also investigates methods' hyperparameters
and visualises defined methods. All these methods use
open-source techniques so that they can replace paid approaches.
Keywords
marker scanning, ArUco, OpenCV, optical character recognition, transfer
learning, card detection, EfficientNetV2, MobileNetV2, TensorFlow
v
Contents
Introduction 1
1 Statement of the Problem 2
1.1 Letter Jam Cards 2
1.2 Scanning Application 3
1.3 Scanning Approaches 3
1.4 Testing Metrics 4
1.5 Custom Prepared Datasets 6
1.5.1 Training Dataset 8
1.5.2 Validation Dataset 8
1.5.3 Test Dataset 8
2 Marker Scanning 11
2.1 Related Work 11
2.2 Letter Jam Scanning Procedure 12
2.2.1 Rectangular Scanning 12
2.2.2 New Dictionary Generation 14
2.2.3 Single Output Value 15
2.2.4 Hyperparameters Tuning 15
2.3 Testing 17
2.4 Results 18
3 Letter Recognition 20
3.1 Related Work 20
3.2 Letter Jam Recognition Methods 21
3.2.1 Transfer Learning Methods 22
3.2.2 Background Removal Methods 24
3.2.3 Card Detection Methods 28
3.3 Testing 32
3.4 Results 34
Conclusion 37
Bibliography 39
Appendix 46
vi
List of Tables
1.1 Prepared datasets 7
2.1 Validating marker scanning methods 17
2.2 Testing marker scanning methods on Training Dataset . . 18
2.3 Testing marker scanning methods on sharp images 18
3.1 Validating transfer learned networks 23
3.2 Validating background removal methods 29
3.3 Validating card detection methods 32
3.4 Testing letter recognition methods on sharp images . . . . 33
3.5 Testing letter recognition methods on moving images . . . 33
vii
List of Figures
1.1 Three Letter Jam cards 2
1.2 Scaled training images with the A+ card in several variations 7
1.3 Subcategories of the Test Dataset 10
1.4 Counts of images in each subcategory of the Test Dataset 10
2.1 Different markers 12
2.2 Processing of single image using altered ArUco method
with most prevalent functions 13
2.3 Solving the scaling problem using squares of sides . . . . 13
2.4 Tilting rectangular marker produces a square 14
2.5 All false positive testing images with green detected markers,
red orientation corner and yellow id 19
3.1 Accuracy of methods in each epoch of training; the blue
line is the training accuracy, and the orange is the "control
accuracy"; the accuracy on the unseen part of the dataset. 23
3.2 Background removal methods 25
3.3 Background removal problem: when applying global thresholding,
background fill and small components removal to
the rotated N card becomes indistinguishable to the Z card. 25
3.4 Background removal procedures 26
3.5 Applying various local thresholding kernel sizes 26
3.6 Keras OCR predictions 29
3.7 Training Dataset cropped using CRAFT 31
3.8 Accuracy of card detection methods in each epoch . . . . 32
3.9 Rotated partial image classified as FP by all tested methods. 33
3.10 Visualising EfficientNetV2 36
viii
Introduction
Letter Jam is a cooperative tabletop game that uses unusual playing
cards with markers and letters. A scanning application scans these
cards at the start of the game.
This thesis proposes a fast and accurate scanning algorithm for
Letter Jam cards. Scanning is problematic because it can occur both
during the day and night, and the place can also be different, so the
algorithm needs to be robust enough to withstand all these different
conditions. These cards are unique for the Letter Jam game, and
consequently, there has been no previous work about scanning them.
The thesis aims to design the best algorithm for scanning Letter
Jam cards, not the application itself. It employs several metrics and
custom datasets to assess the best algorithm. It utilises open-source
libraries to provide a cheaper alternative to paid solutions.
Chapter 1 presents the problem, metrics and datasets. Chapter 2
concentrates on scanning markers using C++ and the computer vision
library OpenCV [1]. It builds upon commonly used methods of
scanning square markers, which improves, evaluates and visualises.
Chapter 3 moves to letter recognition using Python and the machine
learning library TensorFlow [2]. It introduces convolutional neural
networks and related optical character recognition and object detection
methods. Thesis designs new approaches for preprocessing and
visualisation. It considers the benefits of a faster and more inaccurate
scanning against the one with higher accuracy and lower speed.
1
1 Statement of the Problem
This chapter introduces the Letter Jam game and its unique playing
cards. It shows how a scanning application is used for the game setup.
It briefly acknowledges differences in two scanning approaches, with
concrete methods described in Chapters 2 and 3. Lastly, it defines
metrics to evaluate scanning quality and datasets for its testing.
1.1 Letter Jam Cards
Letter Jam is a cooperative tabletop game. There are 64 unusual playing
cards containing 21 letters and 64 unique markers; three of these cards
are shown in Figure 1.1; notice the markers are different for the same
letters. Less frequently used letters, like X and F, are not included in
the deck as I use cards from the Czech version of the game. Each card
contains two same markers at the top, two markers at the bottom, and
five same letters, one dominant letter in the middle and four small
letters in the corners. Markers side by side are rotated 180°. The top
and bottom letters are also rotated.
Figure 1.1: Three Letter Jam cards
2
l . STATEMENT OF THE PROBLEM
1.2 Scanning Application
The cards are essential for the game from the beginning. Each player
needs to have a random five-letter word in front of them. Players do
not know what words they have; their goal is to deduce them. They
sequentially uncover individual letters, and in the end, they guess the
final word on the table.
Originally, each player assembled a hidden word for one of their
playmates. It was a manual process in which each player had to come
up with a five-letter word and send it to another player. A mobile
phone application replaces this process.
Nowadays, one player puts their mobile phone with an installed
scanning application on the table. The application constantly takes
snapshots using the front camera. The player then picks up cards from
the deck facing down. The player does not read the letter; they only
move the card in front of the camera. If the application recognises
the card, it shows whom to give the holding card to. This process
continues until a word for each player is assembled. No player knows
the words used, which makes the game more enjoyable. It can also
make the whole game setup faster.
The library currently used by Czech Games Edition is paid and
closed-source VuForia Engine [3]; therefore, if the proposed scanning
application uses open-source libraries, it will be cheaper.
1.3 Scanning Approaches
I distinguish two ways how to scan the cards. The first is to scan the
markers on cards as they were designed for scanning. The second option
is the recognition of letters, which needs careful results handling.
Suppose the scanning application scans one marker corresponding
to the letter " E " . It shows whom to put the card, and after a second,
it scans a second card with a marker again corresponding to the " E "
letter. The player still holds the same card if the marker on the second
card is identical to the first due to no two cards having the same
marker. If they are different, it is a different card. There should also
be a unique background class (label), which is returned when no
markers are found.
3
l . STATEMENT OF THE PROBLEM
Scanning is ambiguous if the application recognises only the letters
instead of markers. The application cannot tell if the second snapshot
with the letter " E " still identifies the same card as in the first snapshot
or if is it a different card.
The background class, which the application returns when no
letter is recognised, solves this ambiguity. Imagine three snapshots,
first classified as the letter "E", second classified as a background and
third also as the letter "E". The application can now assume these are
two different cards; when the player was passing the card to someone,
the application scanned the background. Conversely, if the second
snapshot is also classified as the letter "E", the application can presume
these are snapshots of the same card.
The letter recognition procedure works only if the player does not
show the same card to the algorithm twice, with a background in the
middle. To prevent misclassifications, the player needs to be instructed
to hold the card still and move it only after successful recognition.
1.4 Testing Metrics
Input to the scanning application is a single image taken from mobile
phone camera. The output depends on the scanning approach, the
marker scanning application returns identification number of a found
marker; a number from 0 to 63 as there are 64 cards. Number 64
identifies the background class described in the previous section. The
algorithm that recognises the letters returns single character letter or
a special character "-" corresponding to the background.
There are more ways how to evaluate multiple class classification.
The first way is to treat all classes, including the background class,
the same. Described approach constructs one large confusion matrix
with each class as its row and column [4]. In the second approach,
the application classifies images using confusion matrix for binary
classification. It asks whenever is it an image of a particular class or
not for each image and then combines individual results [5].
I used neither of these approaches, as the background class plays
a different role for each approach. Accordingly, I developed a third
approach for classification. I took inspiration from the binary classification
and used a 2x2 confusion matrix for evaluation. A positive
4
l . STATEMENT OF THE PROBLEM
image contains a card, whereas a negative image does not; true means
correctly assigned label, and the false is the opposite.
1. True Positives (TP) are correctly classified card images.
2. False Positives (FP) are card images classified as another card.
3. False Negatives (FN) are card images classified as a background.
4. True Negatives (TN) are correctly classified background images.
This simplification of the multiclass problem as a binary provides
many benefits; mainly, the background label that plays a different role
in both approaches is separated from the other labels. Second, I can
define standard metrics binary classification uses.
Accuracy (ACC)[5] is the sum of true positives and true negatives divided
by the number of all images in the dataset. It is the percentage of images
that the algorithm labels correctly. Although it hides differences in
false negatives and false positives, it is a good metric to compare both
the scanning approaches.
Precision (PPV)[5] is the number of true positives divided by the sum
of true positives and false positives. It is the percentage of rightfully
assigned card labels of all found card labels. This metric is valuable for
marker scanning algorithms that must be precise to assemble words
without mistakes.
False Omission Rate (F0R)[6] is the number offalse negatives divided
by the sum of false negatives and true negatives. It characterises the
portion of card images which the method misses. It is only a delay for
marker scanners, whereas for letter recognition models missing a card
can make the application assemble a wrong word.
Response Time (RT)[7] measures the processing time for a given
task. For the Letter Jam application, it is the delay between calling
the method for recognition and receiving its result. I used milliseconds
as the measurement unit. I used a personal computer with Intel®
Core™ i5-4590S Processor, NVIDIA GeForce GTX 1060 3GB and
5
l . STATEMENT OF THE PROBLEM
R A M of 16 GB for all measurements. I also used the mobile phone
Huawei Mate 10 Lite algorithm to check if the methods are still usable
on an older Android mobile phone.
Inverse Efficiency Score (IES)[8] divides the response time by accuracy.
I measure it in milliseconds. Studies differ if response time should
also account for the false cases. I choose to include them because the
application needs to be fast on all images, not only on those it correctly
classifies. Although it is a metric from psychology [9], it is valuable
for scanning Letter Jam cards.
I interpret it as an average time until the application recognises a
card or just the ineffectiveness of the algorithm. Consider two applications,
one with 90 % accuracy and response time of 10 ms and the
second with 50 % accuracy and response time of 5 ms. The first application
has the IES of 11.11 ms and the second 10 ms, meaning the latter
is better. The metric enables a fast algorithm to have an advantage
over the one with higher accuracy and vice versa.
Some authors prefer to use plain response time and accuracy instead
of combining them [10]; hence, I distinguish all three metrics.
1.5 Custom Prepared Datasets
I made datasets to train, validate and test the proposed algorithms. I
shot these datasets using mobile phone cameras in various locations
and periods. For example, some images are taken at night and others
during the day. There are also images of cards from close and from far.
I call these differences variations. Each variation has a fixed location,
time and mobile phone type. The algorithm will generalise better
with more variations, especially during the training of a deep neural
network [11]. I sum variations of datasets in Table 1.1.
Images in the datasets show one card and are labelled usually by
the id of a marker for marker scanning algorithms and letter for letter
recognition algorithms. I prefer to use a more readable version of
numerical representation in the Train and Validation dataset. I put
the letter first and optionally add pluses to distinguish different markers
for the same letter. For example, " A + " means the second type
of marker for a card containing the letter "A". Datasets also contain
6
l . STATEMENT OF THE PROBLEM
Figure 1.2: Scaled training images with the A+ card in several variations
the background class described in Section 1.3. A l l images are 1080
pixels wide and 1920 pixels high. I flipped back some images to maintain
human-readable letters and markers due to some mobile phones
horizontally flipping images taken using the front camera.
Table 1.1: Prepared datasets
dataset variations all images backgrounds
Training Dataset 44 2 816 88
Validation Dataset 10 5 864;6 020fl
210;617&
Test Dataset 11 1 560 512
a. This second number describes the number of images used for letter recognition.
b. Some of the background images for letter recognition algorithms contain marker.
7
l . STATEMENT OF THE PROBLEM
1.5.1 Training Dataset
The Training Dataset with 44 variations is the most diverse. Primarily
its purpose is to train convolutional neural networks proposed in
Chapter 3; however, it has marker labels to evaluate marker recognition
algorithms from Chapter 2. Figure 1.2 shows its examples.
I created this dataset rather carefully; each video contains cards
for the same time. After that, I used a command-line framework for
processing videos FFmpeg [12] to take snapshots every n seconds,
where I guessed the n multiple times to have sharp images. In the
end, I removed duplicate cards and added the missing ones. I used
2 904 images from a total number of 383 940 images. I labelled images
using the unused marker scanner algorithm StepArUcolO described
in Chapter 2.2.4 to ensure that marker scanning algorithms are capable
of recognising all types of markers. I labelled manually the remaining
images which were not labelled by the algorithm. Some images do not
contain letters or markers; I call these images backgrounds.
1.5.2 Validation Dataset
This dataset contains recorded videos of cards in motion. It contains
5 864 images, 5 654 of which contain markers shown. Dataset has ten
different variations. The dataset includes 156 additional images for
letter recognition algorithms because some videos contain a portion
of a marker for some part of the video. Dataset does not contain an
image for each card in the deck. It should be used in conjunction with
the Training Dataset for marker scanning algorithms to determine if
all markers are recognisable.
1.5.3 Test Dataset
The Test Dataset is the most important. I use it to compare both the
marker scanning algorithm and the letter recognition methods. It
includes 1 560 images in 11 variations. Whereas the Training Dataset
contained sharp images, this dataset was created without a timer.
Some cards ended up occupying more time in the video, some less.
I took 130 snapshots from each video at equal length intervals. To
better understand the methods, I then divided the dataset into five
subcategories, with examples at Figure 1.3 and counts at Figure 1.4.
8
l . STATEMENT OF THE PROBLEM
1. Sharp images contain legible markers and letters fully in the image.
Marker and letters are legible if humans can unambiguously
distinguish them from other markers and letters. Sharp background
images do not contain any cards or hands. These images
make up approximately 85 % of the dataset.
2. Moving images contain illegible markers but letters fully in the
image. Because markers are not legible for humans, they have
marker labels set to the background, even though the card is
present. Moving background images contain an illegible card.
These images represent about ten per cent of the dataset.
3. Partial images contain a legible marker but not a full letter. The
letter label was used instead of the background label because
there are smaller letters that the letter scanning algorithm should
distinguish. The partial background images contain an illegible
card and a hand. They make less than two per cent of images.
4. Hidden images for other images. These images do not contain
a whole letter or legible marker. These images are classified as
background for marker scanning algorithms and with a letter
for letter recognition algorithms. Hidden background images
contain illegible cards. They describe almost four per cent of all
images of the dataset.
5. Extra images is a subcategory of background images that contain
a special star card from the game, which the player should not
scan. However, I deliberately scanned this card in some of the
source videos. However, after processing all images from videos,
there remained only two such images.
9
l . STATEMENT OF THE PROBLEM
1321
S H A R P
I M A G E S
150
M O V I N G
I M A G E S
26
P A R T I A L
I M A G E S
61
H I D D E N
I M A G E S
E X T R A
I M A G E S
Figure 1.4: Counts of images in each subcategory of the Test Dataset
10
2 Marker Scanning
This chapter shows that marker scanning is an evergreen problem
with many applications. Following, it alters a related marker scanning
algorithm to scan Letter Jam cards. Next, it proceeds to find the best
hyperparameters for scanning. Lastly, it evaluates the methods using
previously defined datasets and visualises their misclassifications.
2.1 Related Work
One of the oldest markers still used today is the barcode. It is used
mainly in shopping malls where dedicated lasers scan the intensities
of the bars. It was patented in 1951 [18].
A marker from 2000 is the QR code. It is a two-dimensional square
marker containing the data, error-correcting codes, version information
and format [19]. Figure 2.1a depicts one QR code example. ZBar
is an open-source scanning algorithm for QR codes and barcodes. It
takes inspiration from lasers; however, it scans the intensities using a
traditional camera [20].
The AprilTag simplifies the QR Code; it has a black border and
contains the data in the middle. Data needs to be asymmetric because
there is no rotation information as in the QR code. Its primary purpose
is a position estimation of an object in the scene. A dedicated
open-source algorithm written in C scans these markers. It computes
gradients of pixels (magnitudes and directions) and then clusters
pixels with a similar gradient. Lastly, it fits lines among those clusters.
Figure 2.1b shows a circle variant of AprilTag marker [21].
A similar method to the AprilTag is called ArUco, which provides
fewer false positives during scanning. A n example can be seen in
Figure 2.1c. It utilises local thresholding, contour detection and perspective
transformations [22].
The most customisable of these examples is Vuforia's VuMark. For
example, it can have a ribbon shape with green code in the middle, as
seen in Figure 2.1e. It is designed for augmented reality applications.
A proprietary scanning library detects them [23].
VuMarks inspired codes on the Letter Jam cards; they are rectangular
and with a light grey colour, as shown in Figure 2.If.
11
2. M A R K E R SCANNING
(a) QR code [13] (b) AprilTag [14] (c) ArUco [15]
1234561
(d) Barcode [16] (e) VuMark [17] (f) Letter Jam marker
2.2 Letter Jam Scanning Procedure
Markers on Letter Jam cards are similar to ArUco markers; the only visible
differences are the rectangular shape and less saturated colours.
Implementation of ArUco scanning in OpenCV is helpful because
the library provides simple algorithms like image scaling and global
thresholding; furthermore, it provides advanced algorithms for contour
detection and perspective transformations essential for ArUco.
However, it is not part of the standard OpenCV library; it is a contributed
module that needs to be built separately.
I adapted its scanning procedure and recreated the ArUco library
to scan markers on Letter Jam cards. A top-level overview is seen in
Figure 2.2 and the source code as an attachment the thesis provides. I
used the C++ version of the library and, for testing, compiled it for
Release, which substantially improved the processing speed.
2.2.1 Rectangular Scanning
Letter Jam markers are rectangles, whereas ArUco markers are squares.
It poses a problem in rotation; if the application scales a marker using
the wrong axis, it produces a stretched marker it does not recognise.
To tackle this issue, I distinguished rectangles application recognises
seen in Figure 2.2d in two categories. First for horizontal markers
and the second for vertical markers. I named each of the four corners
the algorithm recognises as A, B, C, D and their respective sides
Figure 2.1: Different markers
12
2. M A R K E R SCANNING
(a) Original image (b) Greyscale image [24]
(c) Local thresholding [25] (d) Search for borders [26]
(e) Transform perspective [27] [28] (f) Final thresholding [29]
Figure 2.2: Processing of single image using altered ArUco method
with most prevalent functions
wrong scaling
correct scaling
Figure 2.3: Solving the scaling problem using squares of sides
13
2. M A R K E R SCANNING
Figure 2.4: Tilting rectangular marker produces a square
a, b, c, d. I assume it is a horizontal marker if a2
+ c2
< b2
+ d2
. Figure
2.3 shows procedure visually. This solution is fast; however, it
might not work when the recognised rectangle is a perfect square,
which happens if I highly tilt the card; an example is in Figure 2.4.
2.2.2 New Dictionary Generation
The described algorithm needs to identify the marker after the final
thresholding step Figure 2.2f. It needs to compare the final thresholded
image with all saved markers and find the most similar marker. Each
marker can be represented with a logical matrix (two-dimensional
boolean array) of four rows and eight columns. Bytes in C+ + contain
eight boolean values; therefore, representation can be simplified from
array to four bytes. It could be then translated to just a single integer;
however, the C++ standard does not enforce it to have 4 bytes [30].
This simplification of representation improves speed, as the algorithm
needs only to compare four instead of 32 numbers. However, it
lowers the readability of markers for future algorithm developers. I created
a script to translate images of markers to the described byte array.
Script also computes markers' rotations to improve speed further.
This script is used once by a programmer who wants to update
the dictionary of markers, for example, when creating a scanning algorithm
for the Russian or Chinese Letter Jam game. In these versions,
cards are almost the same. They contain different letters; however,
markers have the exact dimensions and colours.
14
2. M A R K E R SCANNING
2.2.3 Single Output Value
ArUco algorithm can produce multiple outputs for one image, one for
each found marker. It can also return an empty list when it does not
find any marker. I modified this behaviour to choose only the most
frequent marker in the image. When it contains no marker, it returns a
background class. This class is helpful for the classification of images
that do not contain a card at all or when the image contains only a
blurry and illegible marker.
2.2.4 Hyperparameters Tuning
I investigated algorithm hyperparameters for the scanning ArUco
markers. I also experimented with the size of the processed image.
1. Scanning tolerance. It modifies the number of correction bits that
are used when a slightly different marker is found. For example,
scanning tolerance is 50 %, and there are two correction bits in the
dictionary; using simple multiplication, I obtain the actual number
of bits that can be different. It is one bit so that the scanned
marker can differ from the closest match in the dictionary only
at one position.
I created a script that calculates the minimum Hamming distance
between all marker pairs. The more diverse the markers are, the
better [31]. The calculated distance is 14; therefore, if the algorithm
sees a marker that differs from the marker in the library in
less than 14 bits, it can assume it has found a suitable marker. The
bits are flipped randomly in an ideal situation, although uneven
lighting can drastically affect recognition in practice. Therefore,
I used 7 bits which is then lowered using scanning tolerance.
2. Number of thresholding steps. The scanning application uses, by
default, three local thresholding steps of different kernel sizes
to detect markers. Larger kernel sizes are generally suitable for
closer and larger markers and smaller ones for smaller and farther
markers. I altered only the maximum size of the kernel,
meaning if I use more thresholding steps, larger markers can be
recognised.
15
2. M A R K E R SCANNING
3. Constant value for thresholding. This value is subtracted from the
mean when using adaptive threshold function [25].
4. Size of the image. This new hyperparameter not defined in the
ArUco library controls the downsampling of the processed image.
For example, if I set the size to 25%, the algorithm first scales
images to a quarter its original size, then uses other functions as
normal.
I tried affecting other algorithm steps to improve contour detection
and perspective transformation; however, without significant improvement
to accuracy or response time. Lastly, there were some parameters
which I specifically removed, namely for accurate corner detection
and pose estimation. Summary of methods:
1. FaithfulArUco uses default parameters for ArUco scanning with
zero scanning tolerance.
2. CorrArUco7 modifies FaithfulArUco to sets the correction bits in
the dictionary to 7 as described in the Section 2.2.2. The number
of bits difference is multiplied using 60 % scanning tolerance,
meaning an actual number of correction bits is only 4.
3. ConstArUco uses CorrArUco7 but lowers a constant value for the
thresholding step to 3.5.
4. StepArUco5 uses two more thresholding steps than the default
by selecting a higher bound for the thresholding kernel size.
5. StepArUcol 0 uses ten thresholding steps, meaning five more than
StepArUco5 and also lowered the constant value for thresholding
as in ConstArUco.
6. SmallArUco5 resizes the image to half its size and then uses parameters
as in StepArUco5.
7. SmallArUcolO resizes the image to half its size and then uses
parameters as in StepArUcolO.
8. QuarterArUco resizes the image to quarter its size and then uses
one more thresholding step than StepArUco5.
16
2. M A R K E R SCANNING
Table 2.1 shows validation results. I want to choose the three best
methods to compare on the Test Dataset. Comparing the IES and the
response time, the method QuarterAruco is the best. The most precise
method is SmallArUco5, which is also the method with the secondlowest
IES and higher overall accuracy than QuarterAruco. Lastly, I
choose a method that does not perform any resizing, which might be
beneficial for images where the card is further away from the camera.
Although StepArUcolO is the most accurate, I choose StepArUco5
because it is more than two times faster and has higher precision.
Table 2.1: Validating marker scanning methods
name A C C PPV FOR RT IES
FaithfulArUco 33.32 % 99.89 % 95.46 % 20.81 ms 62.44 ms
CorrArUco7 34.47 % 99.84 % 95.38 % 20.05 ms 58.17 ms
ConstArUco 60.91 % 99.88 % 92.48 % 25.99 ms 42.66 ms
StepArUco5 81.36 % 99.91 % 85.41 % 27.53 ms 33.84 ms
StepArUcolO 87.60 % 99.86 % 79.47 % 65.52 ms 74.79 ms
SmallArUco5 84.93 % 99.96 % 82.58 % 7.50 ms 8.83 ms
SmallArUcolO 85.64 % 99.88 % 81.80 % 17.27 ms 20.17 ms
Quarter ArUco 79.04 % 99.93 % 86.83 % 3.68 ms 4.65 ms
2.3 Testing
First, I will be testing methods using the Training Dataset described
in Chapter 1.5.1. All three chosen methods have a precision of almost
100 %. Although StepArUco5 classified the images with the highest
accuracy, its processing time is also the highest. QuarterAruco is the
exact opposite, having the lowest accuracy and highest speed. SmallArUco5
stands in the middle of these two methods, having average
speed and accuracy. Although the dataset contains mostly sharp images,
the average accuracy of the methods is only 70 %. Details are
depicted in Table 2.2.
Next, I move to the Test Dataset from Chapter 1.5.3. Methods tested
on moving, extra and hidden images are equally good. They classify all
these images as backgrounds, which is the desired behaviour. For
17
2. M A R K E R SCANNING
partial images, they are almost identical; they classified half of the
images and StepAruco5 one less image than the rest.
The results are similar to the validation and the training dataset for
sharp images and can be seen in detail in Table 2.3. The QuarterAruco
achieves higher accuracy than in the training dataset because many
images have backgrounds label, which all algorithms classify well.
StepAruco5 has the lowest precision, with two false positives reported,
but one of these markers is falsely labelled. It is evident from the
visualisation in Figure 2.5a that shows that all four markers are found
in the right spots. Although StepArUco has the lowest false omission
rate, its response time and inverse efficiency rate are the highest.
Table 2.2: Testing marker scanning methods on Training Dataset
Model A C C PPV FOR RT IES
QuarterAruco
SmallAruco5
StepArUco5
53.96 %
75.69 %
81.06 %
99.93 %
100.0%
100.0%
93.82 %
88.92 %
86.21 %
3.07 ms
6.78 ms
30.14 ms
5.70 ms
8.96 ms
37.18 ms
Table 2.3: Testing marker scanning methods on sharp images
Model A C C PPV FOR RT IES
QuarterAruco
SmallAruco5
StepArUco5
66.92 %
75.85 %
77.67 %
100.00%
100.00%
99.68 %
51.65 %
43.82 %
41.80 %
4.01 ms
9.03 ms
37.43 ms
6.00 ms
11.90 ms
48.19 ms
2.4 Results
Tested methods have high precision; however, their accuracy is at best
81%. Scanning lowered version of the image improves scanning speed
more than worsens the accuracy or false omission rate. Moreover,
SmallArUco5 is a method that has 100 % precision on both datasets.
I visualised all images that produced false positives in Figure 2.5.
Visualisation is possible because the altered ArUco method still returns
pixel positions of found markers. From the visualisation, I can
see that methods sometimes try to assign one marker to both markers
18
2. M A R K E R SCANNING
side by side or assign a marker using a tiny portion of the image. If
it is a recurring problem in the future, I suggest that the algorithm
needs to find at least two markers to classify the image and both with
some minimum area.
A l l in all, QuarterAruco is the fastest and has the lowest accuracy.
StepAruco5 is the exact opposite, with the highest accuracy and lowest
speed. For a good balance of high speed and high accuracy, I suggest
using SmallAruco5 which worked well for both testing datasets.
(a) Cropped image (b) Background im- (c) Card image miscorrectly
classified age missclassified by classified by Quarterby
StepAruco5. StepAruco5. Aruco.
Figure 2.5: A l l false positive testing images with green detected markers,
red orientation corner and yellow id
19
3 Letter Recognition
This chapter explores different ways how to recognise letters on Letter
Jam cards. First, it describes optical character recognition (OCR) techniques
and the current state of the art. Furthermore, it proposes new
methods for preprocessing. Lastly, it uses previously defined metrics
and datasets to assess their capabilities and visualises the best method.
3.1 Related Work
Many OCR algorithms for handwritten letters exist; for example, fully
connected neural networks, kernel methods, statistical methods, and
pattern matching [32]. However, the MNIST dataset containing handwritten
digits [33] is quite different from the images recognised by the
scanning application of the Letter Jam game. It contains handwritten
digits on a uniform background, whereas Letter Jam characters have
a unified font on backgrounds that can change drastically based on
lighting conditions and shooting location.
The convolutional neural networks are also capable of recognising
different fonts [34]. One method recognising non-Latin characters
utilised discrete curvelet transform, fc-NN classifier, and statistics [35].
Tesseract OCR is a popular letter scanner that uses LSTM [36] neural
network and can recognise text in books [37] or simple scenes.
However, Tesseract is not much useful for recognising texts in photos
containing text partially transformed or occluded [38]. A l l these
mentioned methods require an easily separable background.
The convolutional neural networks [39] gained attention with
AlexNet architecture to classify ImageNet dataset [40]. ImageNet is
a large dataset containing real photos of various objects like ships,
cars and animals. Although this dataset does not contain any text
recognition tasks, a branch of neural networks called transfer learning
can be used. Transfer learning is a method that takes a large model
trained on a vast dataset and uses it for another task with minimum
architecture changes [41]. TensorFlow Hub is an online repository
with many pre-trained models [42].
Keras OCR [38] is an open-source algorithm that can detect text
in real-world photos, for example, on posters, buildings or road signs.
20
3. LETTER RECOGNITION
It uses a combination of two techniques. First uses the CRAFT [43]
convolutional neural network to find text in the image. Second, it uses
a combination of convolutional neural network and recurrent neural
network called C R N N [44] to read the text. It is written in Python
using TensorFlow. It was trained using COCO-Text [45] dataset. This
dataset, unlike the MNIST, has texts in various scenes and contexts.
In the field of object recognition methods, some methods determine
the presence of a given object in an image. Others localise the object
with an accurate pixel position; some recognise all the objects in an
image, not only the most prevalent. Some of these methods employ
volumetric parts or automatic programming or geometric invariants,
or formal grammars [46]. Moreover, some experimental methods use
support vector machines and local image texture matrices [47]. YOLO
is a famous detector of many different objects, including cars, animals
and furniture, based on convolutional neural networks [48].
One can draw inspiration from traditional playing cards for poker
or blackjack for card detection. One can combine Hotelling transform,
discrete cosine transform, and run-length techniques to detect these
cards [49]. The blackjack method uses OpenCV with functions for
thresholding and edge detection with component analysis [50]. A
convolutional neural network can replace these traditional approaches
and detect cards on poker tables [51]. Another field for card detection
is business-oriented, with methods detecting credit cards or id cards.
One approach combines a corner searching, perspective transform
method and neural networks for text recognition on these cards [52].
3.2 Letter Jam Recognition Methods
I used Python and TensorFlow library. I recognised letters using three
state of the art architectures for the ImageNet dataset downloaded
using TensorFlow Hub. I also experimented with preprocessing using
OpenCV thresholding functions. Lastly, I attempted to crop the processed
image to a detected text and then recognise letters afterwards.
21
3. LETTER RECOGNITION
3.2.1 Transfer Learning Methods
I trained the following three architectures EfficientNetV2 [53], MobileNetV2
[54] and BiT-S R50xl [55] using transfer learning.
I used the following architecture to unite all models to the letter
recognition problem.
1. Input layer of size 224x224 with 3 dimensions (colours).
2. Chosen model architecture without the last layer.
3. Densely connected layer with 22 neurons - 21 letters and background
as described in Chapters 1.1 and 1.3.
4. 20 % Dropout and 0.0001 L2 regulariser - dropout [56] and regularises
are two standard techniques to improve accuracy [11].
I augmented [57] images during training to further improve accuracy
and reduce overfitting [11]. Pixels outside the border were filled
in with a mirror reflection of the image. I used the following three
augmentations:
1. Random rotation up to 360°.
2. Random translation up to 10 % in each axis.
3. Random zoom up to 10 % in or out.
I trained these networks on the Training Dataset described in Chapter
1.5.1. The neural network did not see 20 % of this dataset during
training; these images were selected randomly and were used
to control the training procedure. I call this part of the dataset Control
Dataset to not confuse it with the Validation Dataset described in
Chapter 1.5.2. After 100 epochs of training, I observed that all methods
achieved accuracy higher than 90% on the Control Dataset. Figure 3.1
shows the training of these methods.
Even though the training pipeline was relatively simple, all methods
performed well on the Validation Dataset. Their average accuracy
is almost 94 %, and time is less than 22 ms, with details shown in
the Table 3.1.1 chose the EfficientNetV2 as the best network; it has
22
3. LETTER RECOGNITION
Number of epochs
(a) BitNet
Number of epochs
(b) EfficientNetV2
Number of epochs Number of epochs
(c) MobileNetV2 (d) MobileNetV2
Figure 3.1: Accuracy of methods in each epoch of training; the blue
line is the training accuracy, and the orange is the "control accuracy";
the accuracy on the unseen part of the dataset.
Table 3.1: Validating transfer learned networks
architecture A C C PPV F O R R T IES
BitNet
EfficientNetV2
MobileNetV2
92.59 %
96.40 %
91.88 %
93.31 %
96.75 %
93.25 %
16.02 %
7.13%
19.11 %
52.97 ms
23.91 ms
14.73 ms
57.21 ms
24.81 ms
16.03 ms
23
3. LETTER RECOGNITION
the highest accuracy and second-lowest response time. BitNet has
higher accuracy than BitNet; however, its response time is the highest.
MobileNetV2 has the highest speed and lowest accuracy.
I tried training MobileNetV2 for an additional 100 epochs because
the graph of control accuracy showed slower improvement than the
other two methods, and validation accuracy was lower than the other
two; perhaps due to underfitting. However, extended training did not
help MobileNetV2. Similarly, EfficientNetV2 showed a fast training
time, so I tried reducing number of epochs to 50 to reduce overfitting.
Nonetheless, lowered training time did not improve the network.
The average accuracy could be further improved, for example, with
a more complex augmentation pipeline, more training examples or
training algorithm for precisely the best epochs. A better hardware
would improve scanning time or pruning the neural network, a process
in which we remove unimportant weights [58].
3.2.2 Background Removal Methods
The Training Dataset is diverse; it has different colours because cards
are under various lighting conditions. This diversity might confuse
neural networks to try to find patterns in this noise. In order to minimise
these differences, I tried to preprocess the images before a neural
network saw them. I created three preprocessing methods. Figure 3.2
shows preprocessing methods on four training images.
However, the background for card images can not be removed
entirely. Without background, letters Z and rotated N are almost unrecognisable,
as demonstrated in Figure 3.3. Thus I need to preserve
information about rotation, for example, by assigning a grey colour to
small components instead of black.
I created preprocessing methods and named them ThreshPrep
(thresholding preprocessing), BackPrep (background removal preprocessing)
and SepPrep (background separation preprocessing). I
used Python bindings for the OpenCV library for easier integration
with the TensorFlow library. Methods operate on resized images of
224x244 pixels and produce images of the same size.
ThreshPrep uses four grey colours to differentiate the important parts
of the image: the background, the card, the details and the hand. It
24
3. LETTER RECOGNITION
(a) Original images (b) ThreshPrep images
A l A
•A
(c) BackPrep images (d) SepPrep images
Figure 3.2: Background removal methods
Zl
•IS®1
Z
•TS)'
z-"S37
Figure 3.3: Background removal problem: when applying global thresholding,
background fill and small components removal to the rotated
N card becomes indistinguishable to the Z card.
25
3. LETTER RECOGNITION
+
(a) TreshPrep procedure
+
(b) BackPrep procedure
(c) SepPrep procedure
Figure 3.4: Background removal procedures
15 29 43 57
71 85 99 113
....
mm _ : «
m..•
127 141 155 169
Mm Mm m Mm
183 197 211 225
Figure 3.5: Applying various local thresholding kernel sizes
26
3. LETTER RECOGNITION
first utilises Gaussian blur with kernel size 5 to attenuate noise [59].
Next, it performs twice local (adaptive) thresholding [29]. First has a
larger kernel radius of 67 to separate the image into important image
components. The second thresholding uses a smaller radius of 15 to
separate the letter on cards.
I picked kernel sizes by hand that best separated the images in the
Training Dataset; an example of adjusting values for local thresholding
is in Figure 3.5. Lastly, the technique sums these two thresholded
images together, first multiplied by 191 and the second by 64.1 chose
not to multiply both using 128 because it would divide the image into
three parts, whereas using described scalars distinguish the image
into four different parts with intensities 0, 64,191 and 255.
Zero intensities are the borders of cards or hands; 64 is the value
of enclosed areas like cards, hands or planks on the ceiling. Value 191
contains minor details, these are mostly unimportant, but they can
provide finer borders around letters. Last are white regions with an
intensity of 255, which depict the background. Closer letters have an
intensity of 255, further away and smaller cards of 64.
BackPrep preprocesses the image to assign the colour of the background
around the card to grey instead of white. It uses a similar
technique to ThreshPrep. However, after each local thresholding, it
detects connected components [60]. Both thresholding steps set a flag
for components with at least one pixel on the image border. Lastly,
both steps set the colour of all flagged components to a grey value of
32 and 96, respectively.
The procedure produces similar looking images to the ThreshPrep;
however, the white colour is now mainly exclusive to letters, whereas
it was previously in the background. Nonetheless, the method does
not handle the cases in which the letter is further away and grey and
white enclosed spots around the fingers and card.
SepPrep enlarges small letters on images and makes them white. It
works similarly to BackPrep; however, it thresholds the image only
once, using a radius of size 67 without Gaussian blur. Afterwards,
it sets the flagged background components to black, using the same
component technique as BackPrep. Following, it considers the three
27
3. LETTER RECOGNITION
largest connected components in the image and makes them white.
These components are the letter, top and bottom markers in an ideal
situation. It assigns the colour of other components to grey to prevent
the discussed background removal problem. Lastly, it makes a square
bounding box around the white components and uses this bounding
box to scale all inner pixels to fit the image using the nearest neighbour
technique [61].
I can use ThreshPrep and BackPrep immediately during training,
validation and testing as they do not produce illegible images. SepPrep
needs a new dataset of training images since the preprocessing produces
about 5% of illegible markers. It happens, for example, when the
letter is the fourth most dominant connected component. To prevent
training on illegible markers, I altered the whole Training Dataset once
before training, distinguished illegible images manually and moved
them to the background class.
I train preprocessing methods using transfer learned MobileNetV2
architecture and the EfficientNetV2 with the same architecture as in the
Section 3.2.1 and evaluate them in the Table 3.2. Interestingly enough,
neither preprocessing helped architectures to have higher accuracy
on the Validation Dataset than unprocessed variants had. It might be
due to methods more likely returning background class, making their
false omittance rate higher. Unsurprisingly, the preprocessing makes
methods slower and consequently their IES higher.
Also, the same preprocessing on different architectures yields different
results. For MobileNetV2 architecture, the best preprocessing
is BackPrep; for Efficient-NetV2 architecture, the best is ThreshPrep
preprocessing. EfficientNetV2 with preprocessing is not better than
unprocessed variant in any metric; whereas MobBackNet and MobSepNet
are better in precision. I choose MobBackNet, the MobileNetV2
architecture with BackPrep for later and more extensive testing because
it has higher accuracy from the two methods.
3.2.3 Card Detection Methods
Many images from the Training Dataset have cards far from the camera.
It means a neural network needs first to find the card and not
be confused by other factors like ceiling colour, hand position, and
28
3- LETTER RECOGNITION
Table 3.2: Validating background removal methods
architecture ACC PPV FOR RT IES
MobThreshNet 88.84 % 90.23 % 25.09 % 18.95 ms 21.33 ms
MobBackNet 90.58 % 93.95 % 35.15 % 46.56 ms 51.40 ms
MobSepNet 84.10 % 95.22 % 55.11 % 51.91 ms 61.72 ms
EffiThreshNet 94.27 % 94.88 % 11.56 % 28.40 ms 30.13 ms
EffiBackNet 89.37 % 90.54 % 23.97 % 58.82 ms 65.81 ms
EffiSepNet 84.67 % 92.96 % 51.52 % 60.86 ms 71.88 ms
Figure 3.6: Keras OCR predictions
29
3. LETTER RECOGNITION
rotation. To combat this issue, I developed one preprocessing method
that crops the image to detect the letter in the image. I call it CropPrep.
It takes inspiration from Keras O C R mentioned in Chapter 3.1.
Keras OCR as a standalone application is not strong enough to read
letters from cards, as demonstrated in Figure 3.6. The rotated cards
are the most problematic; for example, it classifies "A" rotated 180° as
" V " . Moreover, it sometimes tries to find the text in markers resulting
in nonsense. Lastly, the further away cards are not even recognised.
Keras OCR uses a Character-Region Awareness For Text detection
(CRAFT) algorithm [43] to find letters. I lowered its confidence thresholds
for finding text and polygons to find text on cards more often. It
now recognises text on cards that are further away and crops images
that do not contain letters. To not train a neural network on falsely
cropped images, I decided to use the CRAFT method to crop all the
images from the Training Dataset and label them afterwards manually.
I proceeded with each image in the Training Dataset as follows:
1. Split the image into smaller text regions using the parameterised
CRAFT algorithm.
2. Pick the text regions algorithmically with the maximum area,
discarding the other images.
3. Manually check if there is a legible letter in the cropped image.
(a) If yes, assign the image label class as the original.
(b) Otherwise, assign the image label background label.
This procedure shrank the Training Dataset by six per cent, most
notably letter I, where only 48 % of the training examples were correctly
cropped. It also changed the background label; the previous
backgrounds were usually uniform ceilings; now, it contains markers
or texts from posters or boxes. Figure 3.7 shows some of the images
with the "A" label in the Cropped Dataset (cropped training dataset).
It is not enough to crop images; the algorithm also needs to recognise
their letters. I choose EfficientNetV2 and MobileNetV2 with the
same transfer learning architecture as in Section 3.2.1 to train on the
new Cropped Dataset. Learning curves of both architectures show an
increase in overall accuracy and faster learning; the control accuracy
30
3. LETTER RECOGNITION
A•eg* •^ ^ ^ ^ ^ ^ ^
TO L v
j ! A
m ''4
1 1
\
| . r ^ j
A 1 1
Figure 3.7: Training Dataset cropped using CRAFT
for EfficientNetV2 with CropPrep (EffiCropNet) is almost one, as seen
in Figure 3.8.
I validated both methods using the Validation Dataset. Methods
used following automatic procedure for validation and testing:
1. Split the image into smaller text regions.
2. If there is no found text region, return the background label.
3. If there is at least one text region, choose the largest and use the
pretrained neural network to assign the label.
Even though the dataset contains cards in motion, both methods
recognise the card in 90 % of the cases, with EffiCropNet (EfficientNetV2)
being about four per cent better than MobCropNet
(MobileNetV2) as described by Table 3.3. The downside of this approach
is the dependence on the CRAFT separation algorithm, which
increases overall processing time, even with a utilised G P U card for
faster CRAFT processing. EffiCropNet average time is 629 ms, almost
25 times more than the method without preprocessing. The second
downside is that the letter "I" is less often recognised, being just one
line in the image.
31
3. LETTER RECOGNITION
cr 1.0
2 0.8
B 0.6
ID
m 0.4
I 0.2
is
iz 0.0 0 20 40 60 80 100
Number of epochs
(a) EffiCropNet
& 1.0
| 0.8
u 0.6
10
ci 0.4
| 0.2
10
£ 0.0
0 20 40 60 80 100
Number of epochs
(b) MobCropNet
Figure 3.8: Accuracy of card detection methods in each epoch
Table 3.3: Validating card detection methods
architecture ACC PPV FOR RT IES
EffiCropNet
MobCropNet
94.55 %
90.17 %
97.72 %
97.06 %
26.39 %
41.96 %
594.91 ms
586.14 ms
629.19 ms
650.06 ms
The EffiCropNet more often assigns a background label than the
unprocessed variant. It assigns it correctly more often when it finds a
letter, as described by a higher false omission rate and higher precision
than the unprocessed variant. I use EffiCropNet in the final testing
due to the promising learning curve and higher precision.
3.3 Testing
I tested three previously defined methods:
1. EfficientNetV2 which does not use any preprocessing technique.
2. MobBackNet first removes background from image, then uses
MobileNetV2.
3. EffiCropNet first uses CRAFT to find the card, then crops the
image and labels it with EfficientNetV2.
When tested on the sharp images, EfficientNetV2 has the highest
accuracy, lowest false omission rate and response time. Second in accuracy
and speed is MobBackNet; however, this method is the least
32
3. LETTER RECOGNITION
Table 3.4: Testing letter recognition methods on sharp images
architecture A C C PPV FOR RT IES
EfficientNetV2
MobBackNet
EffiCropNet
97.81 %
91.82 %
91.67 %
97.68 %
94.16 %
97.69 %
1.92 %
12.47 %
18.20 %
28.04 ms
53.23 ms
652.0 ms
28.67 ms
57.98 ms
711.0 ms
Table 3.5: Testing letter recognition methods on moving images
architecture A C C PPV FOR RT IES
EfficientNetV2
MobBackNet
EffiCropNet
76.00 %
79.33 %
64.00 %
84.31 %
80.00 %
92.59 %
28.28 %
21.11 %
42.28 %
26.35 ms
54.38 ms
603.9 ms
41.63 ms
82.27 ms
1132.9 ms
precise. EffiCropNet is not as fast or accurate as its competitors, although
it is the most precise method. A l l details are in Table 3.4.
Testing on the moving images is also interesting; methods were
not trained for this kind of data. Overall accuracy dropped from an
average of 94 % to just 73 %. Unexpectedly, the most accurate method
is MobBackNet. Table 3.5 describes the details.
The most frequent answer for images in the extra images was K,
which makes sense, being the letter quite similar to the star symbol. EfficientNetV2
guessed the most letters correctly of the hidden images, but
still only 3 out of 40. Other methods guessed a single image correctly.
Interesting results provide the partial images. MobBackNet has
three true positives, EffiCropNet six and EfficientNetV2 seven. That is
a relatively high number, considering that part of the letter is out of
the image. However, for the image in Figure 3.9 all methods produce
false positives, meaning networks are suitable only when most of the
letter is in the image.
Figure 3.9: Rotated partial image classified as FP by all tested methods.
33
3. LETTER RECOGNITION
3.4 Results
Neural networks work well only when the testing images are similar
to the training images. Testing on moving images teaches us that we
need to add moving images to the training data if we want to use the
neural network in production. The blur of the moving images makes it
hard for methods to assign letters correctly. The letters on cards need
to be fully visible or not at all for neural networks to assign the correct
label. Also, methods judge unexpected star card as the closest match,
not as a background.
Although MobBackNet is more accurate than EfficientNetV2 on
moving images and EffiCropNet more precise, still I choose EfficientNetV2
as the best overall network due to its high accuracy and low
response time on all images.
I visualise how EfficientNetV2 works. In Figure 3.10a, I picked
four images from the Test Dataset that EfficientNetV2 classified as a
true positive, false positive, false negative, and true negative. I use the
occlusion technique [62] on these images. The occlusion technique
partially covers the input image and analyses the response from the
neural network.
I modified the occlusion technique to show concrete regions the
neural network considers important. I divide each image into a 4x4
grid. Next, I create 16 new images with one region from the grid
occluded (pixels here have zero intensity). Following, I classify these
partially occluded images using EfficientNetV2. Finally, I use the same
image as the original; however, I remove regions from occluded images
which the neural network classifies the same as the whole image. The
occlusion technique produces an image with occluded regions that
are unimportant for the neural network; removing them does not
change the network's return value. Figure 3.10b shows the result for
cherry-picked images.
The visualisation for true positive shows the neural network finding
the card; there are three regions, one containing the letter. Interestingly,
not all regions that do not have the card are occluded. In the
second, the neural network is unsure where to search for the letter;
when occluding most regions, it changes its response. In the third, the
neural network does not find the card; its region stays occluded. In
the last image, not all background image regions are occluded.
34
3. LETTER RECOGNITION
The visualisation technique teaches us that the neural network
model is not an intelligent machine, just a good approximator. When
EfficientNetV2 returns a card label, at least one important region contains
the card; however, not all. Similarly, when it returns background,
regions with the card are occluded; however, not all regions are occluded.
The neural network can incorrectly classify images that are
recognisable to humans. Still, the neural network is not useless; its
overall accuracy on sharp images is 98 %.
Going forward, I think the accuracy of EfficientNetV2 can be improved
with more careful post-processing. I could analyse the probabilities
of the image containing each letter and classify the image only
if the confidence is above a certain threshold, skipping other images. I
also advise using a more complex augmentation pipeline that would
include motion blur of the images to classify better moving images from
the Test Dataset.
35
3. LETTER RECOGNITION
A A
(a) Cherry-picked images from the Test Dataset
A T
^ ^^^^^^
f-WT
(b) Occluded regions of images with predicted labels
Figure 3.10: Visualising EfficientNetV2
36
Conclusion
This thesis tackled a complex and previously unsolved problem of
scanning markers and recognising letters on Letter Jam cards. It developed
several methods for its scanning application.
The thesis described the differences between scanning the markers
and recognising letters in the context of Letter Jam cards. It facilitated
the results from both algorithms to a simple two by two confusion
matrix and further simplified it to five concrete metrics, including
accuracy and response time (speed). It presented three datasets to
train, validate and test the algorithms.
The thesis investigated how to alter the ArUco library, which is
OpenCV contributed module, to detect markers on cards. It achieved
higher accuracy and speed with altered hyperparameters than the
original library proposed. It introduced an extra hyperparameter that
controls the scaling of an image which significantly improved response
time and did not substantially worsen overall accuracy.
The thesis explored letter recognition on cards. The best of the
three chosen neural networks used transfer learning and no other preprocessing
except simple augmentation of input images. Background
preprocessing helped a second network to assign blurry images. Third
cropped the taken image to the detected text, which was the most
precise but slowest method.
The best algorithm overall depends on the concrete needs. A neural
network based on EfficientNetV2 has an accuracy of 98 %, meaning it
can distinguish images in most scenes. However, to use it in production,
one needs to train it to distinguish blurry images because its accuracy
on them is only 76%. O n the other hand, SmallAruco5 is faster and
more precise. It might not recognise images in all situations because
its accuracy is only 78 %. As the result of scanning the markers, it
cannot recognise illegible markers on blurry images, which make up
about 10 % of all images. Nevertheless, unlike EfficientNetV2, it has
not assigned any wrong labels and might be the best method that
needs no additional effort.
Future work for marker scanning algorithms is an investigation
of how to recognise more images; for letter recognition algorithms, it
includes making datasets for moving images or augmenting the train-
37
CONCLUSION
ing dataset and making the algorithms overall faster. Better response
time of marker scanners might be due to C++ code being compiled to
a fast machine code, whereas Python for letter recognition algorithms
needs to be interpreted. A n ambitious project might even combine
both approaches and develop a single method that is faster and more
accurate than its components.
The thesis demonstrated that scanning Letter Jam cards using
open-source techniques is possible. All methods achieved the highest
accuracy on still and sharp images, with more accurate letter recognition
algorithms and faster marker scanning algorithms. Right now,
the best algorithm for production is the marker scanning algorithm
SmallAruco5, whereas the letter recognition algorithm EfficientNetV2
needs future work.
38
Bibliography
1. BRADSKI, G. The OpenCV Library. Dr. Dobb's Journal of Software
Tools. 2000.
2. M A R T I N ABADI; ASHISH A G A R W A L ; PAUL B A R H A M ; E U GENE
BREVDO, et al. TensorFlow: Large-Scale Machine Learning
on Heterogeneous Systems. 2015. Available also from: https: //www.
tensorf low. org/. Software available from tensorflow.org.
3. PTC INC. Vuforia Engine [online]. 2022 [visited on 2022-05-17].
Available from: https://developer.vuforia.com/.
4. K U N C H E V A , Ludmila I. Combining Pattern Classifiers: Methods
and Algorithms, 2nd Edition. In: COMBINING PATTERN
CLASSIFIERS: METHODS AND ALGORITHMS, 2ND EDITION.
2014, pp. 14-15. ISBN 978-1-118-91456-4; 978-1-118-31523-1.
5. SOKOLOVA, Marina; L A P A L M E , Guy. A systematic analysis of
performance measures for classification tasks. INFORMATION
PROCESSING & MANAGEMENT. 2009, vol. 45, no. 4, pp. 427-
437. ISSN 0306-4573. Available from DOI: 10 .1016/j . ipm. 2009 .
03.002.
6. OLLIARO, Piero; TORREELE, Els. Managing the risks of making
the wrong diagnosis: First, do no harm. International Journal of
Infectious Diseases. 2021, vol. 106, pp. 382-385. ISSN 1201-9712.
Available from DOI: 10.1016/j . ijid.2021.04.004.
7. L O H M A N , David F. Individual differences in errors and latencies
on cognitive tasks. Learning and Individual Differences. 1989, vol. 1,
no. 2, pp. 179-202. ISSN 1041-6080. Available from DOI: https :
//doi.org/10.1016/1041-6080(89)90002-2.
8. LIESEFELD, Heinrich R.; JANCZYK, Markus. Combining speed
and accuracy to control for speed-accuracy trade-offs (?) Behav
Res Methods. 2019, vol. 51, no. 1, pp. 40-60. Available from DOI:
https://doi.org/10.3758/sl3428-018-1076-x.
9. HEITZ, Richard P. The speed-accuracy tradeoff: history, physiology,
methodology, and behavior. Frontiers in Neuroscience. 2014,
vol. 8. ISSN 1662-453X. Available from DOI: 10.3389/f nins. 2014.
00150.
39
BIBLIOGRAPHY
10. BRUYER, Raymond; BRYSBAERT, Marc. Combining Speed and
Accuracy in Cognitive Psychology: Is the Inverse Efficiency Score
(IES) a Better Dependent Variable than the Mean Reaction Time
(RT) and the Percentage Of Errors (PE)? Psychologica Belgica.
2011, vol. 51, pp. 5-13. Available from DOI: 10.5334/pb-51-l-5.
11. GOODFELLOW, Ian; BENGIO, Yoshua; COURVILLE, Aaron.
Deep Learning. MIT Press, 2016. Available from http : / /www .
deeplearnirigbook. org.
12. About FFmpeg [online] [visited on 2022-03-27]. Available from:
https://ffmpeg.org/about.html.
13. S C H E E P M A N S , Lionel. Lien Wikimedia Foundation staffand contractors
[online]. 2022 [visited on 2022-05-18]. Available from:
https://commons.wikimedia.org/wiki/File:Lien_Wikimedia_
Foundation_staff_and_contractors.svg.
14. APRIL ROBOTICS LABORATORY. AprilTag [online]. 2022 [visited
on 2022-05-18]. Available from: https : / / a p r i l . eecs .umich.
edu/software/apriltag.
15. OPENCV. Detection of Arllco Markers [online]. 2022 [visited on
2022-05-18]. Available from: https : //docs . opencv . org/4 . x/
d5/dae/tutorial_aruco_detection.html.
16. GAVRILUK, Alexandr. Generated Matrix 2 of 5 barcode [online].
2020 [visited on 2022-05-18]. Available from: https: //commons.
wikimedia.org/wiki/File:Matrix_2_of_5_barcode.png.
17. INC, PTC. VuMark [online]. 2021 [visited on 2022-05-18]. Available
from: https : //web . archive . org/web/20220307233306/
https : / / l i b r a r y . vuf oria . com/f eatures/objects/vumark .
html.
18. W O O D L A N D , Norman J.; BERNARD, Silver. Classifying apparatus
and method. 1952. Available also from: https : / /patents .
google.com/patent/US2612994A/en.
19. WAVE, Denso. QR Code [online] [visited on 2022-03-20]. Available
from: https: //www. qrcode. com/en/.
20. BROWN, Jeff. Zbar Bar Code Reader [online]. 2010 [visited on
2022-03-20]. Available from: http://zbar.sourceforge.net/.
40
BIBLIOGRAPHY
21. O L S O N , Edwin. AprilTag: A robust and flexible visual fiducial
system. In: Proceedings of the IEEE International Conference
on Robotics and Automation (ICRA). IEEE, 2011, pp. 3400-3407.
22. ROMERO-RAMIREZ, Francisco; MUNOZ-SALINAS, Rafael; MEDINA-CARNICER,
Rafael. Speeded Up Detection of Squared
Fiducial Markers. Image and Vision Computing. 2018, vol. 76. Available
from DOI: 10.1016/j .imavis.2018.05.004.
23. PTC INC. Vumark [online]. 2021 [visited on 2022-03-20]. Available
from: https://library.vuforia.com/features/objects/
vumark. html.
24. OPENCV. RGB to GRAY. 2022. Version 4.5.0. Available also from:
docs.opencv.org/4.5.0/de/d25/imgproc_color_conversions.
html#color_convert_rgb_gray.
25. OPENCV. adaptiveThreshold(). 2022. Version 4.5.0. Available also
from: docs . opencv . org/4 . 5 . 0/d7/dlb/group__imgproc_
_misc.html#ga72b913f352e4alblb397736707afcde3.
26. OPENCV. findContours (). 2022. Version 4.5.0. Available also from:
docs . opencv . org/4 . 5 . 0/d3/dc0/group__imgproc__shape .
html#gadfIad6a0b82947falfe3c3d497f260e0.
27. OPENCV. getPerspectiveTransform (). 2022. Version 4.5.0. Available
also from: docs . opencv. org/4.5.0/da/d54/group imgproc_
_transform.html#ga20f62aa3235d869c9956436c870893ae.
28. OPENCV. warpPerspective(). 2022. Version 4.5.0. Available also
from: docs . opencv . org/4 . 5 . 0/da/d54/group__imgproc_
_transform.html#gaf73673a7e8el8ec6963e3774e6a94b87.
29. OPENCV. threshold^). 2022. Version 4.5.0. Available also from:
docs . opencv . org/4 . 5 . 0/d7/dlb/group__imgproc__misc .
html#gae8a4al46dlca78c626a53577199e9c57.
30. MICROSOFT. Built-in types (C++) [online] [visited on 2022-
03-22]. Available from: https : / /docs . microsoft . com/en-
us/cpp/cpp/fundamental-types-cpp?view=msvc-170.
41
BIBLIOGRAPHY
31. GARRIDO-JURADO, Sergio; M U N O Z - S A L I N A S , Rafael; M A DRID-CUEVAS,
Francisco; MEDINA-CARNICER, Rafael. Generation
of fiducial marker dictionaries using Mixed Integer Linear
Programming. Pattern Recognition. 2015, vol. 51. Available from
DOI: 10.1016/j .patcog.2015.09.023.
32. M E M O N , Jamshed; SAMI, Maira; K H A N , Rizwan Ahmed; U D DIN,
Mueen. Handwritten Optical Character Recognition (OCR):
A Comprehensive Systematic Literature Review (SLR). IEEE Access.
2020, vol. 8, pp. 142642-142668. Available from DOI: 10.1109/
ACCESS.2020.3012542.
33. L E C U N , Yann; CORTES, Corinna; BURGES, Christopher J.C. THE
MNIST DATABASE [online] [visited on 2022-04-24]. Available
from: http: //yann. lecun. com/exdb/mnist/.
34. W A N G , Zhangyang; Y A N G , Jianchao; JIN, Hailin; SHECHTM
A N , Eli; AGARWALA, Aseem; BRANDT, Jonathan; H U A N G ,
Thomas S. Deep Font: Identify Your Font from A n Image. In: Proceedings
of the 23rd ACM International Conference on Multimedia.
Brisbane, Australia: Association for Computing Machinery, 2015,
pp. 451-459. M M '15. ISBN 9781450334594. Available from DOI:
10.1145/2733373.2806219.
35. R., Mamatha H . ; S., Sucharitha; K., Srikanta Murthy. Article:
Multi-Font and Multi-Size Kannada Character Recognition based
on the Curvelets and Standard. International Journal of Computer
Applications. 2011, vol. 35, no. 11, pp. 1-8.
36. HOCHREITER, Sepp; SCHMIDHUBER, Jürgen. Long short-term
memory. NEURAL COMPUTATION. 1997, vol. 9, no. 8, pp. 1735-
1780. ISSN 0899-7667. Available from DOI: 10.1162/neco. 1997.9.
8.1735.
37. SMITH, Ray; PODOBNY, Zdenko. Tesseract OCR [online]. 2022
[visited on 2022-03-20]. Available from: https : //github. com/
tesseract-ocr/tesseract.
38. M O R A L E S , Fausto. Keras OCR [online]. 2021 [visited on 2022-
03-20]. Available from: https : //github . com/f austomorales/
keras-ocr.
42
BIBLIOGRAPHY
39. L E C U N , Yann; BENGIO, Yoshua; HINTON, Geoffrey. Deep learning.
NATURE. 2015, vol. 521, no. 7553, pp. 436^44. ISSN 0028-0836.
Available from DOI: 10.1038/naturel4539.
40. KRIZHEVSKY, Alex; SUTSKEVER, Ilya; H I N T O N , Geoffrey E.
ImageNet Classification with Deep Convolutional Neural Networks.
In: PEREIRA, R; BURGES, C. J. C ; BOTTOU, L.; WEINBERGER,
K. Q. (eds.). Advances in Neural Information Processing
Systems. Curran Associates, Inc., 2012, vol. 25. Available also
from: https : //proceedings . neurips . cc/paper/2012/f i l e /
C399862d3b9d6b76c8436e924a68c45b-Paper.pdf.
41. P A N , Sinno J.; Y A N G , Qiang. A Survey on Transfer Learning.
IEEE Transactions on Knowledge and Data Engineering. 2010, vol. 22,
no. 10, pp. 1345-1359. Available from DOI: 10.1109/TKDE. 2009.
191.
42. TensorFlow Hub [online] [visited on 2022-04-24]. Available from:
https://www.tensorflow.org/hub.
43. BAEK, Youngmin; LEE, Bado; H A N , Dongyoon; Y U N , Sangdoo;
LEE, Hwalsuk. Character Region Awarenessfor Text Detection. arXiv,
2019. Available from DOI: 10.48550/ARXIV. 1904.01941.
44. QIANG, Guo; D A N , Tu; GUOHUI, Li; JUN, Lei. Memory Matters:
Convolutional Recurrent Neural Networkfor Scene Text Recognition.
arXiv, 2016. Available from DOI: https : / / d o i . org/10 .48550/
arXiv.1601.01100.
45. COMPUTER VISION GROUP AT CORNELL TECH. COCO-Text
[online]. 2018 [visited on 2022-05-18]. Available from: https :
//bgshih.github.io/cocotext/.
46. A N D R E O P O U L O S , Alexander; TSOTSOS, John K. 50 Years of
object recognition: Directions forward. Computer Vision and Image
Understanding. 2013, vol. 117, no. 8, pp. 827-891. ISSN 1077-3142.
Available from DOI: https : / / d o i . org/10 .1016/j . cviu. 2013 .
04.005.
47. GLEIBMAN, Andrew. Object Recognition System Design in Computer
Vision: a Universal Approach. CoRR. 2013, vol. abs/1310.7170
Available from arXiv: 1310.7170.
43
BIBLIOGRAPHY
48. R E D M O N , Joseph; FARHADI, Ali. YOLOv3: A n Incremental
Improvement. CoRR. 2018, vol. abs/1804.02767. Available from
arXiv: 1804.02767.
49. C H E N , Wen-Yuan; C H U N G , Chin-Ho. Robust poker image recognition
scheme in playing card machine using Hotelling transform,
DCT and run-length techniques. Digital Signal Processing. 2010,
vol. 20, no. 3, pp. 769-779. ISSN 1051-2004. Available from DOI:
https://doi.org/10.1016/j.dsp.2009.09.008.
50. ZUTIS, Krists; HOEY, Jesse. Who's Counting? Real-Time Blackjack
Monitoring for Card Counting Detection. In: FRITZ, Mario;
SCHIELE, Bernt; PIATER, Justus H . (eds.). Computer Vision Systems.
Berlin, Heidelberg: Springer Berlin Heidelberg, 2009, pp. 354-
363. ISBN 978-3-642-04667-4.
51. C H E N , Qianmin; RIGALL, Eric; W A N G , Xianglong; F A N , Hao;
DONG, Junyu. Poker Watcher: Playing Card Detection Based on
EfficientDet and Sandglass Block. In: 2020 11th International Conference
on Awareness Science and Technology (iCAST). 2020, pp. 1-6.
Available from DOI: 10.1109/iCAST51195.2020.9319468.
52. O U , Zhonghong; XIONG, Baiqiao; XIAO, Fenrui; SONG, Meina.
ERCS: A n efficient and robust card recognition system for camerabased
image. China Communications. 2020, vol. 17, no. 12, pp. 247-
264. Available from DOI: 10.23919/JCC.2020.12.018.
53. TAN, Mingxing; LE, Quoc V. EfficientNetV2: Smaller Models and
Faster Training. CoRR. 2021, vol. abs/2104.00298. Available from
arXiv: 2104.00298.
54. SANDLER, Mark; HOWARD, Andrew G.; Z H U , Menglong; Z H MOGINOV,
Andrey; C H E N , Liang-Chieh. Inverted Residuals
and Linear Bottlenecks: Mobile Networks for Classification, Detection
and Segmentation. CoRR. 2018, vol. abs/1801.04381. Available
from arXiv: 1801.04381.
55. KOLESNIKOV, Alexander; BEYER, Lucas; ZHAI, Xiaohua; PUIGCERVER,
Joan; YUNG, Jessica; GELLY, Sylvain; HOULSBY, Neil.
Large Scale Learning of General Visual Representations for Transfer.
CoRR. 2019, vol. abs/1912.11370. Available from arXiv: 1912.
11370.
44
BIBLIOGRAPHY
56. SRIVASTAVA, Nitish; HINTON, Geoffrey; KRIZHEVSKY, Alex;
SUTSKEVER, Ilya; SALAKHUTDINOV, Ruslan. Dropout: A Simple
Way to Prevent Neural Networks from Overfitting. JOURNAL
OF MACHINE LEARNING RESEARCH. 2014, vol. 15, pp. 1929-
1958. ISSN 1532-4435.
57. SHORTEN, Connor; KHOSHGOFTAAR, Taghi M . A survey on
Image Data Augmentation for Deep Learning. JOURNAL OF BIG
DATA. 2019, vol. 6, no. 1. Available from DOI: 10 .1186/s40537-
019-0197-0.
58. REED, R. PRUNING ALGORITHMS - A SURVEY. IEEE TRANSACTIONS
ON NEURAL NETWORKS. 1993, vol. 4, no. 5, pp. 740-
747. ISSN 1045-9227. Available from DOI: 10.1109/72.248452.
59. OPENCV. GaussianBlurQ. 2022. Version 4.5.0. Available also
from: docs . opencv . org/4 . 5 . 0/d4/d86/group__imgproc_
_filter.html#gaabe8c836e97159a9193fb0bllac52cf1.
60. OPENCV. connectedComponents (). 2022. Version 4.5.0. Available
also from: docs . opencv. org/4.5.0/d3/dc0/group imgproc_
_shape.html#gaedef8c7340499ca391d459122e51bef5.
61. OPENCV. ResizeQ. 2022. Version 4.5.0. Available also from: docs.
opencv . org/4 . 5 . 0/da/d54/group__imgproc_^transform .
html#ga47a974309e9102f5f08231edc7e7529d.
62. ZEILER, Matthew D.; FERGUS, Rob. Visualizing and Understanding
Convolutional Networks. In: FLEET, David; PAJDLA,
Tomas; SCHIELE, Bernt; TUYTELAARS, Tinne (eds.). Computer
Vision - ECCV 2014. Cham: Springer International Publishing,
2014, pp. 818-833. ISBN 978-3-319-10590-1.
63. INITIATIVE, Open Source. The MIT License [online]. 2022 [visited
on 2022-05-18]. Available from: https : //opensource. org/
licenses/MIT.
45
Appendix
I publish all source files I used to do this thesis under the MIT l i cence
[63] and include them in the attachment files the thesis provides.
I include installation information in readme files. I decided not to publish
source videos for Training and Testing Datasets due to their large
size. I divide all files into several folders, the most important being:
1. datasets as described in Chapter 1.5.
(a) train images to train neural networks and evaluate marker
scanning algorithms.
(b) validation videos to validate both neural networks and marker
scanning algorithms.
(c) test images to test methods on various categories that they
might encounter in production.
(d) extra datasets that were trained background separation methods
and card detection methods.
2. marker scanning algorithms and results from in Chapter 2.
(a) markers saved as vector graphics.
(b) source algorithms for marker scanning and its evaluation.
(c) results from validation and testing.
3. letter recognition algorithms and results from Chapter 3.
(a) trained models that can be loaded without the need for
training.
(b) source algorithms for transfer learning, background separation,
card detection and their evaluation.
(c) results from validation and testing.
46