MASARYK
UNIVERSITY
FACULTY OF INFORMATICS
Analysis of Logs for Anomaly
Detection in Manufacturing
Master's Thesis
JAKUB KRAKOVSKY
Brno, Spring 2026
MASARYK
UNIVERSITY
FACULTY OF INFORMATICS
Analysis of Logs for Anomaly
Detection in Manufacturing
Master's Thesis
JAKUB KRAKOVSKY
Advisor: RNDr. Michal Batko, Ph.D.
Department of Machine Learning and Data Processing
Brno, Spring 2026
Declaration
Hereby I declare that this paper is my original authorial work, which
I have worked out on my own. A l l sources, references, and literature
used or excerpted during elaboration of this work are properly cited
and listed in complete reference to the due source. During the preparation
of this thesis, I used Grammarly for grammar checks, and both
Gemini and ChatGPT to brainstorm, confirm my ideas, and improve
my writing style. Additionally, I used Gemini during implementation
to write trivial code and generate initial table versions.
Jakub Krakovský
Advisor: RNDr. Michal Batko, Ph.D.
iii
Acknowledgements
First and foremost, I would like to thank my supervisor, RNDr. Michal
Batko, Ph.D., for his willingness to consult on the various problems of
this thesis, and for his valuable advice. Of course, I would like to thank
my colleagues for providing the dataset, answering my questions
about the devices, and offering professional insights to help with the
analysis. Finally, I would like to thank my family and girlfriend for
supporting me throughout my studies.
iv
Abstract
Manufacturing companies generate large amounts of textual data that
can be processed to detect production problems as they occur. These
problems can be costly, especially when the resources required for rootcause
analysis and system repairs are high due to device complexity.
This thesis aims to improve RACOM's anomaly detection pipeline by
leveraging data mining and machine learning techniques on Linux
system logs. The proposed solution focuses on identifying anomalous
behavior in real time, reducing diagnostic effort, and improving the
reliability and maintainability of the production system.
Keywords
machine learning, deep learning, Autoencoder, log parsing, Linux logs,
embedded systems, manufacturing diagnostics, anomaly detection
v
Contents
1 Introduction 1
2 Problem statement 3
3 Related work 5
3.1 Deep learning for anomaly detection in log data: A survey 5
3.2 Deep Learning or Classical Machine Learning? A n
Empirical Study on Log-Based Anomaly Detection . . . 7
3.3 Semi-supervised Log-based Anomaly Detection via
Probabilistic Label Estimation 8
4 Dataset 11
4.1 Data description 11
4.1.1 Log Types 13
4.2 Data analysis 18
4.2.1 Dataset statistics 18
4.3 Analysis of log types 22
4.3.1 Analysis of lower-frequency log types 23
4.3.2 Analysis of higher-frequency log types 25
5 Methods 32
5.1 Data preprocessing 32
5.1.1 Drain parser 33
5.1.2 Vectorization and Embeddings 35
5.2 Model architecture 35
5.2.1 Prevalence filter 37
5.3 Infrastructure 38
5.3.1 Training infrastructure 39
5.3.2 Inference pipeline 40
6 Evaluation and results 41
6.1 Evaluation setup 41
6.2 Results 43
6.3 Discussion 46
7 Conclusion 52
vi
Bibliography 55
A Electronic Attachments 5 8
vii
List of Tables
4.1 Five-line window example 13
4.2 Statistical Distribution of the Log Corpus 20
4.3 Statistical Distribution of Log Lengths 21
4.4 Temporal Span of Log Sessions (seconds) 22
4.5 Anomalous Group Analysis by Log File Type 23
4.6 Event Template Distribution for Event Logs 25
4.7 Event Template Distribution for SSH daemon Logs 26
4.8 Event Template Distribution for Rrbootlog Logs 28
4.9 Event Template Distribution for Main Logs 29
4.10 Frequency Distribution of Event Templates in Messages
Logs 30
6.1 Anomaly detection performance of the previous approach
compared to the new ones 43
6.2 Detailed Anomaly Detection Performance: Type-Specific
Breakdown 43
6.3 Number of devices flagged by different methods across 19
log types, including unique contributions by the Hybrid
Approach 45
6.4 Representative Validation Set Anomalies 50
6.5 Representative Test Set Anomalies 50
viii
List of Figures
3.1 The framework of the PLELog model showing the stages
of the model training[3] 9
4.1 Top 5 Rrbootlog Length Groups 27
4.2 Proposed Method to Log Type mapping 31
5.1 Example of Parsing 33
5.2 Structure of Parse Tree in Drain (depth = 3) [14] 34
6.1 Correlations between log types for the validation set . . . 44
6.2 Correlations between log types for the test set 44
ix
1 Introduction
Modern factories produce massive amounts of textual log messages
during the manufacturing of their products. These newly created
devices and systems can occasionally be faulty, and the generated
textual data can serve as a good indicator of such malfunctions. It is
crucial to ensure that the factory's final product is fully functional,
especially as lengthy repair processes can require significant resource
costs.
R A C O M specializes in creating such complex devices, including
radio modems, cellular routers, microwave links, and embedded data
radios. Production in this company involves multiple checkpoints,
where at each, a block of information about the device is generated and
stored. This data is currently mostly processed manually using simple
techniques, one of which is identifying regular patterns in the data.
However, this approach can be improved, as unseen errors may go
undetected by the current system. Additionally, most error detection
work is handled by quality assurance professionals, where repetitive
manual inspection can lead to fatigue, possibly allowing some errors
to go undetected.
The primary aim of this thesis is to improve the company's faultdetection
process by designing an automated anomaly detection system
capable of identifying irregularities that traditional regular expression
(regex) based methods miss. Additionally, this work seeks
to leverage data mining techniques to derive interpretable rules that
help technicians perform root-cause analysis. To evaluate the practical
impact, a selected model is tested in a production environment to determine
its effectiveness compared to the previous approach. Finally,
the deployed solution should minimize manual inspection fatigue
while enhancing the reliability of the quality assurance process.
The remainder of this thesis is organized as follows. Chapter 2
defines the problem and presents the motivations of this thesis. Chapter
3 explores the current methods used for log anomaly detection,
ranging from classical machine learning to deep learning architectures.
Chapter 4 provides an extensive analysis of the R A C O M dataset, categorizing
logs by frequency and statistical properties to determine the
most effective detection strategy for each category. In Chapter 5, the
1
1. INTRODUCTION
data preprocessing steps, specifically the use of the Drain parser and
semantic embeddings, are discussed alongside the implementation of
the frequency-based autoencoder and the inference pipeline. Chapter
6 presents the experimental results, comparing the proposed approach
with the baseline and evaluating its effectiveness in a production environment.
Finally, Chapter 7 summarizes what was done by this thesis
and its results.
Thesis contribution. The main contribution of this thesis is the design
and deployment of an automated anomaly-detection system for
a production environment. The work includes analysis and categorization
of production logs to select suitable detection approaches for
different log types, design of the model architecture together with the
supporting deployment infrastructure, and preparation of the dataset
including label refinement and evaluation split creation. The proposed
solution is compared with the existing regex-based system and
shows improved detection of previously unseen anomalies. The system
was also validated in production, where it successfully detected
real operational faults and issues.
2
2 Problem statement
R A C O M manufactures sophisticated communication devices with
firmware based on embedded Linux. It uses a combination of opensource
libraries and proprietary software, including modifications to
the kernel and bootloader. Most system components generate various
types of logs during runtime. These logs are a crucial data source
for diagnostics of the product life cycle, from development to endcustomer
service.
Currently, the approach in the company is to match a collection
of regex containing strings like alert, warning, and others to detect
these keywords and identify anomalies. However, this approach is not
effective because many more types of anomalies may be present, such
as novel lines, broken sequences, keywords with the same semantic
meaning as some word in the regex pattern set, and temporal anomalies
resulting from too-long or too-short temporal differences in the log
data. A state-of-the-art model can help detect some of aforementioned
anomalies and help technicians with finding the root cause of the
problem.
Log data is saved along with the exact position of every mounted
part on the printed circuit board (PCB). Occasionally, batches of certain
parts from suppliers can malfunction, compromising batches of
devices that use them. A n advanced model can detect such defective
machines, and technicians can then perform further analysis on devices
with the same mounted parts on the circuit board as the defective
device.
Technicians do not have to go through the logs manually, but a
model can detect problems by itself and only flag the issues to them.
It is important not to overwhelm the professionals by producing too
many false positives, as this can lengthen the quality assurance process
by requiring checks for every flagged device. Unwanted false
positives include, for example, Linux version changes and new service
messages.
One of the primary motivations from the beginning was flagging
non-frequent or completely novel lines, such as firmware changes.
However, it could lead to the false positives problem I described above,
which could result in technicians not wanting to use the proposed sys-
3
2. PROBLEM STATEMENT
tern or a significant slowdown in their work. As the design of online
algorithms is not trivial even in today's publications [1] designed on
known datasets, a different approach has to be utilized. The problem
involves creating a mechanism where technicians can easily distinguish
between "good" messages and actual anomalies. Without a way
to filter these known messages from the detection pipeline between
model training cycles, the proposed system risks being rejected by the
professionals.
4
3 Related work
In this section, I present a review of state-of-the-art solutions for log
anomaly detection methods.
This literature research was conducted on articles published from
2020 to 2026 using Google Scholar as search engine. Firstly, I searched
for survey papers that could lead me to the best existing solutions, using
queries such as "log anomaly detection survey" and "log anomaly
detection overview" [1] [2]. For these types of papers, I aimed to
find the most recent, high-quality publications that could provide an
overview [1] of the current state of this particular task and compare
older classical methods with newer deep learning techniques [2]. This
approach led me to multiple other papers, from which I tried to identify
the most optimal for this thesis's problem. Finally, the optimality
of a specific model was based not only on the F l score reached, but
also on compatibility with the dataset.
3.1 Deep learning for anomaly detection in log data: A
survey
Landauer et al. [1] published an overview of deployed models, their
data preprocessing methods, and the different anomaly detection techniques
used. They do not compare the models based on performance
or results, but rather focus on providing a broad overview of the topic,
which is helpful to gain insight into the problem domain.
Their search started at the beginning of 2022, and they reviewed
62 papers, where 58 of them were published after 2019 [1]. Some of
the most cited approaches in this setting were in descending order
DeepLog, LogRobust, LogAnomaly, Logsy, and LogBERT[l].
Deep learning methods used in the literature include Convolutional
Neural Networks (CNNs), multilayer perceptrons (MLPs), and
recurrent neural networks (RNNs). RNNs are used the most, appearing
in 36 out of 62 papers. They process inputs step-by-step and retain
hidden states over time, making them a highly suitable for detecting sequential
anomalies, which are essential for log anomaly detection [1].
Specifically, R N N variations, Long Short-Term Memory (LSTM) with
5
3. RELATED WORK
input, output, and forget gates, and Gated Recurrent Unit (GRU) with
update and reset gates, are used for their ability to capture long-term
dependencies and mitigate gradient vanishing in R N N s [ l ] . While
these are mostly used for supervised or semi-supervised tasks, architectures
such as Autoencoders (AEs) are also used for unsupervised
anomaly detection. They encode the input into a compressed representation
within the network and then reconstruct the input while
minimizing the reconstruction error. In this way, the neural network
learns to reconstruct the normal inputs as it sees them frequently
during training, and it flags the anomalous inputs because it did not
learn to reconstruct them from few or no samples. Other architectures
used in the surveyed literature are Transformers (TFs), Graph Neural
Networks (GNNs), and Generative Adversarial Networks (GANs).
Log templates and log messages often change, which highlights
the need to adapt to these changes in an automated way [1]. However,
as developing online learning algorithms is significantly more
challenging, only 7 of the 62 surveyed papers use online learning to
some extent [1].
Landauer et al. [1] also attributed a category of supervised, semisupervised,
or unsupervised for each of the papers. Although many
publications claimed to use an unsupervised approach, they actually
used a semi-supervised approach because they required datasets of
normal behavior data [1]. They found 28 semi-supervised, 26 supervised,
and 8 fully unsupervised approaches [1].
To group log lines into clusters with unique identifiers, extract
dynamic information such as IPs, ssh keys and other variables, log
parsers are used. The most used approaches for parsing include Drain
and Spell [1]. Alternatively, some papers use token-based strategies,
which split the log messages into tokens, for example, using whitespaces
[1].
Simple outlier detection typically does not require log grouping,
as it focuses on individual lines rather than groups. However, for
more complicated anomalies such as broken sequences, the literature
often proposes grouping logs into time windows [1]. Main grouping
strategies are sliding time windows, fixed time windows, and session
windows. Sliding time windows are windows of a certain size, shifted
by a given constant at each step, generally smaller than the window
size. Fixed-time windows are special cases of sliding time windows,
6
3. RELATED WORK
where the window size and step size are equal. Session windows
are determined by the process that generated the log and are mostly
grouped by this process identifier.
The surveyed methods detect both singular and collective anomalies,
where the latter is based on sequences, frequencies, and statistical
properties [1].
Supervised methods focus on binary or multi-class classification.
Semi-supervised and unsupervised methods calculate a threshold
between anomalies and normal samples, or output a probability distribution
where elements with the highest or lowest probability represent
anomalies, depending on the used algorithm [1].
The vast majority of publications use one of the four most common
datasets to evaluate their model's performance: Hadoop Distributed
File System (HDFS), BlueGene/L (BGL), Thunderbird, and Spirit [1].
The evaluation metrics in the researched papers are typically represented
by precision, recall, F l score, true positive rate, and false
positive rate. Less common metrics in the surveyed literature are accuracy,
receiver operator curves (ROC), and precision recall curves [1].
Landauer et al. [1] concluded that the appearance of previously
unknown events is the main challenge addressed by the reviewed
approaches [1]. The fundamental idea is to use semantic vector embeddings
to map novel events to old ones. Also, they state that the
explainability of deep models is challenging because, for example, it
is difficult to interpret false positives and false negatives [1], which
makes root cause analysis harder.
3.2 Deep Learning or Classical Machine Learning? An
Empirical Study on Log-Based Anomaly Detection
Yu et al. [2] demonstrated that classical methods such as K-Nearest
Neighbors, decision trees, or simple multilayer perceptrons outperform
deep learning (DL) methods both in accuracy and efficiency.
Reasons for this were the simplicity of the dataset and data leakage,
mainly represented by repetitive sequences appearing in both the train
and test sets [2].
Yu et al. [2] focused only on supervised methods. This represents
a limitation of their work because they require labeled datasets. How-
7
3. RELATED WORK
ever, as we have seen in Landauer et al. [1], unsupervised datasets
are more prevalent in the literature and in real-world settings because
supervised methods are less applicable due to their need for manual
labeling, which is often difficult and expensive [1].
The authors suggest critically analyzing the dataset first and using
DL methods only after concluding they are necessary [2]. Furthermore,
they propose using these simpler methods as baselines.
3.3 Semi-supervised Log-based Anomaly Detection via
Probabilistic Label Estimation
Yang et al. [3] developed a semi-supervised anomaly detection framework
combining an initial Probabilistic Label Estimation (PLE) phase
with a subsequent supervised training stage. They used a G R U architecture
with semantic embeddings of the input. Their approach was
deployed and evaluated across multiple real-world systems, including
a car manufacturing corporation and a university system.
While designing the model, they wanted to utilize the strengths
of unsupervised systems, such as the absence of manual labeling,
alongside those of supervised systems, such as knowledge of prior
anomalies [3].
Yang et al. [3] used a strategy derived from Positive-Unlabeled
(PU) learning, which traditionally lies in using known positive instances
to predict unlabeled instances. However, the authors use a
variant of this approach utilizing the negative instances, which represent
normal log sequences. In this way, the model knows what normal
instances look like, and it estimates labels of mixed instances [3].
To overcome the problem of estimating the labels of unknown
instances, the HDBSCAN clustering method was used. It partitions all
log messages into groups based on the premise that logs in the same
group tend to share the same label [3]. So, the model estimates labels
of unlabeled instances based on whether the group already contains
normal instances.
To reduce the noise introduced by clustering, as it can be imperfect,
H D B S C A N measures the probability that an instance belongs to each
cluster, and then the model assigns a probabilistic label to it. Based on
labels created by this PLE, a supervised dataset is created, and a model,
8
3. RELATED WORK
which they called PLELog, is trained on it. The overall architecture of
this framework is illustrated in Figure 3.1.
No-rral
Log Seq.
in Unlabeled
t Log Seq.
B
ra Upcoming
ro Loq Seq.
1 Q
Figure 3.1: The framework of the PLELog model showing the stages
of the model training[3]
Inputs to the G R U are semantically embedded vectors, created
through a three-step process: log parsing, word embedding, and TFIDF-based
aggregation [3]. Drain is used to extract log events, as
is the case in most other publications [1]. After extraction of these
events, PLELog first cleans them by removing non-character tokens,
removing stop words, and splitting composite tokens according to
Camel Case [3]. Afterwards, PLELog uses FastText [4] to embed each
word into a 300-dimensional vector to extract semantic information.
Additionally, TF-IDF is used to obtain the semantic vector for the entire
log event. The Term Frequency(TF) isolates the local density of a
word w within an event, calculated as TF(w) = JJ, where fw denotes
the frequency of w in the event, and N denotes the total word count of
the event. To measure global rarity, the Inverse Document Frequency
is defined as IDF(w) = log (^)/ where |L| is the total volume of log
events and \LW\ is the subset of events containing w. The composite
weight for the z'-th word is simply coi = TF(Wj-) x JDF(wt). Finally, the
event's overall semantic vector v is computed via a weighted average
of its word vectors vt as v = ^ 2_i=i <^Pi- This TF-IDF aggregation
ensures that the semantic vectors accurately capture the core meaning
of the log events, enhancing the model's robustness against the
instabilities appearing in log data [3].
The PLELog model was evaluated on two of the most popular
datasets, HDFS and BGL, which are also widely used in other pub-
9
Semantic
Embedding
Embedding
On !*[•••]
i
Unlabeled
Log Vectors
- [ • • • ] Anomaly
Detection
Model Building
Attention-basec GRU Network
Predict
Log Seq.
3. RELATED WORK
lications [1]. In comparison to other models, they outperformed
DeepLog [5], LogAnomaly [6], LogCluster [7], and PCA, all of which
are unsupervised or semi-supervised methods. It failed to achieve
as high a score as supervised LogRobust [8], but the difference was
marginal, indicating the usefulness of PLE in the process. Finally, they
evaluated all the previously mentioned non-supervised models on the
two real-world datasets, and PLELog performed best [3], achieving
F l scores of 0.947 on the first dataset and 0.984 on the second.
10
4 Dataset
The data was provided by the R A C O M company, and it originates from
the last stage of the device lifecycle - the final inspection checkpoint.
At this checkpoint, multiple concluding testing scripts are run, and if
they are successful, the device leaves the company and is sent to the
end user.
Throughout the testing process, the Linux operating system running
on the device generates logs and saves them in the /var/log
directory.
4.1 Data description
The dataset consists of 4000 protocols, each representing one run of
the final tests. As each protocol is monitored for mistakes by the
aforementioned regex solution, there are 61 unique IDs marked as
erroneous, and 3939 marked as not erroneous. However, because
the labels originate from manually designed regex rules, they do not
necessarily exhaustively represent all anomalies. Consequently, some
relevant anomalies may remain unlabeled in the dataset.
This collection was created by selecting a specific major turning
point in the production history of RipEX devices and collecting all
protocols from that date onward. RipEX is a radio modem used in
critical applications like SmartGrid power networks, transaction networks,
and ATM's [9]. Apart from log messages, the protocols include
various metadata about the device currently undergoing testing.
Metadata is collected because of the need for root cause analysis.
For example, we can look at the entire board assembly in a separate
database table using the Serial 1 attribute. The constructed dataset has
the following metadata attributes:
• ID protokolu - Unique protocol identifier.
• Limity - Numerical limits identifier for the current type of the
device.
• DPS - PCB type. In this case, only M B M - Main Board Module,
representing the primary circuit board that hosts the central pro-
11
4. DATASET
cessor and manages the core system initialization and resource
allocation.
• Serial 1 - Serial number of the first PCB. Can be present in multiple
protocols creating multiple combinations with Serial 2 at­
tribute.
• Serial 2 - Serial number of the second PCB. It can also be present
in multiple protocols with Serial 1 attribute.
• Final serial - Serial number representing the final assembled
device after putting together the PCBs marked with specific
Serial 1 and Serial 2.
• Typ - Type of the device. This represents the major versions of
the device, which are updated when something important is
added to its functionality.
• Kód - More specific internal type of the device. The production
is made-to-order, and each customer has different needs and
requirements, so the devices differ in specific ways. For example,
some customers need keys uploaded to the device, some need
their own firmware uploaded, and some need to operate on
different wavelengths than the default. This creates a need to
distinguish these devices beyond the Typ attribute alone.
• Výsledek testu - Result of the test depending on the manual
regex matching. Either Ok or Error.
• Výsledek (ručne zadáno) - Manual correction of Výsledek testu.
Mostly NaN.
• Chybové řádky - Number of error lines marked by the manual
regex matching.
The main set of data used for anomaly detection and data mining
in this thesis focuses on the raw log message data. Log messages
originate from different files in /var/log directory, where some chosen
subset of files is taken, and the lines from all of these files are
concatenated with respect to timestamp attribute. A five-line example
window for a single protocol is provided in Table 4.1. Each saved line
contains:
12
4. DATASET
• ID protokolu - Unique protocol identifier.
• Timestamp - Time of log message creation. In some cases NaN.
• Zprava - Raw log message.
• Log Soubor - Name of the file the log message originates from.
There are multiple such files, hereafter always referenced to as
log types.
Table 4.1: Five-line window example
Protocol ID Log Type Message Time
52203707 Messages usbcore: registered interface... 10:26:10
52203707 Main ext_net.evt.sh: Extension... 10:26:11
52203707 Rrdevevent Starting handler... 10:26:11
52203707 Event InformationaLSystem boot... 10:26:15
52203707 Main perform.sh: Performing action... 10:26:19
4.1.1 Log Types
To fully understand the dataset, it is important to know from where
the logs originate. In this subsection, I will name each of the 19 log
types in the R A C O M dataset and provide a brief description of each.
SSH daemon log
Its logs track how users connect to the device. They show when a
connection starts, whether the login worked, and how the session
ended. These logs are useful for seeing if someone is trying to break
into the system or if the SSH service is crashing.
Event log
In the case of RipEX devices [9], the event logs are proprietary telemetry
logs. These logs contain a broad spectrum of system states, ranging
from basic initialization sequences and network interface transitions
13
4. DATASET
to critical security alerts, such as hardware tamper detection and configuration
integrity failures. They are highly structured and always
start with one of the keywords Informational, Notice, Warning, Emergency,
or C r i t i c a l .
HTTPD log
HTTPD logs originate from the lighttpd server, which is again suitable
for embedded devices [10]. Lighttpd logs are service-level logs that
document the web server daemon's lifecycle. Unlike HTTP access
logs, which record per-request data, these services focus on process
initialization and termination signals. Furthermore, these logs capture
security and runtime exceptions, such as expired certificates and
unexpected character sequences.
Main log
The Main log contains logs from various scripts that set up the device.
They track events from the initial boot sequence through the
activation of specific services such as WiFi, radio modules, and many
others. Furthermore, the main logs contain information about network
transitions and radio link states, and they record whether internal
configurations were loaded correctly or if the system had to revert to
default settings. As they show the boot process step by step, they can
also highlight issues such as services that boot repeatedly and startup
failures.
Messages log
The Messages log contains logs that are equivalent to Linux dmesg
output [11]. These logs record the Linux kernel's activity from the
moment the device powers on. They document how the hardware,
including the CPU, memory, and USB ports, is initialized and how the
different device drivers are loaded. Because these logs capture lowlevel
system events, they can be a good source for detecting hardware
failures, disk errors, and critical software crashes.
14
4. DATASET
Qsshd log
These logs track the operation of the Qsshd service. They document
whether the daemon starts and whether the process runs in the back-
ground.
Rngd log
The Rngd log contains logs for the Random Number Generator Daemon
[12]. This service is responsible for providing the system with
properly random data, which is essential for secure encryption and
cryptographic key generation. The logs are very simple: they record
the specific command used to start the daemon and the technical
parameters assigned to it.
Rrbootlog log
At the start of the device, this log type acts as a detailed summary
of the entire boot process. However, the Main log takes over this
responsibility of capturing detailed information after the Main logging
daemon is run. After this, Rrbootlog records the execution and
completion status of different scripts and tracks the progress of the
initialization sequence. Because it covers everything from the start to
the end of the boot, it is one of the best sources to see why the device
did not start correctly. Additionally, it is the only log type with initially
missing timestamps because the timer is not enabled in some early
testing scripts.
Rrchronos log
The Rrchronos log contains records for the proprietary Rrchronos
service, which manages the device's internal timers and recovery
schedules. The logs show the exact settings used to start the timer
service, including the paths to scripts that should run if the system
becomes unresponsive.
15
4. DATASET
Rrdeveventlog
The Rrdevevent log tracks the system's response to changes in peripheral
hardware. It acts as a bridge between the physical device and the
software, executing specific setup scripts whenever a USB device is
connected or a network interface changes state. By monitoring these
logs, we can verify that the RipEX [9] correctly recognizes and configures
its physical attachments. Any missing signals or corrupted
entries in this file suggest that the device is failing to communicate
with its external ports or has suffered an unexpected power loss.
Rrdiag log
The Rrdiag log tracks the invocations of the proprietary diagnostic
tool rrdiag, which generates a diagnostic snapshot file of the device.
Rrhstby log
The Rrhstdby log tracks the operational state of the Hot Standby redundancy
protocol. The records show when the daemon is initialized and
document its behavior during system failures, specifically its refusal
to start while the device is in a recovery state.
Rrioctrl log
Logs in the Rrioctrl log provide a trace of the device's Input/Output
control operations. This service manages hardware monitoring tasks,
including voltage checks and internal messaging between system
components. Any recurring error messages in this file, such as socket
connection failures or watchdog command errors, could indicate a
breakdown in communication between the software and the physical
hardware.
Rrlp log
These logs track the Rrlp service, which handles radio links and data
encryption on the device. The records show the startup parameters
for the radio hardware, such as M A C addresses and encryption key
paths. Monitoring this file lets us see whether the radio service is stuck
16
4. DATASET
in a crash loop or has successfully recovered from a communication
failure.
Rrmoneth log
Data in the Rrmoneth log record a proprietary daemon that shows
the startup of the Rrmon ethernet service.
Rrmon log
The Rrmon is another proprietary service present in our dataset. The
logs connected to the service only contain its starting command with
different arguments.
Rrsectoken log
Data in the Rrsectoken log tracks the lifecycle of the device's custom
Authentication Token service. Its records document the initialization
of the security database and provide detailed error reporting for filesystem
issues or database format conflicts.
Rrsp log
The Rrsp log tracks the execution of a background service responsible
for system-wide settings, including time-zone synchronization.
These records are essential for identifying internal communication
breakdowns.
Udhcpd log
Data in the Udhcpd log show the startup of a basic D H C P server
daemon, along with the path to the server's configuration file.
Summary
Putting all of the information above together, the 19 log types come
from different layers of the Linux operating system present on RipEX
devices. Messages capture the low-level information coming from the
Linux kernel. System initialization and booting are logged in detail
17
4. DATASET
first with Rrboot log, and then with Main. Rrdevevent records whether
physical peripherals work and communicate with the machine as they
should, such as when a USB device is connected. Logs tied to networking
are generated by multiple Linux tools and proprietary services.
SSH daemon log tracks SSH server activity and monitors user activity,
while the HTTPD log captures the web server's signals and security
alerts. The startup configuration of the DHCP server is recorded in the
Udhcpd log. For recording the specific device's radio communications,
the Rrlp log is used. The majority of the remaining types are proprietary
services of various kinds. Event log captures system state and
security alerts as highly structured messages. Other security-related
logs include the Rngd log, which tracks the random number generator,
and the Rrsectoken log, which monitors the initialization of an
authentication database. Tracing I/O hardware messaging is done by
the R r i o c t r l log, while Rrhstby log tracks the Hot Standby protocol.
Furthermore, the Rrchronos log tracks the internal timers, while the
Rrsp log tracks system wide settings like time zone synchronization.
Finally, the dataset includes startup records for specific internal tools
like Qsshd service, Rrdiag snapshot tool, and the proprietary Rrmon
and Rrmoneth logs.
4.2 Data analysis
For a successful anomaly detection system, it is important to analyze
the dataset first and only select suitable methods afterwards, as Yu et
al. [2] suggest. It is common knowledge in the machine learning field
that complex models can overfit to simple data, leading to poor generalization,
whereas simple models can fail to learn essential patterns in
sufficiently diverse data [13]. That is why, at the beginning, I create
dataset statistics to assess the properties of the dataset and determine
how complex the used models should be. Afterwards, I analyze all
log types to figure out a suitable method for each.
4.2.1 Dataset statistics
At this stage, each log type is considered as a separate entity, as different
methods can be used for each. For example, networking logs may
18
4. DATASET
require frequency analysis to detect an unusual number of connections,
and log types related to booting may require sequence analysis
to verify that all scripts ran in the correct order and no scripts were left
out. Simpler formats, such as Event log, may only require an existing
regular expression-based approach due to their predictable structure.
Another valid strategy is to keep a single master log, joined from
all log types, sorted by timestamp. This way, for example, different
types of correlations among log types can be discovered. However,
the focus is on deriving rules and finding fitting A D methods for each
log type, which would be harder in this setup.
The first step was using the Drain parser to extract unique log
templates from the data. For the final inspection use case, we are not
interested in dynamic log variables, such as IPs, M A C addresses, etc.,
so they are extracted and no longer used.
Next, I calculated general statistics for the dataset, along with
statistics on log length and temporal dependencies. They can be seen
in Table 4.2, Table 4.3, and Table 4.4 respectively.
Table 4.2 shows that the three most frequent log types together
form more than 93% of the dataset. Considering that the dataset has
4000 protocols, it follows that, on average, there are at most 6 lines per
protocol across all log types below Rrlp. For the last 8 log types, the
average number of lines is 2. This fact hints at very short sequences,
where the more complex models might overfit.
The overfit hypothesis is at least partially confirmed by UM and
UT columns for some of the lower-frequency logs. As we can see,
Rrmoneth has only one unique message. Similarly, 16 out of 19 log
types have < 10 parsed unique log templates. Therefore, this suggests
that only lightweight algorithms can be used for the majority of the
corpus.
By summing the E M column, we get 76 marked devices, indicating
that multiple logs mark at least some of the same devices, since there
are only 61 unique flagged protocols. The table shows that most errors
are caught in the Rrbootlog type, followed by Messages, R r i o c t r l ,
and Rrsectoken. O n the other hand, 13 log types do not contain
any marked errors. For example, although the Event log has not
directly caught errors, devices with logs containing the keywords
Alert, Emergency, and Warning are all marked as erroneous in other
log types. This shows that the erroneous mark is sometimes not placed
19
4. DATASET
on lines where it should be, and that there are anomalous correlations
between various log types.
Table 4.2: Statistical Distribution of the Log Corpus
Log Source Lines RF U M UT EM
Messages 2,229,543 0.575 10,388 212 11
M a i n 850,906 0.219 166 48 0
R r b o o t l o g 535,553 0.138 253 46 47
SSH daemon 62,126 0.016 38,381 8 0
E v e n t 33,533 0.009 12 9 0
R r l p 24,069 0.006 3,784 10 2
H t t p d 20,023 0.005 13 7 0
R r d e v e v e n t 16,054 0.004 4 3 0
Qsshd 16,018 0.004 7,990 5 0
Udhcpd 16,002 0.004 2 2 0
R r s e c t o k e n 12,013 0.003 6 6 7
R r i o c t r l 8,109 0.002 12 9 8
Rngd 8,069 0.002 2 2 0
R r d i a g 8,030 0.002 4 4 0
R r c h r o n o s 8,009 0.002 3 2 0
Rrmon 8,007 0.002 3,771 2 0
R r h s t d b y 8,007 0.002 3 3 0
R r s p 8,003 0.002 2 2 1
Rrmoneth 7,999 0.002 1 1 0
RF: Relative Frequency U T : Unique Templates
UM: Unique Messages EM: Error-Marked IDs
Table 4.3 even further reinforces the concept of simplicity mentioned
above. The standard deviation column illustrates the almost
completely static length of all log types below R r l p in the table. This
finding points to only auditing the lengths of all these log types, since
they are all expected to have fixed lengths.
Furthermore, because most low-frequency logs are nearly the same
length, the few protocols that differ stand out. This suggests we need
to look closer at those specific cases to understand what makes them
20
4. DATASET
different. The number of different length groups is shown in the
Unique column.
Table 4.3: Statistical Distribution of Log Lengths
Log Type Unique Mean Std. Dev. Min Median Max
Messages 61 557.39 39.56 525 561 1505
M a i n 22 212.73 6.74 199 213 447
R r b o o t l o g 26 133.89 8.22 44 134 376
SSH daemon 8 15.53 2.19 13 16 27
E v e n t 11 8.39 2.23 4 8 26
R r l p 4 6.02 0.89 6 6 60
H t t p d 4 5.01 0.18 5 5 15
R r d e v e v e n t 4 4.01 0.28 4 4 16
Q s s h d _ s v c 3 4.00 0.20 4 4 16
Udhcpd 2 4.00 0.05 4 4 6
R r s e c t o k e n 4 3.00 0.11 3 3 9
R r i o c t r l 7 2.03 0.32 2 2 12
Rngd 4 2.02 0.16 2 2 8
R r d i a g 5 2.01 0.14 2 2 8
R r c h r o n o s 3 2.00 0.10 2 2 8
Rrmon 3 2.00 0.10 2 2 8
R r h s t d b y 3 2.00 0.10 2 2 8
R r s p 3 2.00 0.08 2 2 7
Rrmon e t h 2 2.00 0.03 2 2 3
Judging by the Max column value in Table 4.4, there is at least one
large outlier that also skews the Mean column. As the Max values are
all approximately 14 days and 17 hours, this points to either a data
error or a very special device that spent two weeks at the final inspection
checkpoint, which is highly improbable. 99th Pet strengthens this
claim, as 99% of devices are at the final inspection checkpoint for a
maximum of 681 seconds.
Some log types run for the same amount of time, as shown in the
Median and Mean columns. This temporal alignment implies that they
started and ended simultaneously.
The first such group contains Main and Messages, indicating that
Main remains alive throughout the whole Linux kernel's lifecycle.
21
4. DATASET
Two more groups can be identified with durations of 84 and 66
seconds. This tells us that these logs are temporally coupled together,
and that they should usually take almost the same time to complete.
Consequently, a rule can be created to address temporal dependencies
within these groups.
Table 4.4: Temporal Span of Log Sessions (seconds)
Log Type Mean Min Median 99th Pet Max
Messages 817.50 92 157 681 1,272,869
Main 817.41 90 157 681 1,272,867
R r b o o t l o g 810.91 83 150 674 1,272,861
SSH daemon 743.20 69 106 118 1,272,840
Event 712.70 9 75 133 1,272,804
R r l p 723.77 30 87 95 1,272,816
Httpd 720.39 69 84 91 1,272,812
Rrdevevent 732.08 57 95 107 1,272,822
Qsshd_svc 720.16 26 84 91 1,272,811
Udhcpd 702.96 43 66 73 1,272,794
Rrsectoken 720.28 1 84 91 1,272,812
R r i o c t r l 771.04 1 n o 635 1,272,823
Rngd 772.94 0 112 638 1,272,824
R r d i a g 720.44 26 84 91 1,272,895
Rrchronos 720.12 0 84 91 1,272,811
Rrmon 720.31 0 84 91 1,272,812
Rrhstdby 720.33 0 84 91 1,272,812
Rrsp 703.04 44 66 73 1,272,795
Rrmon_eth 703.03 44 66 73 1,272,794
4.3 Analysis of log types
In this section, I initially look at the lower-frequency log types from
the length point of view, inspired by the suggestions I made when
analyzing Table 4.3.
Subsequently, I examine the remaining 5 higher-frequency log
types, which have higher standard deviations and may not be pro-
22
4. DATASET
cessed solely by a simple length rule. I find a suitable method to
analyze them.
4.3.1 Analysis of lower-frequency log types
Firstly, all log types below Rrlp in Table 4.3 are taken, and the different
length groups are reviewed to confirm the length-based rule
approach. In this analysis, only groups differing from the median
are examined, as the largest median group is taken as normal. Additionally,
I check whether the log IDs with unusual sizes are already
marked as erroneous, and if not, I examine why they are not.
Table 4.5 illustrates the usefulness of a length-based detector for
lower-frequency types accurately.
Table 4.5: Anomalous Group Analysis by Log File Type
Log Type Length Groups Error / Total Devices Error Ratio
H t t p d 7,8,15 6/7 85.7%
Qsshd 6,16 3/4 75.0%
Rngd 3,4,8 49/61 80.3%
Rrchronos 3,8 3/4 75.0%
Rrdevevent 6,8,16 3/14 21.4%
R r d i a g 3 , 4 , 5 , 8 10/22 45.5%
Rrhstdby 3,8 3/4 75.0%
R r i o c t r l 3,4, 6, 7,8,12 53/65 81.5%
R r l p 9, 21, 60 3/4 75.0%
Rrmon_eth 3 2/3 66.7%
Rrmon 3,8 3/4 75.0%
Rrsectoken 4, 5,9 9/10 90.0%
Rrsp 3,7 2/3 66.7%
Udhcpd 6 2/3 66.7%
Total - 151/208 72.6%
By calculating the difference between the total devices and flagged erroneous
devices from all length groups differing from the median (e.g.,
7 — 6 = 1 for Httpd, 4 — 3 = 1 for Qsshd), it can be seen that 10 of our 14
inspected log types have a missing erroneous mark on only one device.
Apparently, this is the same device, and its protocol ID is 61964703.
23
4. DATASET
After further inspection, it is the device with the longest time at the
final inspection checkpoint in Table 4.4. I decided to consult the status
of this device with technicians.
Quality assurance (QA) professionals clarified that one protocol
for wiping previous logs and configurations was not run before the
final inspection protocol, leaving logs from earlier checkpoints. The
present explanation solves both the long time span problem and its
presence across all length-anomalous groups due to a larger number
of lines. We decided to change the label on this device to anomalous
due to contamination from previous checkpoints.
Considering all of this information about types from the Table 4.5,
a rule based on the median gives us a 100% accurate solution for 10
log types length-wise. Now, there are 4 lower-frequency log types left:
Rngd, Rrdevevent, Rrdiag, R r i o c t r l .
For Rngd, there are three length groups with proportions erroneous
to all of {46/57,2/3,1/1}, respectively. In the second group, there is
61964703, which we take as erroneous. We are left with 11 devices not
marked erroneous in the first group.
Rrdevevent has again three groups in the table with ratios {2/3,
0/10,1/1}. As 61964703 is the device belonging to the first group,
we can take it as 100% erroneous. The group with 8 lines and 0/10
erroneous protocols is a batch of devices created one after another,
which may be a special batch for a particular customer with specific
needs. To confirm this statement, I spoke with the trained professionals
again. After an investigation, we found that in the production
history, there are only 10 such devices, and they have identical Kod.
A l l of these second-group devices are of one type and indeed have a
special feature: a WiFi module in the PCI slot.
Rrdiag has four groups, where the proportions are {7/18,0/1,
2/2,1/1}. The second group once again contains the anomalous device
61964703. Ten out of eleven non-marked devices' IDs match those
in Rngd, which reinforces the idea of analyzing them to figure out
why they differ. The one unique Rrdiag anomaly should be marked
anomalous, as it contains a message present only in 3 other erroneous
devices.
R r i o c t r l has six groups with erroneous proportions {42/53,2/3,
4/4,2/2,1/1,2/2}. Again, the 61964703 is present in the second group,
and the 11 non-marked IDs are identical to those for Rngd.
24
4. DATASET
The intersection of the sets of unmarked anomalous devices for
R r i o c t r l and Rngd reveals that these sets are identical, containing the
same subset of devices across both categories. Rrdiag only differs
from these two in the one aforementioned device.
With the help of the technicians, we figured out that the nonerroneous
devices from R r i o c t r l and Rngd require an additional boot
cycle than normal devices. This pattern is also observed for Rrdiag,
except for the one already excluded anomalous device. In these cases,
the extra boot cycle results from either a silent script crash or manual
intervention, both of which trigger a forced hardware reset to restore
the device to a known state. We agreed that this is something we want
to be alerted about.
4.3.2 Analysis of higher-frequency log types
For higher-frequency log types, a different assessment is needed to
select a suitable anomaly-detection approach, as the simple length
rule alone is insufficient. Each type may require a distinct analysis
approach, which is why they are examined individually.
Event log
I display part of the log templates and their frequencies in Table 4.6.
Table 4.6: Event Template Distribution for Event Logs
Event Template Frequency
N o t i c e : l i n k d o w n : s t a r t e d 17,354
N o t i c e : S e r v i c e a c c e s s p l u g g e d . . . 7,996
I n f o r m a t i o n a l : System boot c o m p l e t e d . . . 7,994
A l e r t : C r e d e n t i a l s c o n t e n t l o s t 44
A l e r t : C o n f i g u r a t i o n l o s t 44
W a r n i n g : R e c o v e r y r e s t a r t 8
A l e r t : Tamper open d e t e c t e d t o p : . . . 4
Emergency: System i n r e c o v e r y mode 2
It illustrates the simplicity of the Event log's format. The existing
solution using regular expressions already handles this log type, so
25
4. DATASET
we do not need to propose a new method. However, I would like to
mention that the low frequency of Alerts, Warnings, and Emergencies
suggests the use of a frequency-based model, which would address
one of the primary goals of this thesis: novel lines.
SSH daemon log
SSH daemon templates can be seen in Table 4.7. These are not as
straightforward as for the Event log, where the error strings were
printed at the start. Most templates are common and appear on average
at least once in each protocol.
Table 4.7: Event Template Distribution for SSH daemon Logs
Event Template Frequency
C h i l d c o n n e c t i o n f r o m <IPV4> 19,371
Pubkey a u t h s u c c e e d e d f o r ' r o o t ' w i t h . . . 19,370
E x i t ( r o o t ) f r o m <IPV4>: D i s c o n n e c t . . . 11,366
Not b a c k g r o u n d i n g 8,008
E a r l y e x i t : T e r m i n a t e d b y s i g n a l 4,007
R u n n i n g i n b a c k g r o u n d 2
~@~@~@ [ . . . ] N o t b a c k g r o u n d i n g 1
E x i t b e f o r e a u t h f r o m <IPV4>: E x i t e d n o r m a l l y 1
There are three rare templates, of which the second has null bytes at
the beginning of the message, indicating a power outage or a technician
unplugging the machine from the network during testing. The last
template is present only in one protocol with ID 51976478, which is
marked non-anomalous. Using the length method I used for lowerfrequency
types, I found that this device forms a group of its own
in the SSH daemon type, which is uncommon. Although it does not
pose a problem according to the Q A professionals, we agreed to flag
log lines like these due to their irregularity. The first rare template is
associated with protocol 22075823, which is marked as erroneous.
These three rare templates reinforce the low-frequency hypothesis
from the Event log. Even though some devices flagged by the A D
system are marked non-anomalous, they still differ from the norm
26
4. DATASET
and should be checked. A great example is the last template from
Table 4.7.
Rrbootlog
Rrbootlog is the first larger log type in a group of three that together
account for more than 93% of the datasets' raw logs.
There are 26 unique length groups and the standard deviation is
not as small as for the lower-frequency types, as described in Table 4.3.
This hints at the presence of numerous normal-length groups, not just
one as before, which is confirmed by plotting the multiplicity of each
group in Figure 4.1.
Figure 4.1: Top 5 Rrbootlog Length Groups
Multiple groups of size greater than 150 are observed, indicating
that these groups contain most non-erroneous devices. This excludes
the simple one-length rule approach used in lower-frequency types,
but we could design a rule for multiple lengths. However, there would
be some problems we would need to deal with: add a length to this
group every time a new type of device is created; the more groups
we have, the higher the chance an anomalous device has the same
length as some of the groups; there could be up to 26 groups of normal
27
4. DATASET
lengths we would need to examine and all of them could be mostly
non-anomalous.
The typical length for this type of log is 134 rows, and the number
of unique log templates is 46, which means there are more sequences
of possibly different message orderings to look through than was the
case for lower-frequency types. Because the Rrbootlog logs contain
deterministic booting steps, a chosen complex model can look for a
different arrangement of steps than normal or for a missing script in a
sequence.
To confirm the frequency approach usable for other log types,
Rrbootlog templates are examined. Because of the large number of
these log templates, I present only some of the least frequent ones in
Table 4.8.
Table 4.8: Event Template Distribution for Rrbootlog Logs
Event Template Frequency
S 3 2 0 k e y r i n g . s h : E r r o r : c h e c k _ k e y . . . 258
<*> r t _ c n f p _ c h e c k _ a n d _ r e c o v e r : C o n t e n t s . . . 177
S 3 2 0 k e y r i n g . s h : E r r o r : <*> D e f a u l t k e y . . . 88
S 4 1 0 r r s d k . s h : C r e a t i n g d e f a u l t S D K . . . 18
C h e c k _ c n f _ r u l e s : TODO 7
S 9 9 9 d o n e . s h : RECOVERY mode 2
S 2 8 5 m b p 2 _ p r e p . s h : E r r o r : F a i l e d t o s e t u p . . . 2
r c S : r t _ r e q u e s t _ r e c o v e r y _ r e s t a r t . . . 2
System i s g o i n g down f o r r e b o o t NOW 2
~ @ N o t i f y i n g m i n i t . . . 2
: E r r o r : S c r i p t / e t c / i n i t . d / S 2 8 5 m b p 2 _ p r e p . . . 2
r u n : r r d e v e v e n t _ m a n a g e r EXT_NET / e t c / . . . 2
S t a r t i n g h a n d l e r " / e t c / r r n e t w o r k i n g / . . . 2
S 2 1 0 w d g _ u p g r a d e . s h : To u p d a t e WDG f i r m w a r e . . . 1
: B o o t r e a s o n : wdg u p g r a d e b o o t 1
A s w i t h the o t h e r h i g h - f r e q u e n c y l o g t y p e s , the t e m p l a t e s that
a p p e a r the fewest t i m e s o f t e n c o n t a i n w o r d s that c o u l d i n d i c a t e a n
error. A h i g h n u m b e r of e r r o r strings f o u n d i n these t e m p l a t e s also
correlates w i t h the h i g h n u m b e r of faults f o u n d for this l o g t y p e , seen
i n Table 4.2. H o w e v e r , there are also s o m e u n u s u a l lines, like C h e c k _ -
28
4. DATASET
cnf _rules : TODO, that do not contain any error words but should not
be part of the production logs and should be examined.
All of these findings suggest the use of both a sequence-based and
a frequency-based model.
Main log
The length rule is not possible for this log type because of the mentioned
problems in the Rrbootlog. These two types are similar, which
is why the argument to alert suspicious sequences is the same as for
Rrbootlog.
Like for Rrbootlog, I display some of the log templates of the Main
log in Table 4.9 to see whether the lower-frequency templates exhibit
errors.
Table 4.9: Event Template Distribution for Main Logs
Event Template Frequency
. . . : M o n i t o r i n g c o n f i g u r a t i o n set t o d e f a u l t s . 4,001
. . . o p t i o n s . h t m l : not found 370
b o o t . e v t . s h : P r o d u c t i o n f i l e not f o u n d . . . 54
e x t _ n e t . e v t . s h : P r e p a r i n g W i F i (EXT) i n t e r f a c e . 20
. . . : WWAN (EXT) down event ( W i F i ) . 20
. . . : WWAN (EXT) down event processed ( W i F i ) . 20
r u n : Recovery r e s t a r t . . . 6
<*> S2001ogging_run.sh: BOOT (<IPV4>) 3
. . . parent <*> not f o u n d . . . 2
S 5 0 5 s t a r t u p _ a c t i o n s 2 . s h : . . . not found 2
r e c o v e r y _ o p e n . e v t . s h : Open i n R e c o v e r y . . . 2
r e c o v e r y _ o p e n . e v t . s h : RECOVERY mode 2
r c S : Recovery r e s t a r t . . . 2
The large jump from normal to erroneous templates is evident
between the first and second row. In contrast to Rrbootlog, the rare
log templates in this log type do not have the obvious error messages
easily detectable by regular pattern matching. A lot of them are related
to missing files, recovery restarts, and recovery modes. Some of
them may even be normal, just from devices with a WiFi module in
a different place, as was the case in the Rrdevevent log. Still, most of
29
4. DATASET
them seem unusual and worth examining, which is why the frequency
approach is suitable for this log type.
Messages log
The last log type accounts for more than 57.5% of the dataset. Applying
length-based rules for anomaly detection does not apply to this type,
as there are 61 unique length groups. The nature of these logs makes it
hard to detect sequential anomalies, as many stages of the Linux kernel
are non-deterministic and can execute in varying orders. Therefore,
only a frequency-based model will be utilized, as it has proved effective
for the other high-frequency log types and is highly suitable for this
dataset. Only a few of the least frequent templates are displayed in
the Table 4.10 to see the same pattern as was present in the previous
log types.
Table 4.10: Frequency Distribution of Event Templates in Messages
Logs
Event Template Frequency
. . . O s c i l l a t o r f a i l u r e , d a t a i s i n v a l i d . 12
T r y i n g t o w r i t e t o r e a d - o n l y b l o c k - d e v i c e . . . 10
usb <NUM>-<NUM>: not r u n n i n g at t o p s p e e d . . . 8
rtc-m41t80 <NUM>-<NUM>: h c t o s y s : unable t o r e a d . . . 6
. . . : PLL u n l o c k e d , r e s e t i n g . . . 5
d o _ p a g e _ f a u l t ( ) : sending SIGSEGV t o r r i o c t r l . . . 5
EXT4-fs <*> w a r n i n g : mounting unchecked f s . . . 4
E X T 4 - f s : re-mounted. Quota mode: d i s a b l e d 1
E X T 4 - f s : unmounting f i l e s y s t e m . . . 1
Summary
After the analysis, it is evident that most of the dataset can be solved
using simple methods, such as the proposed rule-based method based
on length. It will be used for 14 of the 19 lower-frequency types,
where it should work reliably. However, for the remaining 5 types, it
is unusable due to the larger complexity of the data.
30
4. DATASET
The analysis showed that the frequency-based method is useful for
all high-frequency types and will likely be useful for lower-frequency
types as well. Therefore, we decided to use an Autoencoder model
on the entire dataset, as detecting novel lines was one of the project's
primary aims.
Event log has a functioning solution with regular patterns, which
the autoencoder will support. Additionally, a PLELog [3] model will
be used to detect anomalies in sequences of Rrbootlog and Main.
The results of the analysis and the methods used for each log type
can be seen in Figure 4.2.
Frequency-based AutoEncoder
Messages Dropbear
Simple length rule
HTTPD Qsshd Rngd
Rrchronos Rrdevevent Rrdiag
Rrhstdby Rrioctrl Rrlp
Rrmoneth Rrmon Rrsectoken
Rrsp Udhcpd
Regex matching
Event
PLELog model
Rrbootlog Main
Figure 4.2: Proposed Method to Log Type mapping
Finally, although I proposed using a rule on device time spans at
the final inspection checkpoint, we decided not to use it after a conversation
with technicians. They pointed out that sometimes the devices
are restarted multiple times, for example, due to repeated failures to
generate keys, and they wait for the boot that will be successful in creating
the keys, and only proceed afterwards. This makes the process
longer, but not necessarily anomalous, as it happens pretty frequently
according to them. Additionally, they are not sure that detecting the
time spans would yield some usable results, as they see the length of
the tests they run, and know how long the tests should take.
31
5 Methods
Based on the analysis in the previous chapter, a multi-model anomaly
detection system is created by combining the selected methods. With
models mapping to log types established, the next steps are data
preprocessing, introducing the selected model architecture, and explaining
the infrastructure implementation.
Data preprocessing includes parsing the raw logs with a parser
and vectorizing templates into embeddings. Furthermore, a scaler is
applied to stabilize the networks' inputs.
The Model architecture section provides a preview of how the
model is initialized for training and used for inference. This includes
showing the number of layers and their neuron counts by log type,
the activation function, the loss function, the threshold setting, the
optimizer, the early stopping procedure, and the batch sizes.
A section on infrastructure details how and where the models are
trained and used. Additionally, I explain how both the simpler rules
and the more complex deep learning methods are integrated into
production. This includes a commentary on how novel log lines are
handled, as managing new production data is not a trivial task.
5.1 Data preprocessing
The first step in preprocessing the data for the main model is to parse
the raw textual logs into structured log messages. Raw logs contain
constant information alongside dynamic information, such as timestamps
or IP addresses. Log parsers extract the dynamic parts called
variables, generating log events. Afterwards, log events are grouped
into log groups using different methods, depending on the parser.
Figure 5.1 shows an example of log parsing input and output.
Choosing a suitable parser for textual logs is an important step for
the following anomaly detection. Since Drain [14] is the most widely
used in today's literature [1] for logs from various systems, including
Linux, I decided to use it as well.
32
5. METHODS
Create table: 'rlp_ifc_staf: 9.0.
Child connection from 10.9.8.14:62408
Link '/mnt/data/db/diag/event.db' is not present, create it.
Create table: 'eth_stat': 9.0.
Log parsing
Create table: <*> <NUM>.<NUM>.
Child connection from <IPV4>:<NUM>
Link <*> is not present, create it.
Create table: <*> <NUM>.<NUM>.
Figure 5.1: Example of Parsing
5.1.1 Drain parser
Drain is a fixed depth tree based online log parsing method [14].
When a new message is passed to Drain, the parser first checks the
lengths of the existing groups in the first layer, selects the message's
length, and then proceeds to the second layer. Here, it matches the
message based on the event's initial tokens, which are likely to be
constants [14]. After this, the parser traverses to a leaf node that
contains a list of log groups. To select the most suitable log group
from the list, a similarity score simSeq is calculated for each group
using the following equation:
EL equ(seq1(i),seq2(i))
simSeq = (5.1)
where seq1 and seq2 represent the log message and the log event respectively,
n is the sequence length of the log message, and seq(i) denotes
the z'-th token of the sequence [14]. The equivalence function equ for
any two tokens t1 and t2 is defined as follows:
equ(t1,t2) = \1 l f
' 1 =
*2
(5.2)
[0 otherwise
After finding the log group with the largest simSeq, it is compared
with a predefined similarity threshold st. If simSeq > st, Drain returns
33
5. METHODS
the group as the most suitable match [14]. Otherwise, Drain returns
a flag to indicate that no suitable log group exists.
Drain adds this log ID to a matching group if it exists. Additionally,
the log event in the log group will be updated as follows. The parser
compares the tokens at the same positions in the log event and the log
message. If they are the same, nothing is updated. If they differ, then
a wildcard usually represented by <*> is inserted into the log event
for the place of that token [14].
If there is no suitable group, the parser creates a new group with
the single new message as its only element.
The tree structure for a better understanding of the matching process
can be seen in Figure 5.2.
Length: 10
Send Receive Starting
Log Group
Log Event: Receive from node *
Log IDs: [1, 23, 25, 46, 345, ...]
Root Node | | Internal Node [ | Leaf Node ^ j Log Group
Figure 5.2: Structure of Parse Tree in Drain (depth = 3) [14]
Drain offers multiple configuration options beyond the alreadymentioned
similarity threshold (st). A user can define regular expression
strings for known patterns to be masked, such as M A C and IP
addresses. The parser removes such strings and replaces them with
strings specified by a user. The depth of the tree can be tuned with
the depth parameter to parse in a more fine-grained or coarse-grained
manner, and a maxChild parameter can be used to prevent an explosion
of some branches.
34
5. METHODS
The next step after mining templates from the messages is to create
vector representations, which serve as inputs to the neural network.
5.1.2 Vectorization and Embeddings
The transformation of the parsed log events to numerical vectors is performed
using a semantic embedding pipeline that captures the contextual
relationships within the log data. The process begins by loading
the all-MiniLM-L6-V2 model from the SentenceTransf ormers library.
While specialized log embedders exist, a general-purpose language
model was chosen for this thesis to prioritize the semantic meaning
of the English non-technical words within the logs. Since most
technical variable terms are intentionally masked or dropped during
parsing, the objective is to analyze the underlying system message
rather than the technical identifiers. This pre-trained model generates
384-dimensional dense vectors, mapping semantically similar log
events closer together in the vector space.
Because neural networks are sensitive to the scale of input features,
and the raw embedding outputs may show significant variance,
a StandardScaler from the s c i k i t - l e a r n library is applied. This
standardization step ensures that the features have zero mean and
unit variance, providing a stable input for the autoencoder.
5.2 Model architecture
The main model used in this thesis is an autoencoder trained on log
events, as one of the primary aims is to detect novel and rare lines.
Inputs for the network are produced by a parser and an embedder as
described above.
The input size is 384, followed by a hidden layer and a latent layer
with dynamic sizes depending on the number of unique templates
of the currently processed log type. For sources with few templates
(< 10), a smaller architecture (16 hidden, 4 latent) is used to prevent
overfitting, while more complex sources like Messages and Main (> 50)
use a larger architecture (128 hidden, 32 latent). Rrbootlog, which is
between this range, that is 10 < t < 50, uses 64 neurons in the hidden
35
5. METHODS
layer, and 16 in the latent layer. After each hidden layer, a ReLU
activation function is used to introduce non-linearity to the model.
The first part of the network, up to the latent layer, is called the Encoder.
It compresses the input into a smaller representation, from
which the second part of the network, the Decoder, learns to reconstruct
the input from the smaller representation. I used the Mean
Squared Error (MSE) objective function to calculate the error of this
process. It has the following equation:
where x is the original vector and x' is the reconstructed one, and n is
the size of the vector.
Once the model is trained, the distribution of reconstruction errors
is analyzed to establish a suitable anomaly threshold. By applying the
Interquartile Range (IQR) outlier rule, the system sets a boundary
beyond which any log line is flagged as a novel or rare event. To ensure
high precision and minimize false alarms, a multiplier of 3 instead of
the traditional 1.5 is applied to the IQR. The final anomaly threshold
T is defined as:
where Qt and Q 3 represent the 25th and 75th percentiles of the reconstruction
errors, respectively. IQR is the interquartile range defined
by Q 3 — Qi- The max function and the constant 0.0001 are used as
a safety measure to prevent the threshold from reaching zero in log
files with extremely low variance. This value was empirically derived
from the reconstruction error data.
To reach the final state, the Adam optimizer is used with a learning
rate of 0.001 for a maximum of 20 epochs. I split the training data
further to create an internal validation split for early stopping. The internal
validation size is 10% of the training data, and the training ends
after 3 epochs without improvement on the validation set. When it
ends early, then the model rolls back to the best-performing weights.
The batch size is adjusted dynamically based on the log type size.
It ranges from 32 for the simpler ones up to 2048 for the complex ones.
The network hyperparameters mentioned in this section were manually
tuned by running tests and evaluating results on the validation
! = 1
(5.3)
T = max(Q3 + (3 x IQR),0.0001) (5.4)
36
5. METHODS
set, as training is outsourced to Google Cloud. I aimed to have a combination
of parameters that would capture as many labeled anomalies
as possible while keeping false alarms to a minimum. In most cases,
the second part of this condition was not satisfied, as the Messages
and Main log types typically flagged more than 200 devices from the
validation set, which is approximately half of them. The used configuration
of neurons and batch size yielded the fewest false positives and
performed best on the used metrics.
Several configurations were tested for (hidden_dim, latent_dim), including:
{(512,64), (256,32), (128,64), (128,32), (64,16)}. The larger
configurations were tested primarily to determine whether they could
address the high false-positive rate. They were not able to improve
the situation, but, on the contrary, the false-positive rate increased
from approximately 100 devices to 200-300 marked devices out of 400.
As enlarging the network capacity and adjusting the batch size did not
help solve the false-positive problem, another approach is presented
in the following subsection.
5.2.1 Prevalence filter
As the training focuses on the frequency of individual lines, the autoencoder
learns the usual templates very well and reconstructs the
rare ones with high error. For high-volume logs, such as Messages,
it can happen that a line or template appears on each tested device
but is not well learned, as its absolute frequency is only maximally
4000, which is approximately 0.2% of the dataset. This makes it a rare
pattern, as other log messages or templates that appear more than
once per log shadow these lines due to their frequency. However, they
are not anomalous, as they appear on every device and should not
trigger an alert.
To address this issue, I present a solution that filters out such
cases by setting an upper bound on the number of devices having
this unique line. It scans the already saved template_ratios . p k l file
during training, where the ratio is calculated for each template t as
follows:
Df
R
t = -FT- (5-5)
u
total
37
5. METHODS
where Dt is the number of devices that generated at least one instance
of template t, and Dtotal is the number of all devices.
When a line is marked anomalous by a model, it is checked whether
the ratio Rt exceeds the manually set upper bound T. If Rt > T, then it
is considered sufficiently prevalent, and its label is changed to nonanomalous.
In the other case of Rt < r, the label stays anomalous.
The T was empirically set to 0.016 = 1.6%, which is approximately 50
devices out of the 3111 in the training dataset. It was observed that
templates occurring in more than approximately 50 devices belong to
the normal message group, including new manufacturer messages.
For example, there is a line with 58 unique device occurences known
to be non-anomalous. Setting the threshold above 50 would therefore
increase the number of false alarms, as such templates would be
marked as anomalies. On the contrary, templates below the threshold
are mostly anomalous and are therefore primary targets for detection.
5.3 Infrastructure
As the detection of the novel lines and anomalies is the primary aim of
the project, frequency-based autoencoder is implemented first. This
model is fully integrated and deployed within the R A C O M infrastructure,
where it automatically processes incoming logs from the final
inspection checkpoint. By analyzing reconstruction errors, the system
identifies deviations from normal behavior, providing technicians with
an automated tool to flag potential hardware or software malfunctions
in real time.
However, the autoencoder is only one component of a hybrid
strategy. Complementary methods, such as whitelists and simple
length-based rules, provide highly accurate monitoring for 14 out of
19 lower-frequency log types.
A sequence-based approach was also considered during the initial
analysis in the form of the PLELog model, but it was not prioritized
at this stage of development. As indicated by the analysis, most log
types exhibit low variability, short sequence lengths, and a limited
number of unique templates. These characteristics suggest that the
dominant anomalies are not primarily driven by sequential dependencies
but rather by deviations at the individual log message level. The
38
5. METHODS
integration of the PLELog model is therefore left as an extension for
future work. Especially, if any sequential anomalies are observed in
the production.
Building an anomaly detection system requires a solution that
addresses model training preparation as well as model deployment
for production. Both are explained in this section, along with the
problems that can arise and the solutions to repair them.
5.3.1 Training infrastructure
The training pipeline begins with the logmodelcreate bash script,
which is run when it is time to train a new model. The script receives
raw log j son data from stdin. It runs a Python script that executes
the following more complex logic.
To prepare the data for training, a local file is uploaded to a Google
Cloud Storage bucket. The system then generates a YAML orchestration
file that specifies the required computational resources, including
memory allocation and the use of GPUs to accelerate embedding and
training. A batch job is run with these specifications.
The YAML file includes a path to the containerized training environment
and runs a t r a i n . py script from it, passing the data input as a
path to the previously uploaded file on a Google bucket. The training
script is created beforehand, containerized with Docker, and uploaded
to the Google Artifact Registry. Containerizing the training script and
the requirements file into a Docker image ensures reproducibility.
In addition to borrowing resources, Google Cloud Batch allows
running jobs in parallel, which speeds up training and provides telemetry
for each job. Also, scalability is ensured for future projects, as more
resources can be used for more complex solutions.
This training part is run manually using the aforementioned bash
script, as the number of protocols and the retraining time period are
variables that need to be considered for each model retraining. For
example, we cannot say we will retrain the models each month, as
there may be a production outage or a new type of device may be
produced in the next month. The former would result in training on
only a few new data points and most old data, and the latter would
result in many unwanted false positives.
39
5. METHODS
5.3.2 Inference pipeline
To detect anomalies during production, a fully automated pipeline is
implemented and deployed.
When a new log is generated on the final inspection checkpoint,
it is inserted into a database. Firstly, it is recorded to the RACOM's
intranet whether the lengths of the simpler log types are equal to
their median. Subsequently, the database has a trigger that runs the
log_model_use bash script with the log data in stdin. This script
checks whether the correct parameters were given and creates a socket
request to an already running server waiting for data.
The server loads the data from the request and prepares a model
with the configurations saved during training. These contain a threshold,
model architecture, template miner, and a scaler. Afterwards,
it runs inference with this initialized model. The last used model
is cached on the server to not load the model each time a request is
recieved, which makes the inference faster.
Each log message is given a label_numeric value by the model,
where 0 indicates a non-anomaly and 1 indicates an anomaly. Furthermore,
each anomalous log template is assigned an internal severity
score on a scale from 1 to 5 to assess the problem's magnitude. The
severity score is assigned only to anomalies, based on their distance
from the threshold.
Finally, the response is sent back to the intranet developers, who
use the combination of (log_id,row,label_numeric,severity) to
assign each row in the dashboard its corresponding label. This system
is used by technicians, who are notified of anomalies during the final
inspection via email.
When a novel line appears in the system, it can cause problems if it
is not an anomaly, as it will be flagged repeatedly until the next model
retraining. For this, a regular expression ignore tool was implemented,
providing a feedback loop for technicians to filter non-anomalous
novel lines. O n the dashboard, they can see a list of already-created
strings, create new ones, or delete old ones.
40
6 Evaluation and results
This chapter first presents the evaluation setup, including details about
the data split, label refinement based on the previous analysis, and
the cleansing process applied to the training dataset. Furthermore, it
introduces the performance metrics used for model assessment.
Next, the chapter showcases the results of the frequency-based
autoencoder model and its combination with the prevalence filter,
referred to as the Hybrid Approach. Their performance is compared
against the existing Regex Baseline, which the proposed methods
should surpass to provide practical value.
Afterwards, the results are discussed in detail. First, the improvements
of the new approach over the previous one are reviewed, and
the decrease in recall caused by the prevalence filter is explained. Then,
the models are compared based on their ability to capture different
types of anomalies present in the dataset. Additionally, the usefulness
of the prevalence filter is demonstrated through an example, and
correlations between flagged anomalies across different log types are
analyzed.
To further evaluate categorization accuracy, an analysis of false
positives and false negatives is conducted. Finally, the chapter presents
an example from the production environment to demonstrate the
practical usefulness of the proposed system.
6.1 Evaluation setup
The dataset contains 4000 protocols, of which 61 are marked as erroneous
by the regex method. I am going to use these labels with the
understanding that there may be many more devices that should be
marked but are not. We saw some such cases in the Data analysis
chapter, where we identified 24 unmarked devices that should have
been flagged because they showed unusual behavior. In summary,
there are 85 anomalous devices we know of. These devices are marked
with an anomalous label.
Splitting the data into training, validation, and test sets is the next
logical step. I decided to use the 80/10/10 ratio, which is common in
the existing literature [1]. The anomalous marked protocols are di-
41
6. EVALUATION A N D RESULTS
vided 50/50 into the validation and test sets, so 42 are in the validation
set and 43 in the test set.
When there are many errors in the training dataset, it can skew
the threshold set at the end of training. This would result in a less
accurate final model, which is why I settled on cleaning it beforehand.
Firstly, I train a model for each log type on the available training data.
Then, the 1% of templates with the highest reconstruction errors are
selected, and their corresponding protocol IDs are removed from the
dataset to ensure a clean training baseline. Using this method for
each log type, 21 unique protocols were removed from the dataset and
excluded from further use, leaving 3111 protocols for training.
During this process, I cleansed the three high-frequency log types:
Messages, Main, and Rrbootlog. However, during analyzing the 1%
highest-error templates for the lower-frequency types, I found that
there are 7 log types with only 1 unique template, 3 with 2,4 with 3,
and SSH daemon with 5. Even before, the number of templates was
very low for training a deep autoencoder, but I wanted to keep the
same approach for all types, as it could be easily automated, and no
other solution would be needed. However, it became evident that
training a neural network model for all these lower-frequency log
types would be impractical, as the trained autoencoders would only
behave like whitelists. I decided to use only the length rule, along
with the template whitelists, for these log types due to their simplicity.
The metrics most commonly used in the existing literature [1] are
also adopted for this thesis: Precision, Recall, and the F l score. These
are defined by the following equations:
X p
Precision = T p + p p (6.1)
R e c a 1 1
= TPTFN ( 6
' 2 )
Precision • Recall
F l score =
2 • (6.3)
Precision + Recall
We want to reach almost perfect Recall, as the anomalies detected by
the previous simple regex method should be easily catchable. However,
we also want acceptable Precision, as we do not want to trigger
many false alarms that would overwhelm human personnel.
42
6. EVALUATION A N D RESULTS
6.2 Results
The results of both new approaches are compared with the Regex
Baseline in Table 6.1. The table is divided on validation set, which I
tuned the hyperparameters on, and the unseen test set used for final
evaluation of the model. The best scores for each set and metric are
highlighted.
Table 6.1: Anomaly detection performance of the previous approach
compared to the new ones
Dataset Model Precision Recall Fl Score
Regex Baseline 1.0000 0.7317 0.8451
Validation Set Autoencoder only 0.2923 0.9048 0.4419
Hybrid Approach 0.9048 0.9048 0.9048
Regex Baseline 1.0000 0.6905 0.8169
Test Set Autoencoder only 0.2786 0.9070 0.4262
Hybrid Approach 0.9024 0.8605 0.8810
Table 6.2 presents the model's ability to catch anomalies of different
types. In the dataset, there are two such types, which originate either
from the legacy regex solution or from the dataset analysis chapter.
Table 6.2: Detailed Anomaly Detection Performance: Type-Specific
Breakdown
Subset Model Regex Analysis
TO CA MI TO CA MI
Regex Baseline 32 32 0 10 0 10
Validation Autoencoder only 32 32 0 10 6 4
Hybrid Approach 32 32 0 10 6 4
Regex Baseline 29 29 0 14 0 14
Test Autoencoder only 29 29 0 14 10 4
Hybrid Approach 29 27 2 14 10 4
TO: Total anomalies; CA: Caught (TP); MI: Missed (FN).
Figure 6.1 and Figure 6.2 describe the relationship between the
protocols caught by log types that identified at least one anomaly.
43
6. EVALUATION A N D RESULTS
Figure 6.2: Correlations between log types for the test set
The correlation in the heatmaps is based on the ratio of the intersection
size to the size of the smaller group. This overlap metric is
defined as:
where A and B represent the sets of anomalous devices flagged by two
different log types. For example, Rrdevevent identified 5 devices and
44
6. EVALUATION A N D RESULTS
Main identified 7 devices as anomalous. Their correlation is 1, since
all devices in the smaller set are also present in the larger set.
Afterwards, Table 6.3 provides a breakdown of the number of
unique devices flagged as anomalous by each detection method across
19 log types. The results are again divided into validation and test sets,
allowing direct comparison of the regex, AE, and hybrid methods. The
HU column shows which log types for the Hybrid Approach added
unique protocols that were not detected in other log types.
Table 6.3: Number of devices flagged by different methods across 19
log types, including unique contributions by the Hybrid Approach
Log Type
Validation Set Test Set
Log Type
RE A E H Y H U RE A E H Y H U
Messages 5 108 15 4 6 121 15 4
R r b o o t l o g 23 28 28 3 24 23 23 1
Event 0 25 25 0 0 23 23 0
Main 0 7 7 0 0 9 9 0
Rrdevevent 0 5 5 0 0 6 6 0
Rrsectoken 3 5 5 0 4 4 4 0
R r i o c t r l 4 4 4 0 4 4 4 0
R r d i a g 0 3 3 1 0 4 4 2
SSH daemon 0 0 0 0 0 3 3 1
Httpd 0 2 2 0 0 2 2 0
Qsshd 0 0 0 0 0 2 2 0
Rrhstdby 0 0 0 0 0 2 2 0
R r l p 1 1 1 0 1 2 2 0
Rngd 0 0 0 0 0 1 1 0
Rrchronos 0 0 0 0 0 1 1 0
Rrmon 0 0 0 0 0 1 1 0
Rrsp 0 1 1 0 1 0 0 0
Rrmon_eth 0 0 0 0 0 0 0 0
Udhcpd 0 0 0 0 0 0 0 0
RE: Regex Baseline; AE: Autoencoder only; HY: Hybrid Approach; HU:
Hybrid Unique.
45
6. EVALUATION A N D RESULTS
6.3 Discussion
From the Table 6.1, it is apparent that the hybrid approach performs
the best on both validation and test sets in terms of F l score. The
filter improves the sole autoencoder's precision from 0.29 to 0.90 on
validation and from 0.28 to 0.90 on test sets. Additionally, it keeps the
validation recall the same, but worsens the test one. This can point to
a bad setting of the prevalence upper bound r, as actual anomalies
should not exhibit a higher prevalence proportion than the defined T
ratio.
After further analysis, I found that the removed templates were
from the Messages log type, and that two unique devices were wrongly
eliminated from the anomaly set. In the train set, this template occurs
in 79 unique protocols, exceeding the current T of 50, so the filter
classifies it as non-anomalous. However, this exact template and even
the exact message appear in the full dataset as normal 214 times and
6 times as erroneous. If a line is marked as erroneous in one protocol,
it should be marked as erroneous in every protocol, as confirmed by
the technicians.
Thus, this line should have been marked erroneous at every occurrence
from the start, which means it would not have been included in
the train set. The model would correctly mark it as anomalous because
it would not know how to reconstruct it, and the filter would not see
any device in the train set with this message. However, currently, even
though the autoencoder does not learn to reconstruct the message
from 158 occurrences in the training set, making it an anomaly, the
filter still removes it from the anomaly set because it is sufficiently
prevalent across the training set. Based on this analysis, the problem
was not in the incorrect setting of the prevalence filter T, but rather in
the inconsistent labeling of the one log message.
The anomalous labels were constructed from the regex errors and
the anomalies found during the dataset analysis, as seen in Table 6.2.
This shows in the regex baseline model's perfect precision, which identifies
only existing errors, leading to no false positives. O n the other
hand, it has lower recall than the other approaches because it cannot
detect size- or frequency-based errors during analysis. Autoencoderbased
approaches detect more truly anomalous devices overall and
also identify all devices, as the previous approach does, as seen in
46
6. EVALUATION A N D RESULTS
Table 6.2. This means we can probably drop the old regular patternmatching
approach and use only the hybrid method, which can detect
anomalies that the regex method misses and catch all the errors that
the original approach does. However, the reliability and consistency
of this approach should be further confirmed through an extended
period of operation in the production environment.
Usefulness of the prevalence filter can be derived from Table 6.3.
Although it helped only for the Messages log type in both the validation
and test sets, it removed 88 false positives for the former and
106 for the latter. If it had not been used, the autoencoder would
flag approximately 1 /3 of both sets as anomalous, which would be
unacceptable in production, as it would exhaust the technicians with
false alarms.
Furthermore, Table 6.3 describes that the most useful log types
when focusing on frequency are Rrbootlog, Event, and Messages. The
first place stays the same as for the regex approach, but there is a large
jump for the Event log type. It may be due to the problem mentioned
while analyzing the Event log, where we saw that templates containing
error words were not marked as erroneous. The templates of the Event
log are presented in Table 4.6. In addition, HU column confirms that
Messages, Rrbootlog, Rrdiag, and SSH daemon should always be used
for anomaly detection, as they detected some protocols, which were
undetected by models for all other log types.
Figure 6.1 and Figure 6.2 show the correlation between the log
types based on the aforementioned overlap metric. The two most
performant log types, Rrbootlog and Event, have a strong correlation,
differing by 1 device in the validation set and 2 in the test set. This
shows that the majority of problems appearing in the Event log also
appear in the Rrbootlog. This redundancy can be an advantage, as
it ensures that if one log type is updated and misses some error, the
anomaly will still likely be captured by the other log type. Furthermore,
it explains why the Event originally lacks erroneous markers,
as the errors typically cascade into Rrbootlog.
On the contrary, the third-most-performing log type, Messages,
shows a low correlation with both of them. This indicates that it
captures low-level Linux kernel faults that typically do not flow into
higher-level proprietary boot sequences and scripts, demonstrating
their complementarity.
47
6. EVALUATION A N D RESULTS
Another significant correlation is between Rrdevevent and Main,
where the former is a perfect subset of the latter in both examined sets.
This indicates that anomalies tied to peripheral devices consistently
cascade into the main system scripts. A similar high correlation is also
prevalent among the low-frequency services, including Rrchronos,
Rrhstdby, Rrmon, Httpd, Qsshd, and Rrlp, which form a highly correlated
block in the test set heatmap. However, this does not represent a
pattern, as all of these are tied to the previously unmarked device with
ID 61964703, which was anomalous in almost every lower-frequency
log type.
Table 6.2 shows the superiority of the new methods. Still, they
missed some known anomalies and flagged some devices labeled
normal. We will look at both such cases.
False negatives
There are 4 missed anomalous protocols for validation and 6 for the test
set. The 2 undetected regex-based anomalies were already discussed
at the start of the discussion, and I found that they had inconsistently
labeled messages. Apart from them, all of the others are tied to the
new anomalies discovered during the data analysis. After examining
their IDs, they all come from the batch of devices with one extra
boot, and none contain any infrequent messages because they failed
silently during one boot. Evidently, frequency-based methods are not
sufficient to capture all types of anomalies in the dataset. This shows
that the length rules should be used for lower-frequency log types,
as they complement frequency-based methods and catch errors that
autoencoders and regex miss.
False positives
Each protocol in this set is analyzed individually using files generated
by running inference on the validation set and the test set. The lines
detected by the model are put into Table 6.4 and Table 6.5 for better
readability. Each anomalous line is analyzed, and it is determined
whether it was a model error or a labeling error in the data set. I also
want to point out that all of these lines originate from the Messages
48
6. EVALUATION A N D RESULTS
file, indicating that the other log types are labeled correctly in the
validation and test sets from a frequency perspective.
Device 11188857 has only one line with a template, which appears
once in only 3 protocols across the entire dataset. The log line indicates
a loss of synchronization that does not propagate to higher-level logs.
Considering the train size of Messages, which is approximately 1.7
million lines, this is an anomaly we want to be alerted about, so the
model was precise there. The label should be changed to anomalous.
Again, for 11051653, there is an infrequent log line that occurs in 8
protocols in the dataset. Again, it is a message we want to be alerted
about, and it could even represent an error, as it says: the usb i s not
running at the top speed.
The next device with ID 91834245 has 5 anomalous messages.
Three of them appear in 14 unique devices, and the template represents
a rare type of flash memory used in these units. Although the
global count of this template seems low, it could appear to be a normal
line based on its semantics. However, during internal stress testing
of a specific batch, technicians noticed inconsistent behavior that was
not trivial to identify because they did not know where to look. Once
they reviewed the logs and the anomalies detected by the autoencoder,
they realized that all affected units shared this new memory component.
This shows that these lines should be flagged to monitor new
or rare hardware components, as the model can successfully isolate
the root cause before the units move further in the production chain.
The remaining anomalous lines for this device, each occurring 7 times,
are tied to ext4 mount events that warn about the "Year 2038 problem"
[15], specifically noting that the filesystem supports timestamps
only until 2038-01-19. While these messages originate from standard
kernel behavior, they should be flagged as anomalous, as they are
rare.
The last device with ID 31834245 has the same lines as the previous
device, so the same conclusion applies.
The anomalies identified in the test set, as presented in Table 6.5,
consist of the same message templates found in the validation set.
Given that these specific lines were determined to be legitimate anomalies
during the validation phase, the same logic applies to the four
protocols in the test set. Therefore, they represent problems that
should be flagged by the hybrid approach.
49
6. EVALUATION A N D RESULTS
Table 6.4: Representative Validation Set Anomalies
Device ID Anomalous lines detected Appearances
11188857 dwc3-octeon: PLL unlocked, reseting.. 3
11051653 usb 1-1: not running at top speed... 8
91834245 mmcblkO: mmc0:0001... 14
mmcblkObootO:... 14
mmcblkObootl:... 14
ext4 remounted at /mnt/cnf... 7
ext4 mounted at /var/tmp/... 7
31834245 mmcblkO: mmc0:0001... 14
mmcblkObootO:... 14
mmcblkObootl:... 14
ext4 remounted at /mnt/cnf... 7
ext4 mounted at /var/tmp/... 7
Table 6.5: Representative Test Set Anomalies
Device ID Anomalous lines detected Appearances
42178215 mmcblkO:... 14
mmcblkObootO:... 14
mmcblkObootl:... 14
ext4 supports timestamps... 7
ext4 supports timestamps... 7
72047612 usb 1-1: not running at top speed.. 8
32040946 mmcblkO:... 14
mmcblkObootO:... 14
mmcblkObootl:... 14
21870765 dwc3-octeon: PLL unlocked... 3
Production evaluation
The hybrid system with the prevalence filter was also tested in production
by running inference on 40 protocols generated after the last
timestamp in the original dataset. The first 8 out of 9 protocols had 0
anomalies. The remaining protocol included lines tied to the unusual
memory, which were identical to those in the false positives.
However, a problem described earlier occurred: a device firmware
update introduced 25 new lines. These messages were present in 28
out of the remaining 31 protocols with a specific Kód of the device.
The technicians reviewed the messages and concluded that they are
non-anomalous. Regex patterns can now be added to ignore these
50
6. EVALUATION A N D RESULTS
messages until the next retraining. Together, we decided not to use
this option, since the new messages are present on every device and
enough new protocols are already available for retraining.
Additionally, I ran the model on one protocol containing errors
detected by the regex method. The baseline regex method detected
50 anomalous lines, whereas the hybrid model detected 60. The additional
10 messages were rare and should have been flagged for inspection.
This pattern can be seen in most erroneous protocols: when the
regex baseline detects errors, the hybrid method detects them all, and
additional suspicious lines.
To summarize, the deployed system performs as expected, accurately
detecting novel and rare lines while improving on the original
regex-based solution. It also provides the option to ignore newly introduced
non-anomalous messages until the next retraining, temporarily.
Overall, the production tests confirmed the practical usefulness of the
proposed approach.
51
7 Conclusion
Throughout this thesis, I focused on improving the anomaly detection
system at final inspection for RACOM. The system is deployed on their
intranet, where technicians can use it to detect immediate problems
and perform root-cause analysis for broken devices.
Firstly, the problem was clearly defined and the project's motivations
were presented. These included creating the system to detect
novel and rare log lines, applying more advanced methods than simple
regular-expression matching to identify hidden anomalies, assisting
in the detection of faulty batches of parts, minimizing the number of
false alarms to avoid overwhelming technicians, and using detected
anomalies for subsequent root-cause analysis.
Afterwards, the latest literature related to log anomaly detection
was reviewed to gain insight into the field. Based on this review, one
suitable paper was selected for the given use case and analyzed in
detail.
Furthermore, a dataset analysis was conducted. The metadata and
raw log data were first described, after which the logs were divided
into individual log types based on the system components from which
they originated. Statistical analysis of the dataset was then performed,
and each log type was examined to derive a mapping of suitable
anomaly-detection methods.
All components of one selected model were described in the Methods
chapter, including the model inputs and preprocessing pipeline,
parameters and hyperparameters, and the prevalence filter used as
a correction mechanism for reducing false positives. In addition, the
training pipeline was explained from an infrastructure perspective,
together with the newly deployed automatic inference pipeline integrated
within RACOM.
To prepare the dataset, the data was split into training, validation,
and test sets using both the original labels and the additional
anomalous labels identified during the dataset analysis. To reduce contamination
in the training data, the top 1% of samples with the highest
reconstruction errors were removed separately for each log type. During
this step, it became clear that 16 out of 19 log types contained
fewer than 5 unique templates, making deep learning unnecessarily
52
7. CONCLUSION
complex for these cases. For this reason, whitelist-based detection
was used for these log types, while autoencoders were trained only
for the remaining ones. In addition, simple length-based rules were
applied to 14 out of 19 lower-frequency log types, complementing
the frequency-based approach and providing reliable detection for
low-variability logs. The models were then trained both with and
without the prevalence filter and compared against the original regex
baseline, where the proposed approaches achieved better results.
Additionally, the anomalies detected by each method were presented
together with correlations between different log types based on
the unique devices they captured. The mistakes of the best-performing
model were further analyzed through an examination of false positives
and false negatives. Finally, the usefulness of the prevalence
filter was demonstrated.
The results showed that the proposed autoencoder-based anomaly
detection system can improve the detection of hidden and previously
unseen anomalies compared to the original regex-based solution while
maintaining a manageable number of false positives. When combined
with the length-based rules, it can catch all the anomalies we know
of, and even more, as it detected some originally normal labels that
were supposed to be anomalous. Additionally, the correlation analysis
revealed several redundant log types and relationships between
anomalies captured across different logs. It also helped identify the
log types necessary for anomaly detection, as some protocols were
marked as anomalous by only a single log type. Finally, a real example
from production was presented, which shows the usefulness of the
deployed system.
Despite the achieved results, the proposed system still has several
limitations. The original dataset labels were generated using
regex-based matching and may therefore not have captured all anomalies,
introducing uncertainty into the evaluation process. In addition,
sequence-based methods were analyzed but not integrated into the
deployed solution, as frequency-based anomalies were the primary
focus. Finally, temporal-based rules were investigated during the
dataset analysis phase. However, at the end they were not deployed
because the observed temporal variations were too common within the
production process to provide sufficiently reliable anomaly indicators.
53
7. CONCLUSION
Future work could focus on integrating more advanced sequencebased
models. However, because the dataset has low variability and
the logs are highly deterministic, it is questionable whether there
are enough sequence-based anomalies to justify such methods, or
whether the currently proposed approaches are already sufficient for
the production setting. Additionally, a Large Language Model (LLM)
could be deployed at the end of the pipeline to provide natural language
explanations for detected anomalies. This would be particularly
valuable for log types where errors are not straightforward, further
assisting technicians in the root-cause analysis process.
54
Bibliography
1. L A N D A U E R , Max; O N D E R , Sebastian; SKOPIK, Florian;
WURZENBERGER, Markus. Deep learning for anomaly detection
in log data: A survey. Machine Learning with Applications.
2023, vol. 12, p. 100470. ISSN 2666-8270. Available from DOI:
https://doi.org/10.1016/j.mlwa.2023.100470.
2. YU, Boxi; YAO, Jiayi; FU, Qiuai; Z H O N G , Zhiqing; XIE, Haotian;
WU, Yaoliang; M A , Yuchi; HE, Pinjia. Deep Learning or Classical
Machine Learning? A n Empirical Study on Log-Based Anomaly
Detection. In: Proceedings of the IEEE/ACM 46th International Conference
on Software Engineering. Lisbon, Portugal: Association for
Computing Machinery, 2024. ICSE '24. ISBN 9798400702174. Available
from DOI: 10.1145/3597503.3623308.
3. YANG, Lin; C H E N , Junjie; WANG, Zan; WANG, Weijing; JIANG,
Tiajun; DONG, Xuyuan; Z H A N G , Wenbin. Semi-Supervised LogBased
Anomaly Detection via Probabilistic Label Estimation. In:
2021 IEEE/ACM 43rd International Conference on Software Engineering
(ICSE). 2021, pp. 1448-1460. Available from DOI: 10 . 1 1 0 9 /
I C S E 4 3 9 0 2 . 2 0 2 1 . 0 0 1 3 0 .
4. BOJANOWSKL Piotr; GRAVE, Edouard; JOULIN, Armand;
MIKOLOV, Tomas. Enriching Word Vectors with Subword Information.
Transactions of the Associationfor Computational Linguistics.
2017, vol. 5, pp. 135-146. ISSN 2307-387X. Available from DOI:
10.1162/tacl_a_00051.
5. D U , M i n ; LI, Feifei; Z H E N G , Guineng; SRIKUMAR, Vivek.
DeepLog: Anomaly Detection and Diagnosis from System
Logs through Deep Learning. In: Proceedings of the 2017 ACM
SIGSAC Conference on Computer and Communications Security.
Dallas, Texas, USA: Association for Computing Machinery, 2017,
pp. 1285-1298. CCS '17. ISBN 9781450349468. Available from DOI:
10.1145/3133956.3134015.
6. M E N G , Weibin; LIU, Ying; ZHU, Yichen; Z H A N G , Shenglin; PEI,
Dan; LIU, Yuqing; C H E N , Yihao; Z H A N G , Ruizhi; TAO, Shimin;
SUN, Pei; Z H O U , Rong. LogAnomaly: Unsupervised Detection
of Sequential and Quantitative Anomalies in Unstructured Logs.
55
BIBLIOGRAPHY
In: Proceedings of the Twenty-Eighth International Joint Conference on
Artificial Intelligence, IJCAI-19. International Joint Conferences on
Artificial Intelligence Organization, 2019, pp. 4739-4745. Available
from DOI: 10.24963/i j cai . 2019/658.
7. V A A R A N D I , Risto; PIHELGAS, Mauno. LogCluster - A data
clustering and pattern mining algorithm for event logs. In: 2015
11th International Conference on Network and Service Management
(CNSM). 2015, pp. 1-7. Available from DOI: 10.1109/CNSM.2015.
7367331.
8. Z H A N G , Xu; X U , Yong; LIN, Qingwei; QIAO, Bo; Z H A N G ,
Hongyu; D A N G , Yingnong; XIE, Chunyu; Y A N G , Xinsheng;
C H E N G , Qian; LI, Ze; C H E N , Junjie; H E , Xiaoting; YAO, Randolph;
L O U , Jian-Guang; CHINTALAPATI, Murali; SHEN,
Furao; Z H A N G , Dongmei. Robust log-based anomaly detection
on unstable log data. In: Proceedings of the 2019 27th ACM
Joint Meeting on European Software Engineering Conference and
Symposium on the Foundations of Software Engineering. Tallinn,
Estonia: Association for Computing Machinery, 2019, pp. 807-
817. ESEC/FSE 2019. ISBN 9781450355728. Available from DOI:
10.1145/3338906.3338931.
9. RipEX Radio modem Router [online]. R A C O M , 2026. Available
also from: https : //www . racom . eu/eng/products/m/ripex/
index. html. cit. 2026-03-27.
10. Lighttpd [online]. Lighttpd, 2026. Available also from: https :
//redmine . l i g h t t p d . net/projects/lighttpd/wiki. cit. 2026-
03-31.
11. dmesg [online]. Linux manual page, 2026. Available also from:
https : //man7 . org/linux/man-pages/manl/dmesg. 1. html. cit.
2026-03-31.
12. rngd [online]. Linux manual page, 2026. Available also from:
https : //linux. die .net/man/8/rngd. cit. 2026-03-31.
13. Overfitting vs. underfitting: Finding the balance [online]. IBM, 2026.
Available also from: https : / /www . ibm . com/think/topics /
overfitting-vs-underfitting. cit. 2026-04-08.
56
BIBLIOGRAPHY
14. HE, Pinjia; ZHU, Jieming; ZHENG, Zibin; LYU, Michael R. Drain:
A n Online Log Parsing Approach with Fixed Depth Tree. In: 2017
IEEE International Conference on Web Services (ICWS). 2017, pp. 33-
40. Available from DOI: 10.1109/ICWS. 2017.13.
15. Year 2038 problem [online]. Wikipedia, 2026. Available also from:
https : / / en . wikipedia . org / wiki / Year _ 2038 _ problem, cit.
2026-05-13.
57
A Electronic Attachments
The electronic attachment contains three folders, README. md, and a
LICENSE file. The folders names and contents are:
• Text - Folder with the text source code.
• Notebooks - Jupyter notebooks showcasing the training and
evaluation code for this thesis.
• Infrastructure - Contains my work on the internal infrastructure.
58