MASARYK U N I V E R S I T Y FACULTY OF INFORMATICS Evaluation of Multiple Clustering Algorithms on Insurance Datasets Bachelor's Thesis KRYŠTOF SUCHÁNEK Brno, Spring 2023 MASARYK U N I V E R S I T Y FACULTY OF INFORMATICS Evaluation of Multiple Clustering Algorithms on Insurance Datasets Bachelor's Thesis KRYŠTOF SUCHÁNEK Advisor: Bruno Rossi, PhD Department of Computer Systems and Communications Brno, Spring 2023 Declaration Hereby I declare that this paper is my original authorial work, which I have worked out on my own. A l l sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source. Kryštof Suchánek Advisor: Bruno Rossi, PhD iii Acknowledgements I would like to thank my advisor Bruno Rossi, Ph.D. for his guidance, support, swift and insightful responses to all of my questions in particular. I would like to thank the company Adacta for the support and opportunity to work on this topic, especially my colleague and consultant Mgr. Marcel Mraz. iv Abstract Clustering is a fundamental data analysis technique focused on uncovering meaningful groupings within datasets. It has found an application in providing valuable insights within the insurance domain. This thesis aims to survey, analyse, and compare clustering algorithms on a publicly available insurance dataset using free, open-source data analytics tools. This thesis provides a theoretical background for common clustering algorithms and clustering evaluation methods. Describes a process of data preparation and describes the data used. A n essential part of the thesis is an experiment on the data prepared using a selection of the algorithms described (K-Means++, Gaussian Mixture Models, Agglomerative clustering). Agglomerative clustering did not prove to be a suitable algorithm for large datasets, while the K-Means++, Gaussian Mixture Models delivered results highly dependent on the selection of dimensions analysed. Keywords clustering, data analysis, insurance, unsupervised learning, machine learning v Contents 1 Preface 2 2 Related works 4 3 Clustering algorithms and evaluation methods 5 3.1 Clustering algorithms 5 3.1.1 Numerical data clustering algorithms 5 3.1.2 Mixed data clustering algorithms 9 3.2 Evaluation methods 10 3.2.1 Methods relying on known ground truth . . . . 10 3.2.2 Methods not relying on known ground truth . 11 4 The Dataset 12 4.1 Dataset terminlogy 12 4.2 Dataset selection 12 4.3 Dataset origin 13 4.4 Dataset description 14 4.5 Data preparation 16 4.5.1 Converting the data 16 4.5.2 Data cleansing 16 4.5.3 Transforming and extending the dataset 17 4.5.4 Outlier detection 18 4.5.5 Subsampling 18 4.6 Data overview 19 4.6.1 Insurer gender 19 4.6.2 Driver age 20 4.6.3 Vehicle production year 20 4.6.4 Sum insured average 21 4.6.5 Total exposure 23 4.6.6 Claim amount 24 5 The experiment 26 5.1 Experiment overview 26 5.1.1 Technology 26 5.1.2 Dimension selection 27 v i 5.1.3 Algorithm selection 28 5.1.4 Hyperparameter selection 29 5.1.5 Evaluation methods 30 5.1.6 Measurement methodology 30 5.2 Results 31 5.2.1 Sum insured, total premium and claim amount sum 32 5.2.2 Total exposure, sum insured and premium . . . 38 5.3 Discussion 44 6 Conclusion 46 Bibliography 47 A Terminology and definitions 54 A . l Insurance terminology 54 A.2 Datatypes 54 A.3 Data analytics 55 v i i List of Tables 4.1 Dataset description 14 4.2 Dataset extension description 17 5.1 Sum insured, total premium and claim amount sum Rand Index matrix on 50 000 record sample 35 5.2 Sum insured, total premium and claim amount average execution time 36 5.3 Sum insured, total premium and claim amount average memory use 37 5.4 Total exposure, sum insured and premium Rand Index matrix on 50 000 record sample 41 5.5 Total exposure, sum insured and premium average execution time 42 5.6 Total exposure, sum insured and premium average memory use 43 viii List of Figures 1.1 Example of a possible use case. Points are weighted averages of cluster members, size scaled by to number of members 3 4.1 Number of records by insurer gender barplot 19 4.2 Number of records by driver age barplot 20 4.3 Number of records by vehicle production year barplot . . 20 4.4 Number of records by sum insured average distribution plot 21 4.5 Sum insured average boxplot 21 4.6 Sum insured average inverse gamma distribution 22 4.7 Number of records by total exposure distribution plot . . 23 4.8 Total exposure boxplot 23 4.9 Number of records by claim amount distribution plot . . 24 4.10 Claim amount boxplot 24 4.11 Claim amount fitted Power Log-Normal distribution. . . . 25 4.12 Claim amount by claim type boxplot 25 5.1 Example of clustering by individual categories 28 5.2 Clusteval [45] - using KMeans and silhouette score. (Red line at two represents the estimated number of clusters.) . 29 5.3 Elbow method based on Silhouette score for KMeans clustering 32 5.4 Silhouette coefficient graph 33 5.5 Davies-Bouldin score comparison. (Lower is better.) . . . 34 5.6 Calinski-Harabasz score comparison. (Higher is better.) . 34 5.7 Sum insured, total premium and claim amount sum average Rand Index score for various sample sizes 35 5.8 Execution time comparison on sum insured, total premium and claim amount sum dimensions, (logarithmic scale) 36 5.9 Memory usage comparison on sum insured, total premium and claim amount sum dimensions, (logarithmic scale) 37 5.10 Elbow method based on Silhouette score for KMeans clustering 38 5.11 Silhouette coefficient graph 39 ix 5.12 Davies-Bouldin score comparison. (Lower is better.) . . . 40 5.13 Calinski-Harabasz score comparison. (Higher is better.) . 40 5.14 Total exposure, sum insured and premium average Rand Index score for various sample sizes 41 5.15 Execution time comparison on total exposure, sum insured and premium dimensions, (logarithmic scale) . . . 42 5.16 Memory usage comparison on total exposure, sum insured and premium dimensions, (logarithmic scale) . . . 43 5.17 Anisotropical clusters G M M , K-Means++ 45 x Introduction In today's vast and still rapidly developing data economy, there is an increasing need for data mining across industries as more growth is projected to come [1] [2]. Insurance, being traditionally a data-intensive domain, is no exception. The opportunities are immense for all parts of an insurer's business, including but not limited to consumer engagement, product innovation, and claims management [3]. Therefore most leading P & C (property and casualty) insurers in Europe and North America are heavily investing in data analytics [4]. The goals of this thesis are to explore, overview, and evaluate commonly used clustering techniques on publicly available data sets based on real-world insurance data. The thesis also includes preparation and analysis of the data since clustering algorithms are often sensitive to incorrectly recorded or missing data and outliers commonly present in real-world data [5]. This thesis is divided into six chapters. Chapter 3 covers clustering algorithms and clustering performance evaluation methods. In Chapter 4,1 detail dataset processing steps and its characteristics. In Chapter 5 the experiment is described, conducted, and evaluated. There's also an Appendix containing definitions for the terminology used. 1 1 Preface Clustering is a broadly applicable approach nowadays in use in a broad set of domains [6]. This is in large part because of the relative simplicity of implementation and the wide availability of existing implementations for a wide range of programming languages [6]. Within the domain of insurance, the application can be divided into two broad categories, these are market segmentation and actuarial science where simple clustering approaches can be a more practical alternative to more traditional methods [7, section 3.5.2]. The applications in this thesis are focusing on market segmentation. And while the dataset used for evaluation focused on policy-level data, the use cases go well beyond applications in the consumer-facing side of business such as identifying groups of policies and customers Figure 1.1. Examples of these applications lie within the area of corporate governance providing insights into metrics such as the performance of insurance agents. The general guiding principle behind the selection of used methods is the "simplicity-first" approach recommended by the literature for analyzing real-world datasets. As it is suggested that even common techniques can yield good results [8, chapter 4]. 2 i . PREFACE Vehicle production year 2010 o Figure 1.1: Example of a possible use case. Points are weighted averages of cluster members, size scaled by to number of members. 3 2 Related works There is an ample amount of research done on the application of clustering techniques to insurance data and other financial data that has many of the same characteristics [9, Chapter Reader Guidelines]. Yet the vast majority of these are focused on the area of outlier and fraud detection and are relying on labelled data [10] [11]. I have found much fewer publications focused on market segmentation within the insurance area. Often presenting the possibility of clustering the data and showing an example of an application yet not providing much, if any, comparison of various techniques [12] [13]. Wuthrich and Buser did show a host of methods yet did not show any comprehensive comparison and did so on a synthetic dataset [14]. Recently Jamotton et al. have compared multiple methods and even demonstrated improvements, but all of these were based on the popular K-Means method [15]. 4 3 Clustering algorithms and evaluation methods In this chapter, I provide definitions and a brief overview of clustering algorithms and evaluation methods. 3.1 Clustering algorithms This section contains a brief overview of clustering techniques. While the subsection 3.1.1 on numerical clustering methods provides examples of commonly used clustering methods. The subsection 3.1.2 on mixed data clustering explains the possible approaches to handling mixed data. The terminology used is explained in the Appendix A . 3.1.1 Numerical data clustering algorithms The algorithms mentioned in this section were picked as commonly used clustering methods using widely differing approaches. This section aims to provide a high-level overview rather than an exhaustive description. K-Means++ algorithm K-Means algorithm (also referred to as Lloyd's algorithm) is one of the oldest and most straightforward clustering approaches, being representative of the group of combinatorial cluster analysis algorithms. This algorithm is simple, scalable, and usable for general-purpose applications. The drawbacks include the inherent assumption that the clusters are convex and isotropic. Furthermore, it is not deterministic, performs poorly in very high-dimensional space, and the number of clusters must be chosen beforehand. It can also perform poorly on data clusters of varying density, and outliers can disproportionally affect the results [16]. 5 3. CLUSTERING ALGORITHMS A N D EVALUATION METHODS The algorithm is performed by choosing initial centroids followed by looping between assigning points to the closest centroid and assigning a mean value of previously assigned points to the centroid until convergence (the algorithm will always converge on a finite set of objects). The original version of the K-Means algorithm has randomly initialised centroids causing results and performance to be highly dependent on the initialisation. K-Means++ initialisation largely alleviates these issues [17], as its scheme randomly picks only the first point and then proceeds to pick all other centres based on probability distribution [18, section 2.3.2]. It is to be noted that there are many other versions and adaptations of this algorithm, including batch variant, incremental variant, and Spherical K-Means Algorithm, among others. However, these are beyond the scope of this thesis [19, section 3.1.5]. EM algorithm for Gaussian Mixture Models The E M (Expectation-Maximisation) algorithm used for fitting G M M (Gaussian Mixture Models) belongs to the class of hill-climbing algo- rithms. Despite vastly different approaches, there are experiments showing that on some datasets, the results are comparable in terms of quality to the results of the K-Means algorithm. These two algorithms share other properties, too, such as relying on initialisation and general unsuitability for high-dimensional data [19, section 3.2]. Another major drawback is that the E M method is not scalable [18, section 2.3.1]. The algorithm relies on repeating a sequence of two steps. The expectation step in which are computed expected values of unknown objects from known values and parameters of models are estimated. Second is the maximisation step, in which the parameters of a model are estimated from both known and expected values using probability maximisation. These two steps are repeated until convergence criteria are met or the maximum number of iterations has been reached [19, section 3.2] [ 18, section 2.1.1]. There have been attempts to remedy some of the issues resulting in a number of modifications. Examples of these include greedy E M and stochastic E M [19, section 3.2]. 6 3. CLUSTERING ALGORITHMS A N D EVALUATION METHODS DBSCAN The DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is one of the popular density-based methods. DBS C A N clusters data points by areas of high density separated by lowdensity areas, where the dense areas are defined by the minimum number of samples and the eps parameter [18, section 2.3.7]. Here lies perhaps the most significant limitation of using the D B S C A N algorithm - selecting the minimum number of samples and eps pa- rameters. The other major limitation is that the clusters ought to have similar densities, and reliance upon the definition of distance proves troublesome again, especially with highly-dimensional data [19, section 2.4]. Regardless, the D B S C A N algorithm is robust towards noise and can discover arbitrarily shaped clusters and can be parallelised [20, table 2] The algorithm is performed as follows. First, the core points are identified. Then the neighbouring core points are joined into clusters; then each non-core point is assigned to a core point; otherwise, it is considered noise [21, algorithm 2]. Just as with the beforementioned methods, there are modifications to this algorithm created to improve upon some of its shortcomings. Relevant examples of such are G D B S C A N (Generalized DBSCAN) and L D B S C A N (Local DBSCAN) [19, section 2.4]. Affinity propagation Affinity propagation is an example from a class of relational algorithms. Affinity propagation algorithm selects some of the data points as prototypes, and the other values are assigned to one of the prototypes based on a similarity measure (commonly Euclidean distance) with the goal of maximising similarity where the prototypes are representatives of their individual clusters. Such approach is quite insensitive to outliers, and the number of clusters does not need to be set in advance. However, the algorithm is sensitive to other parameters and has high time complexity. 7 3. CLUSTERING ALGORITHMS A N D EVALUATION METHODS The main part of the algorithm disregarding initiation is a while loop that runs until the termination condition is reached. Where messages informing about availability (object informing another object that it's ready to become its prototype) and responsibility (object informing another object that it became its prototype) are iteratively sent based on the criteria [22] [19, section 3.4]. As the main drawback of this approach is time complexity, improved versions aim mostly at improving the performance example of such is fast sparse affinity propagation (FSAP) [23]. Agglomerative cluster analysis The Agglomerative cluster analysis belongs to the broader category of hierarchical clustering methods. It starts with treating every single record as a separate cluster and then combines similar clusters until a criterion is met. This can be a specified number of clusters or a linkage distance threshold. Hierarchical methods, in general, have the advantage of determinism and flexibility. Their known shortcomings include the limitation of the possibility of reassigning clusters for records as the algorithm progresses and results being highly reflective of the structure of the algorithm selected [24, section 4.3] [19, section 2.3]. There are many methods of calculating distances (Single linkage method, Complete linkage method, Ward method,...) that significantly impact the characteristics of results. The algorithm itself consists of the following steps—computation of the similarity matrix between all records. And iterative combining of similar clusters and updating the similarity matrix until the criterion is reached [24, section 4.3] [19, section 2.3]. The hierarchical approaches have another major limitation, that is 0(n2 ) space and O(n2 log(n)) time complexity for optimized solutions. For this reason, various methods have been developed combining agglomerative hierarchical clustering and partitional clustering to reduce the overall computational cost [25]. 8 3. CLUSTERING ALGORITHMS A N D EVALUATION METHODS 3.1.2 Mixed data clustering algorithms Mixed data clustering algorithms are often, in principle, generalized versions of numerical algorithms employing one of the various techniques mentioned below [26]. Redefining distance One of the possibilities is to not rely on Euclidean distance and use different distance metrics instead. A n example of such is K-Prototypes that uses Manhattan distance [26]. Converting the data Some proposed methods convert mixed data into purely numerical data, while others convert it into purely categorical data [26]. Other commonly used methods falling into this category are dimensionality reduction techniques such as Principal Component Analysis (PCA) [27]. Other There are many advanced methods that do not fall into the categories mentioned above [26]. However, these are due to their complexity beyond the scope of this thesis. 9 3. CLUSTERING ALGORITHMS A N D EVALUATION METHODS 3.2 Evaluation methods This section contains an overview of some of the common partition quality indexes divided into two categories by the necessity of knowledge about ground truth, a known clustering assignment. 3.2.1 Methods relying on known ground truth In this subsection, I present quality indexes requiring knowledge about ground truth. These methods are, in essence, a measure of the similarity between two cluster assignments. To provide insight into the quality of partition, one of them must be a partition that we assume to be correct. The following variables defined in this paragraph are used throughout the following subsections. N n (true positives) is the number of points clustered together in both clusterings and NIQ (false positives) is the number of points belonging to the same cluster in the first but not in the second set, NQI (false negatives) signifies the opposite situation and NQQ (true negatives) is the number of points clustered separately. Rand Index Rand Index can be understood as a probability that the points are clustered similarly in both cluster assignments. Calculated as ^"^i)/2 where n is the number of samples [28, section 3]. Fowlkes-Mallows index Fowlkes-Mallows index is calculated as y N ^ N w • N ^ + N m • Higher score is better [29, section 4.3.]. V-measure V-measure is based on homogeneity criteria h and completeness criteria c described in the original paper in greater detail. V-measure is computed as ^^j^ where /3 parameter specifies the weight of the respective scores in the final score. Values are between 0 and 1, where higher values are better [28]. 10 3. CLUSTERING ALGORITHMS A N D EVALUATION METHODS 3.2.2 Methods not relying on known ground truth The methods in this subsection require no knowledge about the ground truth. Their results are only based on the characteristics of the partitioned dataset without any other inputs. Davies-Bouldin score The Davies-Bouldin (DB) score is based on measuring compactness and separation. The score is calculated as follows \ Y$=i Ri where k is the number of clusters and Ri = maxRij, i 7^ /, Ru = (Si + Sj)/Mij, Si is the sum of average distances form each point in cluster i to the centroid of cluster i and M y is the distance between centres of clusters i and j. Lower score is better [30, section 2]. Silhouette score The Silhouette method is also based on compactness and separation, but unlike DB score, the distances are calculated for individual points \ Ef=o s Xi where n is the number of points and sXi is the score of a given point based on the difference between that point and every other point in its cluster and the average distance of the point and points in all other clusters divided by normalizing term. The best possible score is 1 [30, section 2]. Calinski-Harabasz score The Calinski-Harabasz (CH) score is based on "between cluster scatter matrix" (BCSM) and a "within-cluster scatter matrix" (WCSM). As foll ° w s ^raceiwcSM ' F^f w n e r e trace(BCSM) is the sum of the distances between each cluster and the dataset's centre, weighted by the size of the clusters. And trace(WCSM) is the sum of squares of distances between the centres of each cluster and each of its points. Higher score is better [30, section 2]. 11 4 The Dataset In this chapter, I will cover the reasoning behind the selection of this particular dataset, its origin and an overview of its contents. As well as the processing steps to make it suitable for my application. 4.1 Dataset terminlogy This section provides an explanation for the unusual meaning of some terms within the context of this dataset. The insurance terminology used is explained in section A . l of the Appendix. Gender In this dataset, gender refers to a value corresponding to a policyholder's sex or stating that the policyholder is a corporate entity. Driver age In this dataset, driver age refers to a range of policyholder's age. In the case of a corporate entity, this value is not defined. 4.2 Dataset selection First and above all, I was looking for a reasonably recent (roughly from 2000 onwards) real-world insurance dataset with both a large number of records and dimensions, with a license that is not overly restrictive for my use case—preferably a well-documented one. 12 4. T H E DATASET 4.3 Dataset origin The original dataset is a brvehinsl dataframe from the collection of CASdatasets: Insurance datasets [31]. This dataframe was compiled from the data available in the AUTOSEG (Sistema de Estatisticas de Automoveis, Statistical System for Automobiles), a motor insurance database created by Susep (Superintendencia de Seguros Privados, Superintendence of Private Insurance), Brazilian insurance regulator. This database contains aggregated data from all insurance companies. Records contain policy information for the year 2011 [7, section 5.5]. While newer data is available on the AUTOSEG website [32], I decided not to rely on the existing aggregation mentioned above. However, doing so might be an intriguing option for any reader needing a larger or more recent dataset, possibly providing insight into changes between different years. 13 4- T H E DATASET 4.4 Dataset description The dataset contains 1 965 355 records with dimensions represented by columns described in the table below. Table 4.1: Dataset description Column name Description Dimension type Gender Gender represented by a Categorical string. (Male, Female, Corpo- rate) DrivAge Driver age group, (e.g. 36- Ordinal 45) VehYear Manufacture year of the ve- Discrete hicle. numeric VehModel Vehicle model, (e.g. G m - Categorical Chevrolet - Kadett G l 2.0 Categorical Mpfi / Efi) VehGroup Vehicle group, (e.g. G m Categorical Chevrolet Kadett) Area Local area name. (e.g. Inte- Categorical rior) State State name. (e.g. Rio de Categorical Janeiro) StateAb Abbreviated state name. (e.g. Categorical RJ) Categorical ExposTotal Total exposure. Continuous numeric ExposFireRob Total exposure for fire and Continuous robbery guarantees. numeric PremTotal Total premium. Continuous numeric PremFireRob Total premium for fire and Continuous robbery guarantees. numeric SumlnsAvg Average of the sum insured. Continuous numeric 14 4. T H E DATASET ClaimNbRob Number of robbery claims Discrete during the exposure period. numeric ClaimNbPartColl Number of partial collision Discrete claims during the exposure numeric period. ClaimNbTotColl Number of total collision Discrete claims during the exposure numeric period. ClaimNbFire Number of fire claims during Discrete the exposure period. numeric ClaimNbOther Number of other claims dur- Discrete ing the exposure period. numeric ClaimAmountRob Claim amount during the ex- Continuous posure period for robbery numeric claims. ClaimAmountPartColl Claim amount during the ex- Continuous posure period for partial col- numeric lision guarantees. ClaimAmountTotColl Claim amount during the ex- Continuous posure period for total colli- numeric sion guarantees. ClaimAmountFire Claim amount during the ex- Continuous posure period for fire guaran- numeric tees. ClaimAmountOther Claim amount during the ex- Continuous posure period for other guar- numeric antees. This description is based on the CAS datasets manual [31], do note that VehModel and VehGroup columns are labelled differently in the manual (as FullVehCode and VehCode, respectively). 15 4. T H E DATASET 4.5 Data preparation Real-world data usually brings many challenges, often being incomplete, noisy and inconsistent. This dataset is no exception. Therefore data must be prepared. This process is usually quite demanding in terms of engineering effort. It is generally estimated that the effort in this area is somewhere in the range of 50 % to up to 80 % of total project time [33]. All of these issues can be mitigated in various ways depending on the nature of the data and the use case. For incomplete and inconsistent records, values can be interpolated based on other records, or these records can be filtered out. I decided to simply filter out as the most straightforward option [34] [8, chapter 6]. I dealt with this process using the pipeline described below. 4.5.1 Converting the data As the original dataset comes as an R D A file, I converted it into a widely supported and platform-independent format. For this, I chose CSV the conversion was done by an R code. 4.5.2 Data cleansing For the rest of the processing, I used Python, heavily relying on Pandas [35] version 2.0.0. In this stage, I filtered out incomplete and inconsistent data, also with records that were outside the studied range, also with records that included too old vehicles. Here as incomplete, I consider not only records where some column is missing but also data where there is no meaning from the business standpoint, such as records with no or extremely small premiums, no exposure to any of the risks, no sum insured etc. In this step, 473 143 records were removed, leaving 1 492 212 to be further processed. 16 4- T H E DATASET 4.5.3 Transforming and extending the dataset Based on the data, I decided to create new dimensions and transform them to speed up processing and simplify further process. These are described in the table below. Table 4.2: Dataset extension description Column name Description Dimension type claimSum Sum of claim amounts for all Continuous types of claims. numeric claimNbSum Sum of the number of claims Discrete for all types of claims. numeric Location Unique string identifier for Categorical location, a combination of Categorical area and state, (e.g. Goias Sudeste de Goias) DrivAge Here, compared to the original column, missing values were converted to Ordinal VehicleManufacturer Vehicle manufacturer, (e. g. Categorical Chevrolet) Categorical The vehicle manufacturer info is obtained from VehGroup column based on regular expression matching from a list of known manufacturers. This list does not cover every manufacturer, and records with unknown manufacturers are filtered out. This is by design because, in a sense, these (577) records are outliers. 17 4. T H E DATASET 4.5.4 Outlier detection Practical data mining can be significantly hampered by data quality. One of the possible ways to mitigate this issue is outlier detection, as it is a suitable method for cases where it is impractical to have all records checked by a person such it this one due to the dataset size [8, section 7.5]. For performing outlier detection, I've chosen the k-Nearest Neighbours algorithm (k-NN) from the Pyod library [36] version 1.0.9 using default parameters (5 nearest neighbours, using the distance to the kth neighbour as the outlier score). While simple, there's evidence that k - N N performs reasonably well on auto insurance data [37]. This method is frequently used within the domain of financial fraud detection [10]. Even though there are many newer and more advanced anomaly detection techniques [38], this is not a focus of my thesis. The outlier detection was applied to all numerical dimensions even though this inherently does not leverage all available information; numerical dimensions are numerous, and encoding other dimensions, particularly categorical, would present a challenging task on its own [39]. After performing outlier detection, there are 1 378 412 records left in the dataset used for the rest of the analysis. 4.5.5 Subsampling To simplify and reduce the overhead of the experiment, I created various subsamples limited both in the number of records and the dimensions available. 18 4- T H E DATASET 4.6 Data overview This section gives the reader a basic idea about the most important dimensions of the processed dataset. I have chosen graph types individually for each dimension after experimenting with multiple types to be as informative as possible rather than using the same type for all dimensions, as their characteristics vary significantly Dimensions are ordered as they appear in the description table above. The distributions were fitted using distfit [40] version 1.6.10. 4.6.1 Insurer gender Male and female policyholders are comparably represented, while there are relatively fewer corporate policyholders, as apparent in Figure 4.1. Figure 4.1: Number of records by insurer gender barplot 19 4. T H E DATASET 4.6.2 Driver age We can see a similar story with driver age groups; the only significantly under-represented group are policyholders aged 18 to 25. Figure 4.2: Number of records by driver age barplot 4.6.3 Vehicle production year The dataset is biased towards newer vehicles. Number of records by vehicle production year CTlClOlOlOlOlOlOlOlOlOlOlCTlOlOl Vehicle production year Figure 4.3: Number of records by vehicle production year barplot 20 4. T H E DATASET 4.6.4 Sum insured average Sum insured average roughly follows distributions from normal and gamma families expected in insurance datasets Figures 4.6 [41]. A l though it is to be noted that there are irregularities (notice the local maximums around 17 500 and 25 000), considering the dataset size, these could be caused by multimodality stemming from various representations of demographic groups. Figure 4.4: Number of records by sum insured average distribution plot Sum insured average boxplot 200000 Sum insured average Figure 4.5: Sum insured average boxplot 21 4- T H E DATASET Inverse gamma 0 100000 200000 300000 400000 Sum insured average Figure 4.6: Sum insured average inverse gamma distribution. Inverse gamma distribution in Figure 4.6 uses the following parameters a = 4.33682, loc = -10574.19064, scale = 163361.99589. 22 4- T H E DATASET 4.6.5 Total exposure The rather odd shape of the total exposure distribution plot in Figure 4.7 seems to suggest the discretization of some of the values at 0.5 intervals. Z 150000 E = 100000 Number of records by total exposure (limited to 5) 2 3 Total exposure Figure 4.7: Number of records by total exposure distribution plot Total exposure boxplot 50 100 150 200 250 300 Total exposure Figure 4.8: Total exposure boxplot 23 4- T H E DATASET 4.6.6 Claim amount Total claim amount follows distribution generally expected from insurance data Figure 4.11 [41]. The reader should keep in mind that the dataset is much smaller in this case as only 177 845 out of 1 378 412 records have any claim. ° 2000 Number of records by claim amount for all claims (limited to 10 000) 'iTTfTTTTTTrrm-m - 4000 6000 Claim amount for all claims Figure 4.9: Number of records by claim amount distribution plot Claim amount for all claims boxplot 20000 30000 Claim amount for all claims Figure 4.10: Claim amount boxplot 24 4- T H E DATASET Power Log-Normal • Emperical PDF • Powerlognorm (t • CM low (0.05] i Cll high (0.05) 20000 30000 40000 Claim amount for all claims Figure 4.11: Claim amount fitted Power Log-Normal distribution. The estimated distribution in Figure 4.11 is parametrized as follows c = -0.14186, s = -0.90560, loc = -2.98688, scale = -71.83359. It is to be noted that the distribution of claim amounts varies significantly by claim type. Claim amount by claim type Claim amount for robbery claims- | o Claim amount for partial collision claims- |-j | j - » • »• t * *4 o Claim amount for total collision claims- |- aj -Q £ Claim amount for fire claimsClaim amount for other claims >«•• +*« tt • • • • 20000 30000 40000 Claim amount Figure 4,12: Claim amount by claim type boxplot 25 5 The experiment This chapter covers the experiment environment, methodology and the reasoning behind its design decisions. 5.1 Experiment overview 5.1.1 Technology This section describes the environment in which the measurements were run. Software The performance was measured on Python 3.10.4 interpreter. The source for the implementation of clustering algorithms was scikitlearn [42] version 1.2.2. Hardware A l l experiments were conducted on a Windows 11 PC with A M D Ryzen 1600AF overclocked to 3.6 G H z and 32 GB of DDR4 R A M running at 3200 M H z . While this setup is far from the cutting-edge technology of 2023, its performance is very stable. The cooling solution is more than adequate, assuring there is no reduction in performance even on long-running workloads. 26 5. T H E EXPERIMENT 5.1.2 Dimension selection The dimensions are selected in a way that could provide valuable real-world insights. These are: - Sum insured, total premium and claim amount sum. - Total exposure, sum insured and premium. The first set of dimensions represents a real-world example, something that would be an interesting insight for almost any insurance company. However, it also has a major limitation - the set of records with claims is only a relatively small subset of all records (roughly 13%). For this reason, I have a second set of dimensions that are better suited for performance benchmarks, as all of these dimensions are in all of the records. A reader might ask themselves why no categorical values were selected. The main reason for this is that such results are, in my experience, harder to present and interpret; thus, the practical value is not as significant. When conducting practical experiments leveraging categorical data by simply dividing the datasetby a categorical feature, clustering such (purely numerical) partitions separately and subsequently combining the set of planes into a three-dimensional graph provided valuable insights with little extra complexity Figure 5.1. 27 5. T H E EXPERIMENT Vehicle manufacturer • Chevrolet • Hat • Ford • Renault Volkswagen Figure 5.1: Example of clustering by individual categories. 5.1.3 Algorithm selection The selected algorithms are K-Means++ (subsection 3.1.1), G M M (subsection 3.1.1) and Agglomerative clustering (subsection 3.1.1). The reason for this is that all of these algorithms can be effectively used with only one hyperparameter manually selected, leaving other values at their defaults. Better yet, this parameter is the same for all of these - the number of clusters. When I originally conceived this thesis, I intended to include D B S C A N in the comparison too. However, the results when experimenting were abysmal; even using small samples of the dataset with handpicked parameters, trying various popular heuristics did nothing to improve the results either. This is, in my opinion, caused by the large differences in the densities of data. A l l algorithms were run on standardized data. 28 5. T H E EXPERIMENT 5.1.4 Hyperparameter selection Hyperparameter selection is not an easy task, especially within the context of unsupervised learning. For this reason, for each model, only the most important parameter (the number of clusters) is estimated. While there are many frameworks intended for heuristics and various AutoML applications, these are usually quite complicated to use [43]. Even though the results on small synthetic datasets are promising, on a large real-world dataset such as this one, the results provided no valuable insight. Often times tending to use the highest or lowest of the results Figure 5.2, non-reproducible seemingly random results even when setting constant seed [44]. Silhouette v s . nr.clusters 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 #CIListers Figure 5.2: Clusteval [45] - using KMeans and silhouette score. (Red line at two represents the estimated number of clusters.) I used the elbow method in combination with silhouette analysis [46] to estimate the number of clusters. Optimizing silhouette score using Yellowbrick [47] library methods with K-Means-r-+ model [48]. While this could constitute an advantage for the K-Means-r--r-, it is to be noted that the results are in line with other methods. 29 5. T H E EXPERIMENT 5.1.5 Evaluation methods Rather importantly, different criteria are used for the clustering performance evaluation than for the hyperparameter estimation to avoid overfitting. As there is no known ground truth about clustering assignments Davies-Bouldin score and Calinski-Harabasz score are used. To provide more insight into how similar the results are between various clustering methods, I also added a matrix of similarity using the Rand Index metric. Other measurements focus on resource usage, namely R A M usage and execution time. A l l of these results are presented in graphs comparing various data set sizes. 5.1.6 Measurement methodology A l l of the metrics obtained are from running a Python script as a subprocess in a Python script. Therefore the performance metrics include startup time and memory usage of the whole environment, not just the performance metrics of the algorithm execution itself. A l l of these tests were run ten times to minimize the effect of variations between runs. Memory usage is measured as Resident Set Size (RSS). 30 5. T H E EXPERIMENT 5.2 Results This section contains a graph representing the results of the experi- ment. Do note that agglomerative clustering wasn't run on larger subsets as its memory requirements make it completely impractical on the hardware used for running the experiment. This is due to a 0(n2 ) space complexity. Based on the results below and assuming a secondorder polynomial for the complexity, the memory necessary for clustering a full 1 378 412 record sample using this method would be around 15 Terabytes of R A M . 31 5. T H E EXPERIMENT 5.2.1 Sum insured, total premium and claim amount sum Results of experiments executed on sum insured, total premium and claim amount sum dimension are presented in this subsection. Hyperparameter estimation Six clusters were estimated using methods discussed in subsection 5.1.4. Distortion Score Elbow for KMeans Clustering «oooc 35OÜO0 R 250000 B 20000C 150000 10000C 5000C {slbow at k = 7, score = 119400.055 ;lbow at k = 7, score = 119400.055 i • - - i slbow at k = 7, score = 119400.055 ;lbow at k = 7, score = 119400.055 i • \ i i i i i \ i 1 / \\ ,A i i i i I i 1 t / > t / / » \ i > JI r Ii \ 1 \ I 1 t 1 1 1 1 \ \ I i f I 1 N 1 1 J* »--< s f t" 0.30 0.25 0.20- 0.15 0.10 2 3 4 5 6 7 6 9 10 11 12 13 14 15 16 17 16 19 k Figure 5.3: Elbow method based on Silhouette score for KMeans clus- tering. 32 5. T H E EXPERIMENT Figure 5.4 displays the Silhouette score (subsection 3.2.2) for every single individual record. These values are grouped by the assigned clusters and ordered within a group. The main property of the graph we look at is that all of the clusters have records that have a score above the red line (mean Silhouette score). While in general, the graph in Figure 5.4 might be of concern because of the uneven sizes of clusters. Based on the dataset and domain knowledge, it is no surprise that the sizes of clusters vary greatly as the distribution of all dimensions is far from uniform. 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 0.0 0.2 0.4 0.6 Figure 5.4: Silhouette coefficient graph. 33 5. T H E EXPERIMENT Clustering performance For these dimensions, K-Means++ and Agglomerative did perform significantly better in terms of the Davies-Bouldin score. We can see significant variance in the result of G M M in particular Figure 5.5. Method comparison using Davies-Bouldin score K-Means++ GMM Agglomerative n = ID ODO n = 25 000 n = 50 000 n = 100 000 n = 177 B45 Model Figure 5.5: Davies-Bouldin score comparison. (Lower is better.) In Figure 5.6, we can see similar results with the Calinski-Harabasz score, except K-Means++ has a slightly better performance than Agglomerative. Interestingly despite both scores being strongly correlated in this case, the variance is higher for K-Means-r-+. Method comparison using Calinski-Harabasz score Model Figure 5.6: Calinski-Harabasz score comparison. (Higher is better.) 34 5- T H E EXPERIMENT Cluster similarity As we can see in Table 5.1 while the clusterings detected by K-Means++ and Agglomerative clustering are quite a bit more similar to each other than to G M M . They are not nearly as similar as based scores above might be assumed at first sight. Table 5.1: Sum insured, total premium and claim amount sum Rand Index matrix on 50 000 record sample. K-Means++ G M M Agglomerative K-Means++ - 0.71 0.85 G M M 0.71 - 0.71 Agglomerative 0.85 0.71 Average Rand Index seems to be fairly consistent across various sample sizes Figure 5.7. Average rand index by sample size x 0 ' 8 £U • a c -o 0.6 c re CC cu 0)0.4 CD t t t 1\ \ \ » i i f k \ \ 1 1 1 1 * >--l i 1 \ \ \ > t l i 1 l 1 I * * r 1 f f ( 3.5 3.0 2.5 on c s a! 2.0 £. B 1.5 1.0 0.5 2 3 4 5 6 7 S 9 10 11 12 13 14 15 16 17 18 19 < Figure 5.10: Elbow method based on Silhouette score for KMeans clustering. 38 5. T H E EXPERIMENT Figure 5.11 is very similar to Figure 5.4 in terms of its interpretation. It does not seem to suggest incorrectly selected number of clusters. Silhouette for 5 iB g Silhouette for 6 ie@ Silhouette for 7 0.0 0.2 0.4 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.0 0.2 0.4 Figure 5.11: Silhouette coefficient graph. 39 5. T H E EXPERIMENT Clustering performance In Figure 5.12 the G M M fared a lot better, outperforming the other two methods in all cases. Compared to the other set of dimensions, a much lower variance in run-to-run scores does seem to suggest more stable clusters. Method comparison using Davies-Bouldin score Model Figure 5.12: Davies-Bouldin score comparison. (Lower is better.) Calinski-Harabasz score confirms the results as it is once again very clear that G M M performed much better than the other methods on this dataset Figure 5.13. Method comparison using Calinski-Harabasz score n - 10 ODO n - 25 ODO n - 50 000 n - 100 000 n - 250 000 n - 500 000 n - 1 378 412 Model Figure 5.13: Calinski-Harabasz score comparison. (Higher is better.) 40 5- T H E EXPERIMENT Cluster similarity For this set of dimensions, observations are similar to the first set of dimensions. Clustering results of K-Means++ and Agglomerative clustering are quite similar while being much more dissimilar to G M M Table 5.4. Table 5.4: Total exposure, sum insured and premium Rand Index matrix on 50 000 record sample. K-Means++ G M M Agglomerative K-Means++ - 0.66 0.88 G M M 0.66 - 0.67 Agglomerative 0.88 0.67 As in the first case average Rand Index is not significantly impacted by sample size Figure 5.14. Average rand index by sample size x 0 ' 8 £U • a c -o 0.6 c ro cc cu 0)0.4 CD R where the X is a set of given objects [19, section 2.2]. Cluster Clusters in this context are defined as homogenous sets by a criterion depending on a clustering algorithm. Any two objects belonging to one cluster are mutually more similar than any two objects picked from different clusters. Formal requirements for clusters are as follows: • Each cluster must contain at least one object. • Each object must belong to a specific cluster. • Each object must belong to exactly one cluster. [19, chapter 2] Hyperparameter Parameter that cannot be directly estimated from data [52]. AutoML Tools that automatically choose learning algorithm and set its hyperparameters [53, chapter 1]. 55 A. TERMINOLOGY A N D DEFINITIONS Natural cluster Natural clusters are mutually exclusive subsets with members that are sufficiently related to one another and sufficiently unrelated to nonmembers allowing for accurate generalization or prediction about the attribute values of cluster members [54]. Cluster analysis Cluster analysis, also referred to as clustering, is the process of dividing a set of data into clusters. As no class labels are present in training data, it falls into the category of unsupervised learning methods. These clusters can be interpreted as an implicit class. Cluster analysis is within this context, sometimes called automatic classification [8, section 10.1]. Machine learning Machine learning explores the possibilities of computers learning from data. The focus is commonly on the recognition of complex patterns [50, section 1.5.2]. Supervised learning Supervised learning is within the domain, to a great extent, synonymous with classification. The learning is done on a dataset containing labelled examples. These labels are used for the supervision of the learning model [50, section 1.5.2] [19, chapter 2]. Unsupervised learning Unsupervised learning, also known as learning without a teacher, is a type of machine learning where the input data is not labelled. Generally speaking, it is equivalent to cluster analysis. Unsupervised learning is typically used to identify underlying patterns in the data. However, it cannot provide any explanation of the semantic meaning of the clusters [50, section 1.5.2]. 56 A. TERMINOLOGY A N D DEFINITIONS Semi-supervised learning Semi-supervised learning combines both machine learning techniques combining, labelled and examples, when creating a model [50, section 1.5.2]. Active learning Active learning is a machine learning method in which users play an active role. In this paradigm, a learning machine can request a label for data from an expert user [50, section 1.5.2]. 57