Under review as a conference paper at ICLR 2022 SELF-SUPERVISE, REFINE, REPEAT: IMPROVING UN-SUPERVISED ANOMALY DETECTION

INTRODUCTION Anomaly detection (AD), the task of distinguishing anomalies from normal data, plays a crucial role in many real-world applications such as detecting faulty products using visual sensors in manufacturing, fraudulent behaviors at credit card transactions, or adversarial outcomes at intensive care units. AD has been considered under various settings based on the availability of negative (normal) and positive (anomalous) data and their labels at training, as overviewed in Sec. 2. Each application scenario is dominated by different challenges. When entire positive and negative data are available along with their labels (Fig.  1a ), the problem can be treated as supervised classification and the dominant challenge becomes the imbalance in label distributions  [4, 16, 20, 27, 32, 35] . When only negative labeled data are available (Fig.  1b ), the problem is 'one-class classification'  [23, 26, 33, 45, 47, 51, 53] . Various works have also extended approaches designed for these to settings with additional unlabeled data (Fig.  1c,d,e )  [10, 18, 24, 46, 58]  in a semi-supervised setting. While there exist many prior works in these settings, they all depend on some labeled data, which is not desirable in all application scenarios. Unsupervised AD, on the other hand, poses unique challenges in the absence of any labeled data information, and a straightforward adaption of methods developed with the assumption of labeled data would be suboptimal. For example, some recent studies  [7, 60]  have applied one-class classifiers (OCCs) that are known to yield impressive performance when trained on negative samples  [7, 23, 26, 33, 51]  to unsupervised AD, but their performance for unsupervised AD has been quite sub-optimal. Fig.  2  illustrates this, showing the unsupervised AD performance of state-of-the-art Deep OCCs  [51]  with different anomaly ratios in unlabeled training data -the average precision significantly drops even when a small portion (2%) of training data is contaminated with anomalies. Our framework SRR (Self-supervise, Refine, Repeat), overviewed in Fig.  3 , brings a novel approach to unsupervised AD with the principles of self-supervised learning without labels and iterative data refinement based on the agreement of OCC outputs. We propose to improve the state-of-the-art performance of OCCs, e.g.  [33, 51] , by refining the unlabeled training data so as to address the fundamental challenges elaborated above. SRR iteratively trains deep representations using refined Figure  1 : AD problem settings. Blue and red dots are for labeled negative (normal) and positive (anomalous) samples, respectively. Grey dots denote unlabeled samples. While previous works mostly focus on supervised (a, b) or semi-supervised (c, d, e) settings, we tackle an AD problem using only unlabeled data (f) that may contain both negative and positive samples. data while improving the refinement of unlabeled data by excluding potentially-positive (anomalous) samples. For the data refinement process, we employ an ensemble of OCCs, each of which is trained on a disjoint subset of unlabeled training data. The samples are declared as normal if there is a consensus between all the OCCs. The refined training data are used to train the final OCC to generate the anomaly scores in the unsupervised setting. Most prior unsupervised AD works  [23, 26, 33, 51]  assume that the data contains entirely negative samples, which makes them not truly unsupervised as they require having humans to do the data filtering. Similar to ours, some prior unsupervised AD works  [7, 45, 60]  considered evaluating on an unsupervised setting where there exist a small percentage of anomalous samples in the training data, i.e. operating in 'truly' unsupervised setting without having the need for humans to do any filtering in the training data. However, these methods often suffered from significant performance degradation as the ratio of anomalous sample ratio has increased (see Sec. 4.2). We would like to highlight that our method distinguishes from the literature by bringing a data-centric approach (refining the unlabeled data) to unsupervised anomaly detection beyond the model-centric approaches (improving the model itself). Our framework SRR aims to provide robustness in performance as the anomalous sample ratio increases, as shown in Fig.  2 . We conduct extensive experiments across various datasets from different domains, including semantic AD (CIFAR-10 [29], Dog-vs-Cat  [19] ), real-world manufacturing visual AD use case (MVTec  [8] ), and tabular AD benchmarks. We consider methods with both shallow  [36, 47]  and deep  [7, 33, 51]  models. We evaluate models at different anomaly ratios of unlabeled training data and show that SRR significantly boosts performance. For example, in Fig.  2 , SRR improves more than 15.0 average precision (AP) with a 10% anomaly ratio compared to a state-of-theart one-class contrastive representation model  [51]  on CIFAR-10. Similarly, on MVTec SRR retains a strong performance, dropping less than 1.0 AUC with 10% anomaly ratio, while the best existing OCC  [33]  drops more than 6.0 AUC. We further investigate the efficacy of our design choices, such as the number of ensemble classifiers, thresholds, and refinement of deep representations via ablation studies.
 RELATED WORK There are various existing works under the different settings described in Fig.  1 : The positive + negative setting is often considered as a supervised binary classification problem. The challenge arises due to the imbalance in label distributions as positive (anomalous) samples are rare. As summarized in  [12] , to address this, over-/under-sampling  [16, 20] , weighted optimization  [4, 27] , synthesizing data of minority classes  [32, 35] , and hybrid methods  [2, 22]  have been studied. The negative setting is often converted to a one-class classification problem, with the goal of finding a decision boundary that includes as many one-class samples as possible. Shallow models for this setting include one-class support vector machines  [47]  (OC-SVM), support vector data description  [53]  (SVDD), kernel density estimation (KDE)  [31] , and Gaussian density estimation (GDE)  [44] . There are also auto-encoder based models  [59]  that treat the reconstruction error as the anomaly score. Deep learning based OCCs have been developed, such as Deep OCC  [45] , geometric transformation  [23] , or outlier exposure  [26] . Noting the degeneracy or inconsistency of learning objectives of existing end-to-end trainable Deep OCCs,  [51]  proposed a deep representation OCC, a two-stage framework that learns self-supervised representations  [17, 28]  followed by shallow OCCs. That work was extended for texture anomaly localization with CutPaste  [33] . The robustness to very low anomaly ratios of these methods under the unsupervised setting was explored in  [7, 60] . The semi-supervised setting is defined as utilizing a small set of labeled samples and large set of unlabeled samples to distinguish anomalies from normal data. Depending on which labeled samples are given, this setting can be split into three sub-categories. When only some positive/negative labeled samples are provided, we denote that as a PU/NU setting. Most previous works in semi-supervised AD settings focus on the NU setting where only some of the normal labeled samples are given  [1, 41, 52] . The PNU setting is a more general semi-supervised setting where subsets of both positive and negative labeled samples are given. Deep SAD  [46]  and SU-IDS  [39]  are included in this category. We show the significant outperformance of our proposed SRR framework compared to Deep SAD, for multiple benchmark datasets when not using any labeled data (see Sec. 4.2). The unlabeled setting has received relatively less attention despite its significance in automating machine learning. The popular methods for this setting include isolation forest  [36]  and local outlier factor  [13] . However, they are difficult to scale, and less compatible with recent advances in representation learning. While OCCs, such as OC-SVM, SVDD, or their deep counterparts, apply to unlabeled settings by assuming the data is all negative, and the robustness of those methods has also been demonstrated in part  [7, 60] , in practice we observe a significant performance drop with a high anomaly ratio, shown in Fig.  2 . In contrast, our proposed framework is able to maintain high performance across anomaly ratios. Data refinement has been applied to AD in some prior works.  [5, 42]  generate pseudo-labels using binary classification and OC-SVM for data refinement to boost the consequent AD performances in unsupervised settings.  [6, 30, 40, 55, 59 ] used the reconstruction errors of the auto-encoder as an indicator for removing possible anomalies.  [21]  and  [38]  used data refinement for AD in supervised and semi-supervised settings. Additional discussions can be found in Appendix A.9. Self-training  [37, 48]  is an iterative training mechanism using predicted pseudo labels as targets for model training. It has regained popularity recently with its successful results in semi-supervised image classification  [9, 50, 57] . To improve the quality of pseudo labels, employment of an ensemble of classifiers has also been studied.  [14]  trains an ensemble of classifiers with different classification methods to make a consensus for noisy label verification. Co-training  [11]  trains multiple classifiers, each of which is trained on the distinct views, to supervise other classifiers. Co-teaching  [25]  and DivideMix  [34]  share a similar idea in that they both train multiple deep networks on separate data batches to learn different decision boundaries, thus becoming useful for noisy label verification. While sharing a similarity, the proposed framework has clear differences from the previous works. SRR performs iterative training with data refinement (with robust ensemble methods) and self-supervised learning for unsupervised learning of an anomaly detector.
 PROPOSED FRAMEWORK Self-supervise, Refine, and Repeat (SRR) is an iterative training framework, where we refine the data (Sec. 3.1) and update the representation with the refined data (Sec. 3.2), followed by OCCs on refined representations. Fig.  3  overviews the framework and Algorithm 1 provides the pseudo code. Notation. We denote the training data as D = {x i } N i=1 where x i ∈ X and N is the number of training samples. y i ∈ {0, 1} is the corresponding label to x i , where 0 denotes normal (negative) and 1 denotes anomaly (positive). Note that labels are not provided in the unsupervised setting. Let us denote a feature extractor as g : X → Z. g may include any data preprocessing functions, an identity function (if raw data is directly used for one-class classification), and learned or learnable representation extractors such as deep neural networks. Let us define an OCC as f : Z → [−∞, ∞] that outputs anomaly scores given the input features g(x). The higher the score f (g(x)), the more anomalous the sample x is. The binary anomaly prediction is made after thresholding: 1 f (g(x)) ≥ η .
 DATA REFINEMENT A naive way to generate pseudo labels of unlabeled data is to construct an OCC on raw data or learned representations as in  [51]  and threshold the anomaly score to obtain a binary label for normal vs. anomalous. As we update the model with refined data that excludes samples that are predicted to be anomalous, it is important to generate pseudo labels of training data as accurately as possible. To this end, instead of training a single classifier (unlike most previous works on data refinement for AD  [5, 42, 59] ), we train an ensemble of K OCCs and aggregate their predictions to generate pseudo labels. We illustrate the data refinement block in Fig.  3  and as REFINEDATA in Algorithm 1. Specifically, we randomly divide the unlabeled training data D into K disjoint subsets D 1 , ..., D K , and train K different OCCs (f 1 , ..., f K ) on corresponding subsets (D 1 , ..., D K ). Then, we estimate a binary pseudo-label of the data x i ∈ D as follows: ŷi = 1 − K k=1 1 − 1 f k (g(x i )) ≥ η k (1) η k = max η s.t. 1 N N i=1 1 f k (g(x i )) ≥ η ≥ γ (2) where 1(•) is the indicator function that outputs 1/0 if the input is True/False. f k (g(x i )) represents an anomaly score of x i for an OCC f k . η k in equation 2 is a threshold determined as a γ percentile of the anomaly score distribution {f k (g(x i ))} N i=1 . To interpret equation 1, x i is predicted as normal, i.e. ŷi = 0, if all K OCCs predict it as normal. While this may be too strict and potentially reject many true normal samples in the training set, we find that empirically, it is critical to be able to exclude true anomalous samples from the training set. The effectiveness of the employment of an ensemble of classifiers is empirically shown in Sec. 4.3.
 REPRESENTATION UPDATE SRR follows the idea of deep representation OCCs  [51] , where in the first stage a deep neural network is trained with self-supervised learning (such as rotation prediction  [23] , contrastive  [51] , or CutPaste  [33] ) to obtain meaningful representations of the data, and in the second stage OCCs are trained on these learned representations. Such a two-stage framework is shown to be beneficial as it prevents the 'hypersphere collapse' of the deep OCCs by the favorable inductive bias it brings with the architectural constraints  [45] . Here, we propose to conduct self-supervised representation learning jointly with data refinement. More precisely, we train a feature extractor g using D = {x i | ŷi = 0}, a subset of unlabeled data Algorithm 1 SRR: Self-supervise, Refine, Repeat. Input: Training data D = {x i } N i=1 , Ensemble count (K), threshold (γ) Output: Refined data ( D), trained OCC (f ), feature extractor (g) 1: function REFINEDATA(D, g, K, γ) 2: Train OCC models {f k } K k=1 on {D k } K k=1 , K disjoint subsets of the training data D.
 3: Compute thresholds η k 's for γ percentile of anomaly distributions (equation 2).
 4: Predict binary labels ŷi (equation 1).
 5: Return D = {x i : ŷi = 0, x i ∈ D}. 6: end function 7: function SRR(D, K, γ) 8: Initialize the feature extractor g.
 9: while g not converged do 10: D = REFINEDATA(D, g, K, γ). 11: Update g using D with self-supervised learning objectives. 12: end while 13: D = REFINEDATA(D, g, K, γ).
 14: Train an OCC model (f ) on refined data ( D). 15: end function D that only includes samples whose predicted labels with an ensemble OCC from Sec. 3.1 are negative. We also update D as we proceed with representation learning. The proposed method is illustrated in Algorithm 1 as SRR. In contrast to previous works  [33, 51]  that use the entire training data for learning self-supervised representation, we find it necessary to refine the training data even for learning deep representations. Without representation refinement, the performance improvements of SRR are limited, as shown in Sec. 4.3.2. Last, for test-time prediction, we train an OCC on refined data D using updated representations by g as in line 13-14 in Algorithm 1.
 UNSUPERVISED MODEL SELECTION As SRR is designed for unsupervised AD, labeled validation data for hyperparameter tuning is typically not available and the framework should enable robust model selection without any reliance on labeled data. Here, we provide insights on how to select important hyperparameters, and later in Sec. 4.3.1 perform sensitivity analyses for these hyperparameters. Data refinement of SRR introduces two hyperparameters: the number of OCCs (K) and the percentile threshold (γ). There is a trade-off between the number of classifiers for the ensemble and the size of disjoint subsets for training each classifier. With large K, we aggregate prediction from many classifiers, each of which may contain randomness from training. This comes at a cost of reduced performance per classifier as we use smaller subsets to train them. In practice, we find K = 5 works well across different datasets and anomaly ratios. γ controls the purity and coverage of refined data. If γ is large, and thus classifiers reject too many samples, the refined data could be more pure and contain mostly the normal samples; however, the coverage of the normal samples would be limited. On the other hand, with a small γ, the refined data may still contain many anomalies and the performance improvement with SRR would be limited. We empirically observe that SRR is robust to the selection of γ when it is chosen from a reasonable range. In our empirical experiments, we find ∼ 1 − 2× of the true anomaly ratio to be a reasonable choice. In other words, it is safer to use γ higher than the expected true anomaly ratio. In some cases, the true anomaly ratio may not be available at all; for such scenarios, we propose Otsu's method  [49]  to estimate the anomaly ratio of the training data for determining the threshold γ (experimental results are in the Appendix A.5).
 EXPERIMENTS We evaluate the efficacy of our proposed framework for unsupervised AD tasks on tabular (Sec. 4.1) and image (Sec. 4.2) data types. We experiment varying ratios of anomaly samples in unlabeled training data and with different combinations of representation learning and OCCs. In Sec. 4.3, we provide performance analyses to better explain major constituents of the performance, as well as sensitivity to hyperparameter values. Implementation details: To reduce the computational complexity of the data refinement block, we utilize a simple OCC such as GDE in the data refinement block. In a two-stage model, we only update the data refinement block at 1st, 2nd, 5th, 10th, 20th, 50th, 100th, 500th epochs instead of every epoch. After 500 epochs, we update the data refinement block per each 500th epoch. Each run of experiments requires a single V100 GPU. Additional discussions can be found in Appendix A.10.
 EXPERIMENTS ON TABULAR DATA Datasets. Following  [7, 60] , we test the efficacy of SRR on a variety of tabular datasets, including KDDCup, Thyroid, or Arrhythmia from the UCI repository  [3] . We also use KDDCup-Rev, where the labels of KDDCup are reversed so that an attack represents anomaly  [60] . To construct data splits, we use 50% of normal samples for training. In addition, we hold out some anomaly samples (amounting to 10% of the normal samples) from the data. This allows us to simulate unsupervised settings with an anomaly ratio of up to 10% of entire training set. The rest of the data is used for testing.  1  We conduct experiments using 5 random splits and 5 random seeds, and report the average and standard deviation of 25 F1-scores (with scale 0-100) for the performance metric. Models. We mainly compare with GOAD  [7]  (the state-of-the-art AD model in the tabular domain) and implement SRR on top of it. GOAD utilizes random transformation classification as the pretext task of self-supervised learning, and the normality score is determined by whether transformations are accurately included in the transformed space of the normal samples. We re-implement GOAD  [7]  with a few modifications. First, instead of using embeddings to compute the loss, we use a parametric classifier, similarly to augmentation prediction  [51] . Second, we follow the two-stage framework  [51]  to construct deep OCCs. For the clean training data setting, our implementation achieves 98.0 for KDD, 95.0 for KDD-Rev, 75.1 for Thyroid, and 54.8 for Arrhythmia F1-scores, which are comparable to those reported in  [7] . Please see Appendix A.3 for formulation and implementation details. Figure  4 : Unsupervised AD performance (F1-score) using OC-SVM (with rbf kernel), GOAD  [7] , and GOAD with the proposed method SRR on various tabular datasets. Shaded areas represent the standard deviation. Results. We show results of GOAD (the baseline) and GOAD with SRR in Fig.  4 . The ranges of the noise ratio are set to 0% for the anomaly ratios in the original dataset (if anomaly ratios are larger than 10%, we set the maximum anomaly ratio as 10%). For KDD-Rev, we set the maximum anomaly ratio to 2.5% because the performance of GOAD (without SRR) drops significantly even with a small ratio of anomalies in the training data. Fig.  4  shows that integrating SRR significantly improves GOAD (the state-of-the-art methods for tabular OCC). The improvements are more significant especially at higher anomaly ratios. This underlines how SRR can achieve significant improvements for both small (Thyroid & Arrhythmia) and large-scale datasets (KDD & KDD-Rev).
 EXPERIMENTS ON IMAGE DATA Datasets. We evaluate SRR on visual AD benchmarks, including CIFAR-10 [29], f-MNIST  [56] , Dog-vs-Cat  [19] , and MVTec  [8] . For CIFAR-10, f-MNIST, and Dog-vs-Cat datasets, samples from one class are set to be normal and the rest from other classes are set to be an anomaly. Similar to the experiments on tabular data in Section. 4.1, we swap a certain amount of the normal training data with anomalies given the target anomaly ratio. For MVTec, since there are no anomalous data available for training, we borrow 10% of the anomalies from the test set and swap them with normal samples in the training set. Note that 10% of samples borrowed from the test set are excluded from evaluation. For all datasets, we experiment with varying anomaly ratios from 0% to 10%. We use area under ROC curve (AUC) and average precision (AP) metrics to quantify the performance for visual AD (with scale 0-100). When computing AP, we set the minority class of the test set as label 1 and majority as label 0 (e.g., normal samples are set as label 1 for CIFAR-10 experiments as there are more anomaly samples that are from 9 classes in the test set). We run all experiments with 5 random seeds and report the average performance for each dataset across all classes. Per-class AUC and AP are reported in Appendix A.4.
 Models. For semantic AD benchmarks, CIFAR-10, f-MNIST, and Dog-vs-Cat, we compare the SRR with two-stage OCCs  [51]  using various representation learning methods, such as distributionaugmented contrastive learning  [51] , rotation prediction  [23, 28]  and its improved version  [51] , and denoising autoencoder. For MVTec benchmarks, we use CutPaste  [33]  as a baseline and compare to its version with SRR integration. For both experiments, we use the ResNet-18 architecture, trained from random initialization, using the hyperparameters from  [51]  and  [33] . The same model and hyperparameter configurations are used for SRR with K = 5 classifiers in the ensemble. We set γ as twice the anomaly ratio of training data. For 0% anomaly ratio, we set γ as 0.5. Finally, a Gaussian Density Estimator (GDE) on learned representations is used as OCC. Results. Fig.  5  shows a significant performance drop with increased anomaly ratio, regardless of representation learning. For example, the AUC of distribution-augmented contrastive representation  [51]  drops from 92.1 to 83.6 when anomaly ratio becomes 10%. Similarly, the improved rotation prediction representation  [28]  drops from 90.0 to 84.1. On the other hand, SRR effectively handles the contamination in the training data and achieves 89.9 AUC with 10% anomaly ratio, reducing the performance drop by 74.1%. An 'oracle' upper bound would be the removal of all anomalies from the training data (which is the same as the performance at 0% anomaly ratio for the same size of the data). As Fig.  5  shows, the performance of SRR is similar to this oracle upper bound performance (less than 2.5 AUC difference) even with high anomaly ratios (10%). The results are also similar in other metrics, such as AP in Fig.  5  or Recall at Precision of 70 and 90 in Fig.  9  in the Appendix. We repeat experiments on 3 additional visual AD datasets and report results in Fig.  6 . We observe consistent and significant improvements over the baseline across different datasets and different one-class classification methods. Note that the improvement is more significant at higher anomaly ratios. For instance, on MVTec dataset, SRR improves AUC by 4.9 and AP by 7.1 compared to the state-of-the-art CutPaste OCC with an anomaly ratio of 10%. In the Appendix (Sec. A.4), we also illustrate per-class performance of SRR compared with the state-of-the-art. with varying anomaly ratios. We use state-of-the-art one-class classification models for baselines, such as distribution-augmented contrastive representations  [51]  for f-MNIST and Dog-vs-Cat, or CutPaste  [33]  for MVTec, and build SRR on top of them.
 PERFORMANCE ANALYSES In this section, we conduct sensitivity analyses on two hyperparameters of SRR, namely, the number K for ensemble OCCs and the percentile threshold γ that determines normal and anomalies in the data refinement module. In addition, we show the importance of updating the representation with refined data. Ablation studies are conducted on two visual AD benchmarks, CIFAR-10 and MVTec. More experimental results and discussions can be found in the Appendix.
 SENSITIVITY TO HYPERPARAMETERS SRR is designed for unsupervised AD and it is important to ensure robust performance against changes in the hyperparameters as model selection without labeled data would be very challenging. Fig.  7  presents the sensitivity analyses of SRR with respect to various hyperparameters. In Fig.  7a , we observe the performance improvement as we increase the number of classifiers for ensemble. This is particularly effective on CIFAR (top in Fig.  7a ), where the number of samples in the training data is large enough that even with large K, the number of samples to train each OCC (N/K) would be sufficient. In Fig.  7b , we observe that SRR performs robustly when γ is set to be larger than the actual anomaly ratio (10%). When γ is less than 10%, however, we see a significant drop in performance, all the way to the baseline (γ = 0). Our results show that SRR improves upon baseline regardless of the threshold. In addition, it suggests that γ could be set to be anywhere from the true anomaly ratio and and 2x the anomaly ratio to maximize its effectiveness. When the true anomaly ratio is unknown, as discussed in Appendix A.5, Otsu's method could be used.
 ITERATIVELY UPDATING REPRESENTATIONS WITH REFINED DATA It is possible to decouple representation learning and data refinement of SRR, which would result in a three-stage framework, where we learn representations until convergence without data refinement, followed by the data refinement and learning OCC. Fig.  7c  shows that SRR using a pre-trained then fixed representation (i.e. when data refinement is used for OCC only) already improves the performance upon the baseline with no data refinement at any stage of learning representation or classifier. Improvements on CIFAR-10 are 4.9 in AUC and 8.0 in AP; and on MVTec are 1.
 QUANTIFYING THE REFINEMENT EFFICACY We evaluate how many normal and anomalies are excluded by the proposed data refinement block. As in Fig.  8 (a, b), with data refinement, we can exclude more than 80% of anomalies in the training set without removing too many normal samples. For instance, among 4% anomalies in CIFAR-10 data, SRR is able to exclude 80% anomalies while removing less than 20% normal samples. Such a high recall of anomalies of SRR is not only useful for unsupervised AD, but also could be useful for improving the annotation efficiency when a budget for active learning is available. Fig.  8(c,  d ) demonstrates the removed normal and abnormal samples by the data refinement module over training epochs. It shows that better representation learning (as training epochs increase) consistently improves the efficacy of the data refinement.
 CONCLUSION AD has wide range of practical use cases. A challenging and costly aspect of building an AD system is that anomalies are rare and not easy to detect by humans, making them difficult to label. To this end, we propose a novel AD framework to enable high performance AD without any labels called SRR. SRR can be flexibly integrated with any OCC, and applied on raw data or on a trainable representations. SRR employs an ensemble of multiple OCCs to propose candidate anomaly samples that are refined from training, which allows more robust fitting of the anomaly decision boundaries as well as better learning of data representations. We demonstrate the state-of-the-art AD performance of SRR on multiple tabular and image data. ETHICS STATEMENT SRR has the potential to make significant positive impact in real-world AD applications where detecting anomalies is crucial, such as for financial crime elimination, cybersecurity advances, or improving manufacturing quality. We note a potential risk associated with using SRR: when representations are not sufficiently good, there will be a negative cycle of refinement and representation updates/OCC. While we rely on the existence of good representations, in some applications they may be difficult to obtain (e.g., cybersecurity). This paper focuses on the unsupervised setting and demonstrates strong AD performance, opening new horizons for human-in-the-loop AD systems that are low cost and robust. We leave these explorations to future work.
 REPRODUCIBILITY STATEMENT The source code of SRR will be published upon acceptance. For reproducibility, we only use public tabular and image datasets to evaluate the performances of SRR. Complete descriptions of those datasets can be found in Datasets subsections in Sec. 4.1 and 4.2. Detailed experimental settings and hyperparameters can be found in Models subsections in Sec. 4.1 and 4.2. The implementation details (including hyperparameters that we used) on GOAD  [7]  are described in Sec. A.3.
 A APPENDIX A.1 ADDITIONAL RESULTS WITH DIFFERENT METRICS There are various performance metrics of OCCs. In the main manuscript, we mainly use AUC, Average Precision (AP) and F1-score as the evaluation metrics. In this subsection, we report the performances of the proposed model (SRR) and baselines in terms on Recall at Precision 70 and 90 as the additional metrics on CIFAR-10 dataset. As in Fig.  9 , we observe a similar trends with these two additional metrics as well. For example, the performances of SRR are robust across various anomaly ratios. On the other hand, all the other OCCs show consistent and significant performance degradation as the anomaly ratio increases. SRR performs 15.8 and 10.9 better than the state-of-the-art OCC  [51]  in terms of recall at precision 70 and 90, respectively.  SRR is also applicable on raw tabular features or learned image representation without representation update using data refinement. In this section, we demonstrate the performance improvements by SRR without representation update to verify the effectiveness of data refinement block of SRR for shallow OCCs. Fig.  10  (upper) demonstrates consistent and significant performance improvements when we apply SRR on top of raw tabular features. Specifically, the Average Precision (AP) improvements are 10.2, 29.0, and 4.1 with KDD-Rev, Thyroid, and Arrhythmia tabular datasets, respectively. We also apply SRR on top of various learned image representations. As can be seen in Fig.  10  (lower), the performance improvements of SRR are consistent across various different learned image representations (without representation update). For instance, the AP improvements are 9.2, 1.0, and 12.1 with learned image representations using RotNet  [23] , Rotation  [28] , and Contrastive  [51] , respectively. A.3 IMPLEMENTATION DETAILS ON GOAD  [7]  FOR TABULAR DATA EXPERIMENTS A classification-based AD method, GOAD  [7] , has demonstrated strong AD performance on tabular datasets. Unlike previous works  [23, 26]  that formulate a parametric classifier for multiple transformation classification, GOAD employs distance-based classification of multiple transformations. For the set of transformations T m : X → D, m = 1, ..., M , the loss function of GOAD is written as in equation 3 with the probability defined in equation 4. L = −E m,x log P (m|T m (x)) , (3) P ( m|T m (x)) = exp(− f (T m (x)) − c m 2 ) n exp(− f (T m (x)) − c n 2 ) , (4) where the centers c m 's are updated by the average feature over the training set. While it is shown to perform well  [7] , we find that the distance-based formulation is not necessary, and we achieve the similar performance, if not worse, to  [7]  using a parametric classifier when computing the probability: P ( m|T m (x)) = exp w mf (T m (x)) + b m n exp (w n f (T m (x)) + b n ) (5) The formulation in equation 5 is easier to optimize than its original form in equation 4 as it can be fully optimized with backpropagation without alternating updates of feature extractor f and centers c m . Once we learn a representation by optimizing the loss in equation 3 using equation 5, we follow a two-stage one-class classification framework of  [51]  to construct a set of Gaussian density estimation OCCs for each transformation. Finally, we aggregate a maximum normality scores from a set of classifiers as the normality score. In Table  1 , we summarize the implementation details, such as network architecture or hyperparameters, and AD performance under clean training data setting that reproduces the results in  [7] . Table  1 : The AD performance under clean only data setting of GOAD in  [7]  and our implementation. Our implementation demonstrates comparable, if not worse, performance to those reported in  [7] . Our implementation also shares most hyperparameters across datasets except the M , the number of transformations, and the train steps, which are closely related to the size of training data.
 A.4 PER-CLASS AUC AND AP In the main manuscript, we report the mean and standard deviation of AUC and AP across all classes in each dataset. In this section, we report the mean and standard deviation of AUC and AP for each class in each dataset (including CIFAR-10 (Table  2 ), MVTec (Table  3 ), fMNIST (Table  4 ), and Dog-vs-Cat (Table  5 ) datasets).
 A.4.1 PER-CLASS AUC AND AP ON CIFAR-10 DATASET The key idea of the Otsu's method is to find the threshold that minimizes the intra-class variance. This is defined as the weighted sum of variances of the two classes. Let us denote the normality scores as {s i } N i=1 and threshold as η. Then, we try to find the threshold (η) that minimizes the weighted sum of the variance (w 0 (η) × σ 0 (η) + w 1 (η) × σ 1 (η)) where w 0 (η) = N i=1 I(s i < η)/N and w 1 (η) = N i=1 I(s i ≥ η)/N . σ 0 (η) and σ 1 (η) are the variances of each class. The optimal threshold (η * ) is determined as η * = min η w 0 (η) × σ 0 (η) + w 1 (η) × σ 1 (η). We use the twice of η * as the hyperparameter (γ) in SRR. We evaluate the performances of Otsu's method (on top of SRR) in comparison to the state-of-the-art OCC  [33]  and original SRR with the knowledge of the true anomaly ratio using MVTec dataset. Fig.  11  demonstrates that even without true anomaly ratio, the performance of SRR can be significantly better than the state-of-the-art OCC  [33]  with Otsu's method. Also, Fig.  11  shows that the knowledge of true anomaly ratio is crucial information for maximizing the performance of SRR in fully unsupervised settings. We further extend the experimental results using Otsu's method with other datasets such as CIFAR-10 and Thyroid datasets.   [51]  for CIFAR-10 dataset and GOAD  [7]  for Thyroid dataset. We introduce 6% noise on MVTec dataset and 1.5% noise on the Thyroid dataset. Metrics are (AUC/AP) for CIFAR-10 dataset and F1 score for Thyroid dataset.
 A.6 ADDITIONAL ABLATION STUDIES To better understand the source of gains, we include comparisons between the final ensemble model on the converged self-supervised extractor (SRR without final OCC) with the proposed SRR (using an additional final OCC). Also, to further support the novelty of SRR, we report additional experimental results only with ensemble model without data refinement (Ensemble Only). As can be seen in Table  7 , the proposed version of SRR (with an additional final OCC) outperforms significantly, which is attributed to the fact that while fitting the individual OCC models in the ensemble, we do not exclude the possible anomaly samples for diversity of the trained submodels, so the anomaly decision boundaries can be fitted robustly. Therefore, the ensemble model can be somewhat less accurate for classifying difficult samples (near the decision boundary between normal and abnormal) compared to the final OCC used (that is trained with only normal samples). Also, employment of ensemble learning without data refinement (Ensemble only), yields much worse performance than the proposed method (SRR), underlining the importance of the core data refinement idea of SRR. Datasets   [33]  for MVTec dataset and  [51]  for CIFAR-10 dataset. We introduce 6% noise on both datasets. Metrics are (AUC/AP).
 A.7 ADDITIONAL BASELINES We add extra baselines from robust anomaly detection literature: Standard PCA  [54] , Robust PCA  [15]  and Local Outlier Factor (LOF)  [13]  for tabular data and Robust autoencoder (Robust AE)  [59]  for image data. As can be seen in Table  8 , the performance of PCA and LOF are highly degraded even with a small amount of anomalies in the training data. For Robust PCA and Robust AE, the performance degradation is less but still significant in comparison to SRR. Overall, SRR outperforms other benchmarks in fully unsupervised settings, underlining the importance of data refinement in improving the robustness to anomaly ratio in training data, as the core constituent of SRR framework. Datasets    Figure 2 : 2 Figure 2: Performance of our proposed model and a baseline OCC using contrastive representation [51] on CIFAR-10 with different anomaly ratios in the training data.
 Figure 3 : 3 Figure 3: Block diagram of SRR composed of representation learner (Sec. 3.2), data refinement (Sec. 3.1), and final OCC blocks. The representation learner updates the deep models using refined data from the data refinement block. Data refinement is done by an ensemble of OCCs, each of which is trained on K disjoint subsets of unlabeled training data. Samples predicted as normal by all classifiers are retained in the refined data, and are used to update the representation learner and final OCC. The process is repeated iteratively until convergence. Convergence graphs can be found in the Appendix A.8.
 Figure 5 : 5 Figure 5: Unsupervised AD performance with various OCCs on CIFAR-10. For SRR we adapt distribution-augmented contrastive representation learning [51]. (Left) AUC, (Right) Average Precision (AP).
 Figure 6 : 6 Figure6: Unsupervised AD performance on (a) MVTec (b) f-MNIST, and (c) Dog-vs-Cat datasets with varying anomaly ratios. We use state-of-the-art one-class classification models for baselines, such as distribution-augmented contrastive representations [51]  for f-MNIST and Dog-vs-Cat, or CutPaste [33]  for MVTec, and build SRR on top of them.
 Figure 7 : 7 Figure 7: Ablation studies on (top) CIFAR-10 and (bottom) MVTec under 10% anomaly ratio setting with respect to (a) ensemble count K, (b) percentile threshold γ, and (c) data refinement with or without representation update.
 Figure 8 : 8 Figure 8: Percentage of excluded anomalous and normal samples by data refinement (a, b) with different anomaly ratios in training data and (c, d) over training epochs for 10% anomaly ratio.
 Figure 9 : 9 Figure 9: Performance of various OCCs on CIFAR-10 dataset. SRR is applied on top of Contrastive [51]. (Left) Recall at Precision 70, (Right) Recall at Precision 90.
 A. 2 2 SRR ON RAW TABULAR FEATURES / LEARNED IMAGE REPRESENTATIONS
 Figure 10 : 10 Figure10: Performance of SRR on (top) raw tabular features and (lower) learned image representations. SRR consistently outperforms baseline and in some cases (e.g., Thyroid, and Contrastive [51] ), the performance improvements are significant.
 Figure 11 : 11 Figure 11: Unsupervised anomaly detection performances with Otsu's method on top of SRR on MVTec dataset. (Left) AUC, (Right) Average Precision (AP).
 Figure 12 : 12 Figure 12: Convergence graphs of SRR with (left) MVTec dataset, (right) CIFAR-10 dataset.
 Table 6 6 are the overall results -Otsu's method yields only slight degradation compared to SRR with true anomaly ratio; however, it still significantly outperforms SOTA OCC baselines. Datasets / Method SOTA OCC SRR with Otsu's method SRR CIFAR-10 0.855 / 0.585 0.906 / 0.703 0.910 / 0.709 Thyroid 0.506 0.623 0.639
 Table 6 : 6 Additional ablation studies. SOTA OCC methods are
 Table 7 : 7 / Method SOTA OCC Ensemble Only SRR without final OCC SRR Additional ablation studies. SOTA OCC methods are CutPaste MVTec 0.905 / 0.845 0.911 / 0.849 0.922 / 0.870 0.937 / 0.887 CIFAR-10 0.855 / 0.585 0.862 / 0.599 0.890 / 0.677 0.910 / 0.709
 Table 8 : 8 Additional experiments with extra baselines from robust anomaly detection literature. We introduce 6% noise on CIFAR-10 and KDD datasets. For Thyroid dataset, we introduce 1.5% noise. Metrics are (AUC/AP) for image data and F1 score for tabular data.A.8 CONVERGENCE GRAPHSThe proposed SRR framework is converged when the iterative training of the self-supervised learning is converged. Depending on the data refinement model training, corresponding self-supervised models are also trained with differently refined training data. Usually, the data refinement model and self-supervised models converge after a similar number of epochs. Fig.12illustrate the convergence graphs of SRR with MVTec and CIFAR-10 datasets. / Method PCA Robust PCA LOF Robust AE SRR CIFAR-10 - - - 0.636 / 0.174 0.910 / 0.709 Thyroid 0.299 0.377 0.338 - 0.506 KDD 0.836 0.893 0.873 - 0.942
			 Note that the experimental settings with contaminated training data in GOAD [7]  and DAGMM [60]  are slightly different from ours. Our contamination ratio is defined as the anomaly ratio over the entire training data, while their contamination ratio is the anomaly ratio over all the anomalies in the dataset.

Abstract: Anomaly detection (AD) -separating anomalies from normal data -has many applications across domains, from manufacturing to healthcare. While most previous works have been shown to be effective for cases with fully or partially labeled data, that setting is in practice less common due to labeling being particularly tedious for this task. In this paper, we focus on fully unsupervised AD, in which the entire training dataset, containing both normal and anomalous samples, is unlabeled. To tackle this problem effectively, we propose to improve the robustness of one-class classification trained on self-supervised representations using a data refinement process. Our proposed data refinement approach is based on an ensemble of oneclass classifiers (OCCs), each of which is trained on a disjoint subset of training data. Representations learned by self-supervised learning on the refined data are iteratively updated as the refinement improves. We demonstrate our method on various unsupervised AD tasks with image and tabular data. With a 10% anomaly ratio on CIFAR-10 image data / 2.5% anomaly ratio on Thyroid tabular data, the proposed method outperforms the state-of-the-art one-class classification method by 6.3 AUC and 12.5 average precision / 22.9 F1-score.


Lecture Notes in Computer Science: Safe Exploration Techniques for Reinforcement Learning -An Overview

Introduction Reinforcement learning (RL) as a machine learning method has been thoroughly examined since 80's. In 1981, Sutton and Barto  [3]  inspired themselves in the reinforcement learning discoveries in behavioral psychology and devised the Temporal Difference machine learning algorithm that had to simulate psychological classical conditioning. In contrast with supervised learning, reinforcement learning does not need a teacher's classification for every sample presented. Instead, it just collects rewards (or punishment) on-the-go and optimizes for the expected long-term reward (whereas supervised learning optimizes for the immediate reward). The key advantage is that the design of the rewards is often much simpler and straight-forward than classifying all data samples. Reinforcement learning proved to be extremely useful in the case of statespace exploration -the long-term reward corresponds to the value of each state  [17] . From such values, we can compose a policy which tells the agent to always take the action leading to the state with the highest value. As an addition, state values are easily interpretable for humans. Since the early years, a lot of advanced methods were devised in the area of reinforcement learning. To name one, Q-learning  [25]  is often used in connection with safe exploration. Instead of computing the values of states, it computes the values of state-action pairs, which has some simplifying consequences. For example, Q-learning doesn't need any transition model (i.e. dynamics model) of the examined system. A completely different approach is policy iteration. This algorithm starts with a (more or less random) policy and tries to improve it step-by-step  [16] . This case is very valuable if there already exists a good policy and we only want to improve it  [11] . What do all of these methods have in common, is the need for rather large training data sets. For simulated environments it is usually not a problem. But with real robotic hardware, the collection of training samples is not only lengthy, but also dangerous (be it mechanical wear or other effects). Another common feature of RL algorithms is the need to enter unknown states, which is inherently unsafe. As can be seen from the previous paragraph, safety is an important issue connected with reinforcement learning. However, the first articles focused on maintaining safety during exploration started to appear much later after the "discovery" of RL. Among the first, Heger  [15]  "borrowed" the concept of a worstcase criterion from control theory community. In 1994 he created a variant of Q-learning where maximization of long-term reward is replaced with maximization of minimum of the possible rewards. That basically means his algorithm prefers to never encounter a bad state (or, at least to choose the best of the bad states). This approach has one substantial drawback -the resulting policies are far from being optimal in the long-term-reward sense  [10] . In this paper we show the various approaches to safe exploration that have emerged so far. We classify the methods by various criteria and suggest suitable use cases for them. To better illustrate some of the practical details, we use the UGV (Unmanned Ground Vehicle) robotic platform from EU FP7 project NIFTi  [6]  (see Figure  1 ) as a reference agent. It may happen that in these practical details we assume some advantages of UGVs over UAVs (Unmanned Aerial Vehicles), like the ability to stand still without much effort, but it is mostly easy to convert these assumptions to UAVs, too. Further organization of this paper is the following: in Section 2 we discuss some basics of reinforcement learning (the reader may skip it if he is familiar with reinforcement learning); Section 3 is an overview of the safety definitions
 Reinforcement learning basics
 Markov Decision Processes Markov Decision Processes (MDPs) are the standard model for deliberating about reinforcement learning problems. They provide a lot of simplifications, but are sufficiently robust to describe a large set of real-world problems. The simplest discrete stochastic MDP comprises of:  [17]  a finite set of states S a finite set of actions A a stochastic transition model P : P t (s, a, s ) = P r(s t+1 = s | s t = s, a t = a) for each s, s ∈ S, a ∈ A, where P r stands for probability and the immediate reward function R : S × A → R (or R : S × A × S → R if the reward depends on the stochastic action result) To interpret this definition, we say that the at every time instant t the agent is in a state s, and by executing action a it gets to a new state s . Furthermore, executing a particular action in a particular state may bring a reward to the agent (defined by R). The most important and interesting property of MDPs is the Markov property. If you have a look at the definition of the transition model, the next state only depends on the current state and the chosen action. Particularly, the next state is independent of all the previous states and actions but the current one. To give an example, the robot's battery level cannot be treated implicitly by counting the elapsed time, but rather it has to be modeled as a part of the robot's state. Once the model is set up, everything is ready for utilizing an MDP. "The agent's job is to find a policy π mapping states to actions, that maximizes some long-run measure of reinforcement"  [17] . The "long-run" may have different meanings, but there are two favorite optimality models: the first one is the finite horizon model, where the term J = h t=0 r t is maximized (h is a predefined time horizon and r t is the reward obtained in time instant t while executing policy π). The dependency of r t on the policy is no longer obvious from this notation, but this is the convention used in literature when it is clear which policy is used. This model represents the behavior of the robot which only depends on a predefined number of future states and actions. The other optimality model is called discounted infinite horizon, which means we maximize the discounted sum J = ∞ t=0 γ t r t with γ ∈ (0, 1) being the discount factor. The infinite horizon tries to find a policy that is the best one taking into account the whole future. Please note the hidden dependency on the policy π (and the starting state s 0 ) -it is the policy that decides on which action to take, which in turn specifies what will the reward be. Other extensions of MDPs to continuous states, time or actions are beyond the scope of this overview. However, some of the referenced papers make use of these continuous extensions, which proved to be useful for practical applications.
 Value iteration Value iteration is one of the basic methods for finding the optimal policy. To describe this algorithm, it is first needed to define the essential notion of the optimal value of a state. In this whole subsection we suppose the discounted infinite horizon model, but analogous results can be shown for finite horizon, too. "The optimal value of a state is the expected infinite discounted sum of reward that the agent will gain if it starts in that state and executes the optimal policy."  [17]  Given a policy π, the induced value function is therefore defined as V π (s) = E ∞ t=0 r k γ k , (1) where E denotes the expected value and r k are the rewards for executing policy π. Taking the best value function over all policies then yields the optimal value function V * : [17] V * (s) = max π V π (s) . (2) Inversely, if we have the value function given, we can derive a policy from that. It is a simple policy that always takes the action leading to the most profitable neighbor state (with the highest value). One useful formulation of the properties of the optimal value function is the formulation using the recurrent Bellman equations which define a dynamic system that is stable for the optimal value function. We can say a state's optimal value is the best immediate reward plus its best neighbor's optimal value: [17] V * (s) = max a R(s, a) + γ s ∈S P(s, a, s )V * (s ) . (3) Analogously, we can find the optimal policy using the same Bellman equation: π * (s) = argmax a R(s, a) + γ s ∈S P(s, a, s )V * (s ) . (4) The Value iteration algorithm is based on trying to compute the solution of Equation 4 using iterative Bellman updates (refer to Algorithm 1). In the algorithm, we use a structure called Q to store the "value" of state-action pairs. In Value iteration it is just a structure to save intermediate results, but it is the core of the Q-learning algorithm (described in Section 2.3). The stopping criterion of the Value iteration algorithm is not obvious, but Williams and Baird  [26]  derived an easily applicable upper bound on the error of the computed value function. That said, after a sufficient number of those simple iterations, we can compute the almost optimal value function. The number of iterations needed for Value iteration to converge may be impractically high, but it is shown that the optimal policy converges faster  [4] , thus making Value iteration practical.
 Q-learning Just a small change to the Value iteration algorithm results in Q-learning. The basic algorithm is the same as Value iteration, just the update step is done differently (refer to Algorithm 2). The consequence of this change is that no model of the system (transition function P) is needed. It is sufficient to execute all actions in all states equally often, and Watkins  [25]  proved that if Q-learning were run for an infinite time, the computed Q would converge to the optimal Q * (an analogue of V * ).
 Policy iteration Policy iteration is a completely different approach to computing the optimal policy. Instead of deriving the policy from the Value or Q function, Policy iteration Algorithm 1 The Value iteration algorithm  [17]  Input: an MDP (states S, actions A, rewards R, transition model P) Output: the optimal value function V * , resp. the optimal policy π * derived from the value function 1. V(s) := arbitrary function 2. π := the policy derived from V 3. while π is not good enough do 4. for all s ∈ S do 5. for all a ∈ A do Update: 6. Q(s, a) := R(s, a) + γ s ∈S P(s, a, s )V(s ) 7. end for 8. V(s) := max a Q(s, a) 9. end for 10. π := the policy derived from V 11. end while 12. V * := V, π * := π Algorithm 2 The Q-learning algorithm (only the parts that differ from Value iteration when V is substituted with Q)  [17]  Input: an MDP (states S, actions A, rewards R, transition model may be unknown) Output: the optimal state-value function Q * , resp. the optimal policy π * derived from the state-value function 6. Q(s, a) := Q(s, a)+ α R(s, a) + γ max a Q(s , a ) − Q(s, a)
 line left out works directly with policies. In the first step, a random policy is chosen. Then a loop consisting of policy evaluation and policy improvement repeats as long as the policy can be improved  [17]  (refer to Algorithm 3 for details). Since in every step the policy gets better, and there is a finite number of different policies, it is apparent that the algorithm converges  [23] . Policy iteration can be initialized by a known, but suboptimal policy. Such policy can be obtained e.g. by a human operator driving the UGV. If the initial policy is good, Policy iteration has to search much smaller subspace and thus should converge more quickly than with a random initial policy  [11] . Algorithm 3 The Policy iteration algorithm  [17]  1. π = arbitrary policy 2. repeat 3. π := π Policy evaluation: (system of linear equations) 4. Vπ(s) = R(s, π(s)) + γ s ∈S P(s, π(s), s )Vπ(s ) Policy improvement: 5. π (s) := argmax a∈A R(s, a) + γ s ∈S P(s, a, s )Vπ(s ) 6. until π = π
 Defining safety To examine the problems of safe exploration, it is first needed to define what exactly is the safety we want to maintain. Unfortunately, there is no unified definition that would satisfy all use cases; thus, several different approaches are found in the literature. An intuitive (but vague) definition could be e.g.: "Statespace exploration is considered safe if it doesn't lead the agent to unrecoverable and unwanted states." It is worth noticing here that unwanted doesn't necessarily mean low-reward. In the next subsections we present the main interpretations of this vague definition.
 Safety through labeling The largely most used definition of safety is labeling the states/actions with one of several labels indicating the level of safety in that state/action. What varies from author to author is the number and names of these labels. To start with, Hans  [14]  has the most granular division of state/action space. His definitions are as follows (slightly reformulated): an (s, a, r, s ) tuple (transition) is fatal if the reward r is less than a certain threshold (s is the original state, a is an action and s is the state obtained after executing a in state s, yielding the reward r), an action a is fatal in state s if there is non-zero probability of leading to a fatal transition, state s is called supercritical if there exists no policy that would guarantee no fatal transition occurs when the agent starts in state s, Since we will compare other definitions the the Hans', it is needed to define one more category. A state s is called fatal if it is an undesired or unrecoverable state, e.g. if the robot is considered broken in that state. The fatal transition can then be redefined as a transition ending in a fatal state. Opposite to the precisely defined terms in Hans' definition, the meaning of words "undesired" and "unrecoverable" here is vague and strongly task-dependent. - Continuing on, Geibel  [12]  defines only two categories -fatal and goal states. "Fatal states are terminal states. This means, that the existence of the agent ends when it reaches a fatal state"  [12] . This roughly corresponds to our defined set of fatal states. Goal states are the rest of final states that correspond to successful termination. Since Geibel only considers terminal states for safety, his goal states correspond to a subset of safe states. The other Hans' categories need not be represented, since they are meaningless for final states. An extension of Geibel's fatal and goal states is a division presented by García  [10] . His error and non-error states correspond to fatal and goal states, but García adds another division of the space -the known and unknown states, where known states are those already visited (and known have empty intersection with error ). He then mentions a prerequisite on the MDP that if an action leads to a known error /non-error state, then its slight modification must also lead to an error /non-error state (a metric over the state space is required). In Ertle's work  [9] , again the two basic regions are considered -they are called desired and hazardous (corresponding to safe and fatal). However, due to the used learning technique, one more region emerges -the undesired region. It contains the whole hazardous region and a "small span" comprising of desired states, and denotes the set of states where no training (safe) samples are available, because it would be dangerous to acquire those samples. In particular, he says that "The hazards must be 'encircled' by the indications of the undesired approaching so that it becomes clear which area [. . . ] is undesired"  [9] . A summary of the labeling-based definitions is shown in Figure  3 . We examined the apparent imbalance between the number of categories Hans defines, and the other definitions, and that led us to the following observations. The first observation is that creating labels for actions or transitions is unnecessary. If we need to talk about the "level of safety" of an action, we can use the worst label out of all possible results of that action (which retains compatibility with Hans' definitions). Moreover, as "it is impossible to completely avoid error states"  [22] , we can ignore the effects of the action which have only small probability (lower than a safety threshold) -we will call such effects the negligible effects. A second remark is that the fatal and supercritical sets can be merged. In Hans' work we haven't found any situation where distinguishing between supercritical and fatal would bring any benefit. Specifically, in his work Hans states that: "Our objective is to never observe supercritical states"  [14] , which effectively involves avoiding fatal transitions, too. And since we avoid both supercritical and fatal, we can as well avoid their union. Third, safety of a state does not necessarily depend on the reward for getting to that state. E.g. when the UGV performs a victim detection task, going away from the target area may be perfectly safe, but the reward for such action should be small or even negative. Putting these observations together, we propose a novelty definition of safety for stochastic MDPs, which is a simplification of Hans' model and a generalization of the other models: -A state is unsafe if it means the agent is damaged/destroyed/stuck. . . or it is highly probable that it will get to such state regardless of further actions taken. -A state is critical if there is a not negligible action leading to an unsafe state from it. -A state is safe if no available action leads to an unsafe state (however, there may be an action leading to a critical state). To illustrate the definition on a real example, please refer to Figure  2 . In 2(a), the UGV is in a safe state, because all actions it can take lead again to safe states (supposing that actions for movement do not move the robot for more than a few centimeters). On the other hand, the robot as depicted in 2(b) is in a critical state, because going forward would make the robot fall over and break. If the robot executed action "go forward" once more, it would come to an unsafe state. Right after executing the action it would still not be broken; however, it would start falling and that is unsafe, because it is not equipped to withstand such fall and therefore it is almost sure it will break when it meets the ground.
 Safety through ergodicity An MDP is called ergodic iff for every state there exists a policy that gets the agent to any other state  [20] . In other words, every mistake can be remedied in  such MDP. Moldovan  [20]  then defines δ-safe policies as policies guaranteeing that from any state the agent can get to the starting state with probability at least δ (using a return policy, which is different from the δ-safe one). Stated this way, the safety constraint may seem intractable, or at least impracticalit is even proved, that expressing the set of δ-safe policies is NP-hard  [20] . An approximation of the constraint can be expressed in the terms of two other MDP problems which are easily solved  [20] ; that still leads to δ-safe policies, but the exploration performance may be suboptimal. In our view, safety through ergodicity imposes too much constraints on the problems the agent can learn. It sometimes happens that a robot has to learn some task after which it is not able to return to the initial state (e.g. drive down a hill it cannot go upwards; a human operator then carries the robot back to the starting position). But the inability to "return home" in no means indicates the robot is in an unsafe state.
 Safety through costs Another definition of safety is to define a cost for taking an action/being in a state and minimize the worst-case cost of the generated policies (up to some failure probability). Such approach is presented in  [15] . However, unless a threshold is set, this definition leads only to the safest possible policies, which are not necessarily safe. Expressing the safety using costs is natural for some RL tasks (e.g. when learning the function of a dynamic controller of an engine, the engine's temperature can be treated as a cost). Unfortunately, not all unsafe states can be described using such costs in general. In addition, specifying the right costs may be a difficult task.
 Safety as variance of the expected return An alternative to safety as minimization of a cost (either worst-case or expected) is minimizing both the cost and its variance. This approach is called expected value-variance criterion  [15]  and is used mainly in works prior 2000, e.g.  [7] . A safe policy by this criterion can be viewed as a policy that minimizes the number of critical actions (because fatal transitions are expected to yield much larger costs than safe transitions, increasing the variance significantly). As stated in  [10] , the worst-case approach is too restrictive and cautious. The other expected value-variance criteria suffer from the same disadvantages as safety through costs -mainly from the general difficulty to tune up the costs.
 Safe exploration approaches Finally, when the theoretical concepts have been shown and the various safety definitions have been presented, we can focus on the main part of this overview. Our categorization of safe exploration techniques is based on the work of García  [10] . The basic division is as follows: approaches utilizing the expected return or its variance (Sec. 4.1), labeling-based approaches (Sec. 4.2) and approaches benefiting from prior knowledge (Sec. 4.3).
 Optimal control approaches Techniques in this category utilize variations of the expected value-variance safety criterion. The most basic one is treating the rewards as costs (when a reward is denoted by r t , the corresponding cost is denoted by c t ). Standard RL methods can then be used to solve the safe exploration task, as described e.g. in  [7]  for discounted infinite horizon. The RL objective function J = E ∞ t=0 γ t c t ( 5 ) is called the risk-neutral objective. To make this objective risk-sensitive, we specify a risk factor α and rewrite the objective as: [15] J = 1 α log E [exp (αγ t ∞ t=0 c t )] (6) E [ ∞ t=0 γ t c t ] + α 2 V ar [ ∞ t=0 γ t c t ] , which is also called the expected value-variance criterion. This approach is a part of theory using exponential utility functions, which is popular in optimal control  [19] . To complete this section, the worst-case objective function (also called the minimax objective) is defined as J = sup ∞ t=0 γ t c t . (7) As can be seen, the objective functions containing expectations cannot in fact assure that no unsafe state will be encountered. On the other hand, the minimax objective provides absolute certainty of the safety. However, it may happen that some of the unsafe states can only be reached with a negligible probability. In such cases, the α-value criterion defined by  [15]  can be used -it only takes into account rewards that can be reached with probability greater than α. In the work of Mihatsch  [19] , a scheme is presented that allows to "interpolate" between risk-neutral and worst-case behavior by changing a single parameter. Delage's work  [8]  takes into account the uncertainty of parameters of the MDP. It is often the case that the parameters of the MDP are only estimated from a limited number of samples, causing the parameter uncertainty. He then proposes a possibility that the agent may "invest" some cost to lower the uncertainty in the parameters (by receiving some observations from other sources than exploration). A completely new research area then appears -to decide whether it is more valuable to pay the cost for observations, or to perform exploration by itself. An approximation scheme for dealing with transition matrix uncertainty is presented in  [21] . It considers a robust MDP problem and provides a worst-case, but also robust policy (with respect to the transition matrix uncertainty). A theory generalizing these approaches can be found in  [24] . The theory states that the optimal control decision is based on three terms -the deterministic, cautionary and probing terms. The deterministic term assumes the model is perfect and attempts to control for the best performance. Clearly, this may lead to disaster if the model is inaccurate. Adding a cautionary term yields a controller that considers the uncertainty in the model and chooses a control for the best expected performance. Finally, if the system learns while it is operating, there may be some benefit to choosing controls that are suboptimal and/or risky in order to obtain better data for the model and ultimately achieve better long-term performance. The addition of the probing term does this and gives a controller that yields the best long-term performance.  [24]  To conclude this section, we think that these methods are not well suited for safe exploration -the expected value-variance and similar criteria provide no warranties on the actual safety. On the other hand, the worst-case approaches seem to be too strict.
 Labeling-based approaches The approaches utilizing some kind of state/action labeling (refer to Section 3.1 for the various labeling types) usually make use of two basic components -a risk function and a backup policy. The task of the safety function is to estimate the safety of a state or action. In the simplest case, the safety function can just provide the labeling of the given action; or it can return a likelihood that the action is safe; and in the best case, it would answer with a likelihood to be safe plus a variance (certainty) of its answer. The backup policy is a policy that is able to lead the agent out of the critical states back to the safe area. It is not obvious how to get such a policy, but the authors show some ways how to get one. In the work of Hans  [14] , the most granular labeling is used, where fatal transitions are said to be the transitions with reward less than a given threshold. The safety function is learned during the exploration by collecting the so-called min-reward samples -this is the minimum reward ever obtained for executing a particular action in a particular state. The backup policy is then told to either exist naturally (e.g. a known safe, but suboptimal controller), or it can also be learned. To learn the backup policy, an RL task with altered Bellman equations is used: Q * min (s, a) = max s min R(s, a, s ), max a Q * min (s , a ) . A policy derived from the computed Q * min function is then taken as the backup policy (as it maximizes the minimum reward obtained, and the fatal transitions are defined by low reward). He defines a policy to be safe, if it executes only safe actions in safe states and produces non-fatal transitions in critical states. To learn such safe policy, he then suggests a level-based exploration scheme (although he gives no proofs why it should be better than any other exploration scheme). This scheme is based on the idea that it is better to be always near the known safe space when exploring. All unknown actions from one "level" are explored, and their resulting states are queued to the next "level". For exploration of unknown actions he proposes that the action should be considered critical until proved otherwise, so the exploration scheme uses the backup policy after every unknown action execution. A disadvantage of this approach is that the agent needs some kind of "path planning" to be able to get to the queued states and continue exploration from them. García's PI-SRL algorithm  [10]  is a way to safeguard the classical policy iteration algorithm. Since the labels error /non-error are only for final states, the risk function here is extended by a so called Case-based memory, which is in short a constant-sized memory for storing the historical (s, a, V(s)) samples and is able to find nearest neighbors for a given query (using e.g. the Euclidean distance). In addition to the error and non-error states, he adds the definition of known and unknown states, where known states are those that have a neighbor in the case-based memory closer than a threshold. A safe policy is then said to be a policy that always leads to known non-error final states. To find such policy, the policy iteration is initialized with the safe backup policy and exploration is done via adding a small amount of Gaussian noise to the actions. This approach is suitable for continuous state-and action-spaces. Another approach is presented in the work of Geibel  [12] , where the risk and objective functions are treated separately. So the risk function only classifies the states (again only final states) as either fatal or goal, and the risk of a policy (risk function) is then computed as the expected risk following the policy (where fatal states have risk 1 and goal states have risk 0). The task is then said to be to maximize the objective function (e.g. discounted infinite horizon) w.r.t. the condition that the risk of the considered policies is less than a safety threshold. The optimization itself is done using modified Q-learning, and the optimized objective function is a linear combination of the original objective function and the risk function. By changing the weights in the linear combination the algorithm can be controlled to behave more safely or in a more risk-neutral way. A generalization of Geibel's idea to take the risk and reward functions separately can be found in the work of Kim  [18] . In this work, the constrained RL task is treated as a Constrained MDP and the algorithm CBEETLE for solving the Constrained MDPs is shown. The advantage of this work is that it allows for several independent risk (cost) functions and doesn't need to convert them to the same scale. A similar approach of using constrained MDP to solve the problem can be found in the work of Moldovan  [20] . He does, however, use the ergodicity condition to tell safe and unsafe states apart (that is, safe are only those states from which the agent can get back to the initial state). Moreover, this approach is only shown to work for toy examples like the grid world with only several thousands of discrete states, which may not be sufficient for real robotics tasks. The idea of having several risk functions is further developed by Ertle  [9] . The agent is told to have several behaviors and a separate safety function is learned for each behavior. This approach allows for modularity and sharing of the learned safety functions among different types of agents. More details on this work will be provided in the next section, because it belongs to learning with teachers. An approach slightly different from the previously mentioned in this section is using the methods of reachability analysis to solve safe exploration. Gillula in his work  [13]  defines a set of keep-out states (corresponding to unsafe in our labeling) and then a set called P re(τ ) is defined as a set of all states from which it is possible to get to a keep-out state in less than τ steps. Reachability analysis is used to compute the P re(τ ) set. Safe states are then all states not in P re(τ ) for a desired τ . This approach, however, doesn't utilize reinforcement learning, it computes the optimal policy using standard supervised learning methods with one additional constraint -that the system must use safe actions near the P re(τ ) set. On the other hand, the system is free to use whatever action desired when it is not near P re(τ ). As was presented in this section, the labeling-based approaches provide a number of different ways to reach safety in exploration. They are, however, limited in several ways -some of them make use of the (usually hard-to-obtain) transition matrix, the others may need to visit the unsafe states in order to learn how to avoid them, or need the state-space to be metric.
 Approaches benefiting from prior knowledge The last large group of safe exploration techniques are the ones benefiting from various kinds of prior knowledge (other than the parameters of the MDP). We consider this group the most promising for safe exploration, because "it is impossible to avoid undesirable situations in high-risk environments without a certain amount of prior knowledge about the task"  [10] . The first option how to incorporate prior knowledge into exploration is to initialize the search using the prior knowledge. In fact, several works already mentioned in previous sections use prior knowledge -namely the approaches with a backup policy  (Hans [14] , García  [10] ). Also, García suggests that the initial estimate of the value function can be done by providing prior knowledge, which results in much faster convergence (since the agent does no more have to explore really random actions, the estimate of the value function already "leads it" the right way)  [10] . Another option how to incorporate prior knowledge is by using Learning from Demonstration (LfD) methods. Due to the limited space, we will not give the basics of LfD -a good overview of the state-of-the-art methods is for example in  [2] . For our overview, it is sufficient to state that LfD methods can derive a policy from a set of demonstrations provided by a teacher. What is important, is that the teacher does not necessarily have to have the same geometrical and physical properties as the trainee (although it helps the process if possible). It is therefore possible to use LfD to teach a 5-joint arm to play tennis, while using 3-joint human arm as the source of demonstrations (but the learned policy may be suboptimal; RL should then be used to optimize the policy). In Apprenticeship Learning  [1] , the reward function is learned using LfD. The human pilot flies a helicopter at his best, and both system dynamics and the reward function are learned from the demonstrations. It is however apparent that the performance of the agent is no longer objectively optimal, but that it depends on the abilities of the human pilot. Another way of incorporating prior knowledge into the learning process is to manually select which demonstrations will be provided, as in the work of Ertle  [9] . In the work it is suggested that more teacher demonstrations should come from the areas near the unsafe set, in order to teach the agent precisely where the border between safe and unsafe is located. The last technique described in our overview is interleaving autonomous exploration with teacher demonstrations. As in the previous case, some teacher demonstrations are provided in advance, and then the exploration part starts utilizing the teacher-provided information. After some time, or in states very different from all other known states, the agent requests the teacher to provide more examples  [2, 5] . The idea behind this algorithm is that it is impossible to think out in advance what all demonstrations will the agent need in order to learn the optimal policy. Finishing this section, the algorithms utilizing prior knowledge seem to be the most promising out of all the presented approaches. They provide both a speedup of the learning process (by discarding the low-reward areas) and a reasonable way to specify the safety conditions (via LfD or interleaving).
 Conclusion In our work we have given a short introduction on the basics of Markov Decision Processes as well as the basic Reinforcement Learning methods like Value Iteration, Q-learning and Policy Iteration. In Section 3 we have summarized many recent approaches on how to define safety in the framework of optimal control and reinforcement learning. We have also proposed a novelty definition of safety, which divides the state space to safe, critical and unsafe states. We have shown that all other labeling-based safety definitions are covered by our new definition. In Section 4 many different safe exploration methods are categorized into three basic groups -algorithms from optimal control theory, reinforcement learning algorithms based on state labeling, and algorithms utilizing extra prior knowledge. We have shortly summarized the advantages and disadvantages of the particular approaches. We have also stated that at least for difficult real-world problems, safe exploration without prior knowledge is practically impossible, and prior knowledge almost always helps to achieve faster convergence. Another observation has been that some of the safe exploration algorithms need to visit unsafe states to correctly classify them later, which might discard them from some usage scenarios where the unsafe states are really fatal. It seems to us that the field of safe exploration in reinforcement learning has been very fragmented and lacks an all-embracing theory. However, the question is, if it is even possible to find such theory -the main problem may be the fragmentation and differences of various RL methods themselves. At least, the safe exploration community would benefit from a unification of the terminology (and our proposal of the novelty safety labeling would like to help that). Other ways of possible future research are for example the following. New ways of incorporating prior knowledge into methods not utilizing it yet could bring interesting speed-up of those algorithms. There is also a bottleneck in the estimation of the results of unknown actions -some advanced function approximation methods should be explored (we aim to investigate Gaussian Processes this way). There are not enough experiments from difficult continuous real-world environments, which would show for example how large problems can be solved using safe exploration. The interleaved learning needs some guidelines on how to cluster the queries for the teacher to some larger "packs" and "ask" them together, possibly increasing the fully autonomous operating time. Last, but not least, the possibility to share some learned safety functions among different kinds of robots seems to be an unexplored area with many practical applications (maybe robot-to-robot LfD could be used). Fig. 1 . 1 Fig. 1. NIFTi UGV robotic platform
 (a) A safe state. (b) A critical state -if the robot went still forward, it would fall down and probably break.
 Fig. 2 .Fig. 3 . 23 Fig. 2. An illustration of safe and critical states.
 action a is supercritical in state s if it can lead to a supercritical state, state s is called critical if there is a supercritical or fatal action in that state (and the state itself is not supercritical), action a is critical in state s if it leads to a critical state (and the action itself is neither supercritical nor fatal in s), state s is called safe if it is neither critical nor supercritical, action a is safe in state s if it is neither critical, nor supercritical, nor fatal in state s, and finally a policy is safe if for all critical states it leads to a safe state in a finite number of non-fatal transitions (and if it only executes safe actions in safe states).

Abstract: We overview different approaches to safety in (semi)autonomous robotics. Particularly, we focus on how to achieve safe behavior of a robot if it is requested to perform exploration of unknown states. Presented methods are studied from the viewpoint of reinforcement learning, a partially-supervised machine learning method. To collect training data for this algorithm, the robot is required to freely explore the state space -which can lead to possibly dangerous situations. The role of safe exploration is to provide a framework allowing exploration while preserving safety. The examined methods range from simple algorithms to sophisticated methods based on previous experience or state prediction. Our overview also addresses the issues of how to define safety in the real-world applications (apparently absolute safety is unachievable in the continuous and random real world). In the conclusion we also suggest several ways that are worth researching more thoroughly.


Feasibility of Training an AGI using Deep Reinforcement Learning, A Very Rough Estimate.tei.xml