Chongli Qin
James Martens
Sven Gowal
Dilip Krishnan
Google Krishnamurthy
Dvijotham Deepmind
Alhussein Fawzi
Soham De
Robert Stanforth Deepmind

Adversarial Robustness through Local Linearization

Introduction In a seminal paper, Szegedy et al.  [22]  demonstrated that neural networks are vulnerable to visually imperceptible but carefully chosen adversarial perturbations which cause them to output incorrect predictions. After this revealing study, a flurry of research has been conducted with the focus of making networks robust against such adversarial perturbations  [14, 16, 17, 25] . Concurrently, researchers devised stronger attacks that expose previously unknown vulnerabilities of neural networks  [24, 4, 1, 3] . Of the many approaches proposed  [19, 2, 6, 21, 15, 17] , adversarial training  [14, 16]  is empirically the best performing algorithm to train networks robust to adversarial perturbations. However, the cost of adversarial training becomes prohibitive with growing model complexity and input dimensionality. This is primarily due to the cost of computing adversarial perturbations, which is incurred at each step of adversarial training. In particular, for each new mini-batch one must perform multiple iterations of a gradient-based optimizer on the network's inputs to find the perturbations.  1  As each step of this optimizer requires a new backwards pass, the total cost of adversarial training scales as roughly the number of such steps. Unfortunately, effective adversarial training of ImageNet often requires large number of steps to avoid problems of gradient obfuscation  [1, 24] , making it significantly more expensive than conventional training. One approach which can alleviate the cost of adversarial training is training against weaker adversaries that are cheaper to compute. For example, by taking fewer gradient steps to compute adversarial examples during training. However, this can produce models which are robust against weak attacks, but break down under strong attacks -often due to gradient obfuscation. In particular, one form of gradient obfuscation occurs when the network learns to fool a gradient based attack by making the loss surface highly convoluted and non-linear (see  Fig 1) , an effect which has also been observed by Papernot et al  [18] . This non-linearity prevents gradient based optimization methods from finding an adversarial perturbation within a small number of iterations  [4, 24] . In contrast, if the loss surface was linear in the vicinity of the training examples, which is to say well-predicted by local gradient information, gradient obfuscation cannot occur. In this paper, we take up this idea and introduce a novel regularizer that encourages the loss to behave linearly in the vicinity of the training data. We call this regularizer the local linearity regularizer (LLR). Empirically, we find that networks trained with LLR exhibit far less gradient obfuscation, and are almost equally robust against strong attacks as they are against weak attacks. The main contributions of our paper are summarized below: • We show that training with LLR is significantly faster than adversarial training, allowing us to train a robust ImageNet model with a 5× speed up when training on 128 TPUv3 cores  [9] . • We show that LLR trained models exhibit higher robustness relative to adversarially trained models when evaluated under strong attacks. Adversarially trained models can exhibit a decrease in accuracy of 6% when increasing the attack strength at test time for CIFAR-10, whereas LLR shows only a decrease of 2%. • We achieve new state of the art results for adversarial accuracy against untargeted white-box attack for ImageNet (with = 4/255 2 ): 47%. Furthermore, we match state of the art results for CIFAR 10 (with = 8/255): 52.81% 3 . • We perform a large scale evaluation of existing methods for adversarially robust training under consistent, strong, white-box attacks. For this we recreate several baseline models from the literature, training them both for CIFAR-10 and ImageNet (where possible).  4  2 Background and Related Work We denote our classification function by f (x; θ) : x → R C , mapping input features x to the output logits for classes in set C, i.e. p i (y|x; θ) = exp (f i (x; θ)) / j exp (f j (x; θ)), with θ being the model parameters and y being the label. Adversarial robustness for f is defined as follows: a network is robust to adversarial perturbations of magnitude at input x if and only if argmax i∈C f i (x; θ) = argmax i∈C f i (x + δ; θ) ∀δ ∈ B p ( ) = {δ : δ p ≤ }. (1) In this paper, we focus on p = ∞ and we use B( ) to denote B ∞ ( ) for brevity. Given the dataset is drawn from distribution D, the standard method to train a classifier f is empirical risk minimization (ERM), which is defined by: min θ E (x,y)∼D [ (x; y, θ)]. Here, (x; y, θ) is the standard cross-entropy loss function defined by (x; y, θ) = −y T log (p(x; θ)) , (2) where p i (x; θ) is defined as above, and y is a 1-hot vector representing the class label. While ERM is effective at training neural networks that perform well on heldout test data, the accuracy on the test set goes to zero under adversarial evaluation. This is a result of a distribution shift in the data induced by the attack. To rectify this, adversarial training  [17, 14]  seeks to perturb the data distribution by performing adversarial attacks during training. More concretely, adversarial training minimizes the loss function E (x,y)∼D max δ∈B( ) (x + δ; y, θ) , (3) where the inner maximization, max δ∈B( ) (x + δ; y, θ), is typically performed via a fixed number of steps of a gradient-based optimization method. One such method is Projected-Gradient-Descent (PGD) which performs the following gradient step: δ ← Proj (δ − η∇ δ (x + δ; y, θ)) , (4) where Proj(x) = argmin ξ∈B( ) x − ξ . Another popular gradient-based method is to use the sign of the gradient  [8] . The cost of solving Eq (3) is dominated by the cost of solving the inner maximization problem. Thus, the inner maximization should be performed efficiently to reduce the overall cost of training. A naive approach is to reduce the number of gradient steps performed by the optimization procedure. Generally, the attack is weaker when we do fewer steps. If the attack is too weak, the trained networks often display gradient obfuscation as shown in Fig  1 . Since the introduction of adversarial training, a corpus of work has researched alternative ways of making networks robust. One such approach is the TRADES method  [27] , which is a form of regularization that optimizes the trade-off between robustness and accuracy -as many studies have observed these two quantities to be at odds with each other  [23] . Others, such as work by Ding et al  [7]  adaptively increase the perturbation radius by find the minimal length perturbation which changes the output label. Some have proposed architectural changes which promote adversarial robustness, such as the "denoise" model  [25]  for ImageNet. The work presented here is a regularization technique which encourages the loss function to be well approximated by its linear Taylor expansion in a sufficiently small neighbourhood. There has been work before which uses gradient information as a form of regularization  [20, 17] . The work presented in this paper is closely related to the paper by Moosavi et al  [17] , which highlights that adversarial training reduces the curvature of (x; y, θ) with respect to x. Leveraging an empirical observation (the highest curvature is along the direction ∇ x (x; y, θ)), they further propose an algorithm to mimic the effects of adversarial training on the loss surface. The algorithm results in comparable performance to adversarial training with a significantly lower cost.
 Motivating the Local Linearity Regularizer As described above, the cost of adversarial training is dominated by solving the inner maximization problem max δ∈B( ) (x + δ). Throughout we abbreviate (x; y, θ) with (x). We can reduce this cost simply by reducing the number of PGD (as defined in Eq (4)) steps taken to solve max δ∈B( ) (x + δ). To motivate the local linearity regularizer (LLR), we start with an empirical analysis of how the behavior of adversarial training changes as we increase the number of PGD steps used during training. We find that the loss surface becomes increasingly linear (as captured by the local linearity measure defined below) as we increase the number of PGD steps.
 Local Linearity Measure Suppose that we are given an adversarial perturbation δ ∈ B( ). The corresponding adversarial loss is given by (x + δ). If our loss surface is smooth and approximately linear, then (x + δ) is well approximated by its first-order Taylor expansion (x) + δ T ∇ x (x). In other words, the absolute difference between these two values, g(δ; x) = (x + δ) − (x) − δ T ∇ x (x) , (5) is an indicator of how linear the surface is. Consequently, we consider the quantity γ( , x) = max δ∈B( ) g(δ; x), (6) to be a measure of how linear the surface is within a neighbourhood B( ). We call this quantity the local linearity measure.   6 )) is large (on the order of 10) when we train with just one or two steps of PGD for inner maximization, (2a). In contrast, γ( , x) becomes increasingly smaller (on the order of 10 −1 ) as we increase the number of PGD steps to 4 and above, (2b). The x-axis is the number of training iterations and the y-axis is γ( , x), here = 8/255 for CIFAR-10.
 Empirical Observations on Adversarial Training We measure γ( , x) for networks trained with adversarial training on CIFAR-10, where the inner maximization max δ∈B( ) (x + δ) is performed with 1, 2, 4, 8 and 16 steps of PGD. γ( , x) is measured throughout training on the training set  5  . The architecture used is a wide residual network  [26]  28 in depth and 10 in width (Wide-ResNet-28-10). The results are shown in Fig  2a and 2b . Fig  2a  shows that when we train with one and two steps of PGD for the inner maximization, the local loss surface is extremely non-linear at the end of training. An example visualization of such a loss surface is given in Fig A1a  .  However, when we train with four or more steps of PGD for the inner maximization, the surface is relatively well approximated by (x) + δ T ∇ x (x) as shown in Fig 2b  .  An example of the loss surface is shown in Fig A1b  .  For the adversarial accuracy of the networks, see Table  A1 .
 Local Linearity Regularizer (LLR) From the section above, we make the empirical observation that the local linearity measure γ( , x) decreases as we train with stronger attacks  6  . In this section, we give some theoretical justifications of why local linearity γ( , x) correlates with adversarial robustness, and derive a regularizer from the local linearity measure that can be used for training of robust models.
 Local Linearity Upper Bounds Adversarial Loss The following proposition establishes that the adversarial loss (x + δ) is upper bounded by the local linearity measure, plus the change in the loss as predicted by the gradient (which is given by |δ T ∇ x (x)|). Proposition 4.1. Consider a loss function (x) that is once-differentiable, and a local neighbourhood defined by B( ). Then for all δ ∈ B( ) | (x + δ) − (x)| ≤ |δ T ∇ x (x)| + γ( , x). (7) See Appendix B for the proof. From Eq  (7)  it is clear that the adversarial loss tends to (x), i.e., (x + δ) → (x), as both |δ ∇ x (x)| → 0 and γ( ; x) → 0 for all δ ∈ B( ). And assuming (x + δ) ≥ (δ) one also has the upper bound (x + δ) ≤ (x) + |δ T ∇ x (x)| + γ( , x).
 Local Linearity Regularization (LLR) Following the analysis above, we propose the following objective for adversarially robust training L(D) = E D (x) + λγ( , x) + µ|δ T LLR ∇ x (x)| LLR , (8) where λ and µ are hyper-parameters to be optimized, and δ LLR = argmax δ∈B( ) g(δ; x) (recall the definition of g(δ; x) from Eq (5)). Concretely, we are trying to find the point δ LLR in B( ) where the linear approximation (x) + δ T ∇ x (x) is maximally violated. To train we penalize both its linear violation γ( , x) = (x + δ LLR ) − (x) − δ T LLR ∇ x (x) , and the gradient magnitude term δ T LLR ∇ x (x) , as required by the above proposition. We note that, analogous to adversarial training, LLR requires an inner optimization to find δ LLR -performed via gradient descent. However, as we will show in the experiments, much fewer optimization steps are required for the overall scheme to be effective. Pseudo-code for training with this regularizer is given in Appendix E.
 Local Linearity Measure γ( ; x) bounds the adversarial loss by itself Interestingly, under certain reasonable approximations and standard choices of loss functions, we can bound |δ ∇ x (x)| in terms of γ( ; x). See Appendix C for details. Consequently, the bound in Eq  (7)  implies that minimizing γ( ; x) (along with the nominal loss (x)) is sufficient to minimize the adversarial loss (x + δ). This prediction is confirmed by our experiments. However, our experiments also show that including |δ ∇ x (x)| in the objective along with (x) and γ( ; x) works better in practice on certain datasets, especially ImageNet. See Appendix F.3 for details.
 Experiments and Results We perform experiments using LLR on both CIFAR-10 [13] and ImageNet  [5]  datasets. We show that LLR gets state of the art adversarial accuracy on CIFAR-10 (at = 8/255) and ImageNet (at = 4/255) evaluated under a strong adversarial attack. Moreover, we show that as the attack strength increases, the degradation in adversarial accuracy is more graceful for networks trained using LLR than for those trained with standard adversarial training. Further, we demonstrate that training using LLR is 5× faster for ImageNet. Finally, we show that, by linearizing the loss surface, models are less prone to gradient obfuscation.
 CIFAR-10: The perturbation radius we examine is = 8/255 and the model architectures we use are Wide-ResNet-28-8, Wide-ResNet-40-8  [26] . Since the validity of our regularizer requires (x) to be smooth, the activation function we use is softplus function log(1 + exp(x)), which is a smooth version of ReLU. The baselines we compare our results against are adversarial training (ADV)  [16] , TRADES  [27]  and CURE  [17] . We recreate these baselines from the literature using the same network architecture and activation function. The evaluation is done on the full test set of 10K images.
 ImageNet: The perturbation radii considered are = 4/255 and = 16/255. The architecture used for this is from  [11]  which is ResNet-152. We use softplus as activation function. For = 4/255, the baselines we compare our results against is our recreated versions of ADV  [16]  and denoising model (DENOISE)  [25] .  7  For = 16/255, we compare LLR to ADV  [16]  and DENOISE  [25]  networks which have been published from the the literature. Due to computational constraints, we limit ourselves to evaluating all models on the first 1K images of the test set. To make sure that we have a close estimate of the true robustness, we evaluate all the models on a wide range of attacks these are described below.
 Evaluation Setup To accurately gauge the true robustness of our network, we tailor our attack to give the lowest possible adversarial accuracy. The two parts which we tune to get the optimal attack is the loss function for the attack and its corresponding optimization procedure. The loss functions used are described below, for the optimization procedure please refer to Appendix F.1.
 Loss Functions: The three loss functions we consider are summarized in Table  1 . We use the difference between logits for the loss function rather than the cross-entropy loss as we have empirically found the former to yield lower adversarial accuracy.
 Attack Name Loss Function Metric Random-Targeted max δ∈B( ) f r (x + δ) − f t (x + δ) Attack Success Rate Untargeted max δ∈B( ) f s (x + δ) − f t (x + δ) Adversarial Accuracy Multi-Targeted [10] max δ∈B( ) max i∈C f i (x + δ) − f t (x + δ) Adversarial Accuracy Table  1 : This shows the loss functions corresponding to the attacks we use for evaluation and also the metric we measure on the test set for each of these attacks. Notation-wise, s = argmax i =t fi(x + δ) is the highest logit excluding the logits corresponding to the correct class t, note s can change through the optimization procedure. For the Random-Targeted attack, r is a randomly chosen target label that is not t and does not change throughout the optimization. C stands for the set of class labels. For the Multi-Targeted attack we maximize fi(x + δ) − fT (x + δ) for all i ∈ C, and consider the attack successful if any of the individual attacks on each each target class i are successful. The metric used on the Random-Targeted attack is the attack success rate: the percentage of attacks where the target label r is indeed the output label (this metric is especially important for ImageNet at = 16/255). For the other attacks we use the adversarial accuracy as the metric which is the accuracy on the test set after the attack. For CIFAR-10, the main adversarial accuracy results are given in Table  2 . We compare LLR training to ADV  [16] , CURE  [17]  and TRADES  [27] , both with our re-implementation and the published models  8  . Note that our re-implementation using softplus activations perform at or above the published results for ADV, CURE and TRADES. This is largely due to the learning rate schedule used, which is the similar to the one used by TRADES  [27] .
 Results for Interestingly, for adversarial training (ADV), using the Multi-Targeted attack for evaluation gives significantly lower adversarial accuracy compared to Untargeted. For ImageNet, we compare against adversarial training (ADV)  [16]  and the denoising model (DE-NOISE)  [25] . The results are shown in Table  3 . For a perturbation radius of 4/255, LLR gets 47% adversarial accuracy under the Untargeted attack which is notably higher than the adversarial accuracy obtained via adversarial training which is 39.70%. Moreover, LLR is trained with just two-steps of PGD rather than 30 steps for adversarial training. The amount of computation needed for each method is further discussed in Sec 5.2.1. Further shown in Table  3  are the results for = 16/255. We note a significant drop in nominal accuracy when we train with LLR to perturbation radius 16/255. When testing for perturbation radius of 16/255 we also show that the adversarial accuracy under Untargeted is very poor (below 8%) for all methods. We speculate that this perturbation radius is too large for the robustness problem. Since adversarial perturbations should be, by definition, imperceptible to the human eye, upon inspection of the images generated using an adversarial attack (see Fig  F4 ) -this assumption no longer holds true. The images generated appear to consist of super-imposed object parts of other classes onto the target image. This leads us to believe that a more fine-grained analysis of what should constitute "robustness for ImageNet" is an important topic for debate.
 Runtime Speed For ImageNet, we trained on 128 TPUv3 cores  [9] , the total training wall time for the LLR network (4/255) is 7 hours for 110 epochs. Similarly, for the adversarially trained (ADV) networks the total wall time is 36 hours for 110 epochs. This is a 5× speed up.
 Accuracy Degradation: Strong vs Weak Evaluation The resulting model trained using LLR degrades gracefully in terms of adversarial accuracy when we increase the strength of attack, as shown in Fig 3  .  In particular, Fig  3a  shows that, for CIFAR-10, when the attack changes from Untargeted to Multi-Targeted, the LLR's accuracy remains similar with only a 2.18% drop in accuracy. Contrary to adversarial training (ADV), where we see a 5.64% drop in accuracy. We also see similar trends in accuracy in Table  2 . This could indicate that some level of obfuscation may be happening under standard adversarial training. As we empirically observe that LLR evaluates similarly under weak and strong attacks, we hypothesize that this is because LLR explicitly linearizes the loss surface. An extreme case would be when the surface is completely linear -in this instance the optimal adversarial perturbation would be found with just one PGD step. Thus evaluation using a weak attack is often good enough to get an accurate gauge of how it will perform under a stronger attack.  We use either the standard adversarial training objective (ADV-1, ADV-2) or the LLR objective (LLR-1, LLR-2) and taking one or two steps of PGD to maximize each objective. To train LLR-1/2, we only optimize the local linearity γ( , x), i.e. µ in Eq. (  8 ) is set to zero. We see that for adversarial training, as shown in Figs 4a, 4c, the loss surface becomes highly non-linear and jagged -in other words obfuscated. Additionally in this setting, the adversarial accuracy under our strongest attack is 0% for both, see Table  F3 . In contrast, the loss surface is smooth when we train using LLR as shown in Figs  4b, 4d . Further, Table  F3  shows that we obtain an adversarial accuracy of 44.50% with the LLR-2 network under our strongest evaluation. We also evaluate the values of γ( , x) for the CIFAR-10 test set after these networks are trained. This is shown in
 Conclusions We show that, by promoting linearity, deep classification networks are less susceptible to gradient obfuscation, thus allowing us to do fewer gradient descent steps for the inner optimization. Our novel linearity regularizer promotes locally linear behavior as justified from a theoretical perspective. The resulting models achieve state of the art adversarial robustness on the CIFAR-10 and Imagenet datasets, and can be trained 5× faster than regular adversarial training. Figure 1 : 1 Figure 1: Example of gradient obfuscated surface. The color of the surface denotes the prediction of the network.
 Figure 2 : 2 Figure 2: Plots showing that γ( , x) (Eq (6)) is large (on the order of 10) when we train with just one or two
 2 Figure 4 : 24 Figure 4: Comparing the loss surface, (x), after we train using just 1 or 2 steps of PGD for the inner maximization of either the adversarial objective (ADV) max δ∈B( ) (x + δ) or the linearity objective (LLR) γ( , x) = max δ∈B( ) (x + δ) − (x) − δ T ∇ (x) . Results are shown for image 126 in test set of CIFAR-10, the nominal label is deer. ADV-i refers to adversarial training with i PGD steps, similarly with LLR-i.
 Fig F3. The values of γ( , x) are comparable when we train with LLR using two steps of PGD to adversarial training with 20 steps of PGD. By comparison, adversarial training with two steps of PGD results in much larger values of γ( , x).
 Table 2 : 2 Model accuracy results for CIFAR-10. Our LLR regularizer performs the best under the strongest attack (highlighted column). (S) denotes softplus activation; (R) denotes ReLU activation; and models with (S, R) are our implementations. Robustness CIFAR-10: Wide-ResNet-28-8 (8/255) Methods Nominal FGSM-20 Untargeted Multi-Targeted Attack Strength Weak Strong Very Strong ADV[16] 87.25% 48.89% 45.92% 44.54% CURE[17] 80.76% 39.76% 38.87% 37.57% ADV(S) 85.11% 56.76% 53.96% 48.79% CURE(S) 84.31% 48.56% 47.28% 45.43% TRADES(S) 87.40% 51.63 50.46% 49.48% LLR (S) 86.83% 54.24% 52.99% 51.13% CIFAR-10: Wide-ResNet-40-8 (8/255) ADV(R) 85.58% 56.32% 52.34% 46.89% TRADES(R) 86.25% 53.38% 51.76% 50.84% ADV(S) 85.27% 57.94% 55.26% 49.79% CURE(S) 84.45% 49.41% 47.69% 45.51% TRADES(S) 88.11% 53.03% 51.65% 50.53% LLR (S) 86.28% 56.44% 54.95% 52.81%
 Table 3 : 3 The accuracy obtained are 49.79% and 55.26% respectively. Evaluation using Multi-Targeted attack consistently gave the lowest adversarial accuracy throughout. Under this attack, the methods which stand out amongst the rest are LLR and TRADES. Using LLR we get state of the art results with 52.81% adversarial accuracy. LLR gets 47% adversarial accuracy for 4/255 -7.30% higher than DENOISE and ADV. For 16/255, LLR gets similar robustness results, but it comes at a significant cost to the nominal accuracy. Note Multi-Targeted attacks for ImageNet requires looping over 1000 labels, this evaluation can take up to several days even on 50 GPUs thus is omitted from this table. The column of the strongest attack is highlighted. ImageNet: ResNet-152 (4/255) Methods PGD steps Nominal Untargeted Random-Targeted Accuracy Success Rate ADV 30 69.20% 39.70% 0.50% DENOISE 30 69.70% 38.90% 0.40% LLR 2 72.70% 47.00% 0.40% ImageNet: ResNet-152 (16/255) ADV [25] 30 64.10% 6.30% 40.00% DENOISE [25] 30 66.80% 7.50% 38.00% LLR 10 51.20% 6.10% 43.80%
 The annotations on each node denotes no. of PGD steps × no. of random restarts (see Appendix F.1). (3a), background color denotes whether the attack is Untargeted (blue) or Multi-Targeted (orange). (3b), we only use Untargeted attacks. Adversarial Accuracy 48 50 52 54 56 58 60 62 1 × 1 1 × 1 10 × 1 10 × 1 Stronger Attack → Untargeted 20 × 1 20 × 1 50 × 1 50 × 1 100 × 1 100 × 1 200 × 1 200 × 1 Multi Targeted 1000 × 20 1000 × 20 1000 × 1 1000 × 1 200 × 20 200 × 20 LLR ADV Adversarial Accuracy 38 40 42 44 46 48 50 1 × 1 1 × 1 10 × 1 10 × 1 Stronger Attack → 20 × 1 50 × 1 20 × 1 50 × 1 100 × 1 100 × 1 1000 × 1 200 × 1 1000 × 1 200 × 1 LLR ADV (a) CIFAR-10 (8/255) (b) ImageNet (4/255) Figure 3: Adversarial accuracy shown for CIFAR-10, (3a), and ImageNet, (3b), as we increase the strength of attack. (3a) shows LLR's adversarial accuracy degrades gracefully going from 53.32% to 51.14% (-2.18%) while ADV's adversarial accuracy drops from 54.43% to 48.79% (-5.64%). (3b) LLR remains 7.5% higher in terms of adversarial accuracy (47.20%) compared to ADV (39.70%). For ImageNet, see Fig3b, the adversarial accuracy trained using LLR remains significantly higher (7.5%) than the adversarially trained network going from a weak to a stronger attack.
			 While computing the globally optimal adversarial example is NP-hard [12] , gradient descent with several random restarts was empirically shown to be quite effective at computing adversarial perturbations of sufficient quality. 2  This means that every pixel is perturbed independently by up to 4 units up or down on a scale where pixels take values ranging between 0 and 255. 3  We note that TRADES [27]  gets 55% against a much weaker attack; under our strongest attack, it gets 52.5%. 4  Baselines created are adversarial training, TRADES and CURE [17] . Contrary to CIFAR-10, we are currently unable to achieve consistent and competitive results on ImageNet at = 4/255 using TRADES.
			 To measure γ( , x) we find max δ∈B( ) g(δ; x) with 50 steps of PGD using Adam as the optimizer and 0.1 as the step size. 6  Here, we imply an increase in the number of PGD steps for the inner maximization max δ∈B( ) (x + δ).
			 We attempted to use TRADES on ImageNet but did not manage to get competitive results. Thus they are omitted from the baselines.
			 Note the network published for TRADES [27]  uses a Wide-ResNet-34-10 so this is not shown in the table but under the same rigorous evaluation we show that TRADES get 84.91% nominal accuracy, 53.41% under Untargeted and 52.58% under Multi-Targeted. We've also ran ∞ DeepFool (not in the table as the attack is weaker) giving ADV(S): 64.29%, CURE(S): 58.73%, TRADES(S): 63.4%, LLR(S): 65.87%.

Abstract: Adversarial training is an effective methodology to train deep neural networks which are robust against adversarial, norm-bounded perturbations. However, the computational cost of adversarial training grows prohibitively as the size of the model and number of input dimensions increase. Further, training against less expensive and therefore weaker adversaries produces models that are robust against weak attacks but break down under attacks that are stronger. This is often attributed to the phenomenon of gradient obfuscation; such models have a highly non-linear loss surface in the vicinity of training examples, making it hard for gradient-based attacks to succeed even though adversarial examples still exist. In this work, we introduce a novel regularizer that encourages the loss to behave linearly in the vicinity of the training data, thereby penalizing gradient obfuscation while encouraging robustness. We show via extensive experiments on CIFAR-10 and ImageNet, that models trained with our regularizer avoid gradient obfuscation and can be trained significantly faster than adversarial training. Using this regularizer, we exceed current state of the art and achieve 47% adversarial accuracy for ImageNet with ∞ adversarial perturbations of radius 4/255 under an untargeted, strong, white-box attack. Additionally, we match state of the art results for CIFAR-10 at 8/255.


AI transparency: a matter of reconciling design with critique

In the age of ubiquitous computing, we are surrounded by objects that incorporate artificial intelligence solutions. We interact with different kinds of AI without realizing itusing online banking systems, searching for YouTube clips, or consuming news through social media-not really knowing how and when AI systems operate. Corporate strategies of secrecy and user interfaces that hide traces of AI-driven personalization combine with the inherent opacity of deep learning algorithms (whose inner workings are not directly comprehensible to human interpreters) to create a marked lack of transparency associated with all aspects of emerging technologies. It is in response to the widespread application of AI-based solutions to various products and services in the late 2010s that multiple expert groups-both national and international-have voiced the demand to 'open' the algorithmic black box, to audit, expound, and demystify AI. They claim that to ensure that the use of AI is ethical, we must design emerging systems to be transparent, explainable, and auditable.  1  The opening of the algorithmic black box, however, cannot be seen only as an engineering challenge. It is critique, as the underside of making, that prioritizes unboxing, debunking the illusion, seeing through-to reveal how an object really works. Critique-grounded in the tradition of Critical Theory and practiced by cultural studies, critical race theory, queer theory, as well as decolonial theory scholars, among others-moves beyond the technical detail to uncover the desires, ideologies, and social relations forged into objects, opening the black boxes of history, culture, and progress. In what follows, I argue that the calls for technological transparency demand that we combine the practice of design with critique. I relate the question of AI transparency to the broader challenge of responsible making, contending that future action must aim to systematically reconcile design-as a way of concealing-with critique-as a manner of revealing.
 Levels of opacity Many companies sell simple analytics tools as artificial intelligence, as something that supposedly supplants human intelligence to deliver better results. What is advertised as an 'AI solution' often relies on simple data analysis performed by human analysts. The use of metaphors and simplifications obfuscates human labor, labor that is outsourced, hidden away, in an invisible and immaterial factory, in a different part of the globe. According to Ian  Bogost (2015) , the 'metaphor of mechanical automation' is nothing more than a welldirected, but misleading, masquerade (n. pag.). Although the metaphor is only an approximation, a distortion, or even a 'caricature,' it convincingly plays the role of an accurate depiction of the whole. The term artificial intelligence, elusive, misleading and with a definition that has changed over time, forms the basis of the marketing stunt. Technology companies rely on abstract, overly schematic representations to simplify reality and arrive at an easily digestible, prepacked idea of the object, one that misrepresents the object's essence and overlooks its true composition, but also satiates the end user's curiosity. Other systems, as a matter of deliberate practice, incorporate complex data processing and machine learning surreptitiously. Shoshana  Zuboff (2019)  observes that the influence these systems have on our decision-making is 'designed to be unknowable to us  ' (p.11) ; that company strategies of misdirection serve as 'the moat that surrounds the castle and secures the action within' (p.65.)-a way for corporations like Google or Facebook to protect their secrets and mislead the public. These systems could, in theory, be designed to be more 'knowable.' Elements of the user interface could, for example, flag up when algorithmic operations are influencing the user's decision-making. Just like labels inform the consumer about the product's contents, the interfaces of Facebook and YouTube could announce to users that the information delivered by the platforms is algorithmically curated. Considering that only 24% of US college students realize Facebook automatically prioritizes certain posts and hides others  (Powers 2017) , such a feature would definitely be relevant. But the challenge of transparency at a time of unprecedented technological complexity cannot be approached only as a matter of failed (or indeed successful, depending on your position) communication. The 'opacity' of machine learning algorithms refers, after all, not only to 'institutional self-protection and concealment,' but also, as  Burrell (2016)  points out, to 'the mismatch between mathematical optimization in highdimensionality characteristic of machine learning and the demands of human-scale reasoning and styles of semantic interpretation' (p.2). The fundamental lack of transparency of systems that incorporate AI solutions relates not only to convoluted storytelling devised by marketing teams, or misleading interfaces and user experience design, but also-and most importantly-an emerging form of making brought about by the automation of cognitive tasks themselves. Considering this level of opacity of AI systems, a team of researchers from MIT's Intelligence and Decision Technologies Group developed a neural network named, suggestively, Transparency by Design Network  (Mascharka et al. 2018)  that not only performs 'human-like reasoning steps' to answer questions about the contents of images, but also 'visually renders its thought process' as it solves problems, allowing human analysts to interpret its decision-making. It is a deep learning system carrying out a sort of unboxing on itself, incorporating explainability in its very operations: a realization of the most common understanding of 'transparency' in the context of contemporary AI research.
 Transparency and critique The challenge of technological transparency that has now attracted the attention of policymakers has constituted a concern for cultural studies scholars, media theorists, and design philosophers for decades. In the early 1990s, the philosopher Paul Virilio (1994) noted that once 'synthetic vision' becomes a reality and the human subject is excluded from the process of observation of images 'created by the machine for the machine,' the power of statistical science to appear objective (and thus persuasive) will be 'considerably enhanced, along with its discrimination capacities' (p.75). Another prominent media theorist Friedrich  Kittler (2014)  expressed concerns about modern media technologies that are 'fundamentally arranged to undermine sensory perception.' Kittler wrote about a 'system of secrecy' based on user interfaces that 'conceal operations necessary for programming' and thus 'deprive users of the machine as a whole,' to suggest that perceiving the imitation, penetrating through the illusion of software 'that appears to be human,' is a fundamental challenge in the age of global-scale computing  (pp. 221-24) . Transparency, as Adrian Weller (2017) has poignantly noted, is an ambiguous term that will mean different things to different people. For the user, a transparent system will 'provide a sense for what [it] is doing and why,' while for an expert or regulator, it will enable auditing 'a prediction or decision trail in detail' (p.3). The calls for technological transparency have thus been filtered down to reach various stakeholder groups with different meanings and representing different interests. But what seems consistent in all of Footnote 1 (continued) 2019). Another interdisciplinary group of experts working under the auspices of IEEE (the world's largest professional association for electronic and electrical engineers) published, in 2019, the first edition of Ethically Aligned Design, a set of actionable recommendations on how to align design practices with society's values and principles, stressing that the 'standards of transparency, competence, accountability, and evidence of effectiveness should govern the development of autonomous and intelligent systems' (p. 5). these diverse takes on the development, use, and regulation of technology is that transparency is framed as a matter of design. In what follows, I problematize this claim, arguing that design, in the most fundamental sense, relies on concealment and obfuscation. I contend that only the sort of transparency that arises from critique-a method of theoretical examination that, by revealing pre-existing power structures, aims to challenge them-can help us produce technological systems that are less deceptive and more just.
 Design as blackboxing
 Art and artifice Coined in 1956 by John McCarthy, the term 'artificial intelligence' had its critics among those who attended the Dartmouth Conference (which famously established the field of AI); Arthur Samuel argued that 'the word artificial makes you think there's something kind of phony about this,  […]  or else it sounds like it's all artificial and there's nothing real about this work at all  ' (in: McCorduck 2004, p. 115) . The historian of AI Pamela McCorduck notes that while other terms, such as 'complex information processing,' were also proposed, it was 'artificial intelligence' that endured the trial of time. According to her, it is 'a wonderfully appropriate name, connoting a link between art and science that as a field AI indeed represents' (p. 115). She is referring indirectly here to the origins of the word artificial; in Latin, artificialis means 'of or belonging to art,' while artificium is simply a work of art, but also a skill, theory, or system. When the philosopher Vilém Flusser traced the etymology of the word 'design' in his The Shape of Things: A Philosophy of  Design (1999) , he referred to this relationship between art and artifice to argue that all human production, all culture, can be defined as a form of trickery. Flusser rejects the distinction between art and technology, and goes back to these ancient roots: the Greek for 'trap' is mechos (mechanics, machine); the Greek techne corresponds to the Latin ars; an artifex means a craftsman or artist, but also a schemer or trickster-to demonstrate that in their essence all forms of making are meant to help us 'elude our circumstances,' to cheat our own nature. Culture itself becomes a delusion brought about by means of design-a form of selfdeception that makes us believe we can free ourselves from natural restrictions by producing a world of artifice. From doors to rockets, from tents to computer screens, from pencils to mechanized intelligences, Flusser selects his examples to show that, ultimately, any involvement with culture is based on deception: sometimes 'this machine, this design, this art, this technology is intended to cheat gravity, to fool the laws of nature' (ch.1, n. pag.)-and sometimes to trick ourselves into thinking we control both gravity and the laws of nature. In that sense, art and technology are representative of the same worldview in which cultural production must be deceptive/artful enough to enable humans to go beyond the limits of what is (humanly) possible. Flusser refers to the act of weaving to explain the 'conspiratorial, even deceitful' (ch.18) character of design. In the process of carpet production, he points out, knotting is meant to deny its own warp, to hide the threads behind a pattern, so that anyone stepping on the finished rug perceives it as a uniform surface, according to the designer's plan. He offers weaving as one of the primordial forms of cultural production to embody trickery, but the same holds true for any form of design. The trick is always based on misdirection, shifting the end user's attention from the material to the application, from the current state of things to emerging possibilities and new futures. Designing is a methodical way of crafting alternative realities out of existing materials-a process of casting the intended shape onto the chosen fabric so as to create a new possibility. The material used in that process must, so to speak, dematerialize: it has to disappear from view and give way to the new object-to abstract the end result from the point of origin and the labor process. By obfuscating some components while exhibiting others, 'ideal' design enables an end user's cognitive efficiency.
 Patterns, layers, and repetitions For Flusser, any product of human making is both an object and an obstacle-Latin objectum, Greek problema-or, more specifically, any object is also an 'obstacle that is used for removal of obstacles' (ch.9). To move forward, we solve problems that lie ahead and obstruct our way; we produce objects that help us remove these obstacles; but the same objects turn into obstacles for those that come after us. In other words, since the results of human problem-solving are stored in objects, progress involves obfuscation and forgetting. We come up with a solution and, with time, this singular idea turns into a pattern; others use the already established template to produce new, more complex structures and these structures turn into new patterns, covering up previous layers of design with new design. To expedite the process of production, to advance, to move faster, the designer turns to these conventions and templates, choosing from a menu of preprogrammed options-or abstracting new rules based on previous patterns. And as the complexity of the production process increases, the reliance on patterns grows too. New design always depends on previous design, and this ultimate dependence on patterns and abstractions complicates understanding the process in its totality. In the age of ubiquitous computing, speaking of obfuscation by design becomes of particular importance. In 2015, Benjamin Bratton called his model of the new kind of layering brought about by planetary-scale computation 'the Stack': 'an accidental megastructure, one that we are building both deliberately and unwittingly and is in turn building us in its own image' (p.5). New technologies 'align, layer by layer, into something like a vast, if also incomplete, pervasive if also irregular, software and hardware Stack' (p.5). This makes it hard to perceive the Stack's overarching structure, indeed, to see it as design, however incidental. Today, we produce new technologies, new objects, to see, know, and feel more, to register what is normally hidden from our view, meanwhile, creating complex systems based on multiple, invisible layers and algorithmic operations whose effects are not always comprehensible even to the designers themselves.
 Automations and automatisms In her comprehensive account of what she calls 'surveillance capitalism,' Shoshana Zuboff points out the dangers of technological illusion-'an enduring theme of social thought, as old as the Trojan horse' (p.16)-that serves the new economic project in rendering its influence invisible. Surveillance capitalism claims 'human experience as free raw material for translation into behavioral data,' and turns that data into 'prediction products' that can later be sold to advertisers (p.8). Echoing the work of philosophers such as Bernard  Stiegler (2014 Stiegler ( , 2015  or Antoinette  Rouvroy (2016) , Zuboff argues that the ultimate goal of this new form of capitalism is 'to automate us,' by reprogramming our behavior and desires. Various internet platforms that dominate the market prompt us to action, influence our decision making, relying on big data analyses of our preference trends online. Automated systems create statistical models to profile users, tracing any emerging patterns in countless interactions with digital products; patterns turn into further abstractions, new models that are later reflected in new products and solutions, which end up 'automating' us, guiding our decision-making without our knowing. But is this process specific to AI-enhanced personalization under surveillance capitalism? Bratton has recently argued that what 'at first glance looks autonomous (self-governing, set apart, able to decide on its own) is, upon closer inspection, always also decided in advance by remote ancestral agents and relays, and is thus automated as well  ' (2019, loc.345, n. pag.) . Any decision taken now relies on multiple decisions taken in the past; new design depends on previous design; a new object coalesces from an aggregation of old solutions. Culture is an amalgamation of such objectsobjects that, ironically, become obstacles because they are meant to enable our cognitive efficiency. A tool becomes an obstacle because the results of our problem-solving and labor are already stored within it; a tool must never be seen as a tool, as its use must be intuitive-it must remain imperceptible; any new tool meant to advance the process is made with existing tools, and so the emerging layering of design in the Anthropocene makes it harder to distinguish between tool and fabric. Extending this to the ongoing automation of cognitive tasks in the age of ubiquitous computing, the phenomenon takes on new scale. This is why the emerging need for transparency refers not so much to company politics of disinformation or algorithmic black boxes, as to the very essence of our culture, as a process of knowledge production, pattern formation, and concealment. Particular problems caused by the widespread adoption of automated decision-making systems, such as algorithmic bias, can have specific, targeted, solutions in the form of new policy, engineering standards, or better education. But a shift of focus from the particular to the total is more than an exercise in theory-it makes us realize that transparency has never been at the heart of our making, that design has always been a form of blackboxing. There is, in that sense, something deeply anti-cultural about transparency. Or, putting it differently, there is nothing natural about transparency by design: we have been programmed to cover up as we make, not the opposite. The ongoing transformation of lived experiences into data is a new analytical paradigm that demands our intervention, truly calls for an 'unboxing,' an excavation of processes and data trails. But the opening of the algorithmic black box cannot be viewed only as a technical issue-precisely because any solution is, first and foremost, a result of cultural blackboxing. While contemporary debates on AI focus on transparency as a direct response to the opacity of algorithms, what we are in need of are approaches that aim to 'unbox' new technologies as objects-obstacles, solutions that aim towards cognitive automation, products that store the results of problem-solving performed 'by remote ancestral agents,' and that can thus perpetuate injustices via automatically accepted patterns and norms.
 Critique as unboxing
 Apparent transparencies Among entries on subjects such as theology, economics, and medicine, Denis Diderot and Jean le Rond d'Alembert included in their Dictionary of the Sciences, Arts, and Crafts, entries on artisanal practices that detail the individual steps in the processes of production adopted in clockmaking, tailoring, woodworking, and many others. One such entry focuses on the making of artificial flowers: the first plate (Fig.  1 ) depicts a dozen workers scattered across the main workshop area, performing different tasks at various stages of manufacturing, while following pages of illustrations showcase the most popular templates used to emboss specific petal shapes onto fabric, with a final plate celebrating the finished commodity. By bringing to view the backstages of production, the Encyclopedia was essentially undesigning, reversing the process of 'conspiratorial weaving' described by Flusser. Now, in an age of growing technological complexity, shaped by significant degrees of cognitive automation, there is a need for a similar undesigning of new technologies. The artist Todd McLellan's photographs (Fig.  2 ) that document his multiple attempts at taking various objects apart are a suggestive illustration of this challenge in the age of extreme technological complexity. We might dissemble our smartphone, but learning what is hidden beneath the interactive surface of the touchscreen will never give us an indication of how the device really works and, more significantly, in whose interest. The meaningless innards of the device become symbolic of the contestable quality that transparency really is-if we think of it as a condition for, or indeed a guarantee of, understanding. Critical undesigning cannot be confused with a simple act of reverse engineering. There can be transparency without critique, or apparent transparency: but a sort of transparency that does not arise from critical processes of unboxing is unlikely to advance comprehension. In his lecture on black boxes, Galloway (2010) relates Marx's idea of descent into 'the hidden abode of production' (p.7), as a means of uncovering capital relations forged into commodities, to 'traditions of critical inquiry, in which objects were unveiled or denaturalized to reveal their inner workings-from Descartes's treatise on method […] to the Freudian plumbing of the ego' (p.5). Based on the assumption that the surface is merely a superficial facade to be penetrated by means of critique, these theories prioritized the interior and perceived objects as 'mystical black boxes waiting to be deciphered to reveal the rationality (of history, of totality) harbored within' (p.3). For the purpose of this article, critique is understood as a broad set of methodologies, grounded in the tradition of Critical Theory, that perform a metaphorical dismantling of objects to reveal how hidden and immaterial layers of design reflect social and economic structures-and how the power relations these structures generate become the sources of injustice, oppression, and exploitation. Critique's ultimate goal is to uncover and challenge the system(s) that objects of design engender; revelation is conceived of as the condition necessary for resistance-and systemic transformation. Looking beyond individual design flaws (and fixes), critique points to those 'ancestral relays' that automate our thinking-to patterns, repetitions, and automatisms so deeply ingrained, weaved into the fabric of our culture, that they become imperceptible-in particular to those who do not experience the injustices resulting from the adoption of already established patterns.
 Critique in the age of AI A former YouTube employee, Guillaume Chaslot, has coined the term Algotransparency to describe an experiment in which he investigates the terms appearing most frequently in the titles of videos recommended by YouTube. A program developed by Chaslot and his team traces thematic patterns in YouTube recommendations to prove there exists a systemic bias that promotes controversial clips. His research suggests Google's platform indeed 'systematically amplifies videos that are divisive, sensational and conspiratorial'  (Lewis 2018) -that the recommendations are not related to the individual user's interests (as the company claims), but rather-exploit controversy to boost clickability. Algotransparency attempts to unbox the logic of YouTube's copyrightprotected recommendation algorithm without directly looking into the system's black box, concentrating only on the effects of its activity. This specific experiment gives a good indication of where we should be directing our attention: focusing not so much on how YouTube operates, as on why it works at all. This question extends beyond the technicality of the algorithm, to more widely interrogate the forces orchestrating our consumption of digital goods and whose interests they serve-what Zuboff calls surveillance capitalism, or what Stiegler refers to as hyperindustrialism. Ian Bogost (2015) has argued that the illusion of automation in technology-the trick that misdirects our attention from essential questions about human decision-making incorporated into emerging systems-breaks down 'once you bother to look at how even the simplest products are really produced.' In 2014 he collaborated with Alexis Madrigal to analyze Netflix's recommendation system and demonstrate that the platform's operations are distributed among so many different agents-including human curators who hand-tag all Netflix content-'that only a zealot would call the end result an algorithm'(Bogost 2015). Many experiments and critical projects try to achieve something similar: debunk the illusion of software by exposing AI as processual and collaborative, tracing the results of data analysis back to human decisions, biases, and labor. In their Anatomy of an AI System, for instance, Crawford and Joler (2018) present a figurative dissection of Amazon's Echo device that brings to view the invisible mechanisms and dynamisms that the product encapsulates (Fig.  3 ). The detailed mapping of various objects and agents, as well as multiple layers of interaction between those elements, constitutes a representation of the system as composed not only of hardware and software, data and computation, but also human labor and planetary resources. Critique in the age of extreme technological complexity is as much about dissecting and penetrating, as it is about charting the invisible and immaterial terrains of interaction, analysis, consumption, and computation; mapping wider relations between energies, influences, and resources under surveillance capitalismpatterns of exploitation of both people and environments that the production of objects/obstacles entails. In another project, Crawford teamed up with the artist Trevor Paglen to carry out what they call an archeology of datasets, such as ImageNet, used in machine learning to train AI to recognize specific elements of images-sets that can also become sources of bias inscribed into emerging systems. By excavating the datasets' underlying structures,  Crawford and Paglen (2019)  aim to reveal their implicit meanings: 'we have been digging through the material layers, cataloguing the principles and values by which something was constructed, and analyzing what normative patterns of life were assumed, supported, and reproduced.' The blackboxed Earth, a world deeply transformed by ubiquitous computing, by various layers of what Bratton calls the Stack, demands this form of unearthing. Successful critique in the age of AI exposes the technology as relying on human cognition and decision-making; more broadly, critique reveals the constellation of objects/obstacles as products of layered problem-solving, a flawed process that is necessarily tainted by pre-existing patterns and abstractions, biases and beliefs. Prioritizing reflection over efficiency, critique becomes a methodical way of resisting our reliance on patterns-patterns that allow us to move faster, but that can also harbor previously made assumptions-about gender (Costanza-Chock 2018) or race  (Benjamin 2019)  as particularly poignant examples-and thus perpetuate, rather than challenge, pre-existing forms of injustice.
 Reconciling design with critique In keeping up with the societal demand for transparent AI, the big players of the tech industry have been introducing changes in their engineering standards and organizational structures, hiring ethicists and policy specialists to cooperate with their product development teams. In 2016, Microsoft established the Aether Committee, a body of senior advisors to the company's leadership, that provides guidance 'on rising questions, challenges, and opportunities with the development and fielding of AI technologies' (2020), and oversees the work of other teams 'to ensure that AI products and services align with Microsoft's AI principles'-which include transparency and accountability. In 2017, DeepMind set up its Ethics and Society team 'to guide the responsible development and deployment of AI.' The team composed of 'ethicists and policy researchers' collaborates with the company's AI research team 'to understand how technical advances will impact society, and find ways to reduce risk.' Smaller industry players who decide to follow suit, but cannot afford to establish their own ethics 'departments,' begin to enlist the help of 'ethical auditing' companies; Glassbox, for example, is a tech consultancy startup, founded in 2018, that aims to 'provide clarity to the black box' by analyzing software products for signs of bias and training the client company's employees about systemic injustices. This way, elements of critique that reveal potential implications of human decision-making in design are supposed to become part of the production pipeline. These sites of interaction between 'humanists' and 'technologists' in the industry-even if, in some cases, they amount to nothing more than backdrops for press releasesdeserve our attention. Specifically, they require of us a comprehensive rethinking of what satisfies our desire for transparency in the age of extreme technological complexity. Can a system of checks and balances in the industry, an ongoing negotiation between blackboxing and unboxing, lead to anything more than design that anticipates critique? Is critique from within the industry necessarily a compromise and, therefore, nothing more than another step in the process of production of objects that are also obstacles? Must critique be external to the process of designing to remain genuine? If design is, fundamentally, blackboxing and automation, and critique is unboxing that aims to reverse the process of 'conspiratorial weaving,' then we could conclude that these two sides of human activity are in stark opposition to one another-that design is incompatible with critique. Instituting a real change in the way we move forward must focus precisely on tackling this ostensible impossibility-on reconciling design with critique, progress with suspension, production with reflection. The calls for transparency by design require that the 'making' itself be reinvented to incorporate critique. If a systemic transformation depends on a new-found compatibility between design and critique, then the designers of emerging systems should turn to already existing, alternative design practices to learn what combining the processes of making and critique might entail. Anthony Dunne and Fiona Raby, who have been pioneers in the field of 'critical design' since the late 1990s, have been advocating for design understood as 'critical thought translated to materiality  ' (2013, p.35 ). An object of design in their framing must become a critical challenge-as much for the designer, as for the user: 'it encourages people to question, in an imaginative, troubling, and thoughtful way, everydayness and how things could be different.' (p.189) More recently, Ratto (2011) coined the term 'critical making' to describe a design process that focuses on 'the act of shared construction itself as an activity and a site for enhancing and extending conceptual understandings of critical sociotechnical issues' (p.254)-with those who take part as the agents of critique. For Dunne and Raby, design has become 'so absorbed in industry, so familiar with dreams of industry, that it is almost impossible to dream its own dreams' (p.88). The suggestion is that the challenge lies not so much in the incompatibility between making and critique, as between the futures imagined by the industry and the dreams functioning outside of it. While elements of critique-gender critique and critical race theory in particular-seem to have already penetrated sections of the industry in the form of the mentioned ethics auditing services, reconciling the critique of surveillance capitalism and hyperindustrialism, the forces behind most of today's innovation, with the design of new technologies within corporate structures appears a considerable (and counterintuitive) feat. Perhaps this is where we should direct our attention: more research is needed on how practices such as critical design and critical making can influence the process of AI design; how critique can be operationalized within the industry to challenge industrial values and visons, including the idea of 'progress' itself. In a recently published volume on the practice of 'undesign' (McNamara and Coombs 2019), Cameron Tonkinwise's essay proposes what he calls 'anti-progressive' design as a means of interrogating the designers' internalized desire for 'progress.' While users, as he rightly observes, are willing 'to unlearn and relearn modes of interaction' if what is new is also 'easier and more convenient, and hopefully more effective and pleasurable' (p.76), it is the designers who should learn how 'not to prefer progress, or how to prefer what does not feel like progress' (p.81). Tonkinwise argues it is the designers' duty 'to find a way to pursue the destructively preferable without casting the resulting change as progress: what is preferable are futures that no longer appear to be mere advancements of what currently exists  ' (p.81) . This is to say that the responsibility for future action lies, primarily, with the designers: the human makers who, as products of a specific culture, are being increasingly challenged to become aware of their own biases and automatisms that predetermine their actions and choices. For designers, combining design with critique, rather than an attempt at making things transparent, would constitute an attempt at becoming self-conscious. Flusser argued that a renewed form of culture would have to be a culture 'aware of the fact that it was deceptive' (ch.1). To rethink technological transparency, we should first recognize that the designer has always been a trickster laying out traps, technologizing misdirection to pave the way forward. All aspects of design-practical, political, moral-would have to reflect this awareness: as we move forward, we must acknowledge that any problem solved now will also form a trap for those coming after us. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by/4.0/. Fig. 1 Fig. 2 12 Fig. 1 Maker of artificial flowers, 1765 (https ://hdl.handl e.net/2027/ spo.did22 22.0001.451)
 Fig. 3 3 Fig. 3 Kate Crawford and Vladan Joler, Anatomy of an AI System, 2018 (courtesy of the artists)
			 The High-Level Expert Group on AI convened by the European Commission presented its Ethics Guidelines for Trustworthy Artificial Intelligence in the early 2019. The document identifies several key characteristics of a system that can be deemed trustworthy, which include transparency, defined as a form of traceability of data, operations, and business models that shape the end product (AI HLEG

Abstract: In the late 2010s, various international committees, expert groups, and national strategy boards have voiced the demand to 'open' the algorithmic black box, to audit, expound, and demystify artificial intelligence. The opening of the algorithmic black box, however, cannot be seen only as an engineering challenge. In this article, I argue that only the sort of transparency that arises from critique-a method of theoretical examination that, by revealing pre-existing power structures, aims to challenge them-can help us produce technological systems that are less deceptive and more just. I relate the question of AI transparency to the broader challenge of responsible making, contending that future action must aim to systematically reconcile design-as a way of concealing-with critique-as a manner of revealing.


Bayesian Inverse Reinforcement Learning

Introduction The Inverse Reinforcement Learning (IRL) problem is defined in  [Russell, 1998]  as follows: Determine The reward function that an agent is optimizing. Given 1) Measurement of the agent's behaviour over time, in a variety of circumstances 2) Measurements of the sensory inputs to that agent; 3) a model of the environment. In the context of Markov Decision Processes, this translates into determining the reward function of the agent from knowledge of the policy it executes and the dynamics of the state-space. There are two tasks that IRL accomplishes. The first, reward learning, is estimating the unknown reward function as accurately as possible. It is useful in situations where the reward function is of interest by itself, for example when constructing models of animal and human learning or modelling opponent in competitive games. Pokerbots can improve performance against suboptimal human opponents by learning reward functions that account for the utility of money, preferences for certain hands or situations and other idiosyncrasies  [Billings et al., 1998] . There are also connections to various preference elicitation problems in economics  [Sargent, 1994] . The second task is apprenticeship learning -using observations of an expert's actions to decide one's own behaviour. It is possible in this situation to directly learn the policy from the expert  [Atkeson and Schaal, 1997] . However the reward function is generally the most succint, robust and transferable representation of the task, and completely determines the optimal policy (or set of policies). In addition, knowledge of the reward function allows the agent to generalize better i.e. a new policy can be computed when the environment changes. IRL is thus likely to be the most effective method here. In this paper we model the IRL problem from a Bayesian perspective. We consider the actions of the expert as evidence that we use to update a prior on reward functions. We solve reward learning and apprenticeship learning using this posterior. We perform inference for these tasks using a modified Markov Chain Monte Carlo (MCMC) algorithm. We show that the Markov Chain for our distribution with a uniform prior mixes rapidly, and that the algorithm converges to the correct answer in polynomial time. We also show that the original IRL is a special case of Bayesian IRL (BIRL) with a Laplacian prior. There are a number of advantages of our technique over previous work: We do not need a completely specified optimal policy as input to the IRL agent, nor do we need to assume that the expert is infallible. Also, we can incorporate external information about specific IRL problems into the prior of the model, or use evidence from multiple experts. IRL was first studied in the machine learning setting by  [Ng and Russell, 2000]  who described algorithms that found optimal rewards for MDPs having both finite and infinite states. Experimental results show improved performance by our techniques in the finite case. The rest of this paper is organised as follows: In section 2 we define our terms and notation. Section 3 presents our Bayesian model of the IRL process. Section 4 discusses how to use this model to do reward learning and apprenticeship learning while section 5 discusses the sampling procedure. Sections 6, 7 and 8 then present experimental results, related work and our conclusions respectively.
 Preliminaries We recall some basic definitions and theorems relating to Markov Decision Processes and Reinforcement Learning. A (finite) Markov Decision Problem is a tuple (S, A, T, γ, R) where • S is a finite set of N states. • A = {a 1 , . . . , a k } is a set of k actions. • T : S × A × S → [0, 1] is a transition probability func- tion. • γ ∈ [0, 1) is the discount factor. • R : S → R is a reward function, with absolute value bounded by R max . The rewards are functions of state alone because IRL problems typically have limited information about the value of action and we want to avoid overfitting. A Markov Decision Process (MDP) is a tuple (S, A, T, γ), with the terms defined as before but without a reward function. To avoid confusion we will use the abbreviation MDP only for Markov Decision Processes and not Problems. We adopt the following compact notation from  [Ng and Russell, 2000]  for finite MDPs : Fix an enumeration s 1 . . . s N of the finite state space S. The reward function (or any other function on the state-space) can then be represented as an Ndimensional vector R, whose ith element is R(s i ). A (stationary) policy is a map π : S → A and the (discounted, infinite-horizon) value of a policy π for reward function R at state s ∈ S,denoted V π (s, R) is given by: V π (s t1 , R) = R(s t1 )+E st 1 ,st 2 ,... [γR(s t2 )+γ 2 R(s t3 )+. . . |π] where P r(s ti+1 |s ti , π) = T (s ti , π(s ti ), s ti+1 ). The goal of standard Reinforcement Learning is to find an optimal policy π * such that V π (s, R) is maximized for all s ∈ S by π = π * . Indeed, it can be shown (see for example  [Sutton and Barto, 1998] ) that at least one such policy always exists for ergodic MDPs. For the solution of Markov Decision Problems, it is useful to define the following auxilliary Q-function: Q π (s, a, R) = R(s) + γE s ∼T (s,a,•) [V π (s , R)] We also define the optimal Q-function Q * (•, •, R) as the Qfunction of the optimal policy π * for reward function R. Finally, we state the following result concerning Markov Decision Problems (see  [Sutton and Barto, 1998] ) : Theorem 1 (Bellman Equations). Let a Markov Decision Problem M = (S, A, T, γ, R) and a policy π : S → A be given. Then, 1. For all s ∈ S, a ∈ A, V π and Q π satisfy V π (s) = R(s) + γ s T (s, π(s), s )V π (s ) (1) Q π (s, a) = R(s) + γ s T (s, a, s )V π (s ) 2. π is an optimal policy for M iff, for all s ∈ S, π(s) ∈ argmax a∈A Q π (s, a) (2) 3 Bayesian IRL IRL is currently viewed as a problem of infering a single reward function that explains an agent's behaviour. However, there is too little information in a typical IRL problem to get only one answer. For example, consider the MDP shown in Figure  1 . There are at least three reasonable kinds of reward functions: R 1 (•) has high positive value at s 1 (and low values elsewhere) which explains why the policy tries to return to this state, while R 2 (•) and R 3 (•) have high values at s 2 and s 3 respectively. Thus, a probability distribution is needed to represent the uncertainty. S 0 S 1 S 3 S 2 a 1 a 1 a 1 a 1 0.4 0.3 0.3 a 2 a 2 a 2 a 2 Figure 1: An example IRL problem. Bold lines represent the optimal action a 1 for each state and broken lines represent some other action a 2 . Action a 1 in s 1 has probabilities 0.4,0.3 and 0.3 of going to states s 1 , s 2 , s 3 respectively, and all other actions are deterministic.
 Evidence from the Expert Now we present the details of our Bayesian IRL model (Fig.  2 ). We derive a posterior distribution for the rewards from a prior distribution and a probabilistic model of the expert's actions given the reward function. Consider an MDP M = (S, A, T, γ) and an agent X (the expert) operating in this MDP. We assume that a reward function R for X is chosen from a (known) prior distribution P R . The IRL agent receives a series of observations of the expert's behaviour O X = {(s 1 , a 1 ), (s 2 , a 2 ) . . . (s k , a k )} which means that X was in state s i and took action a i at time step i. For generality, we will not specify the algorithm that X uses to determine his (possibly stochastic) policy, but we make the following assumptions about his behaviour: 1. X is attempting to maximize the total accumulated reward according to R. For example, X is not using an epsilon greedy policy to explore his environment. 2. X executes a stationary policy, i.e. it is invariant w.r.t. time and does not change depending on the actions and observations made in previous time steps. For example, X could be an agent that learned a policy for (M, R) using a reinforcement learning algorithm. Because the expert's policy is stationary, we can make the following independence assumption: P r X (O X |R) = P r X ((s 1 , a 1 )|R)P r X ((s 2 , a 2 )|R) . . . P r X ((s k , a k )|R) The expert's goal of maximizing accumulated reward is equivalent to finding the action for which the Q * value at each state is maximum. Therefore the larger Q * (s, a) is, the more likely it is that X would choose action a at state s. This likelihood increases the more confident we are in X 's ability to select a good action. We model this by an exponential distribution for the likelihood of (s i , a i ), with Q * as a potential function: P r X ((s i , a i )|R) = 1 Z i e αX Q * (si,ai,R) where α X is a parameter 1 representing the degree of confidence we have in X 's ability to choose actions with high X R α X (s1, a1) (s2, a2) (s k , a k ) Figure  2 : The BIRL model value. This distribution satisfies our assumptions and is easy to reason with. The likelihood of the entire evidence is : P r X (O X |R) = 1 Z e αX E(OX ,R) where E(OX , R) = P i Q * (si, ai, R) and Z is the appropriate normalizing constant. We can think of this likelihood function as a Boltzmann-type distribution with energy E(O X , R) and temperature 1 αX . Now, we compute the posterior probability of reward function R by applying Bayes theorem, P r X (R|O X ) = P r X (O X |R)P R (R) P r(O X ) = 1 Z e αX E(OX ,R) P R (R) (3) Computing the normalizing constant Z is hard. However the sampling algorithms we will use for inference only need the ratios of the densities at two points, so this is not a problem.
 Priors When no other information is given, we may assume that the rewards are independently identically distributed (i.i.d.) by the principle of maximum entropy. Most of the prior functions considered in this paper will be of this form. The exact prior to use however, depends on the characteristics of the problem: 1. If we are completely agnostic about the prior, we can use the uniform distribution over the space −R max ≤ R(s) ≤ R max for each s ∈ S. If we do not want to specify any R max we can try the improper prior P (R) = 1 for all R ∈ R n . 2. Many real world Markov decision problems have parsimonious reward structures, with most states having negligible rewards. In such situations, it would be better to assume a Gaussian or Laplacian prior:  P Gaussian (R(s) = r) = 1 √ 2πσ e − r P Beta (R(s) = r) = 1 ( r Rmax ) 1 2 (1 − r Rmax ) 1 2 , ∀s ∈ S In section 6.1, we give an example of how more informative priors can be constructed for particular IRL problems.
 Inference We now use the model of section 3 to carry out the two tasks described in the introduction: reward learning and apprenticeship learning. Our general procedure is to derive minimal solutions for appropriate loss functions over the posterior (Eq. 3). Some proofs are omitted for lack of space.
 Reward Learning Reward learning is an estimation task. The most common loss functions for estimation problems are the linear and squared error loss functions: L linear (R, R) = R − R 1 L SE (R, R) = R − R 2 where R and R are the actual and estimated rewards, respectively. If R is drawn from the posterior distribution (3), it can be shown that the expected value of L SE (R, R) is minimized by setting R to the mean of the posterior (see  [Berger, 1993] ). Similarily, the expected linear loss is minimized by setting R to the median of the distribution. We discuss how to compute these statistics for our posterior in section 5. It is also common in Bayesian estimation problems to use the maximum a posteriori (MAP) value as the estimator. In fact we have the following result: Theorem 2. When the expert's policy is optimal and fully specified, the IRL algorithm of  [Ng and Russell, 2000 ] is equivalent to returning the MAP estimator for the model of (3) with a Laplacian prior. However in IRL problems where the posterior distribution is typically multimodal, a MAP estimator will not be as representative as measures of central tendency like the mean.
 Apprenticeship Learning For the apprenticeship learning task, the situation is more interesting. Since we are attempting to learn a policy π, we can formally define the following class of policy loss functions: L p policy (R, π) = V * (R) − V π (R) p where V * (R) is the vector of optimal values for each state acheived by the optimal policy for R and p is some norm. We wish to find the π that minimizes the expected policy loss over the posterior distribution for R. The following theorem accomplishes this: Theorem 3. Given a distribution P (R) over reward functions R for an MDP (S, A, T, γ), the loss function L p policy (R, π) is minimized for all p by π * M , the optimal policy for the Markov Decision Problem M = (S, A, T, γ, E P [R]). Proof. From the Bellman equations (1) we can derive the following: V π (R) = (I − γT π ) −1 R (4) where T π is the |S|×|S| transition matrix for policy π. Thus, for a state s ∈ S and fixed π, the value function is a linear function of the rewards: V π (s, R) = w(s, π) • R where w(s, π) is the s'th row of the coefficient matrix (I − γT π ) −1 in (4). Suppose we wish to maximize E[V π (s, R)] alone. Then, max π E[V π (s, R)] = max π E[w(s, π)•R] = max π w(s, π)•E[R] By definition this is equal to V * M (s), the optimum value function for M , and the maximizing policy π is π * M , the optimal policy for M . Thus for all states s ∈ S, E[V π (s, R)] is maximum at π = π * M . But V * (s, R) ≥ V π (s, R) for all s ∈ S, reward functions R, and policies π. Therefore E[L p policy (π)] = E( V * (R) − V π (R) p ) is minimized for all p by π = π * M . So, instead of trying a difficult direct minimization of the expected policy loss, we can find the optimal policy for the mean reward function, which gives the same answer.
 Sampling and Rapid Convergence We have seen that both reward learning and apprenticeship learning require the mean of the posterior distribution. However the posterior is complex and analytical derivation of the mean is hard, even for the simplest case of the uniform prior. Instead, we generate samples from these distributions and then return the sample mean as our estimate of the true mean of the distribution. The sampling technique we use is an MCMC algorithm GridWalk (see  [Vempala, 2005] ) that generates a Markov chain on the intersection points of a grid of length δ in the region R |S| (denoted R |S| /δ). However, computing the posterior distribution at a particular point R requires calculation of the optimal Q-function for R, an expensive operation. Therefore, we use a modified version of GridWalk called PolicyWalk (Figure  3 ) that is more efficient: While moving along a Markov chain, the sampler also keeps track of the optimal policy π for the current reward vector R. Observe that when π is known, the Q function can be reduced to a linear function of the reward variables, similar to equation 4. Thus step 3b can be performed efficiently. A change in the optimal policy can easily be detected when moving to the next reward vector in the chain R, because then for some (s, a) ∈ (S, A), Q π (s, π(s), R) < Q π (s, a, R) by Theorem 1. When this happens, the new optimal policy is usually only slightly different from the old one and can be computed by just a few  of R in R |S| /δ. (b) Compute Q π (s, a, R) for all (s, a) ∈ S, A. (c) If ∃(s, a) ∈ (S, A), Q π (s, π(s), R) < Q π (s, a, R) i. π := PolicyIteration(M, R, π) ii. Set R := R and π := π with probability min{1, P ( R,π) P (R,π) } Else i. Set R := R with probability min{1, P ( R,π) P (R,π) } 4. Return R Figure  3 : PolicyWalk Sampling Algorithm steps of policy iteration (see  [Sutton and Barto, 1998] ) starting from the old policy π. Hence, PolicyWalk is a correct and efficient sampling procedure. Note that the asymptotic memory complexity is the same as for GridWalk. The second concern for the MCMC algorithm is the speed of convergence of the Markov chain to the equilibrium distribution. The ideal Markov chain is rapidly mixing (i.e. the number of steps taken to reach equilibrium is polynomially bounded), but theoretical proofs of rapid mixing are rare. We will show that in the special case of the uniform prior, the Markov chain for our posterior (3) is rapidly mixing using the following result from [Applegate and  Kannan, 1993]  that bounds the mixing time of Markov chains for pseudo-logconcave functions. Lemma 1. Let F (•) be a positive real valued function defined on the cube {x ∈ R n | − d ≤ x i ≤ d} for some positive d, satisfying for all λ ∈ [0, 1] and some α, β |f (x) − f (y)| ≤ α x − y ∞ and f (λx + (1 − λ)y) ≥ λf (x) + (1 − λ)f (y) − β where f (x) = log F (x). Then the Markov chain induced by GridWalk (and hence PolicyWalk) on F rapidly mixes to within of F in O(n 2 d 2 α 2 e 2β log 1 ) steps. Proof. See [Applegate and  Kannan, 1993] . Theorem 4. Given an MDP M = (S, A, T, γ) with |S| = N , and a distribution over rewards P (R) = P r X (R|O X ) defined by (3) with uniform prior P R over C = {R ∈ R n | − R max ≤ R i ≤ R max }. If R max = O(1/N ) then P can be efficiently sampled (within error ) in O(N 2 log 1/ ) steps by algorithm PolicyWalk. Proof. Since the uniform prior is the same for all points R, we can ignore it for sampling purposes along with the normalizing constant. Therefore, let f (R) = α X E(O X , R). Now choose some arbitrary policy π and let Note that f π is a linear function of R and f f π (R) = α X i Q π (s, a i , R) IJCAI-07 (R) ≥ f π (R), for all R ∈ C. Also we have, max s,a Q * (s, a) = max s,a,π Q π (s, a) = max s,π V π max (s) ≤ R max 1 − γ Similarly, min s,a Q * (s, a) ≥ − Rmax 1−γ . Therefore, f (R) ≤ αX NRmax 1−γ and f π (R) ≥ − αX NRmax 1−γ and hence f π (R) ≥ f (R) − 2α X N R max 1 − γ So for all R 1 , R 2 ∈ C and λ ∈ [0, 1], f (λR 1 + (1 − λ)R 2 ) ≥ f π (λR 1 + (1 − λ)R 2 ) ≥ λf π (R 1 ) + (1 − λ)f π (R 2 ) ≥ λf (R 1 ) + (1 − λ)f (R 2 ) − 2α X N R max 1 − γ Therefore, f satisfies the conditions of Lemma 1 with β = 2αX NRmax 1−γ = 2N • O( 1 N ) 1−γ = O(1) and α = |f (R 1 ) − f (R 2 )| R 1 − R 2 ∞ ≤ 2α X N R max (1 − γ)O( 1 N ) = O(N ) Hence the Markov chain induced by the GridWalk algorithm (and the PolicyWalk algorithm) on P mixes rapidly to within of P in a number of steps equal to O(N 2 1 N 2 N 2 e O(1) log 1/ ) = O(N 2 log 1/ ). Note that having R max = O(1/N ) is not really a restriction because we can rescale the rewards by a constant factor k after computing the mean without changing the optimal policy and all the value functions and Q functions get scaled by k as well.
 Experiments We compared the performance of our BIRL approach to the IRL algorithm of  [Ng and Russell, 2000]  experimentally. First, we generated random MDPs with N states (with N varying from 10 to 1000) and rewards drawn from i.i.d. Gaussian priors. Then, we simulated two kinds of agents on these MDPs and used their trajectories as input: The first learned a policy by Q-learning on the MDP + reward function. The learning rate was controlled so that the agent was not allowed to converge to the optimal policy but came reasonably close. The second agent executed a policy that maximized the expected total reward over the next k steps (k was chosen to be slightly below the horizon time). For BIRL, we used PolicyWalk to sample the posterior distribution (3) with a uniform prior. We compared the results of the two methods by their average 2 distance from the true reward function (Figure  4 ) and the policy loss with 1 norm (Figure  5 ) of the learned policy under the true reward. Both measures show substantial improvement. Note that we have used a logarithmic scale on the x-axis. We also measured the accuracy of our posterior distribution for small N by comparing it with the true distribution of rewards i.e. the set of generated rewards that gave rise to the same trajectory by the expert. In Figure  6 , we show scatter plots of some rewards sampled from the posterior and the true distribution for a 16-state MDP. These figures show that the posterior is very close to the true distribution.
 From Domain Knowledge to Prior To show how domain knowledge about a problem can be incorporated into the IRL formulation as an informative prior, IJCAI-07 we applied our methods to learning reward functions in adventure games. There, an agent explores a dungeon, seeking to collect various items of treasure and avoid obstacles such as guards or traps. The state space is represented by an m-dimensional binary feature vector indicating the position of the agent and the value of various fluents such as hasKey and doorLocked. If we view the state-space as an m-dimensional lattice L S , we see that neighbouring states in L S are likely to have correlated rewards (e.g. the value of doorLocked does not matter when the treasure chest is picked up). To model this, we use an Ising prior (see  [Cipra, 1987] ): P R (R) = 1 Z exp(−J (s ,s)∈N R(s)R(s ) − H s R(s)) where N is the set of neighbouring pairs of states in L S and J and H are the coupling and magnetization parameters. We tested our hypothesis by generating some adventure games (by populating dungeons with objects from a common sense knowledge base) and testing the performance of BIRL with the Ising prior versus the baseline uninformed priors. The results are in figure  7  and show that the Ising prior does significantly better.
 Related Work The initial work on IRL was done by  [Ng and Russell, 2000]  while  [Abbeel and Ng, 2004]  studied the special case where rewards can be represented as linear combinations of features of the state space and gave a max-margin characterization of the optimal reward function.  [Price and Boutilier, 2003]  discusses a Bayesian approach to imitating the actions of a mentor during reinforcement learning whereas the traditional literature on apprenticeship learning tries to mimic the behaviour of the expert directly  [Atkeson and Schaal, 1997] . Outside of computer science, IRL-related problems have been studied in various guises. In the physical sciences, there is a body of work on inverse problem theory, i.e. infering values of model parameters from observations of a physical system  [Tarantola, 2005] . In control theory,  [Boyd et al., 1994]  solved the problem, posed by Kalman, of recovering the objective function for a deterministic linear system with quadratic costs.
 Conclusions and Future Work Our work shows that improved solutions can be found for IRL by posing the problem as a Bayesian learning task. We provided a theoretical framework and tractable algorithms for Bayesian IRL and our solutions contain more information about the reward structure than other methods. Our experiments verify that our solutions are close to the true reward functions and yield good policies for apprenticeship learning. There are a few open questions remaining: 1. Are there more informative priors that we can construct for specific IRL problems using background knowledge? 2. How well does IRL generalize? Suppose the transition function of the actor and the learner differed, how robust would the reward function or policy learned from the actor be, w.r.t the learner's state space? Figure 4: Reward Loss.
 Figure 6 : 6 Figure 5: Policy Loss.
 Figure 7 : 7 Figure 7: Ising versus Uninformed Priors for Adventure Games
 If the underlying MDP represented a planning-type problem, we expect most states to have low (or negative) rewards but a few states to have high rewards (corresponding to the goal); this can be modeled by a Beta distribution for the reward at each state, which has modes at high and low ends of the reward space: P Laplace (R(s) = r) = 1 2σ e − |r| 2σ , ∀s ∈ S 3. 2 2σ 2 , ∀s ∈ S on αX as well(Fig 2). But it will be simpler to treat αX as just a parameter of the distribution.
 Algorithm PolicyWalk(Distribution P , MDP M, Step Size δ )1. Pick a random reward vector R ∈ R |S| /δ. 2. π := PolicyIteration(M, R) 3. Repeat (a) Pick a reward vector R uniformly at random from the neighbours
			 Note that the probabilities of the evidence should be conditioned IJCAI-07

Abstract: Inverse Reinforcement Learning (IRL) is the problem of learning the reward function underlying a Markov Decision Process given the dynamics of the system and the behaviour of an expert. IRL is motivated by situations where knowledge of the rewards is a goal by itself (as in preference elicitation) and by the task of apprenticeship learning (learning policies from an expert). In this paper we show how to combine prior knowledge and evidence from the expert's actions to derive a probability distribution over the space of reward functions. We present efficient algorithms that find solutions for the reward learning and apprenticeship learning tasks that generalize well over these distributions. Experimental results show strong improvement for our methods over previous heuristic-based approaches.