Stuart Armstrong
Mario Weick
Anders Sandberg
Andrew Snyder-Beattie
Nick Beckstead

The underwriter and the models- solo dances or pas-de-deux? What policy data can tell us about how underwriters use models

Executive Summary Using a collection of data on some of the catastrophe policies written since 2006 by a major reinsurance organisation, this paper explores how tightly underwriters follow the models and under what conditions they deviate from them. The data included underwriter premium and LE (loss estimate), and sometimes included LE from up to four different models (AIR, RMS ALM, RMS DLM, and IHM -the in-house model). We analysed the data in order to see what could be said about the relationship between model LE and premiums (as well as underwriter LEs). Mimicking a common procedure in machine learning, the data was randomly divided into a training set and a testing set, allowing many different theories to be investigated on the training set without risk of overfitting and detecting spurious connections.
 Results There were three main results, all statistically significant and with large effect sizes: 1. The models gave good predictions as to what the underwriter premium and LE would be. In a regression test, 79% of the variance in the premium, and 88% of variance in the LE, was explained by variance in the models. In fact, most of this variance was explained by the mean LE of the four models -78% and 87% respectively, corresponding to correlations of 0.88 and 0.93. 2. As the modelled loses rose, underwriter estimates moved closer to those of the models. This was evident through a variety of measures: the underwriters would report using more models than otherwise, the underwriter LE would become more strongly correlated with the mean model LE, and they would have less extreme premium/ LE ratios (meaning that the LE information was being used more to fix the premium). It should be noted that one effect that we expected to find -that underwriters were more willing to follow the models if these were more closely bunched together -was not present in the data. It seems that the underwriter does not make much use of model spread.
 The role of underwriters As far as can be seen in the data, the underwriters premiums (and LE) were strongly correlated with the models' LE estimates. The higher the expected loss (as seen by model LE), the less likely the underwriters were to deviate from the models. That high correlation may seem to suggest a limited role to the underwriter. However, this conclusion is premature for several reasons. Most significantly, this dataset only included policies that the reinsurance organisation had actually written, so the role of underwriters in rejecting policies could have been very important. There are also issues of yearly variation and changing market conditions (2012, for instance, seems to be an outlier in many ways). This analysis also ignores the effect of underwriters negotiating and interacting with brokers -even if every good underwriter were to sign similar kinds of deals, this does not mean the underwriters were superfluous. It is significant that the models were more predictive of underwriter LE than of premium (which would be influenced by negotiation). Underwriters may also play an important role in correcting erroneous information in the policy, and making sure that the correct models were applied in the first place. Finally, there were no details of outcomes in the data (which policies led to pay-outs, and by how much?), limiting our ability to estimate underwriter expertise. Thus it is possible and likely that the underwriter played (or could play) a more synergistic role with models, focusing on quality control, market insights, and business relations. This thesis can be strongly questioned. First, the social role of the underwriter should not be underestimated: they are involved in negotiations with brokers, and require perceptiveness of the opinions and behaviour of the other market actors. Secondly, when the practical role of the underwriter was analysed in detail, some of the work involved coping with poor data quality and correcting model errors. 5 Though these tasks do not appear on the list of bottlenecks to automation, they are certainly tasks that cannot be easily automated, as they represent a failure of the automation process itself. Currently (and for the foreseeable future), only humans possess the skills to apply these kind of corrections, which often involve deducing what kind of errors have occurred or what kind of extra data could improve the situation. Thirdly, underwriters may be using strategic intelligence to select models and to maintain the sort of overall vision that could resolve larger systemic risks. 6 Therefore it would be incorrect to see the underwriters as necessarily in direct competition with the models, but as occupying (potentially) different and complementary roles. This was one of the approaches advocated in the "autopilot problem" paper: change the "pilot's" (or underwriter's) role.  7  Merely relying on effective automation technology can both weaken the skills of the human and make the whole endeavour vulnerable to situations where the automation fails for some reason. However, making use of particular human abilities to control or complement the automation allows for both greater performance and better robustness. For instance, though simple linear models outperform expert predictions in many domains, these simple linear models could only be constructed thanks to expert knowledge of the important factors.  8  Similarly, experienced underwriters are potentially aware of what information is relevant in a particular type of case -and whether such a case is at hand. The quality of expertise is highly dependent on features of the task, 9 rather than on features of the expert, so it is important that the task the underwriter performs be well designed to make best use of their human qualities (especially as expertise tends to be quite specific to the task performed 10 ). If this can be achieved, the underwriter and models of the future will amplify each other. We thus need to understand how underwriters currently interact with models. To this end, a major reinsurance organisation has made available several years' worth of records on policies priced by its underwriters and by its models so that a comparison could be made and the role of the underwriter teased out. The most useful features of this data were the premiums that were actually charged, the underwriters' Loss Estimate (LE), and the same LEs as given by the models. Using this data, this paper explores how tightly underwriters follow the models and under what conditions they deviate from them.
 Introduction The underwriters have traditionally been the key players in the insurance industry, making the final decision on any particular policy. It is their responsibility to negotiate a price and ultimately accept or decline to insure the risk. But recent years have seen the emergence of another key insurance player: computerised models that give their own estimation of risk, exposure, and other critical features of a policy. In CAT (catastrophe) insurance, these models are now used extensively by insurers, re-insurers, and regulators, and have underpinned the risk-linked securities markets. 1 In this new, model-centric world, what is the current role of the underwriter? More usefully, what will the role of the underwriter become? Some studies suggest that the underwriter is soon to be replaced by automation.  2  The Frey and Osborne study analysed current automation trends and concluded, based on O*NET data (an online job classification service developed for the US Department of Labor 3 ), that underwriting involved none of the skills estimated to be hard to automate such as manual dexterity, strategic intelligence or socially dependent tasks 4 (see Table  1 ).
 Computerisation bottleneck O*NET Variable O*NET Description
 Perception and Manipulation
 Finger Dexterity The ability to make precisely coordinated movements of the fingers of one or both hands to grasp, manipulate, or assemble very small objects.
 Manual Dexterity The ability to quickly move your hand, your hand together with your arm, or your two hands to grasp, manipulate, or assemble objects. How often does this job require working in cramped work spaces that requires getting into awkward positions? Creative Intelligence Originality The ability to come up with unusual or clever ideas about a given topic or situation, or to develop creative ways to solve a problem. Fine Arts Knowledge of theory and techniques required to compose, produce, and perform works of music, dance, visual arts, drama, and sculpture.
 Social Intelligence Social Perceptiveness Being aware of others' reactions and understanding why they react as they do.
 Negotiation Bringing others together and trying to reconcile differences.
 Persuasion Persuading others to change their minds or behaviour.
 Assisting and Caring for Others Providing personal assistance, medical attention, emotional support, or other personal care to others such as coworkers, customers, or patients.
 Results Three primary results were prominent in this initial analysis. First, variance in the model's LE explained the majority of the variance in premium (and in underwriter's LE). Second, underwriters tend to be conservative in estimating losses, more often setting expected losses above those of the models rather than below them. Last, the premiums (and underwriters' LE) moved closer to the models for more expensive policies. The causes and implications of these results are still uncertain -this preliminary analysis is only capable of identifying correlations, not causations. Note that the high R 2 need not mean that the underwriters are explicitly using the mean LE in their estimates -the models are highly correlated with each other, as can be seen in Table  2 . Thus many linear or quasi-linear combinations of models will be highly correlated with the models, with their mean, and hence with premium. Indeed, the correlation between premium and models is quite comparable with the correlation the models have with each other.
 Data and Methods The reinsurance organisation provided a large collection of policies with appropriate premium and LE information. Data was taken from two sources: a reinsurance system used to record catastrophe model outputs; and an underwriting system used to record class of business and other risk details. The data were compiled over several years since 2006, with modellers recording model outputs and underwriting teams recording risk details as business was transacted. The data included premium charged, underwriter LE, general location, year, and potential LEs from up to four different models. LE is the loss estimate -the mean amount of money that the underwriter expects their company will pay out on that policy. The more usual "loss on line" is the LE for a particular "layer" of insurance, divided by the limit for that layer. The four models were AIR Catrader (AIR), RMS RiskLink Aggregate level model (RMS ALM), RMS RiskLink Detailed level model (RMS DLM), and the organisation's in-house model (IHM). After cleaning the data and restricting to US/ Canada policies (where the models are the most reliable), we were left with a collection of 660 policies where all four models were used 11 -no further selection was applied to this set. It was decided to split the data into a training and a testing set. This allows "cross validation", where hypotheses are formed on the training set and tested on the testing set. 12 This prevents overfitting, in which hypotheses are tailored to narrowly to the data, modelling noise rather than signal.  13  The training set would be thoroughly analysed to generate hypotheses; these would then be tested for statistical significance on the testing set. The ideal size of the testing set is 1/3 of the original set.  14  As the splitting into testing and training was done before restricting down to the 660 policies, this subset ended up divided into 457 policies in the training set, and 203 in the testing set. A total of 32 hypotheses were formed on the training set, which were then tested on the testing set, and all were found to be significant at the 5% level 15 , even when accounting for multiple comparisons  16  . Most of these hypotheses were linear regressions, but other comparisons were made as well (see detail of results). Result 2: Conservative Pricing Underwriters tended to set their LEs somewhat conservatively. Underwriter LE were above the minimum model LE 94% of the time (as compared to below the maximum model 83% of the time). See Table  5  in next section for breakdown of this into highest and lowest quartiles.  22  This was especially interesting as the maximum value and the mean were both better predictors of underwriter LE than the minimum (see Figure  1  and Table  4 ). Thus, it seems the minimum value was a rough lower bound for underwriters, but that they used little information from it beyond this. This strong correlation could be a sign of an autopilot problem 20 , if underwriters put too much trust in the models. In the meantime, we can analyse how these correlations change over time. For example, Table  3  demonstrates how in 2012, the fit between premium and model LE declined, as compared to surrounding years. This was around the period when the controversial "RMS 11" windstorm model was in use.    First of all was the simple fact that they used (or at least recorded the use of) more models. The full US/ Canada data (6138 policies) was decomposed into categories dependent on the number of models recorded 23 , and the median premium and underwriter LE of each category was computed (see Figure  2 ). For illustration, the interquartile distance was added in the case of zero and four models. Though the ranges overlap, those policies which recorded two or less models clearly had lower LEs and premiums than those that recorded three models or four. Figure  2  -Median underwriter LE and premiums, depending on how many models were reported for that policy. The inter-quartile range for 4 models and 0 models is also plotted. To see this, the 660 policies with four models were separated into quartiles.  24  The correlation between the underwriter LE and mean, minimum, and maximum model LE was computed in each quartile (Table  5 ). The data suggests underwriters hewed closer to the model information in these quartiles. A last way of looking at the difference in behaviour, is to consider what happens when the underwriter has established their LE, and compare this with the final premium. The data indicates that for high underwriter LE 25 , the premium/LE ratio is likely to be less extreme that for low ones. The linear anti-correlation is weak (-11%), but the effect is clearer graphically (see Figure  3 ). Thus the premium is more influenced by LE estimates when these are large.  To establish the results overleaf, a total of 32 hypotheses were tested. The significance of each result was established using the testing set, with each p-value adjusted -increased 26 -using the Holm-Bonferroni method. See table 6 (where "HBp-value" is the Holm-Bonferroni 27 adjusted p-value). As can be seen, all hypotheses were significant at the 5% level, and all but two were significant at the 1% level (these significance levels were marked in bold).  Of these hypotheses, 22 were regression/correlations with premium or underwriter LE as dependent variable, and either all model LEs or mean model LE as independent variables. We should expect these regressions to show scale invariance to some extent: if all the model LEs double, then, say, the underwriter LE should also double as well. This means that we expect the residuals (the deviations of the underwriter LE from its "theoretic value" as predicted by the regression model) to double as well. Thus we expect residuals to be higher for high expected losses and lower for low expected losses: the data should be heteroscedastic. And indeed it is (see Figure  4 ). Excessive heteroscedasticity precludes the use of the standard F-test to determine the p-values for the model. Instead, we took the logarithms of all the variables, expecting that this would remove the scale variations in the residuals. The residuals that resulted were much closer to being homoscedastic (see Figure  4 ). Thus the p-values of these regressions and correlations were calculated using the log-log regression rather than the standard linear regression. These p-values were sufficient to show that all regression models were significantly different from the null hypothesis. All regression data given in this paper (the R2 and the correlation coefficients), however, came from a standard linear regression, as a linear model is more likely to be closer to the underwriters' behaviour than a logarithmic one. Generally, the logarithmic model had slightly lower R2 than the corresponding linear model, but the values were very close (and on occasion the logarithmic model had a slightly better fit than the linear model). Statistical testing and significance
 Discussion The high correlation between model-estimated LE and premium (and underwriter LE) may seem to suggest a limited role to the underwriter. However, this conclusion is premature for several reasons. For a start, this dataset is likely incomplete, as it had to be put together specifically, and many policies failed to record model estimates.  28  Most importantly, it only included policies that had actually been written; the role of underwriters in rejecting policies could have been very important. The overall result ignores the fact of yearly variation: in particular, 2012 fits very poorly into the general analysis. It is likely that underwriters were aware of changing market conditions (or changes to the models themselves) and were able to react to them accordingly in that year. This analysis also ignores the effect of underwriters negotiating and interacting with brokers. It is significant that the models were more predictive of underwriter LE than of premium 29 (which would be influenced by negotiation). Underwriters may also play an important role in correcting erroneous information in the policy, and making sure that the correct models were applied in the first place 30 . Finally, there were no details of outcomes in the data (which policies led to payouts, and by how much?), limiting our ability to estimate underwriter expertise.  31  Conversely, there could have been a lot of wasted effort on the underwriter's part. The four models are so highly correlated (see Table  4 ) that any attempts by the underwriters to strike a fine balance between them would likely have made little impact. Given all those caveats, the story that the data presents is clear. If this sample is taken to be representative, then there seems little difference between using models to estimate loss, and having the underwriters do the same. More importantly, a similar pattern is true to a lesser extent in premium, where around 80% of the variance in premiums is explained by the models in a completely linear fashion. The remaining 20% is unlikely to represent perfect performance on the part of the underwriter, free of noise and bias. Underwriters, like all humans, are subject to general biases that interfere with their performance 32 , some of which could have a specific impact on their job.  33  The important question is how biases balance against expertise within this 20%. We have done some preliminary exploration of these biases and counteracting expertise with an experimental pilot study.  34  In that study, the underwriters appeared less model-bound than in this paper: the actual conditions of work may play a large role in decisionmaking. A full analysis of the underwriters' work is needed if the aim is to develop expertise that complements models in a robust way. We believe that data analysis such as this paper, an understanding of the cognition of underwriting, and a systemic perspective on the inherent risks of outsourced cognition -whether from autopilot biases or model-induced correlations across the insurance market -can help develop practices that both reduce systemic risk and amplify human capacity. It will become more and more vital for insurance companies to record their own data as this one has, and to analyse it intelligently.  28  In many cases, it is likely that models were used but the data wasn't recorded.  29  The models explain 88% of the underwriter LE variance in the 2010-2013 period, but only 79% of the premium variance.  30  From conversations with people in the industry, this last effect is more likely to be a factor in insurance modelling than in reinsurance modelling, which seems to be a more mechanical process. 31 And even loss data in the short term is not enough to estimate true underwriter expertise, as many risk as of low probability/high return period, and wouldn't show up in the data.  32  The bias literature is vast, but Kahneman, Daniel. Thinking, fast and slow. Macmillan, 2011 provides a good overview, Gigerenzer, Gerd. "How to make cognitive illusions disappear: Beyond "heuristics and biases"." European review of social psychology 2.1 (1991): 83-115 provides a good critique, and Kahneman, Daniel, and Gary Klein. "Conditions for intuitive expertise: a failure to disagree." American Psychologist 64.6 (2009): 515 gives a good synthesis of some opposed views on the subject. 33 Beckstead, Nick. "Biased error search as a risk of modelling in insurance" in "Systemic Risk of Modelling." Joint Future of Humanity Institute-MS Amlin White Paper 3 (2014) Figure 1 - 1 Figure 1 -Underwriter's LE as a function of cat model LE. With a set of models, underwriters are unlikely to set LE below the lowest model, and tend to stick fairly close to the mean model estimate.
 23 5071 policies did not formally record a modelled loss, 27 had one, 85 had two, 295 had 3 and 660 had all four models. Dependency for Biggest Risks Premium and LE medians for varying number of models LE Furthermore, the correlation between underwriter LE and model LE increased for higher model LE.
 Figure 3 - 3 Figure 3 -Ratio of premium to underwriter LE plotted against underwriter LE.
 Figure 4 - 4 Figure 4 -Residuals for linear and log-log regression for underwriter LE versus all four model LEs.
 Table 1 O 1 *NET variables that are bottlenecks to automation, according to Frey and Osborne.
 Table 2 2 The premium is actually more highly correlated with the other models than the in-house model (IHM) is. Or, put another way, premium deviation with model LE is comparable to the noise 19 in the model LEs. Correlations between premium, model LEs, and mean model LE. Premium AIR LE RMS ALM LE RMS DLM LE IHM LE Mean LE Premium 1 0.825 0.808 0.796 0.852 0.881 AIR LE 0.825 1 0.909 0.864 0.878 0.966 RMS ALM LE 0.808 0.909 1 0.854 0.797 0.930 RMS DLM LE 0.796 0.864 0.854 1 0.758 0.915 IHM LE 0.852 0.878 0.797 0.758 1 0.937 Mean LE 0.881 0.966 0.930 0.915 0.937 1
 Though the controversy was mainly around European windstorms, not US/Canada ones, this could have had an impact on underwriter trust in models. Alternatively, the large losses in 2011 could have played a similar role.21     Year # of policies All Model LE to Premium R2 Mean Model LE to Premium R2 All Model LE to Underwriter LE R2 Mean Model LE to Underwriter LE R2 2010 74 91.32% 76.08% 91.65% 89.87% 2011 345 84.94% 82.01% 91.65% 90.94% 2012 145 75.69% 69.56% 80.13% 75.58% 2013 92 95.15% 83.22% 94.83% 88.18% Mean 86.78% 77.72% 89.57% 86.14% Combined 656 78.80% 77.55% 87.63% 86.95%
 Table 3 R 3 2 between premiums and Underwriter LE with models, by year. 20 Armstrong, Stuart. "The Autopilot Problem" in "Systemic Risk of Modelling." Joint Future of Humanity Institute-MS Amlin White Paper 2 (2014). 21 Aon Benfield. "Reinsurance Market Outlook." Aon Benfield Analytics (2013).
 Table 4 4 Correlation coefficients between underwriter LE and the mean, maximum, and minimum of the model LEs. Mean Max Min
 Table 5 5 Correlation coefficients between underwriter LE and mean model LE, maximum model LE, and minimum model LE, separated onto the four quartiles. 1st quartile 2nd quartile 3rd quartile 4th quartile Mean 0.449 0.233 0.598 0.876 Max 0.486 0.387 0.363 0.847 Min 0.231 -0.035 0.329 0.747 Premium/LE ratio vs underwriter LE Premium/LE ratio Underwriter LE
 Table 6 p 6 -values and Holm-Bonferroni corrected p-values (for multiple comparisons) for all 32 hypotheses considered.
			 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 3
			 This was the judgement formed by three of the paper's authors during several periods of immersion at the reinsurance organisation, which
			 US Gov't Accountability Office, GAO-02-941, "Catastrophe Insurance Risks: The Role of Risk-Linked Securities and Factors Affecting Their Use" (2002).2 Frey, Carl Benedikt, and Osborne, Michael. "The future of employment: how susceptible are jobs to computerisation?" Oxford Martin School Working Paper (2013).
			 Or that, when it did involve these skills, that they were not fundamental to the job.4 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 5
			 Of the form U i = β 0 + β 1 M AIRi + β 2 M RMS-ALMi + β 3 M RMS-DLMi +β 4 MGMi +ε i where U is the underwriters' LE, M X is the LE of the X'th model, and ε i is the error term for the i'th policy.18  Of the form U i = β 0 + β 1 M MEANi +ε i where U is the underwriters' LE, M i is the average LE of all four models, and ε i is the error term for the i'th policy.19  Using model LE variation as an informal measure of noise.
			 Picard, Richard R, and R Dennis Cook. "Cross-validation of regression models." Journal of the American Statistical Association 79.387 (1984): 575-583. 13 Hawkins, Douglas M. "The problem of overfitting." Journal of chemical information and computer sciences 44.1 (2004): 1-12. 14 Dobbin, Kevin K, and Richard M Simon. "Optimally splitting cases for training and testing high dimensional classifiers." BMC medical genomics 4.1 (2011): 31.
			 Benjamini, Yoav. "Simultaneous and selective inference: current successes and future challenges." Biometrical Journal 52.6 (2010): 708-721.6 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 7
			 Note that we are using quartile in the sense of a set of data representing a quarter of the policies, not in the sense of the three values (lower quartiles, median, upper quartile) that divide the ranked data into those four sets.
			 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 9
			 That is, the policies were ranked according to their mean LE, and they were split into four sets at lower quartile, the median, and the upper quartile. By an abuse of nomenclature, these four sets are also called quartiles.25  Since the underwriter would set their own LE before setting a premium, it is valid to use underwriter LE on the x-axis at this point.
			 p-values of the individual hypotheses were calculated separately, then the hypotheses were ordered by increasing p-value. These p-values were then multiplied by (32+1-r), where r was the rank of the hypothesis on the list. Thus the hypothesis with the lowest p-value had this value multiplied by 32, that with the next-lowest had its value multiplied by 31, all the way down to the one with the highest p-value, which was multiplied by 1. As long as these adjusted p-values were below the criteria of significance, then the familywise error rate would be below that.27 Holm, Sture. "A simple sequentially rejective multiple test procedure." Scandinavian journal of statistics (1979): 65-70.
			 The underwriter and the models -solo dances or pas-de-deux?The underwriter and the models -solo dances or pas-de-deux? 13

MS Amlin White Paper The underwriter and the models- solo dances or pas-de-deux.pdf.downloadasset.tei.xml

Abstract: The underwriter and the models -solo dances or pas-de-deux? 1 involved detailed conversations with, and questioning of, employees in various roles within the company.