When AI goes bad - allegedly

We recently read of a popular insurance company allegedly discriminating against those with "ethnic" names and even Hotmail addresses. But was AI at play here and was it running amok? Allegedly, of course.

You may have read recently that the insurer Admiral got into some hot water when they were allegedly discriminating against “foreign” names. Barely a week later they were in the press again for apparently discriminating against users of Hotmail email addresses.

What’s going on here?

Not so Admirable?

With over 11% of UK market share, they will have millions of policies and will be quoting on thousands more daily. Surely, in amongst all the data surrounding your type of car, your speeding points and whether you have a garage or not, they are not focusing on apparent trivialities like email domain or name. Of course, those decisions have not been made directly by a human for quite some time. The risk and premiums are calculated using complex, multi-variate models created by expert professionals called actuaries. These models then become the backbone of how a premium is calculated. So, could it be that Admiral’s actuaries purposefully set rules that discriminate in this manner? Are they racist and domainist (if that’s a thing)?

I would like to doubt it, especially as Admiral have already threatened the lawyers on these stories. So, assuming that these outcomes are real and Admiral are not intentionally discriminating then Machine Learning is the one to blame here…

AI don't believe it...

We’ve already heard the stories of Microsoft’s racist bot Tay making waves and causing consternation, but despite these early failures, Machine Learning is set to be a critical part of our future. Its ability to make huge volumes of data accessible and understandable will be critical. By 2020, it’s estimated the average internet user will be consuming 1.5GB of data per data and that autonomous vehicles will be spewing 4 Terabytes of data a day. From this data, insurance companies will have a more complete picture of customers from a variety of sources. They can then better manage risk and create products and services that serve their clients best. The future of car insurance NEEDS Machine Learning.

Unfortunately, these stories from Admiral suggest that it’s a machine making these poor decisions…

Somewhere in an ML algorithm learning from its data, it has picked up on some bias involving Hotmail users. But it doesn’t know that an email domain is necessarily irrelevant, just that it saw a bias. Somewhere in the data there will be a measurable cost benefit in predicting worse outcomes for Hotmail users.

But this is bad thinking. There is nothing innate about using Hotmail that contributes to your risk – you just change your email and that’s that. It’s more than likely that this factor is somehow correlated with other aspects of your life and work which are clearer reasons why you are a greater risk. E.g. – Hotmail users are perhaps of a certain age range and professional level. Those are the factors that risk would typically look to but the algorithm found the bias at a higher level.

The bias within

It’s likely a similar story with the Mohammed example. Are “Mohammeds” more likely to be professional drivers putting more miles on their car? Perhaps using it as an Uber part time? Is that the actual causal relationship? I am not saying this IS the case, merely that it may be the case and where the bias enters the model. But somewhere along the way, drivers called Mohammed have been weighted towards greater risk for factors perhaps not immediately obvious.

This is, to a large part, due to the imbalanced nature of insurance and claims data. For example. consider how often you’ve ever had an accident compared to how often you haven’t – what they’re looking to do is predict risk, but the risk is already very small, in order to run their ML to predict the actual risk they’re going to have to exaggerate the risks – which means you could end up with more false positives than through traditional underwriting methods. (Read more about ML data preparation)

Don't panic!

So, is using artificial intelligence to price risk such a terrible idea? Are we just overcomplicating? No. But like any emerging technology, it needs to be treated with care. As AI theorist Eliezer Yudkowsky once said “By far, the greatest danger of Artificial Intelligence is that people conclude too early that they understand it.” Machine Learning WILL revolutionise the financial service sector in almost every corner, but it’s got a lot to learn. WE ALL have a lot to learn. And here in EQTR\X (Equator’s Innovation team) we’re always learning, and always sharing. Stay tuned for a LOT more on AI and ML.