The Ethics of AI in Education

A presentation given at the Middle East Teaching and Learning Conference on 26th August 2020.

The video is available here:

The slides are available here:

 

This presentation looks at how AI works, how it is being used presently in Education and then outline some concerns about how AI might be used in education in the future. 

In it I argue that AI has a much greater part to play in Education – particularly in making education more widely available in the developing world and in reducing the cost of education. The talk then moves on to discuss general ethical concerns about how AI is being used in society, looking at the issue of how we program autonomous vehicles as a case in point. I then outline five areas of concern about the use (and potential abuse) of AI in education arguing that we need to have a much more informed debate before things go too far. With this in mind, I close with some suggestions for courses and reading that might help colleagues to become better informed about the subject.

Daniel Susskind A World Without Work (2020) Summary of Key Arguments

Daniel Susskind, Economics don at Balliol Oxford, leads the reader through a persuasive well-structured argument that the automation that we are seeing today will be profoundly different from technological change in the past. The consequence of this is that we need to rethink the status of work and look to new ways to shape our society in a ‘world without work’.

The Context 

Ch.1. A History of Misplaced Anxiety 

A survey of the effect of technology on work over the past three centuries is characterised by two rival forces: a harmful substituting force which displaces human beings from performing particular tasks because the technology is faster or cheaper; and a helpful complementing force which raises demand for the work of humans to perform other tasks (e.g. the rollout of ATM machines did not replace bank tellers, but it meant that their role changed to provide customers with a better service p.26)
The helpful complementing force to date has done this in 3 ways:

1. the Productivity Effect: machines have made displaced workers more productive at other activities; 
2. the Bigger Pie Effect: technology has made economies and incomes around the world much bigger; 
3. the Changing Pie Effect: technology changed how consumers spend their incomes and how producers make goods and services available.

“Up until now, in the battle between the harmful substituting force and helpful complementing force, the latter has won out and there has always been large enough demand for the work that human beings do” (p.28)

Ch.2. The Age of Labour 

“a time when successive waves of technological progress have broadly benefited rather than harmed workers.” Autor-Levy-Murnane ALM Hypothesis: Machines could readily perform the ‘routine’ tasks in a job, but would struggle with the ‘non-routine’ ones (p.39) Technological progress is neither skill-biased or unskilled-biased it was task-biased (p.40).

Ch.3. The Pragmatist Revolution and Ch,4. Underestimating Machines

Greek Poet Archilochus: ‘The fox knows many things, but the hedgehog knows one big thing.’ p.64 – “we should be wary not of one omnipotent fox, but of an army of industrious hedgehogs” p.67)
AI Purists (cognitive scientists) closely observe human beings acting intelligently and try to build machines like them. This approach and the quest for Artificial General Intelligence AGI, to date, has failed. (c.f. the omnipotent fox remains illusive)
AI Pragmatists (computer scientists) take a task that requires intelligence when performed by a human being and build a machine to perform it in a fundamentally different way – relying on advances in processing power and data storage. The quest for Artificial Narrow Intelligence ANI is proving quite fruitful (c.f. the army of industrious hedgehogs).

“The temptation is to say that because machines cannot reason like us, they will never exercise judgement; because they cannot think like us they will never exercise creativity; because they cannot feel like us, they will never be empathic. All that may be right. But it fails to recognise that machines might be able to carry out tasks that require empathy, judgement or creativity when done by a human being – but doing them in some entirely different fashion.” (p.72-73)

“We do not need to solve the mysteries of how the brain and mind operate to build machines that can outperform human beings.” (p.74)

The Threat 

5. Task Encroachment 

Technology is encroaching on all areas of work: Manual capabilities (agriculture, driverless cars, car manufacturing, construction industry, 3D printing); Cognitive capabilities (Law, Medicine, Education, Finance, Insurance, Botany, Journalism, Military – “Computational creativity”) Affective Capabilities (“affective computing”, “social robotics”)
Task encroachment will be taken up at different paces. Because

1. “some tasks are far harder to automate than others”; 
2. in some industries the cost of human labour is low and complexity of automation is high (e.g. cleaning, hairdressing, table-waiting); 
3. different cultures and jurisdictions will have different attitudes and regulations. 

Ch.6. Frictional Technical Unemployment 

“There is still work to be done by human beings; the problem is that not all workers are able to reach out and take it up” (p.99).
Three reasons for this:

1. A Skills Mismatch (work available for much more qualified or more skilful); 
2. An Identity Mismatch (work available but doesn’t fit will self-image – not a graduate job, or a man’s job – jobless lorry drivers may not want to do “pink collar” work such as child care); 
3. A Place Mismatch (the work available may be in a different part of the country/ world) 

There will be 3 consequences of Frictional Unemployment: the “technological overcrowding, with people packing into a residual pool of whatever work remains within their reach” (p.109):

1. there will be downward pressure on wages; 
2. there will be downward pressure on the quality of the work; 
3. there will be downward pressure on the status of the work available (rich-poor, master-servant divide). 

Ch.7. Structural Technical Unemployment 

In the future the Complementing Force is likely to be much weaker: 1) The Productivity Effect – “As task encroachment continues, human capabilities will be come irrelevant . . . for more and more tasks” (p.114); 2) The Bigger-Pie Effect – “a growing demand for good may mean not more demand for the work of human beings, but merely more demand for machines” (p.116); 3) The Changing Pie Effect – for consumers, “As task encroachment continues, it becomes more and more likely that changes in demand for goods will not turn out to be a boost in demand for the work of humans, but of machines” (p.119); and for producers, “As task encroachment continues, will it not become sensible to allocate more of the complex new tasks to machines instead?” (p.121).

“It is a mistake to think that there is likely to be enough demand for them [human beings] to keep everyone in work” (p.124). 

For Susskind, at present there is an assumption that human beings are superior to machines; in future we may need to assume that we are inferior.

“Just as today, we talk about ‘horsepower’ harking back to a time when the pulling power of a draft horse was a measure that mattered, future generations may come to use the term ‘manpower’ as a similar kind of throwback, a relic of a time when human beings considered themselves so economically important that they crowned themselves as the unit of measurement” (p.130). 

Ch. 8. Technology and Inequality 

 “The largest economic pies, belonging to the most prosperous nations, are being shared out less equally in the past” (p.137) 

The longstanding relationships between traditional (33.3%) and human capital (66.6%) is changing: Traditional Capital “is everything owned by the residents and governments of a given country at a given point of time, provided that it can be traded on some market” (p.133). Human Capital is “the entire bundle of skills and talents that people build up over their lives and put to use in their work” (p.134).
Susskind identifies three trends (p.146):

1. “human capital is less evenly distributed”; 
2. “human capital is becoming less and less valuable relatively to traditional capital”; 
3. “traditional capital is distributed in an extraordinarily uneven fashion”. 

“Today many people lack traditional capital, but still earn an income from the work that they do, a return on their human capital. Technological unemployment threatens to dry up this latter stream of income as well, leaving them with nothing at all” (p.149). 

The Response 

Ch.9. Education and its Limits 

“’More education’ remains our best response at the moment to the threat of technological employment."

We can do this in 3 ways:

1. What we teach: “do not prepare people for tasks that we know that machines can already do better; or activities that we can reasonably predict will be done better by machines very soon” (p.158) N.B. “Many tasks that cannot yet be automated are found not in the best-paid roles, but in jobs like social workers, paramedics and schoolteachers.” 
2. How we teach: non traditional blended-learning and online learning methods need to become more commonplace. 
3. When we teach: we need to move to a world of life-long learning: “People will have to grow comfortable with moving in and out of education, repeatedly, throughout their lives. We will have to constantly re-educate ourselves” (p.160). 

BUT

“Even the best existing education systems cannot provide the literacy, numeracy and problem-solving skills that are required to help the majority of workers compete with today’s machines” (p.165). 

“Some people may cease to be of economic value: unable to put their human capital to productive use, and unable to re-educate themselves to gain other useful skills” (p.166). 

“Education will also struggle to solve the problem of structural technological unemployment. If there is not enough demand for the work that people are training to do, a world-class education will be of little help” (p.166). 

Ch. 10. The Big State 

The role for the state in the C21 world without work is to deal with the looming disparities and inequalities in society. It will do this through taxation and redistribution of income and wealth. Taxation: 

1. 1) taxing workers who have managed to escape the harmful effects of task encroachment; 
2. 2) taxing capital – taxing “the income that flows to owners of increasingly valuable traditional capital” (p.176); 
3. taxing big business – this needs to be tackled at a global level. 

Redistribution: Susskind rejects the idea of Universal Basic Income (UBI) which has no strings attached. He has an excellent critique of the problems associated with membership criteria for UBI. He argues instead for a Conditional Basic Income (CBI) which requires recipients to contribute in some way (to be defined) to society. This is based on a view of ‘contributive justice’ whereby everyone feels that their fellow citizens are giving back to society. His vision is for a ‘Capital-sharing State’ and a ‘Labour-supporting State’.

Ch. 11. Big Tech 

For Susskind, Big Tech companies are here to stay for two reasons:

1. Expensive Resources: it costs enormous amount to develop new technologies – huge amounts of data, world-leading software, and extraordinarily powerful hardware. Small firms cannot compete and talented ones just get bought out. 
2. Network effects – networks are more rewarding the bigger they get. 

Susskind rejects the economic arguments against large monopolistic firms – most are not abusing their monopoly economically. Instead he questions their political and social influence. Here he argues for new regulatory institutions which can insist on greater transparency and ensure that liberty, democracy and social justice are not under threat.

Ch,12. Meaning and Purpose 

A world without work throws up philosophical questions about how human beings find meaning; and practical questions of how people will spend their leisure time: volunteering, unpaid work, community required activities (see Conditional Basic Income above).

 “A job is not simply a source of income but of meaning, purpose and direction in life as well” (p.215). 

In the C21 “Work is the opium of the People”. Work has meaning beyond the purely economic. “The problem is not simply how to live, but how to live well” (p.236).
Revisiting Education. Spartan King Agesilaus: ‘the purpose of education is to teach children the skills that they will use when they grow up.’ Perhaps schools should prepare young people for a world of leisure: character, virtue, life skills education.

“If free time does become a bigger part of our lives, then it is likely also to become a bigger part of the State’s role as well” (p.234) 

A world without work throws up three fundamental problems:

1. the problem of inequality; 
2. the problem of political power; 
3. the problem of meaning.

This is an important book - essential reading for anyone who is interested in preparing young people for the future.

AI in Education: Four Risks that we need to consider

The Holy Grail of EdTech is an AI-driven autonomous system that provides a personalised learning experience with minimal teacher input. This is not surprising given the potential financial rewards that such a system would bring. Education is very big business. According to IBIS Capital [Global EdTech Industry Report 2016], education was $5 trillion industry globally in 2014 and is growing at $600 billion a year. However, only 2% of it is digitised – education as an industry is a ‘late adopter’ at best. EdTech is increasingly being seen as a ripe market for investors. Three factors make it very attractive: the importance and cost of education to Governments and parents; the global teacher shortage and the unsustainability of the present model of one teacher to 20-30 students; and the abundant scope for new markets (there are currently 263m children in the world not in education). There is scope here to be the ‘Amazon of Education’ - no wonder the Venture Capitalists are turning their attention to EdTech.

However, we are not talking about the automation of the sale of books here - we are talking about the education of young people and there a range of ethical concerns which need to be addressed at an early stage.

1. Skewed Dataset. We are still in the ‘Early Adopter’ phase of the use of Educational AI. It is at present an ‘enhancement’ of existing educational processes and therefore pioneering schools need to be at a relatively high level of digital and pedagogical sophistication to be able to deploy these systems. We need to recognise that these systems are being developed using an unrepresentative dataset which has implications for fairness and wider applicability down the line. 
2. AI Summative Judgements. At present educational platforms are using AI to make formative judgements on individual students (based on what is likely to be a globally unrepresentative dataset) in order to tailor learning to the needs of the individual student. There is a risk down the line that we may see AI educational systems replace the present public examination system, making summative judgments that have a significant impact on life choices such as access to higher education or certain levels of employment. Furthermore, judgments might be made on wider criteria than presently deployed. For example, rather than just assessing whether or not a student has mastered a particular skill or unit of knowledge, it would be possible to assess how fast a student takes to so.
3. Digital Divide: As AI becomes more commonplace, there is a danger of an educational ‘digital divide’ between those generally wealthy countries who can afford to deploy AI educational systems and those who do not. (c.f. There were significantly lower participation from Africa in the recent MIT project Awad et al ‘The Moral Machine Experiment’ Nature Vol 563 November 2018).
4. Monopoly on Education. It is conceivable that, in time, there will be a few large companies, the 'Amazons of Education' if you like, who would dominate the automated-education industry. These providers would be able to control the content of what is taught (in the way that some Governments around the world do today). It would be possible, for example, for these companies to decide that the Holocaust should not be on the C20 history syllabus (as is the case in many Arab states today); or they might promote certain ideologies or lifestyle choices (vegetarianism, hetrosexuality, etc.). We should avoid a world where a minority have a monopoly on education. 

It is time to have the moral debate about AI in education

Horses pulling Automobiles: The Impact of AI on Education

Cars were forbidden in the town of Nantucket, Massachusetts, until 1918 – some 25 years after the Duryea brothers set up the first car manufacturing company in America. The story goes that, during this time, Clinton Folger, the Island’s postman, towed his "gasoline buggy" to the state highway so that he could then drive to Siasconset on the other side of Nantucket Island to deliver the mail. The picture of the horse-drawn automobile is an apt metaphor for what is happening with Artificial Intelligence in Education today. 

Artificial Intelligence (AI) learning platforms began to be launched at the primary and secondary education market a little over a year ago. We are still at the pioneering stage where the early adopters are trialing products, suggesting improvements and trying to evaluate their impact on teaching and learning. 

It would be wrong to see AI learning platforms simply as the next in a long-line of new technologies which have been harnessed to enhance what we are doing in the classroom, although there is little doubt that they can do this. AI is likely to transform nearly every industry; indeed, it will change society as a whole. It goes without saying that it has the potential to disrupt education. To understand the drivers behind this disruptive potential of AI we need to consider Education from a global perspective. 

Education is very big business. According to IBIS Capital [Global EdTech Industry Report 2016], education was $5 trillion industry globally in 2014 and is growing at $600 billion a year. However, only 2% of it is digitised – education as an industry is a ‘late adopter’ at best. So, EdTech is increasingly being seen as a ripe market for investors. Three factors make it very attractive: the importance and cost of education to Governments and parents; the global teacher shortage and the unsustainability of the present model of one teacher to 20-30 students; and the abundant scope for new markets (there are currently 263m children in the world not in education). Thus, the Holy Grail of EdTech is an effective AI platform that will solve these problems in a scalable and affordable way by providing a personalised learning experience with minimal teacher input. There is scope here to be the ‘Amazon of Education’ - no wonder the Venture Capitalists are turning their attention to EdTech.

To date, the AI platforms which have been launched are automating and enhancing aspects of the teaching and learning process. Typically, they conduct a base-line test, they introduce relevant content, which they then test, and using adaptive algorithms provide personalised feedback to the learner, their parents and the teacher. This is helpful but it not really embracing the full transformatory power of AI. In short, we are using AI in education to achieve C20 educational outcomes. It is rather like the situation faced by the postman of Nantucket: we have invented the automobile, but we are being forced by the regulators to pull it along with a horse because it doesn’t fit with their outdated view of the world. There is huge latent potential in the way in we AI is being developed that will change radically the educational landscape globally and in the UK. 

Teacher-Pupil contact time is the most precious resource that schools have. This is particularly true of specialist teacher time (e.g. suitably qualified Physics teachers) which is already a scarce resource. Looking ahead five to ten years, AI learning platforms will not replace teachers, but they will change what teachers do, especially in secondary schools as new technologies, including AI, allow schools to make the most of the limited teacher time available. Looking at things globally, in this Brave New World, the quality of the education available will be driven by cost; and the amount and nature of the human contact time available will determine the price-point. 

As Clayton Christensen has argued, disruptive technologies get their foothold in new and emerging markets and then gradually work their way into the mainstream. On this basis it is possible to predict how AI will transform education. AI learning platforms will have their greatest impact in markets where there currently there is no education available. Budget Secondary Education will not have face-to-face contact with qualified teachers but will be delivered totally through online courses on learning platforms. This is not a form of schooling that is recognisable to western educationalists, but for many young people around the world this will be better than the present situation of receiving no education at all. Moving up to Mid-Range Secondary Education, this will be delivered through blended learning programmes which combine AI learning platforms, subject specialist teaching via Virtual-Reality conferencing, and some face-to-face contact with teachers in a bricks-and-mortar environment. The US Public School system is in the vanguard of this (for an overview see Keeping Pace with K-12 Digital Learning Reports). However, to date, there has been little appetite for adopting this model in the UK as was witnessed when counsellors rejected the plans to use Blended Learning at the Ark Pioneer Academy in Barnet (see Tes 30/01/2017). Finally, it will only be in top fee-paying private schools and in the state sector in the wealthiest countries of the world that Premium Secondary Education that will be delivered by specialist teachers in classrooms. Face-to-face teaching in a class of 20-30 will be a luxury (indeed from a global perspective, it already is). Here AI platforms will enhance and augment the learning process. 

Given the level of investment by the private sector into EdTech, it is clear that AI in Education is here to stay and the days of the Horse-drawn Automobile are coming to an end. So how best to prepare for the future? Perhaps the most important thing that educationalists can do at this time is inform themselves as to how AI works and to join the debate about what constitutes the ethical use of AI in education before it’s too late.

This article was published in Tes as 'Teaching's use of AI is like a horse-drawn automobile' 16/05/2019

The Machines are coming: Automated Systems and AI in Education

A Presentation given at the COBIS Annual Conference in London on Monday 13th May 2019



The presentation looks at:

1. How automated systems can handle data; 
2. How automated systems can provide live data to parents bringing an end to school reports 
3. AI in Education 
4. Will AI ever replace teachers?

Artificial Intelligence, Ethics and Education

What is AI? 

Artificial Intelligence refers to those computer systems which are both autonomous and adaptive; i.e. they are systems which have the ability to perform complex tasks without constant guidance by a user and they also have the ability to improve performance by learning from experience. The process of getting computers to learn without being explicitly programmed is called Machine Learning. 

We are all familiar with the increasing role that AI is playing an increasing part in our lives. Machine-learning is managing our email junk folders; it is suggesting the next word when we are texting; it is labelling and organising our photo albums; and it makes suggestions on what we should buy next from Amazon or watch next on Netflix. Most of these functions rely on ‘Supervised’ Machine Learning algorithms that are developed on the basis of an initial training set of data, which is then supplemented as further information becomes available. 

AI and Ethics 

Ethical concerns about AI revolve around ‘algorithmic bias’ i.e. around the validity of the way in which the algorithm is constructed and usually around the nature of the training dataset. These arguments take three forms: 

1. Concerns about Bias: the training dataset on which the algorithm was originally constructed may not reflect the composition of the wider population. To take an extreme example, a dataset that is based on American billionaires is likely to be white, educated, aged over 45, male and, by definition, rich.
2. Concerns about Fairness: the training dataset is based on accurate historic data, but those data reflect unfair practices. For example, in 2011 the City of Boston MA launched ‘Street Bump App’ which maps the location of potholes that needed repairing around the city by collecting data from the accelerometer in the Smartphone. The app successfully collected data and saved the City time and money in surveying the roads. However, a review of the project after 12 months showed that a disproportionate number the potholes identified and repaired were in affluent middle-class areas, to the detriment of those in poorer areas. This was almost certainly because affluent middle-class residents were more likely to own a smartphone and were more likely to download the app. In similar vein, data scientist
   
   Cathy O’Neill, author of Weapons of Math Destruction, has voiced her concerns about the way in which the algorithms are being used in the US criminal justice system. The police are using historic arrest data as a proxy for crime data to drive preventative policing models. Because of this, the algorithm simply reinforces historic practice by sending the police back to the neighbourhoods which they are already over-policing; and are not sent to neighbourhoods which have crime, but those crimes are found. The irony is that, in these examples, the intention was to create algorithms which were free from human bias, however, because of the way in which they were constructed that had unintended consequence of perpetuating historic inequalities. 
3. Concerns about Unethical Behaviour: the dataset is deliberately skewed or designed to behave in a dishonourable way. AI is fundamentally an ethically neutral platform. It can be used or misused like any other technology. History teaches us that most technologies are misused at some point. 

In order to avoid historic or intentional bias, it is necessary to develop new protocols. Once designers deviate from historic data and endeavour to build an algorithm which is based on data which is deemed both unbiased and fair, they are presented with some quite serious ethical challenges. Here there will be parallels here to the debates about the value of positive discrimination in the workplace. One way to manage AI is to establish some protocols which will ensure that we avoid algorithmic bias - and here diversity is the key. There needs to be a Diversity of Background and a Diversity of Mindset of the team building the algorithms to avoid “group think”; a Diversity of Data that comprises any training set; and Diversity of Algorithmic Models used. 

Looking ahead it is likely that there will need to be formal regulation of algorithm design (rather akin the way in which financial services is regulated) which will entail the development of regulatory function of ethical audit. This role will ensure that algorithms are not subject to intentional or unintended bias. 

The Ethics of AI in Education 

The use of AI in education is in its infancy. We are beginning to see adaptive learning platforms, such as CenturyTech, being used in schools, primarily to supplement and support what teachers are doing in the classroom. Whether or not this is the first tentative step towards the ‘Holy Grail’ of fully adaptive and personalised learning that does not require teacher input is a debate for another day. Looking ahead it is like AI in Education is likely to pose some significant ethical issues. 

1. First, as those who are embroiled in GDPR know only too well, there are a whole range of concerns about the security, ownership and privacy of personal student data that is captured and stored within an AI platform. There will need to policies and protocols in this area. 
2. Secondly, there are concerns about the fairness of access to AI technologies and the potential for AI to increase the ‘Digital Divide’ between those who can afford access to the technology and those who don’t. 
3. Thirdly, there is a danger of having a biased training set on which educational AI technologies are founded reinforce social/ cultural/ etc. stereotypes. For example, it is quite possible that the dataset for an AI learning platform might be skewed because the early adopters all come from affluent fee-paying schools who can afford to provide access. 

Assessment Algorithms. 

Perhaps the greatest ethical issues might come around AI being used to make significant summative assessments of students’ abilities in the allocation of places at university or into the jobs market. We have already seen the ‘Big Four’ accountancy firms preferring their own assessment platforms to consideration of A-level and Degree results in order to find recruits who have the most potential (e.g. ‘Big Four’ look beyond academics – Financial Times 28/02/2016). It is quite possible to conceive of a time when both universities and employers, motivated from the noble intention of assessing potential and facilitating social mobility, will rely on their own assessment recruitment algorithms to identify suitable candidates. If this were to have it would be vital that any algorithm be subject to rigorous ethical audit to ensure that it meets a standard test of fairness. 

Final Remarks 

We have only begun to realise the potential that Artificial Intelligence has to shape C21 society and, sadly, social and ethical debate is struggling to keep up with the development of the technology. There needs to be an informed debate about the place of AI in society, and particularly of how it is going to be applied in education. In order to do this, we need a much greater understanding in society of how AI and Machine Learning work – and that is a challenge which I hope will be taken up by schools over the coming months and years.

This article was published in Digital Strategy Edition 2, March 2019 by the ISC Digital Strategy Group.

The Rise of AI

A keynote presentation given at the ISBA Digital Strategy and Cybersecurity Conference at the BMA in London on Wednesday 27th March 2019.
The presentation looks at the following areas:

1. What is AI?
2. The Ethics of Ai.
3. AI, Education and the #FutureSchool
4. The threat of AI to Fee-paying Education

Transforming Learning in a Millennial World

My keynote presentation at the BSME Annual Conference held at Yas Marina Conference Centre in Abu Dhabi on Wednesday 20th March 2019

Weapons of Math Destruction by Cathy O’Neil - Book Summary

This is a very important book that seeks to call to account algorithm-based automated computer systems which are increasingly making decisions about a whole range of aspects of our lives. With a Harvard Doctorate in algebraic number theory, Cathy O’Neill has an insider’s of view of WMDs, having both lectured on the subject and subsequently worked for a Hedge Fund on Wall Street at the time of the 2008 crash. 

The central theme is that those designing these systems which Cathy O’Neill wittily dubs Weapons of Math Destruction (WMDs) often set out with the best of intentions but that they are often very flawed. WMDs often promise efficiency, fairness and freedom from human prejudice, but in practice sacrifice fairness, justice and equality on the altars on efficiency and profit. 

The book has a simple structure working through how WMDs are making (flawed) decisions about: 

• Banking 
• University Entrance 
• Online Advertising 
• Policing and Justice 
• Job Recruitment 
• Hours of Employment and Shift Patterns 
• Credit Ratings • Getting Insurance 
• Political Elections and Social Media Advertising Campaigns 

Her fundamental criticisms of many of these WMDs are 

1. they are opaque, 
2. they don't take into account feedback to improve the mathematical model, and 
3. they rarely use real data; instead they use proxies. 

The scariest example (of many in the book) was that insurance companies use credit scores as a proxy for careful driving - thus in Florida adults with clean driving records and poor credit scores paid an average of $1,552 more than the same drivers with excellent credit scores and a drunk driving conviction (p.165) Indeed, the whole chapter on credit ratings and how various organisations are using big data to evaluate our credit-worthiness is quite frightening. (Did you know that Facebook patented a type of credit rating based on our social networks? (p.155 – See ‘Could a Bank Deny Your Loan Based on Your Facebook Friends?’ Atlantic Magazine 25/09/2015). 

The losers of so many of these WMDs is that the poor and minorities who are caught in a trap from which they cannot escape, be that policing algorithms which mean that they are more likely to be stop-and-searched; or it recidivism algorithms which give them longer jail sentences, or banking and insurance algorithms which charge them more for credit or insurance cover. 

The book is ultimately a call for greater regulation of WMDs so that there is greater transparency about their assumptions and methods. This is a must read for anyone who believes in a fair civil society.

The Ethics of AI

A theory of Knowledge Lecture given on Monday 4th March 2019 to the Lower Sixth at JESS, Dubai.