The Ethics of AI in Education

A presentation given at the Middle East Teaching and Learning Conference on 26th August 2020.

The video is available here:

The slides are available here:

 

This presentation looks at how AI works, how it is being used presently in Education and then outline some concerns about how AI might be used in education in the future. 

In it I argue that AI has a much greater part to play in Education – particularly in making education more widely available in the developing world and in reducing the cost of education. The talk then moves on to discuss general ethical concerns about how AI is being used in society, looking at the issue of how we program autonomous vehicles as a case in point. I then outline five areas of concern about the use (and potential abuse) of AI in education arguing that we need to have a much more informed debate before things go too far. With this in mind, I close with some suggestions for courses and reading that might help colleagues to become better informed about the subject.

Invisible Women - by Caroline Criado-Perez - Book Summary

Some books change society - Invisible Women by Caroline Criado-Perez sets out to be one of those. It is a book that every legislator, every employer, indeed, everyone who believes in a truly equal society should read.

The core argument is simple:  

• There is a significant gender data gap because most of the data which we collect is not sex-disaggregated.
• This matters because 'data determines how resources are allocated' (p.256)
• This means that 'Male is the default Human' with a consequence that
• The world is designed by men for men and this is not working (and, at times, dangerous) for women.
• Furthermore, 'Gender neutral does not automatically mean gender equal' (p.309)

Criado-Perez demonstrates this thesis by looking in turn at range of issues which illustrate that women have different priorities and needs:

Daily Life

• 'Women's travel patterns tend to be more complicated' than men's because they do more unpaid care work (p.30).
• Women have different needs re toileting (this is a safety issue both in terms of hygiene and attacks from men). This strikes a chord with anyone who has witnessed the queues for women's toilets

Workplace

'Women's work, paid and unpaid, is the backbone of our society and our economy' (p.142).

•  'Women do the majority of unpaid work irrespective of the proportion of household income they bring in' (p.71) - 'Globally, 75% of unpaid work is done by women'. This has implications of women's health and financial status. There is a need for properly paid maternity leave.
• 'Women continue to be disadvantaged by a working culture that is based on the ideological belief that male needs are universal' (p.86). 
• There is a significant gender pay gap, which needs addressing - collecting and analysing data on hiring procedures to see whether these are gender neutral is a good place to start (p.110).
• There are a range of occupational health issues which affect women differently to men (Chapter 5)
• There are significant issues with sexual harassment at work (estimated 50% of women in the EU p.137) which need addressing.
• 'The culture of paid work as a whole needs a radical overhaul. It needs to take into account that women are not the unencumbered workers the traditional workplace has been designed to suit' (p.91).

Design

There is a 'one-size-fits-men approach to supposedly gender-neutral products [which] is disadvantaging women' (p.157).

• The problem is not women, it is with male-based design. 
• Examples: Grand Piano keyboard width (p.157), Smartphones (p.159), Voice-recognition software (p.162 - if it doesn't work for a woman - try lowering your voice!); online datasets (p.164).
• Women find it difficult to get funding for design ideas. One factor is that '93% of VCs are men and men back men' (p.171).  'Given the male domination of VCs, data gaps are particularly problematic when it comes to tech aimed at women' (p.174).
• 'The [male-dominated] tech industry is rife with examples of tech that forgot about women' (p.176).
• 'Car design has a long and ignominious history of ignoring women' (p.185). 'When a women is involved in a car crash she is 47% more likely to be seriously injured than a man, and 71% more likely to be moderately injured . . . She is also 17% more likely to die' (p.186).

Medicine

We have 'a medical system which, from root to tip, is systematically  discriminating against women, leaving them chronically misunderstood, mistreated and misdiagnosed' (p.196)

• There are significant sex differences (pp.198ff):
• fundamental mechanical workings of the heart
• lung capacity
• women are 3x more likely to develop an autoimmune disease
• blood-markers for autism - etc
• 'Because women have largely been excluded from medical research, this data is severely lacking' (p.199) ... 'The failure to include women in medical trials is a historical problem that has its roots in seeing the male body as the default human body' (p.201).
• 'The lack of sex-disaggregated data affects our ability to give women sound medical advice' (p.208).
• There are many drugs and treatments that don't work for women.
• Yentyl Syndrome: 'women are misdiagnosed and poorly treated unless their symptoms or diseases conform to that of men' (p.217).
• There is a lack of research into medical issues that mainly or only affect women (p.229)

'Women are dying, and the medical world is complicit. It needs to wake up.' (p.216)

Public Life

'The failure to measure unpaid household services is perhaps the greatest gender gap of all.' (p.241)

• Unpaid work is excluded from definitions of Gross Domestic Product (GDP). 'The omission of housewives from national income computation distorts the picture' (Paul Studenski, p.241).
• 'The upshot of the failure to capture all this data is that women's unpaid work seems to be seen as a costless resource to exploit' (Sue Himmelweit, p.244)
• Taxation: 'There's a fairly simple reason why so many tax systems discriminate against women, and that is that we don't systematically collect data on how tax systems affect them' (p.260)

'Together with our woman-blind approach to GDP and public spending, global tax systems are not simply failing to alleviate gendered poverty: they are driving it.' (p.264)

• 'The presence of women in politics makes a tangible difference to the laws that get passed' (p.266).
• Discussion of women in politics - and a call for electoral reform.
• 'The evidence is clear: politics as it is practised today is not a female-friendly environment' (p.281).
• 'When you exclude half the population from a role in governing itself, you create a gender gap a the very top' (p.285)
• 'Female politicians are not operating on a level playing field' (p.286)

Disaster Situations

• 'The failure to include women in post-disaster efforts can end up in farce.' (p.290). This is illustrated with examples from Gujurat in 2001; Miami 1992 Hurricane Andrew; New Orleans 2005 Hurricane Katrina
• 'The presence of women at the negotiating table not only makes it more likely that an agreement will be reached, it also makes it more likely that the peace will last' (p.294).
• 'Women are disproportionately affected by conflict, pandemic and natural disaster ... Domestic violence against women increases when conflict breaks out' (p.296).
• 'The data gap when it comes to sexual abuse is compounded in crisis settings by powerful men who blur the lines between aid and sexual assault' (p.306)
• Homelessness: 'women are actually more likely to experience homelessness than men' (p.306)

Afterword

• Criado-Perez identifies three important themes that define women's relationship with the world:

1. The female body: this needs to be taken into account in design medical, technological and architectural.
2. Male sexual violence against women: we need to measure it and design out world to account for it - not to do so is to limit women's liberty.
3. Unpaid care work: we need to measure this and to make it more equitable.

'Failing to collect data on women and their lives means that we continue to naturalise sex and gender discrimination.' (p.314)

One major reason why gender data gap needs to be addressed as a matter of priority is because 

'the introduction of Big Data into a world full of gender data gaps can magnify and accelerate already-existing discriminations.' (p.136)

The Book

This is a very well researched book. It is fully referenced and indexed allowing the reader easily to follow up on the points made. 

Artificial Intelligence, Ethics and Education

What is AI? 

Artificial Intelligence refers to those computer systems which are both autonomous and adaptive; i.e. they are systems which have the ability to perform complex tasks without constant guidance by a user and they also have the ability to improve performance by learning from experience. The process of getting computers to learn without being explicitly programmed is called Machine Learning. 

We are all familiar with the increasing role that AI is playing an increasing part in our lives. Machine-learning is managing our email junk folders; it is suggesting the next word when we are texting; it is labelling and organising our photo albums; and it makes suggestions on what we should buy next from Amazon or watch next on Netflix. Most of these functions rely on ‘Supervised’ Machine Learning algorithms that are developed on the basis of an initial training set of data, which is then supplemented as further information becomes available. 

AI and Ethics 

Ethical concerns about AI revolve around ‘algorithmic bias’ i.e. around the validity of the way in which the algorithm is constructed and usually around the nature of the training dataset. These arguments take three forms: 

1. Concerns about Bias: the training dataset on which the algorithm was originally constructed may not reflect the composition of the wider population. To take an extreme example, a dataset that is based on American billionaires is likely to be white, educated, aged over 45, male and, by definition, rich.
2. Concerns about Fairness: the training dataset is based on accurate historic data, but those data reflect unfair practices. For example, in 2011 the City of Boston MA launched ‘Street Bump App’ which maps the location of potholes that needed repairing around the city by collecting data from the accelerometer in the Smartphone. The app successfully collected data and saved the City time and money in surveying the roads. However, a review of the project after 12 months showed that a disproportionate number the potholes identified and repaired were in affluent middle-class areas, to the detriment of those in poorer areas. This was almost certainly because affluent middle-class residents were more likely to own a smartphone and were more likely to download the app. In similar vein, data scientist
   
   Cathy O’Neill, author of Weapons of Math Destruction, has voiced her concerns about the way in which the algorithms are being used in the US criminal justice system. The police are using historic arrest data as a proxy for crime data to drive preventative policing models. Because of this, the algorithm simply reinforces historic practice by sending the police back to the neighbourhoods which they are already over-policing; and are not sent to neighbourhoods which have crime, but those crimes are found. The irony is that, in these examples, the intention was to create algorithms which were free from human bias, however, because of the way in which they were constructed that had unintended consequence of perpetuating historic inequalities. 
3. Concerns about Unethical Behaviour: the dataset is deliberately skewed or designed to behave in a dishonourable way. AI is fundamentally an ethically neutral platform. It can be used or misused like any other technology. History teaches us that most technologies are misused at some point. 

In order to avoid historic or intentional bias, it is necessary to develop new protocols. Once designers deviate from historic data and endeavour to build an algorithm which is based on data which is deemed both unbiased and fair, they are presented with some quite serious ethical challenges. Here there will be parallels here to the debates about the value of positive discrimination in the workplace. One way to manage AI is to establish some protocols which will ensure that we avoid algorithmic bias - and here diversity is the key. There needs to be a Diversity of Background and a Diversity of Mindset of the team building the algorithms to avoid “group think”; a Diversity of Data that comprises any training set; and Diversity of Algorithmic Models used. 

Looking ahead it is likely that there will need to be formal regulation of algorithm design (rather akin the way in which financial services is regulated) which will entail the development of regulatory function of ethical audit. This role will ensure that algorithms are not subject to intentional or unintended bias. 

The Ethics of AI in Education 

The use of AI in education is in its infancy. We are beginning to see adaptive learning platforms, such as CenturyTech, being used in schools, primarily to supplement and support what teachers are doing in the classroom. Whether or not this is the first tentative step towards the ‘Holy Grail’ of fully adaptive and personalised learning that does not require teacher input is a debate for another day. Looking ahead it is like AI in Education is likely to pose some significant ethical issues. 

1. First, as those who are embroiled in GDPR know only too well, there are a whole range of concerns about the security, ownership and privacy of personal student data that is captured and stored within an AI platform. There will need to policies and protocols in this area. 
2. Secondly, there are concerns about the fairness of access to AI technologies and the potential for AI to increase the ‘Digital Divide’ between those who can afford access to the technology and those who don’t. 
3. Thirdly, there is a danger of having a biased training set on which educational AI technologies are founded reinforce social/ cultural/ etc. stereotypes. For example, it is quite possible that the dataset for an AI learning platform might be skewed because the early adopters all come from affluent fee-paying schools who can afford to provide access. 

Assessment Algorithms. 

Perhaps the greatest ethical issues might come around AI being used to make significant summative assessments of students’ abilities in the allocation of places at university or into the jobs market. We have already seen the ‘Big Four’ accountancy firms preferring their own assessment platforms to consideration of A-level and Degree results in order to find recruits who have the most potential (e.g. ‘Big Four’ look beyond academics – Financial Times 28/02/2016). It is quite possible to conceive of a time when both universities and employers, motivated from the noble intention of assessing potential and facilitating social mobility, will rely on their own assessment recruitment algorithms to identify suitable candidates. If this were to have it would be vital that any algorithm be subject to rigorous ethical audit to ensure that it meets a standard test of fairness. 

Final Remarks 

We have only begun to realise the potential that Artificial Intelligence has to shape C21 society and, sadly, social and ethical debate is struggling to keep up with the development of the technology. There needs to be an informed debate about the place of AI in society, and particularly of how it is going to be applied in education. In order to do this, we need a much greater understanding in society of how AI and Machine Learning work – and that is a challenge which I hope will be taken up by schools over the coming months and years.

This article was published in Digital Strategy Edition 2, March 2019 by the ISC Digital Strategy Group.

Weapons of Math Destruction by Cathy O’Neil - Book Summary

This is a very important book that seeks to call to account algorithm-based automated computer systems which are increasingly making decisions about a whole range of aspects of our lives. With a Harvard Doctorate in algebraic number theory, Cathy O’Neill has an insider’s of view of WMDs, having both lectured on the subject and subsequently worked for a Hedge Fund on Wall Street at the time of the 2008 crash. 

The central theme is that those designing these systems which Cathy O’Neill wittily dubs Weapons of Math Destruction (WMDs) often set out with the best of intentions but that they are often very flawed. WMDs often promise efficiency, fairness and freedom from human prejudice, but in practice sacrifice fairness, justice and equality on the altars on efficiency and profit. 

The book has a simple structure working through how WMDs are making (flawed) decisions about: 

• Banking 
• University Entrance 
• Online Advertising 
• Policing and Justice 
• Job Recruitment 
• Hours of Employment and Shift Patterns 
• Credit Ratings • Getting Insurance 
• Political Elections and Social Media Advertising Campaigns 

Her fundamental criticisms of many of these WMDs are 

1. they are opaque, 
2. they don't take into account feedback to improve the mathematical model, and 
3. they rarely use real data; instead they use proxies. 

The scariest example (of many in the book) was that insurance companies use credit scores as a proxy for careful driving - thus in Florida adults with clean driving records and poor credit scores paid an average of $1,552 more than the same drivers with excellent credit scores and a drunk driving conviction (p.165) Indeed, the whole chapter on credit ratings and how various organisations are using big data to evaluate our credit-worthiness is quite frightening. (Did you know that Facebook patented a type of credit rating based on our social networks? (p.155 – See ‘Could a Bank Deny Your Loan Based on Your Facebook Friends?’ Atlantic Magazine 25/09/2015). 

The losers of so many of these WMDs is that the poor and minorities who are caught in a trap from which they cannot escape, be that policing algorithms which mean that they are more likely to be stop-and-searched; or it recidivism algorithms which give them longer jail sentences, or banking and insurance algorithms which charge them more for credit or insurance cover. 

The book is ultimately a call for greater regulation of WMDs so that there is greater transparency about their assumptions and methods. This is a must read for anyone who believes in a fair civil society.