Press Release 135/2019

The Risk of Being Discriminated by the Algorithm

KIT Study Identifies Many Possibilities of Unequal Treatment and Recommends Preventative Measures
Not always fair: When humans are evaluated by algorithms, care must be taken. (Photo: Patrick Langer, KIT)
Not always fair: When humans are evaluated by algorithms, care must be taken. (Photo: Patrick Langer, KIT)

Not only companies, also state institutions increasingly rely on automated decisions by algorithm-based systems. Their efficiency saves time and money, but also entails many risks of individuals or population groups being discriminated. This is the result of a study made by the Institute for Technology Assessment and Systems Analysis (ITAS) of Karlsruhe Institute of Technology (KIT) on behalf of the Federal Anti-Discrimination Agency.


When granting a loan, selecting new staff members, or making legal decisions - in an increasing number of sectors, algorithms are applied to prepare human decisions or to make these decisions for humans. “Unfortunately, it often is a mistake to think that this inevitably leads to more objective and fairer decisions,” says Carsten Orwat of the Institute for Technology Assessment and Systems Analysis (ITAS) of KIT. “Situations become particularly critical when algorithms work with biased data and rely on criteria that ought to be protected,” the author says. These criteria include, in particular, the age, gender, ethnic origin, religion, sexual orientation, and handicaps.


On behalf of the Federal Anti-Discrimination Agency, Carsten Orwat studied in detail the causes of discrimination, its impact on society, and future options to reduce discrimination risks. His study entitled “Diskriminierungsrisiken durch Verwendung von Algorithmen” (discrimination risks by using algorithms) lists 47 examples to illustrate how algorithms can discriminate people in various ways and how this can be detected and proved.


Real Estates, Loans, Judicial Matters, and More: Various Examples of Discrimination Risks

As examples, Orwat describes situations on the real estate and loan markets or in the court system. In the USA, for instance, several cases have been documented, in which algorithms of social media permitted targetted advertisements to be invisible to persons protected by the “Fair Housing Act,” such as migrants, people with handicaps, or with non-white skin color, the author says. In Finland, a bank was sentenced to pay a fine, because its algorithm for the automatic granting of online loans discriminated men against women and Finnish against Swedish native speakers. This unequal treatment is forbidden by Finnish anti-discrimination law. When deciding on early releases from prison, US judges use a much disputed system that calculates risk scores. Journalists and human rights associations criticize the fact that this system systematically overestimates black people’s risk of re-offending.


“Machine learning systems often have problems when they are trained with data reflecting unequal treatments or stereotypes,” Carsten Orwat explains. “In this case, the algorithms generated will also do so. When processing data containing evaluations of people by other people, unequal treatments and discriminations may even spread or increase.” This happened in the USA in a system for food and health controls that was based on discriminating ratings of restaurants.


Recommendations of Countermeasures

However, society must no longer accept these unequal treatments. The study lists several options to counteract discrimination by algorithms. “Preventative measures appear to be most reasonable,” Carsten Orwat says. Companies may ask antidiscrimination agencies to instruct their staff and IT experts and increase their awareness. Then, these persons will use datasets that do not reflect any discriminating practices or unequal treatments.


According to Orwat, the goal is to make future algorithms “discrimination-free by design.” This means that programs have to be checked during their development already. In the end, it is all about the protection of the society’s values, such as equality or free development of the personality. To guarantee this in spite of the very rapid developments of “big data” and AI, it is necessary to improve anti-discrimination and data protection legislation at some points, Orwat points out.


The complete study in PDF format for download (in German):


Being “The Research University in the Helmholtz Association”, KIT creates and imparts knowledge for the society and the environment. It is the objective to make significant contributions to the global challenges in the fields of energy, mobility, and information. For this, about 9,800 employees cooperate in a broad range of disciplines in natural sciences, engineering sciences, economics, and the humanities and social sciences. KIT prepares its 22,300 students for responsible tasks in society, industry, and science by offering research-based study programs. Innovation efforts at KIT build a bridge between important scientific findings and their application for the benefit of society, economic prosperity, and the preservation of our natural basis of life. KIT is one of the German universities of excellence.

jm, 14.10.2019


Monika Landgraf
Head of Corporate Communications, Chief Press Officer
Phone: +49 721 608-41150
Fax: +49 721 608-43658
presse does-not-exist.kit edu

Contact for this press release:

Jonas Moosmüller
ITAS - Öffentlichkeitsarbeit
Phone: +49 721 608 26796
jonas moosmueller does-not-exist.kit edu
The photo in the best quality available to us may be requested by
presse does-not-exist.kit edu or phone: +49 721 608-41105.

The press release is available as a PDF file.