“If there is a threat, the warning light must flash”

Ranting, incitement to hatred, fake news, and more and more calls for violence – against minorities, politicians or simply people with a different point of view. Hate on social networks has long become a mass phenomenon that threatens democracy and the culture of debate and can often have severe consequences under criminal law. How can potentially criminal offences in hate speech and fake news be recognised, filtered out and punished more quickly? This is what the “BoTox” research project led by Professor Melanie Siegel, a computational linguist at h_da, is examining.

By Astrid Ludwig, 28.2.2024

Social parasites, enemies of the people, bloody Jews – mood and choice of words are becoming increasingly aggressive. Sadly, racist, contemptuous and sexist abuse, shitstorms and threats are an everyday occurrence on social media. “But when sentences such as ‘I’ve bought a gun’ pop up, then the warning light must flash,” says Melanie Siegel, “because this indicates a potentially concrete threat that must be recognised as an offence under criminal law and automatically reported to the right authorities.” It is on precisely this topic that the computational linguist and her research team are currently working.

The professor of semantic technologies has been studying hate speech and fake news on the internet since 2017 and was one of the first researchers to examine this phenomenon in German-speaking countries in depth. “Offensive language on the internet” is one of her research priorities. People do not have to tolerate hate speech or threats. “Something can be done about it,” Siegel is keen to stress.

One of her first research projects was entitled “Detection of Toxicity and Aggression in Postings and Comments on the Net” – in short DeTox. It focused on developing and applying text analysis and filters for hate speech with the help of artificial intelligence. It was also back then that she started working with the Hessian Ministry of the Interior, which set up the “HessenGegenHetze” (“HesseAgainstHate”) reporting office in 2020, which citizens can contact if they are themselves victims of hate speech or want to report such incidents.

Innovative solutions

Since December 2023, the Ministry of the Interior in Wiesbaden has again been funding cybersecurity projects that contribute to “developing innovative solutions that can also serve to protect our digital infrastructures and our security in practice,” explains Stefan Sauer, State Secretary of the Interior. The h_da research project “BoTox – Bot and Context Detection in the Environment of Hate Speech” is one of them. Professor Siegel and her team have €292,000 at their disposal up until 2025. “We are focusing,” she says, “on recognising the relevance of hate speech in terms of criminal law.”

The researchers, who include a research associate and four student assistants, want to develop a system that automatically detects when texts and conversations fulfil possible criteria such as insults, sedition, or incitement to violence. “We have identified twelve different criminal offences,” says Melanie Siegel. With the help of automatic learning and artificial intelligence, the new application will not only detect, analyse and pre-classify such content but also forward it directly to the “HessenGegenHetze” reporting office. The aim is to support the office’s work as well as make it easier and more efficient. “So that action can be taken more quickly if there is a threat,” she says. To do this, the research team needs, of course, to be familiar with the law and with basic legal principles. This expertise is provided by Professor Dirk Labudde from Hochschule Fresenius in Idstein, with whom Melanie Siegel has collaborated on previous projects. Labudde is a legal counsellor and professor of digital forensics.

Bot or human?

The h_da team wants to investigate further aspects. In particular, whether the hate speech was written by a bot, i.e. a computer programme, or a human is also relevant. In times of AI applications such as ChatGPT, identifying the author is quite difficult, says Siegel. Generating texts automatically has become much easier. “We tried it ourselves and had ChatGPT write hate speech.” Although the developers at OpenAI have built ethical guardrails into their software, these can be circumvented “if you ask the right questions and set the right tasks,” she says. “This, too, is an important insight for us.” The team also wants to look at the context in which hate speech pops up. How widespread is it on the internet overall and what happens when users retaliate? Does it make things better or worse? Is it wiser not to “feed” the trolls or should we consciously criticise toxic statements? “What can we do about hate speech?” says Siegel, describing her approach.

All these aspects and training the software to automatically detect hate speech call for a vast amount of data. An important element here is that the researchers need a sufficiently large number of relevant examples of hate speech for the system to learn to differentiate in the first place, but also a lot of irrelevant comments. In addition, the data must reflect reality. She reports that men post hate speech far more often than women and that, according to the findings, ultra-right groups are more active than ultra-left ones. The datasets must reflect this, too. The basis for the analysis is also important. Three of her student assistants read the same comments and then rate them. Is it an extreme opinion, is it insulting, or does it even incite sedition? Do the students rate them in the same way? “A Sisyphean task,” says Siegel.

Twitter no longer available as a data source

In previous research projects, Siegel and her team could make use of the messenger service Twitter. Thanks to a research licence, they were able, for example, to collect around one million Twitter data from comments on political TV shows for the DeTox project. Today, however, Twitter (or rather X, as it is now called) is no longer accessible “because research licences have become so expensive that no one can afford them anymore,” she reports. That is why the team now has to resort to other platforms such as Facebook, Telegram or YouTube, “and access is not easy there either.” However, the data from the “HessenGegenHetze” reporting office, which is meanwhile better known, are helpful. While users initially reported around 2,000 comments per month, the office now receives the same number per week.

However, the team also continues to use the existing training data. Although not all topics have been mapped – COVID-19 was still unknown and the war in Ukraine had not yet erupted – “we are working on making the data transferable,” explains Melanie Siegel. In some cases, this is easy: “There was a similar choice of words for Chancellor Merkel, for example, as there is today for the current government.” Many topics have also stayed the same: anti-Semitism, Holocaust denial, xenophobia, migration, racism, or discrimination against minorities.

The project will also incorporate new scientific findings. For example, Mina Schütz, a doctoral student at h_da, has examined the choice of words used in fake news and found that much more emotional language is used online than in scientific or journalistic texts. Personal pronouns are also used more frequently, as are exclamation marks, emojis and headlines in big letters. Jian Xi, who is also a doctoral candidate, has studied the interplay between images and text in fake news.

For Siegel, it is important to involve her students and incorporate research findings into her current teaching. However, she also wants to act as an adviser and raise awareness. In the near future, she will hold training courses and workshops at a police academy: “The police officers and lawyers there need to be on the alert.”