Forget future existential threats — AI is automating gendered violence and misogyny right now

Debate about “agentic AI”, systems that act independently and might go “rogue”, has prompted fresh calls for a pause in the AI race. But AI harms aren’t hypothetical, and the panic fostered by Big Tech founders themselves may be a distraction. 

Timnit Gebru, the founder of the Distributed AI Research Institute, argues that AI companies stir up extinction fears to avoid discussing AI’s real harms, such as the ecological cost of data centres and autonomous weapons. We would add one more concern to her list: the intensifying violence against women and people of diverse sexual orientations, gender identities and expressions.

Online and offline violence are still often treated as separate. However, harmful ideas that circulate across social media at speed, shaping attitudes about women, masculinity and entitlement, do not remain online. They spill over into offline stalking, harassment, intimidation and physical violence.

Research, including a UN Women report, shows that violence against women is “rooted in gender inequality and discrimination, unequal power relations, and harmful social norms”, and that femicide is its most extreme manifestation." The recent murders of nine women within months in Ekurhuleni, Gauteng, prompted Amnesty International South Africa to call on the state to recognise femicide  as a distinct crime.

Research confirms that norms around gender equality play a significant role in driving intimate partner femicide. These norms are increasingly shaped on social media and in the so-called manosphere, a loose network of right-leaning online spaces promoting misogynistic narratives, popularised by influencers such as Andrew Tate, and spreading across Africa.

So how does AI play into all of this? AI is changing the scale of gendered harms, as a new research report by the Association for Progressive Communications notes. 

“Coordinated harassment campaigns or gender disinformation campaigns before maybe would have required an organised human effort that now can be run at volumes and with a persistence that was simply either not possible before or [was] very costly,” says author Daniela Schnidrig.

AI also lowers the technical threshold for abuse. Generative AI can now produce sexualised content from a single photo, and the majority of victims of sexually explicit deepfakes are women and gender-diverse persons. A 2025 study by the Oxford Internet Institute identified nearly 35,000 publicly downloadable deepfake model variants and found that 96% of those analysed targeted identifiable women.

The infrastructure itself is not neutral either. AI systems learn from training data that reproduces harmful stereotypes and racist ideas, as gender studies professor Safiya Umoja Noble demonstrated as early as 2018 in Algorithms of Oppression. 

Social media and AI are increasingly bound together in a closed loop, as content is generated by AI, amplified by AI-driven algorithms and moderated by AI. These engagement-optimised algorithms push sensational material, while automated tools fail to recognise gendered harm, especially in Global South contexts. With only 22% of the world’s AI professionals being women and hardly any women leading major social media companies, standards are shaped by those who have historically defined what counts as harm, rather than by those who bear the brunt of abuse. 

Double standards

Years after the breastfeeding censorship debate, Meta’s “Nudity and sexual activity” guidelines still restrict images of female nipples but not male ones, a rule whose cultural and health harms range from the automated removal of photos of Himba women in traditional attire in 2024 (a ban that was subsequently overturned), to the flagging of restorative nipple tattoos for breast cancer survivors, as reported in June this year by the BBC.

Double standards are at play. While AI-generated, non-consensual images remain online, platforms shadow-ban content by women and LGBTQIA+ people, hiding their posts without warning, with activists particularly targeted.

This abuse drives women out of public life. A UN Women study found that 70% of women journalists, activists and human rights defenders surveyed had experienced online violence, and nearly one in four had faced AI-assisted abuse. Consequences range from self-censorship on social media and at work to severe mental health impacts, including anxiety and depression.

Solutions do not require rejecting technology. AI offers real opportunities to support survivors and to strengthen the institutions meant to protect them. Chatbots like Zuzi offer survivor-centred information on gender-based violence in local languages such as isiZulu, while the FemAI Policy Playbook by Stellenbosch University’s Policy Innovation Lab shows how AI could connect fragmented data across police, health and justice institutions so that a survivor’s journey through the system can be followed and failures in support identified. 

The profit motive

But the answer is not an abstract slowdown of AI development. A redesign of power structures is required. We must ask why content that fuels GBV is amplified in the first place, and ultimately, the answer is money.

Under digital economist Nick Srnicek’s “Platform Capitalism”, companies profit from engagement, and misogyny generates plenty of it,  turning hate into revenue through algorithms and marketing schemes. Recent studies support this. Regulation that focuses on content removal while leaving these hugely profitable revenue models untouched will fall short. Accountability measures, such as interrupting revenue streams and algorithmic auditing,  deserve attention.

Gender perspectives and harm analysis must be built into AI policy from the start, with those affected by digital abuse represented in the design, regulation and evaluation of these systems. Regulation must also be contextual, and international frameworks must give countries the sovereignty to regulate tech giants and reclaim public spaces. 

Guidelines ignoring Global South realities will not work. While the African Union’s 2024 Continental AI Strategy names Technology-Facilitated GBV as a risk, a commentary for the European Centre for Development Policy Management notes that AU digital policies do not adequately embed gender equality into operational design. With much AI policy still to be written, initiatives like Pollicy’s Afro-feminist handbook and the Feminist AI Research Network show what gender-just governance could look like. 

The stakes are high. If these harms go unaddressed, we risk backsliding on decades of hard-won intersectional justice gains.

The debate over a “pause” should force a more honest question, not whether harm could arrive someday, but why so much of it is already here and why those most affected are still being asked to wait. DM

Academic researcher Janine Deselaers and journalist Janet Heard are attached to the Centre for Information Integrity in Africa (Cinia) at  Stellenbosch University.