Consumers have a right to know when they are communicating with a machine

A few weeks ago, I phoned my insurer. Something about the conversation felt slightly wrong.

The voice on the other end of the line was fluent, responsive and polite. It answered questions quickly. It sounded calm and conversational, and the background noise sounded like a call centre. But, after a few minutes, I realised I was not speaking to a person at all. I was speaking to a machine.

What unsettled me most was not that the company used artificial intelligence (AI). It was that nobody had told me. There was no disclosure at the beginning of the call. No indication that I was interacting with a machine. No obvious option to request a human representative. Had I not been paying close attention, I may never have realised it.

Similar experiences are becoming increasingly common in South Africa, with banks, insurers, retailers and service providers deploying AI voice agents capable of mimicking human conversation with remarkable accuracy.

The commercial rationale for using such AI voice agents is obvious. AI systems are cheaper, faster and available 24 hours a day. For routine customer queries, they may even outperform call centre staff. But efficiency and cost savings are not the same thing as fairness. The issue is not whether companies may use AI systems. The issue is whether consumers have a right to know when they are interacting with a machine. In my view, they do.

Speaking to a human being or a machine is material. It affects how you communicate, what you disclose, how much trust you place in the interaction, and whether you choose to escalate a matter to a person. A customer may explain a medical condition, financial difficulty or dispute differently if they know no human is listening. A consumer may also decide that a complex or sensitive matter should not be handled by an automated system at all. Undisclosed AI systems remove that choice from the consumer entirely.

There is also a deeper concern. Many companies appear to be designing AI voice agents to sound indistinguishable from humans. After all, if customers realise too quickly that they are speaking to a machine, many may immediately ask for a human agent, which may, in turn, affect the cost savings and efficiencies in using the AI voice agent in the first place. The commercial incentive is therefore not merely automation, but seemingly also concealed automation. That, in turn, raises serious legal and ethical questions.

Legal frameworks

South African law does not expressly prohibit the use of undisclosed AI voice agents. However, several existing legal frameworks strongly suggest that companies should not be deploying these systems deceptively.

Take, for example, the Consumer Protection Act 68 of 2008, which prohibits suppliers from engaging in misleading or deceptive conduct and requires them to correct material misunderstandings held by consumers. A company that intentionally deploys a human-like AI system without disclosure may well be allowing consumers to believe something false about the nature of the interaction they are having.

The position becomes even more significant in the financial sector. Although the Consumer Protection Act would not necessarily apply to financial services regulated under financial sector legislation, banks and insurers remain subject to the Financial Sector Conduct Authority’s Treating Customers Fairly framework. That framework requires clear communication with customers and prohibits unreasonable barriers to assistance, complaints and dispute resolution. An undisclosed AI system arguably creates exactly such a barrier if consumers do not even know that a human alternative exists.

Finally, the Protection of Personal Information Act 4 of 2013 (Popia) adds another layer of complexity. Popia places restrictions on certain forms of automated decision-making that produce legal consequences or similarly significant effects for individuals. While a customer-service interaction may not itself constitute an automated “decision” in every case, these systems increasingly gather, classify and process information that feeds directly into underwriting, fraud detection, claims handling and customer-risk profiling.

The concern is therefore not simply that AI systems are speaking to consumers. It is that consumers may have no meaningful visibility into how algorithmic systems are shaping decisions about them behind the scenes.

A global problem

This problem is not unique to South Africa. Globally, regulators are moving toward mandatory transparency obligations for AI systems. The European Union’s AI Act, for example, requires users to be informed when they are interacting with certain AI systems unless this is obvious from the circumstances. International soft-law instruments are moving in the same direction.

Then there’s Unesco’s Recommendation on the Ethics of Artificial Intelligence in 2021, which emphasises transparency and human oversight in AI deployment. The African Union’s Continental AI Strategy similarly stresses disclosure and accountability in AI systems used across the continent. Even South Africa’s now-withdrawn Draft National AI Policy recognised transparency as a foundational principle for AI governance.

There is a broader principle at stake here. In a society increasingly affected by AI usage, people should not be forced to guess whether they are interacting with a machine. Transparency is not a luxury feature of ethical AI deployment. It is the precondition for trust. At a minimum, companies deploying AI voice agents should be required to disclose this at the beginning of an interaction with customers. Consumers should also have the right to escalate to a human representative at any stage of the process.

If companies believe their AI systems improve customer experience, they should have no hesitation in telling customers when those systems are being used. The fact that many do not is perhaps an indication that there is a need for further regulation in this area of consumer protection. DM