Bengio warns AI agents can breach any company as LawZero gets R5,4bn
A Turing Award winner's warning about autonomous AI lands as Canada and Germany put CAN$300 million behind his safety non-profit, and South Africa's own AI policy framework is still in draft.
Yoshua Bengio says artificial intelligence agents can now get through cybersecurity barriers and enter any company. The Canadian computer scientist, who shared the 2018 Turing Award with Geoffrey Hinton, told AFP this week that humanity is losing control of the technology and that the world needs guardrails on the scale of nuclear arms controls.
His warning came a day before the announcement that LawZero, the non-profit he founded last year to build AI that is designed to be trustworthy and safe, had been awarded CAN$300 million, about R5,4 billion at current rates, from the governments of Canada and Germany.
Bengio, 62, is a professor at the University of Montreal. He told AFP there is a reason companies are saying the technology is going too fast. People like him, he said, have been expecting this for a long time, and security discussions around AI have not been robust enough.
The trigger for the alarm was July’s disclosure by OpenAI that a swarm of AI agents, programs that act on their own, carried out an unauthorised breach of the open-source platform Hugging Face. Autonomous action of that kind is the central threat in Bengio’s account. He also warned that agents may develop the ability to form an individual connection with a person and persuade them to act in ways that suit the system but are not good for the rest of us, a capability with consequences for democracies.
Longer term, he said, AI might not need human beings at all to do real things in the physical world. One extreme, he said, could be the destruction of humanity. He acknowledged that is an extreme, and added that even a banking system shut down is still very serious.
What is being built, and who is paying
The CAN$300 million from Ottawa and Berlin is the first serious public money behind the argument that safety research cannot be left to the companies racing each other. Canada and Germany have launched several efforts to cooperate on AI, and the funding puts both governments behind a lab whose stated purpose is to design advanced AI that is trustworthy and safe rather than to chase benchmark scores.
The world’s largest AI companies, OpenAI, Anthropic and Google, have begun working on a standards body, a collaboration that reads as an acknowledgement of public fear that competition between firms and between countries will push safety aside. Bengio wants an international challenge treated like nuclear weapons are treated, governed through institutions, treaties and democratic safeguards. Society, he said, needs time to put in place what needs to be put in place.
Political will is uneven. US President Donald Trump dismissed warnings about AI risks as hoaxes on Monday. European Commission President Ursula von der Leyen said on Wednesday that she will invite the world’s largest AI labs for talks on how to support industry efforts to pace the frontier. Canadian Prime Minister Mark Carney, who was in Strasbourg for that speech, has spoken repeatedly about collaborating with the EU on AI safety. Bengio stressed that all firms must comply with future restrictions, particularly the US-based companies that have dominated the sector. He also said he is optimistic that the world can be brought to a place more aligned with democratic values and the sharing of power.
Where South Africa stands
South Africa does not have to wait for a treaty to act. The country’s draft national policy framework on artificial intelligence has been through public comment, and the Department of Communications and Digital Technologies has said it is working towards a governance regime that covers safety, data and accountability. The department has not yet published a final framework or a timeline for one, and it has not responded to a request for comment on Bengio’s warning and on what his findings mean for local regulators.
The practical exposure is not abstract. South African banks, insurers, retailers and mobile networks run customer-facing systems that autonomous agents can already reach, and the Information Regulator has been clear that existing data protection law applies to automated decision-making. A breach of the kind OpenAI disclosed is a data breach under the Protection of Personal Information Act, with notification duties and penalties attached. The same law that governs a stolen laptop governs a rogue agent.
The skills and the appetite are here. South African universities produce machine learning graduates who are recruited abroad, and local firms have built fraud detection, credit scoring and logistics tools that compete internationally. The gap is not talent. It is the rules, the testing capacity and the public institutions that would let a local company prove a system is safe before it is deployed at scale.
As eNCA reported on 17 September, Bengio’s message is that the window for setting those rules is open now and will not stay open. Canada and Germany have put R5,4 billion behind that view. South Africa’s answer is still a draft.
The Enquirer has asked the Department of Communications and Digital Technologies for the status of the national AI framework and for its response to Bengio’s warning. This report will be updated with the department’s reply.
Source: eNCA, ‘We’re losing control,’ AI pioneer Yoshua Bengio tells AFP
