AI pioneers warn of a coming “intelligence explosion”
Story By Staff Reporter
Two of the figures widely seen as the founding fathers of modern AI have joined senior staff from OpenAI and Anthropic with a warning. They urge governments to prepare for a sudden surge in AI capability. In their view, it could reshape history.
Nobel laureate Geoffrey Hinton and Canadian computer scientist Yoshua Bengio co-wrote a new report. It calls on policymakers to act now, before progress spirals out of control. The authors define an “intelligence explosion” as a sharp, AI-driven speed-up in AI development. Years of advancement could shrink into months or less.
The paper asks, “What if automating AI R&D triggers an intelligence explosion?” More than 20 people contributed. Anthropic co-founder Jack Clark and OpenAI chief scientist Jakub Pachocki are among them.
Their central worry is that AI systems could start improving themselves without human involvement. Experts call this recursive self-improvement, and the paper mostly describes it as automated AI research and development. Once AI matches expert human ability in this field, one developer could run the equivalent of millions of top researchers.
The authors say automated AI R&D is the likeliest trigger. AI already helps improve its own technology. Developers can roll out upgraded systems quickly once they exist.
The paper calls an intelligence explosion potentially the most significant technological shift in human history. It could compress years of progress into months. It could also undermine human control over AI systems. Worse, it could weaken the checks on power within and between states, companies and branches of government. The authors add that once an explosion begins, the window for action may close.
They put forward three policy priorities. First, governments should require transparent reporting on AI R&D, including independent auditors embedded in companies. Second, they should find ways to restrain breakneck development speed. Third, they should prepare to adapt if an explosion happens.
Anthropic and OpenAI have both agreed to let independent evaluators test their models. Both companies have warned that AI development is nearing a critical point for safety. Anthropic says AI now writes 80% of its code. OpenAI uses autonomous AI agents for tasks such as training new models.
The paper says early evidence suggests a software-driven intelligence explosion is possible. That would bring very rapid growth in highly capable, even superhuman, systems. Medical breakthroughs and technological leaps could follow.
The report also names three dangers. Powerful systems could enable biological and cyber threats that emerge faster than defences. Humans who step back from AI R&D may lose the ability to steer these systems. Countries could turn a small lead, in cyberspace for example, into a decisive one.
The authors admit the effects are uncertain. Supply chains for specialised materials and regulatory requirements could slow a breakthrough’s rollout. AI might also speed up how fast people address risks.
If an explosion did happen, AI could quickly surpass human experts in most fields. Technological progress would accelerate radically. The paper says preparation should be an urgent priority, including at the highest levels of government.
Governments could consider several measures:
- Transparent reporting on AI R&D, with independent auditors embedded in companies
- Limits on how fast an AI can improve within a given period
- Agreements with datacentres to pause certain AI R&D projects
- Full isolation of automated AI R&D systems, so they cannot escape human control
- Emergency response plans for the scenarios an explosion could produce
Systems still show glitches, such as ignoring instructions. Even so, AI could fully automate R&D projects that take humans months by 2028. The authors say productivity gains from AI R&D automation have not yet reached the trigger threshold. Gains from newer systems are probably approaching it.
Reporting based on original coverage by Reuters.
