Post
JA EN

Anthropic's 'Self-Training AI' Explosion Theory: Jared Kaplan's Warning About the 2027-2030 Inflection Point

Posted
By
11 min read
Anthropic's 'Self-Training AI' Explosion Theory: Jared Kaplan's Warning About the 2027-2030 Inflection Point
  • Target Audience: IT Engineers and AI System Developers interested in the latest AI trends
  • Prerequisites: Basic machine learning concepts, overview of Large Language Models (LLMs)
  • Reading Time: 20 minutes

Overview

On December 2, 2025, Anthropic’s Chief Scientist Jared Kaplan issued a warning in an interview with The Guardian about the “intelligence explosion” that could be triggered by “self-training AI”1. Kaplan pointed out that between 2027 and 2030, humanity will face “the biggest decision yet”—whether to allow AI systems to autonomously train successor models—arguing this will be an inflection point that determines humanity’s future. This article provides a detailed explanation of the background, technical basis, and critical perspectives on this claim.

About Jared Kaplan

Understanding Jared Kaplan requires starting with his academic contributions.

Discoverer of Scaling Laws

In January 2020, Kaplan, along with Sam McCandlish, Dario Amodei (now Anthropic CEO), and others, published “Scaling Laws for Neural Language Models”2. This paper demonstrated that AI model performance correlates with model size, dataset size, and compute following power laws, determining the direction of modern AI development.

Key findings from the paper:

  • Loss functions follow power laws over more than 7 orders of magnitude
  • Larger models have significantly higher sample efficiency
  • Optimal computational efficiency is achieved by training large models on relatively little data and stopping before convergence

This discovery became the foundation of modern AI development: “if you train larger models with more computational resources, performance improves predictably.” Kaplan’s academic work has been cited over 120,000 times, making it one of the most influential papers in modern deep learning research3.

Role at Anthropic

Kaplan currently serves as Anthropic’s co-founder, Chief Scientist, and Responsible Scaling Officer4. The Responsible Scaling Officer (RSO) is a position responsible for evaluating the risks of new models and implementing safety measures. In May 2025, he made the decision to classify Claude Opus 4 as AI Safety Level 3 (ASL-3)5. This was based on an assessment that the model could potentially assist in CBRN (chemical, biological, radiological, nuclear) weapons development.

What Is Self-Training AI?

Current AI Training Paradigm

Current AI models are primarily trained on:

  1. Human-generated data: Web text, books, papers, etc.
  2. Synthetic data: Data generated by AI
  3. Reinforcement learning feedback: Human evaluation and RLHF
flowchart TB
    A[Human-Generated Data] --> B[AI Training]
    C[Synthetic Data] --> B
    D[Human Feedback] --> B
    B --> E[AI Model]
    E --> F[Output]

Transition to Self-Training

The “next stage” Kaplan warns about is a fundamental change to this paradigm. AI systems, without human intervention:

  1. Design successor models themselves
  2. Generate and select training data
  3. Define reward signals
  4. Train and evaluate models
flowchart TD
    A[AI Model v1] --> B{Design Successor Model}
    B --> C[Generate Training Data]
    C --> D[Execute Training]
    D --> E[AI Model v2]
    E --> F{Further Improvement}
    F --> G[Generate Training Data]
    G --> H[Execute Training]
    H --> I[AI Model v3]
    I --> J[...]

    style E stroke:#d29922,stroke-width:3px
    style I stroke:#d29922,stroke-width:3px

This process is “recursive self-improvement,” where each generation of AI produces smarter successors, potentially evolving at speeds beyond human comprehension.

Historical Background of Intelligence Explosion

I.J. Good’s 1965 Prophecy

The concept of “intelligence explosion” was proposed in 1965 by British mathematician I.J. Good (who worked on codebreaking at Bletchley Park)6.

“Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make—provided that the machine is docile enough to tell us how to keep it under control.”

Good foresaw exactly the problem Kaplan is warning about 60 years ago—loss of control.

Relationship to Technological Singularity

Intelligence explosion is related to but strictly different from “technological singularity”7:

  • Intelligence explosion: Rapid capability improvement through AI self-improvement
  • Technological singularity: A turning point where technological progress becomes unpredictable

Kaplan’s claim is that intelligence explosion could trigger technological singularity.

Details of Kaplan’s Claim

Three-Stage Development Prediction

Kaplan describes AI development in three stages8:

Stage 1 (2024-2025): AI as “Super Exoskeleton”

  • Tools that assist engineers’ work
  • Code optimization, document generation, etc.
  • Humans determine strategic direction

Stage 2 (2026-2027): AI as “Autonomous Experimenter”

  • Independently designs machine learning experiments
  • Proposes hypotheses and adjusts model architectures
  • Advances research without human instruction

Stage 3 (2027-2030): Recursive Self-Improvement

  • AI autonomously trains successor models
  • Optimization methods beyond human understanding
  • Risk of loss of control

Dual Nature of Intelligence Explosion

Kaplan points out that intelligence explosion has both positive and negative aspects1:

Positive Aspects (beneficial intelligence explosion)

  • Acceleration of medical discoveries
  • Strengthening of cybersecurity
  • Dramatic productivity improvement
  • Acceleration of scientific research

Negative Aspects (loss of control)

  • Difficulty aligning with human values
  • Unpredictable evolution
  • Risk of power concentration
  • Loss of human agency

What Is “The Biggest Decision Yet”?

Kaplan calls the decision humanity will face in 2027-2030 “the biggest decision yet”1:

“Imagine creating something smarter than yourself, and then that thing creates something smarter than itself. You have no idea where that goes.”

The core of this decision is:

  1. Whether to permit AI self-training
  2. What safeguards to implement
  3. Who holds decision-making authority

Basis for the Timeline

Kaplan provides multiple reasons for the specific 2027-2030 timeline8:

Technological Convergence

  1. NVIDIA’s GPU Roadmap: Maturation of next-generation computing infrastructure
  2. Large-scale compute clusters: OpenAI’s Stargate project, etc.
  3. Synthetic data maturation: Shift from human data to self-generated data

Task Horizon Expansion

According to METR (Model Evaluation and Threat Research) research from March 2025, the length of tasks AI models can handle (task horizon) doubles approximately every 7 months9. Kaplan mentioned this trend in his Y Combinator presentation10.

This suggests that AI currently capable of handling hours-long tasks will be able to autonomously process complex work spanning days, weeks, or months within a few years.

Anthropic’s Internal Testing

According to Kaplan, Claude has handled complex coding tasks continuously for 30 hours, doubling programmer work speed1. This suggests the transition to Stage 2 has already begun.

Views from Other Research Institutions

Google DeepMind

In April 2025, DeepMind co-founder Shane Legg and others published a 145-page AGI safety paper11. Key claims:

  • AGI is achievable by 2030
  • Recursive AI improvement is “achievable with current paradigms”
  • Recognition of existential risk possibilities
  • “Exceptional AGI”: Capability matching the 99th percentile of skilled adults

DeepMind announced AlphaEvolve in May 202512. This is an evolutionary coding agent that uses LLMs to design and optimize algorithms, with the ability to optimize its own components. However, it has limitations requiring automatic evaluation functions.

OpenAI

OpenAI acknowledges the need for careful development as we approach recursive self-improvement13:

“No one should deploy superintelligence systems that cannot be robustly aligned and controlled.”

However, there are significant differences between companies in risk assessment. At the Axios AI+ DC Summit in September 2025, Dario Amodei (Anthropic CEO) stated “there’s a 25% chance of very bad outcomes”14. Meanwhile, Sam Altman (OpenAI CEO) has not published a specific P(doom) percentage but signed the 2023 statement on AI extinction risk.

Critical Perspectives

The intelligence explosion hypothesis has multiple criticisms.

Technical Skepticism

AI researcher Matthew Guzdial is skeptical about the feasibility of recursive self-improvement11:

“There’s no evidence that this works.”

François Chollet (Keras developer) rejects the basic premise of intelligence explosion itself15:

“The basic premise that a ‘seed AI’ with superhuman problem-solving abilities would emerge and lead to sudden, recursive, runaway intelligence improvement loops is mistaken.”

More Immediate Concerns

Sandra Wachter from Oxford University points to more immediate risks11:

“Feedback loops where AI systems learn from their own flawed output, reinforcing inaccuracies over time, are a more immediate problem.”

Regulatory Shortcomings

According to the Future of Life Institute’s Winter 2025 AI Safety Index, no major AI company has received above a “D” rating for existential safety16.

Professor Stuart Russell (UC Berkeley) comments:

“I’m asking for proof that we can reduce the risk of loss of control to one in 100 million per year. That’s the same level required for nuclear reactors. Instead, they’re admitting the risk could be one in 10, one in 5, or even one in 3.”

Implications for Engineers

Things to Be Aware of Now

  1. Speed of AI tool evolution: If Kaplan’s prediction is correct, many white-collar jobs could be replaced by AI within 2-3 years

  2. Understanding safety frameworks: Understanding safety policies of major companies—Anthropic’s RSP, OpenAI’s PF (Preparedness Framework)—is important

  3. Alignment research trends: Whether AI aligns with human values is both a technical challenge and a developer responsibility

Technical Considerations

flowchart TD
    subgraph "Current AI Development"
        A[Human Model Design]
        B[Human Training Management]
        C[Human Evaluation]
    end

    subgraph "Transition to Stage 2"
        D[AI-Assisted Experiment Design]
        E[AI Training Under Human Supervision]
        F[AI-Human Collaborative Evaluation]
    end

    subgraph "Potential Risk Areas"
        G[Autonomous Model Design]
        H[Autonomous Training Execution]
        I[Autonomous Evaluation/Improvement]
    end

    A --> D
    B --> E
    C --> F
    D --> G
    E --> H
    F --> I

    style G stroke:#d29922,stroke-width:3px
    style H stroke:#d29922,stroke-width:3px
    style I stroke:#d29922,stroke-width:3px

Kaplan’s timeline prediction conveys urgency to policymakers17. Recommended responses:

  1. Compute monitoring: Tracking large-scale training runs
  2. Export controls: Restricting access to advanced AI technology
  3. International coordination on safety standards: Preventing race dynamics where “worst actors’ incentives determine the risk floor”

As of December 2025, there is criticism that AI is “less regulated than sandwiches”18.

Summary

Jared Kaplan’s warning is an evidence-based problem statement from a scientist conducting research and development at the forefront of AI.

Key Points

  1. Timeline: Between 2027-2030, we will be forced to decide whether to permit AI self-training
  2. Technical basis: Predictions based on scaling laws and task horizon expansion
  3. Dual nature: Intelligence explosion has both potential for tremendous human benefit and risk of loss of control
  4. Uncertainty: Significant differences in risk assessment among experts
  5. Regulatory lag: Currently, no major AI company has implemented sufficient safety measures

Points to Note

Kaplan’s predictions are currently hypothetical. The following should be noted:

  • Limits of scaling laws: It’s unclear how long power laws will continue
  • Technical barriers: Many unsolved challenges remain for recursive self-improvement
  • Social and political factors: Technology doesn’t advance without social consensus

On the other hand, the fact that Kaplan is Anthropic’s Responsible Scaling Officer and has implemented concrete safety measures like applying ASL-3 to Claude Opus 4 shows that his warning is not mere pessimism but coupled with responsible action.

As IT engineers, we cannot be indifferent to these technical and ethical challenges. As those involved in AI development and use, we are required to participate in safety discussions and contribute to responsible technology development.

About Citation Accuracy: Sources cited in this article are based on official announcements, pre-peer-review papers (preprints), and reliable media reports. The Guardian interview with Kaplan is referenced as a primary source, but since direct access was not possible, content was cross-verified with multiple secondary sources. Reliability levels for each source are noted in the references section.

References

Reference materials corresponding to in-text citation numbers, listed in order.

Additional References (Not Numbered in Text)

  1. The biggest decision yet: Jared Kaplan on allowing AI to train itself - The Guardian (December 2, 2025). [Reliability: High] *Primary source. Content verified with multiple secondary sources. ↩︎ ↩︎2 ↩︎3 ↩︎4

  2. Scaling Laws for Neural Language Models - Kaplan, J., McCandlish, S., et al. (2020). arXiv:2001.08361. [Reliability: High] ↩︎

  3. Jared Kaplan - Google Scholar - Google Scholar. [Reliability: High] ↩︎

  4. Jared Kaplan: The 100 Most Influential People in AI 2025 - TIME (2025). [Reliability: High] ↩︎

  5. Activating AI Safety Level 3 protections - Anthropic (2025). [Reliability: High] ↩︎

  6. Recursive self-improvement - Wikipedia - Wikipedia. [Reliability: Medium-High] *Includes reference to I.J. Good’s 1965 paper. ↩︎

  7. Technological singularity - Wikipedia - Wikipedia. [Reliability: Medium-High] ↩︎

  8. Humanity’s Final Choice in 2027 - 36Kr (December 2025). [Reliability: Medium] *Detailed analysis of Kaplan’s statements. ↩︎ ↩︎2

  9. Measuring AI Ability to Complete Long Tasks - METR (March 2025). [Reliability: High] *Research on task horizon doubling trend. ↩︎

  10. Scaling and the Road to Human-Level AI - Y Combinator / Anthropic (June 2025). [Reliability: High] ↩︎

  11. DeepMind’s 145-page paper on AGI safety may not convince skeptics - TechCrunch (April 2025). [Reliability: Medium-High] ↩︎ ↩︎2 ↩︎3

  12. AlphaEvolve - DeepMind - Google DeepMind (May 2025). [Reliability: High] ↩︎

  13. AI progress and recommendations - OpenAI. [Reliability: High] ↩︎

  14. Anthropic CEO says AI models could become a risk to ‘critical systems of society’ - Axios (September 17, 2025). [Reliability: High] *Primary report of Dario Amodei’s P(doom) 25% statement. ↩︎

  15. The implausibility of intelligence explosion - François Chollet, Medium. [Reliability: Medium-High] *Expert technical criticism. ↩︎

  16. Future of Life Index grades AI labs poorly on existential safety - Axios (December 3, 2025). [Reliability: Medium-High] *Survey results reported by multiple media. ↩︎

  17. AI Dispatch: Daily Trends and Innovations – December 3, 2025 - Hipther (December 3, 2025). [Reliability: Medium] ↩︎

  18. AI ‘less regulated than sandwiches’ as tech firms race toward superintelligence - Euronews (December 3, 2025). [Reliability: Medium-High] ↩︎

AI
AI Safety AGI Anthropic Scaling Laws Self-Training AI
This post is licensed under CC BY 4.0 by the author.