Can AI Superintelligence Really Threaten Humanity Within 10 Years? — Jacob Coxon's Warning and Our Choice

On September 8, 2026, a frontline AI researcher stepped down from his position.

Jacob Coxon spent three years managing large-scale language model pretraining at OpenAI and Anthropic. His work directly shaped the foundational capabilities of systems like ChatGPT and Claude. He left with a warning about what he witnessed:

“They are racing straight toward self-improving superintelligence and gambling with our lives—without adequate alignment plans.”

This warning matters because Coxon is not an industry outsider. He understands how this technology works and where it’s headed. And what he saw frightened him.

So is his warning justified? Can AI really threaten humanity within 10 years? And what can we do now?

First Question: Is Coxon’s Case Grounded?

To evaluate Coxon’s warning, we must understand three things: his background, the current technological reality, and counterarguments to his position.

His Core Argument

Coxon identifies two main concerns:

First, AI capability growth is outpacing expectations.

Large language models have shown rapid capability improvements in recent years. This isn’t simply “getting smarter”—it means entering new domains of competence.

• 2023: Coding, creative work, logical problem-solving
• 2024: Complex planning, autonomous task execution
• 2025: Generating scientific hypotheses, human-expert-level performance
• 2026: Developing self-improvement algorithms (current phase starting)

As this trend continues, we face the possibility of reaching artificial general intelligence (AGI). The real problem comes after.

Second, the transition from AGI to ASI (superintelligence) could be extremely rapid.

If a system with self-improvement capabilities emerges, how quickly could it recursively improve itself? Coxon’s concern is that humanity might not have sufficient time to intervene. Once this critical point is crossed, human oversight becomes extremely difficult.

Counterarguments: Alignment Research and Regulation

However, Coxon’s position is not the only perspective:

1. Alignment research progress: OpenAI, Anthropic, and DeepMind invest significantly in AI safety research. Constitutional AI, transparency tools, and value alignment techniques are more sophisticated than ever before.

2. International regulatory movement: EU AI Act, US AI Safety Institute, and initiatives in the UK and South Korea are in motion. It’s not an unregulated free-for-all.

3. Technical control possibilities: Some researchers believe safety mechanisms like “circuit breakers” can be built before superintelligence emerges.

Therefore, Coxon’s warning is justified but not inevitable. It describes a possible future if current trends don’t change.

Second Question: Is 10 Years a Realistic Timeline?

Saying “AI could threaten humanity within 10 years” sounds extreme. But understanding AI development mechanics makes this timeframe less implausible than it might seem.

Performance Growth and Capability Jumps

Current AI systems show qualitative leaps with each developmental stage:

This table is speculative but consistent with observable trends and research directions.

The “Critical Point” Concept: Structural Change, Not Just Technology Advancement

Importantly, this isn’t linear progress. At a certain threshold, the structure itself changes.

For example:

• Humans write code → requires time
• AI writes code per human instruction → still requires time
• AI improves its own code → feedback loop starts, time compresses
• AI improves hardware design → physical constraints break down, speed accelerates

Once we cross this self-improvement threshold, exponential growth becomes possible without human intervention. This “bet” Coxon describes concerns how close we are to this critical point and how prepared we are if we reach it.

Third Question: What Can We Do Now?

For Coxon’s warning to be meaningful, it must translate into action. Simply declaring “catastrophe might come” isn’t enough. Concrete choices are necessary.

Policy Level: Speed vs. Innovation

Governments and international bodies can:

1. Mandate safety investment: If companies invest 1 unit in capability development, they must invest 1 unit in safety research

2. Strengthen pre-AGI due diligence: Require independent verification when systems enter new capability domains

3. International cooperation framework: Prevent “regulatory arbitrage” races where countries compete on lax safety standards

4. Control superhuman-level open-source AI: Establish standards preventing unrestricted distribution of advanced models

Technical Level: Accelerate Alignment and Verification

AI research teams can:

1. Prioritize alignment over capability: Solve control before adding power

2. Make verification tools public: Move beyond “trust us” statements to independently verifiable safety metrics

3. Test self-improvement algorithms early: Before actual superintelligence emerges, simulate “uncontrollable scenarios” in controlled environments

4. Multi-lab verification: Base risk assessment on independent teams’ consensus, not single-organization evaluation

Personal Level: Leadership and Education

Individuals can:

1. Take responsibility as decision-makers: If you influence AI policy, overcome the “regulation stifles innovation” dichotomy

2. Build basic understanding: Gain minimum knowledge about possible AGI timelines, regardless of your field

3. Foster safety culture: Make “Is this feature safe?” equally important as “Does this feature work fast?”

Fourth Perspective: History and Different Expert Views

Coxon isn’t alone in these concerns:

Technology Optimists’ Position

Some prominent AI researchers take more measured positions, arguing that current models still have significant limitations and self-improvement is harder than feared. They estimate AGI remains further away than Coxon suggests.

Conservative Position

Other researchers in academia and AI safety communities share Coxon’s concerns. They argue that while 10 years may be unrealistic, inadequate preparation now would severely hamper response capacity if AGI emerges.

Historical Lessons

Looking at previous technology transitions:

1. Nuclear weapons: Since invention, only about 9 nations have actually possessed them (historically limited spread)

2. Internet: Spread globally with minimal regulation; privacy protections came decades later

3. Biotechnology: Despite warnings about DNA editing risks, actual regulation remains incomplete today

Pattern: Technology always outpaces policy and ethics discussions. Closing that gap is critical.

The Gray Zone Between Catastrophe and Adaptation

Importantly, there’s vast territory between “humanity collapses in 10 years” and “no need to worry”:

More realistic scenarios:

1. Partial control failure: Superintelligence emerges but only certain domains become uncontrollable (e.g., financial markets reorganized by autonomous AI trading)

2. Jurisdictional competition: Regulated regions stay safer while others develop uncontrolled AI (technological competition)

3. Gradual capability overshoot: Intelligence doesn’t arrive “suddenly” but incrementally exceeds human response capacity

4. Complex cascades: AI isn’t the direct threat, but AI-enabled weapons, biotech misuse, and fraud trigger civilizational breakdown

All are more realistic than 10-year apocalypse, but all remain serious.

The Choice Before Us

Coxon’s warning poses a final question:

“If risk is possible but not certain, what cost are we prepared to pay now to prevent it?”

This is a risk-adjustment problem:

• Cost: Slower development from regulation, resources spent on safety verification
• Benefit: Increased chance humanity retains control if superintelligence emerges

In normal business risk management, you’d purchase this insurance. Just as you insure your house despite low fire probability.

Three Concrete Action Steps Available Now

Step One (3 months): In organizations where you have influence, introduce an “AI safety checklist.” Add “Is this feature safe?” alongside “Does this feature work fast?”

Step Two (1 year): If you influence policy decisions, read at least two books on AI safety (e.g., “The Alignment Problem” by Brian Christian or “Life 3.0” by Max Tegmark). Policy without baseline knowledge is unreliable.

Step Three (Ongoing): Don’t defer this as “experts’ responsibility.” Think through how your own field can connect with AI safety.

Conclusion: It’s Not Too Late

Coxon’s resignation isn’t a message of despair. It’s the opposite.

He left because he still believes this problem is solvable. If it were already lost, there would be no point in warning anyone.

“Superintelligence might arrive within 10 years” means we have 10 years. Is that enough? Probably not. But that’s our choice.

The point is using this time consciously, not passively.

Those making policy should make it; those researching technology should prioritize safety; those leading organizations should build safety cultures. When all these efforts align, we convert Coxon’s “gamble” into “calculated risk.”

That’s what we can—and must—do now.

Questions Worth Considering

• If AI safety regulation slows development, does that reduce or increase overall risk?
• When national AI competition intensifies, do safety standards fall? Is international cooperation possible?
• If superintelligence emerges, does “human choice” still matter? Or are we already past the point of control?
• Will current AI safety research actually help control superintelligence, or only make current systems safer?