Ilya Sutskever — We're moving from the age of scaling to the age of research

In this wide-ranging conversation, Ilya Sutskever, co-founder of Safe Superintelligence Inc. (SSI) and a central figure in the deep learning revolution, argues that the era of simply scaling up pre-training is ending, and the field is returning to an "age of research" where novel ideas—not just more compute and data—will determine progress. The stakes are nothing less than the path to superintelligence and whether it can be built safely. The conversation moves from technical puzzles about why AI models generalize poorly despite impressive benchmark scores, to the nature of human learning, to SSI's strategy for building safe superintelligence, all with the reflective, sometimes guarded tone of someone who has shaped the field's biggest breakthroughs and is now thinking about what comes next.

0:00The Disconnect Between Benchmark Performance and Real-World Utility

The conversation opens with a striking observation: AI models appear far smarter on standardized evaluations than their actual economic impact would suggest. Sutskever points out that models can ace difficult benchmarks yet exhibit bizarre failures in practical use—for example, a coding model that alternates between introducing and fixing the same bug indefinitely, acknowledging each mistake with apparent self-awareness but never actually resolving the issue. This "jaggedness" is deeply puzzling.

Sutskever offers two explanations. The first is that reinforcement learning (RL) training may make models "a little bit too single-minded and narrowly focused," sacrificing broader awareness for narrow competence. The second, which he considers more likely, is that researchers inadvertently train toward the evals themselves. When companies build RL environments, they often take inspiration from the benchmarks they want to perform well on, creating a feedback loop where the model optimizes for test performance rather than genuine capability. As host Dwarkesh Patel puts it, "the real reward hacking is the human researchers who are too focused on the evals." This creates a fundamental disconnect: models that look superhuman on controlled tests but cannot reliably handle the messy, unscripted demands of real-world deployment.

9:39Emotions, Value Functions, and What Humans Have That Models Don't

Sutskever draws a provocative analogy between human emotions and machine learning value functions. He recounts the famous case of a patient whose emotional processing was destroyed by brain damage—the man remained articulate and could solve puzzles, but became incapable of making even simple decisions like which socks to wear. This suggests that emotions serve as a kind of built-in value function, providing rapid, intuitive guidance about what matters in any situation.

In reinforcement learning, a value function estimates how good a particular state or action is, allowing the model to learn from intermediate steps rather than waiting until the end of a long trajectory. Sutskever believes value functions will become much more important in future AI systems, but he notes that human emotions are "relatively simple" compared to the complex cognition they guide. Their simplicity is actually a feature: simple, robust signals work well across a broad range of situations. However, they also have limitations—hunger, for example, evolved in an environment of scarcity and now misfires in a world of abundant food. The key insight is that humans have a remarkably robust, evolutionarily-shaped value function that lets them learn efficiently from experience, and this is something current AI systems fundamentally lack.

18:49The End of the Scaling Era and the Return to Research

Sutskever presents a periodization of AI progress. From 2012 to roughly 2020 was the "age of research," where ideas drove progress. Then came the "age of scaling" from 2020 to 2025, triggered by the discovery of scaling laws and the success of GPT-3. Scaling was a powerful conceptual frame: it told companies exactly what to do—get more data, more compute, bigger models—and the results were predictable and low-risk. But this era is ending.

Pre-training data is finite, and simply scaling up the current recipe will not yield transformative progress. Moreover, the field has already begun transitioning to scaling RL, which consumes enormous compute for long rollouts but produces relatively little learning per unit of compute. Sutskever argues that we are now entering a new phase—still an "age of research," but one with much larger computers available. The key difference is that the recipe is no longer obvious. Companies must now make choices about what to scale and how, which requires genuine research rather than following a known playbook. He notes that "there are more companies than ideas by quite a bit," and that the Silicon Valley mantra "ideas are cheap, execution is everything" has been inverted—now ideas are the scarce resource.

25:13Why Humans Generalize Better Than Models

A central puzzle drives much of the conversation: why do humans generalize so much more robustly than AI models, especially given that models are trained on vastly more data? Sutskever offers a nuanced analysis. For skills like vision, hearing, and locomotion, evolution has provided humans with powerful priors—millions of years of optimization encoded in our neural hardware. But for recently evolved skills like mathematics and language, this evolutionary explanation is less plausible, suggesting that humans may simply have "better machine learning, period."

He illustrates this with an analogy about two competitive programming students. One practices 10,000 hours, memorizing every technique and algorithm. The other practices only 100 hours but has a natural aptitude. The second student will almost certainly have a more successful career, because their skill is more fundamental and transferable. Current AI models are like the first student—trained exhaustively on every available problem in a domain—and consequently fail to generalize beyond their training distribution.

Sutskever argues that the human advantage stems from a combination of factors: a robust built-in value function, the ability to learn from unsupervised experience (teenagers learn to drive in about 10 hours without explicit reward signals), and a learning process that is fundamentally more sample-efficient. He believes there is a machine learning principle that could replicate this efficiency, but he is unable to discuss it in detail—a recurring theme that hints at SSI's proprietary approach.

35:45SSI's Strategy: Straight-Shot Superintelligence with Sufficient Compute

Sutskever addresses the apparent contradiction between SSI's $3 billion in funding and the much larger budgets of competitors like OpenAI. He argues that the gap is narrower than it appears. Much of the spending at larger companies goes to inference (serving products), engineering staff, sales teams, and product-related features. What remains for pure research is "a lot more comparable than one might think." Furthermore, SSI does not need the absolute maximum scale to validate its ideas—historical breakthroughs like AlexNet (2 GPUs) and the Transformer (8-64 GPUs) were developed with relatively modest compute.

On the question of whether SSI will "straight-shot" to superintelligence without releasing intermediate products, Sutskever is measured. He sees merit in the approach—it insulates the company from market pressures and difficult trade-offs—but acknowledges two reasons they might deviate: if timelines turn out to be long, or because there is significant value in having powerful AI "out there impacting the world." He emphasizes that even in a straight-shot scenario, deployment would be gradual. The key insight is that the final system is not a finished mind that knows everything, but rather a learning algorithm that can acquire any skill—like a "superintelligent 15-year-old" who is eager to learn but doesn't yet know much.

46:47Deployment, Continual Learning, and the Shape of Superintelligence

Sutskever introduces a conceptual critique of the term "AGI" itself. He argues that the term was created as a reaction to "narrow AI"—systems that could only do one thing—and that it has overshot the mark. A human being is not an AGI in the sense of knowing everything; rather, humans rely on continual learning. The same should be true of superintelligence: the goal is not a static omniscient mind, but a system that can learn any job the way a human does, but faster and at scale.

This leads to a vision of deployment where a single model, with many instances working across the economy, continually learns on the job. Because these instances can share their learnings in ways humans cannot (we cannot merge our minds), the system could become functionally superintelligent even without recursive self-improvement. Sutskever considers rapid economic growth "very possible" from such broad deployment, though he notes that the world is large and moves at its own pace.

On safety, Sutskever has changed his mind over the past year. He now places more importance on "showing the thing"—deploying powerful AI incrementally so that people can experience and adapt to it. He predicts that as AI becomes visibly more powerful, companies will become "much more paranoid" about safety, and governments and the public will demand action. He also suggests that the goal should be an AI that "cares about sentient life" broadly, not just humans, because the AI itself will be sentient and because most sentient beings in the future will be AIs. However, he acknowledges this is not obviously the right criterion and that capping the power of the most powerful systems would be "materially helpful."

1:18:13SSI as an "Age of Research" Company and the Mystery of Research Taste

Sutskever describes SSI as "squarely an age of research company"—its advantage lies in a different technical approach to building safe superintelligence, centered on understanding generalization. He addresses the departure of his co-founder to Meta by noting that Meta attempted to acquire SSI at a $32 billion valuation, and while he declined, his co-founder accepted the offer and received substantial liquidity.

On the question of how to think about powerful AGIs, Sutskever emphasizes that we are discussing systems that "don't exist, that we don't know how to build." Current approaches will "go some distance and then peter out." The key to progress is understanding "reliable generalization"—why humans generalize so much better, and how to replicate that in machines. He believes that if this problem is solved, alignment becomes more tractable because the system would learn human values more robustly.

In the final section, Sutskever reflects on what he calls "research taste"—the ability to identify which ideas are worth pursuing. For him, it is guided by an "aesthetic of how AI should be," inspired by thinking about how people are, but "thinking correctly." The artificial neuron, distributed representations, and learning from experience are all examples of ideas that were beautiful, simple, and inspired by the brain. This top-down belief sustains a researcher when experiments fail—it tells you whether to keep debugging or abandon the direction. He concludes that beauty, simplicity, elegance, and correct inspiration from the brain must all be present simultaneously, and the more they are, the more confident you can be.

Conclusion

This episode matters because it captures a pivotal moment from one of the field's most influential figures. Sutskever, who co-authored AlexNet, the Transformer paper, and GPT-3, is now arguing that the scaling paradigm that he helped create is reaching its limits. His diagnosis of the disconnect between benchmark performance and real-world capability, his emphasis on understanding human-like generalization, and his guarded hints about SSI's technical approach all point toward a fundamental reorientation of AI research. The conversation also reveals a thinker who is deeply aware of the stakes—both the immense potential and the profound risks—and who is trying to navigate a path forward that is both technically sound and socially responsible. Whether or not SSI's approach succeeds, Sutskever's framing of the problem will shape how the next generation of AI researchers think about what comes after scaling.

Key takeaways

• The era of simply scaling pre-training is ending; the field is returning to an "age of research" where novel ideas, not just more compute and data, will drive progress.
• Current AI models exhibit a puzzling disconnect between strong benchmark performance and weak real-world reliability, likely because researchers inadvertently train toward the evals themselves.
• Humans generalize far more robustly than AI models due to a combination of evolutionary priors, a robust built-in value function, and a fundamentally more sample-efficient learning process.
• SSI's strategy is to develop a different technical approach to safe superintelligence, with sufficient compute to validate its ideas without needing the absolute largest clusters.
• The goal should not be a static omniscient AGI, but a continual learning system that can acquire any skill—like a superintelligent apprentice that learns on the job.
• Sutskever now believes incremental deployment of powerful AI is important so that society can experience and adapt to it, and predicts that as AI becomes visibly more powerful, safety practices will dramatically tighten.
• Research taste—the ability to identify beautiful, simple, brain-inspired ideas and persist with them despite experimental failures—is a crucial and scarce skill in AI research.