Michael Nielsen – How science actually progresses

In this wide-ranging conversation, Dwarkesh Patel and Michael Nielsen explore the surprisingly mysterious nature of scientific progress, challenging the common textbook narrative that science advances through clean experimental falsification and verification. Nielsen, a pioneer of quantum computing and open science, argues that the real history of science reveals something far messier: verification loops can take centuries, great scientists can remain wrong for decades after their peers have moved on, and the heuristics that drive progress are often inarticulable. The stakes are high because AI researchers are trying to build automated scientific discovery systems, but if we cannot even explain how human science works, how can we hope to automate it? The conversation ranges from Einstein's complicated relationship with the Michelson-Morley experiment to why natural selection took so long to discover, from the nature of scientific explanation in the age of deep learning to the provocative possibility that alien civilizations would develop entirely different technology stacks than our own.

0:00The Michelson-Morley Myth and How Science Really Works

The episode opens by dismantling one of the most famous origin stories in physics. The standard narrative holds that the Michelson-Morley experiment of 1887 disproved the existence of the luminiferous ether, creating a crisis that Einstein resolved with special relativity. Nielsen explains that this is almost entirely wrong. Michelson and Morley were testing specific theories of the ether, not the ether concept itself—and Michelson continued to believe in the ether until his death in 1929, conducting experiments into the 1920s. Another physicist, Miller, even claimed to have detected the ether wind from Mount Wilson in California, prompting Einstein's famous remark, "Subtle is the Lord, but malicious he is not."

The deeper point is that falsification is far messier than philosophy of science textbooks suggest. When an experiment contradicts a theory, scientists have enormous latitude in deciding what to reject. Lorentz developed the Lorentz transformations—the mathematical core of special relativity—but interpreted them as describing the effects of moving through the ether, with length contraction and time dilation caused by physical pressure warping clocks and measuring rods. This interpretation was experimentally indistinguishable from Einstein's for decades. The first decisive evidence came from muon decay experiments in 1940-41, which showed that time dilation was real, not just a mathematical convenience. Yet the scientific community had already adopted Einstein's interpretation well before this experimental confirmation.

Nielsen emphasizes that there is no centralized authority or standard procedure for resolving these disputes. "Great scientists can remain wrong for a very long time after the scientific community has broadly changed its opinion," he notes. Poincaré understood the principle of relativity and the constancy of the speed of light, yet clung to a dynamical interpretation of length contraction. Einstein, by contrast, was a teenager in the 1890s who also believed in the ether but was less attached to it. Nielsen suggests that expertise itself can become a prison: "He knows so much, he understands so much, and then he's not able to let go."

17:51Newton: The Last of the Magicians

Nielsen and Patel discuss John Maynard Keynes's famous essay describing Newton as "the last of the magicians, the last great mind which looked out on the visible and intellectual world with the same eyes as those who began to build our intellectual inheritance rather less than 10,000 years ago." This characterization challenges the image of Newton as the first modern scientist. Newton's unpublished work on alchemy and theology, Keynes argued, was marked by the same "careful learning, accurate method, and extreme sobriety of statement" as the Principia. The same heuristics and biases that drove his astronomical work also drove his esoteric investigations.

This raises a crucial question for Patel: if the aesthetic and heuristic judgments that guide scientific progress are so personal and historically contingent, can they be encoded into AI systems? Nielsen is skeptical. "If you're attempting to reduce science to a process, you're attempting to reduce it to something where there is just a method which you can apply and you turn the crank and out pops insight," he says. "You can do a certain amount of that, but you're going to get bottlenecked at the places where your existing method doesn't apply. But definitionally, there's no crank you can turn."

23:26Why Natural Selection Wasn't Obvious

Patel poses a puzzle: the Principia Mathematica was published in 1687, but the Origin of Species didn't appear until 1859. Yet natural selection seems conceptually simpler than gravity. Thomas Huxley, upon reading Darwin, reportedly said, "How extremely stupid not to have thought of that." No one has ever said that about the Principia. So why did Darwinism take so long?

Nielsen argues that Darwin's genius wasn't in having the idea of selection—animal breeders had understood artificial selection for millennia. It was in understanding just how central the idea was to all of biology and in compiling the overwhelming case for its explanatory power across geology, paleontology, biogeography, and comparative anatomy. The idea required "deep time" to work: Charles Lyell's geological work in the 1830s established that the Earth was millions or billions of years old, without which evolution would seem impossible on a 6,000-year timescale.

Patel notes that the Roman poet Lucretius had an idea that sounds like natural selection in the first century BCE, but it was fundamentally different. Lucretius believed in a one-time generative period in the past where all species emerged, followed by a single filter that left only the fit. He had no concept of ongoing gradual change or a tree of life connecting all organisms. The fact that Darwin and Alfred Wallace independently arrived at the same idea in the 1850s suggests that the necessary building blocks—deep time, paleontology showing intermediate fossils, biogeography from the age of exploration—had only recently fallen into place.

29:52Could Gradient Descent Have Discovered General Relativity?

The conversation turns to AlphaFold and what it means for scientific explanation. Patel observes that AlphaFold is a 100-million-parameter neural network that predicts protein structures, but it doesn't provide anything like the kind of explanation that general relativity does. General relativity can explain why Mercury's orbit precesses—a phenomenon it was never designed to predict. AlphaFold cannot do that kind of thing.

Nielsen offers three perspectives on this. First, the conservative view: AlphaFold is a useful model but not a scientific explanation. Second, an intermediate view: AlphaFold contains many "little explanations" inside it that can be extracted through interpretability work, much like chess players have learned new strategies from studying AlphaZero. Third, the most interesting possibility: these models represent a new type of object altogether. "In the past we haven't had the ability to really do anything with them, and now we're going to have new interesting operations we can do—we can merge them, we can distill them," Nielsen says. He draws an analogy to Mathematica: in 1920, a 100-page equation was useless; today, it's a working tool that can be manipulated and simplified.

But Patel worries about a historical counterexample. If someone in 1500 had trained a model on planetary observations, they would have just kept adding epicycles to the Ptolemaic model. Gradient descent would never have made the leap to heliocentrism because that leap required a global restructuring that temporarily made predictions worse. Nielsen agrees that this is a real concern and points to the history of general relativity itself. Einstein was forced to go through "somewhat ugly intermediate stages" before arriving at the shockingly simple final theory. The forcing function was the realization that Newtonian gravity was incompatible with special relativity—it would allow faster-than-light signaling. This kind of conceptual forcing function is very different from optimizing a loss function.

50:54Why Aliens Will Have a Different Tech Stack

Nielsen introduces a provocative idea: alien civilizations would likely develop entirely different science and technology stacks than humans. This contradicts the common assumption of a linear "tech tree" that all civilizations would converge upon. The argument rests on the observation that the space of possible scientific and technological ideas is vastly larger than any one civilization can explore.

Nielsen points to computer science as an example. The field began in the 1930s with Turing and Church establishing the fundamental theory of computation. Ninety years later, we are still discovering deep new principles hidden within that theory—public key cryptography, cryptocurrencies, and countless others. "The idea that we've discovered all the deep ideas in programming just seems obviously ludicrous," he says. Similarly, physics has gone from teaching three or four phases of matter to discovering superconductors, superfluids, Bose-Einstein condensates, and fractional quantum Hall systems. "It looks like actually there's a lot of phases of matter to discover, and we're going to discover a lot more of them."

Different civilizations might explore this vast space in different ways based on their sensory biases, their physical substrates, and their historical contingencies. A civilization of auditory creatures might develop entirely different mathematics. This has profound implications for interstellar relations: if different civilizations develop genuinely different capabilities, there could be enormous gains from trade. Patel points out that this makes friendliness much more rewarding than domination. Nielsen agrees but notes that comparative advantage has limits—we don't trade with chimpanzees, partly because of power imbalances. The key question is whether future civilizations will be able to establish the institutional conditions for mutually beneficial exchange.

1:15:26Are There Infinitely Many Deep Scientific Principles?

Patel asks whether there are infinitely many deep scientific principles left to discover, or whether we are approaching fundamental limits. He cites the famous "low-hanging fruit" argument and the empirical finding by Nicholas Bloom and colleagues that maintaining Moore's Law required a 9% annual increase in the number of scientists to sustain a 40% annual increase in transistor density.

Nielsen is skeptical of the diminishing returns argument. He offers an analogy: if you go to a dessert buffet with 30 desserts, the best ones go first. But if someone is standing behind the table restocking it with new desserts, the situation changes entirely. Scientific progress has this character—new fields keep opening up, creating new opportunities for young researchers to make breakthroughs without spending 25 years mastering everything that came before. Computer science arose from "abstruse questions in the philosophy of mathematics and logic" that seemed esoteric at the time. Deep learning is a more recent example.

Nielsen acknowledges that deep learning now requires billions of dollars to push the frontier, but he argues this is partly a consequence of how attention and resources get centralized. "At any given time there's always a most successful thing," he says. If deep learning weren't the focus, we might be talking about CRISPR or some other field. The deeper point is that the structure of knowledge itself seems to generate new fields, and we don't understand the dynamics of this process well enough to predict when it will stop.

1:26:25The Birth of Quantum Computing

Patel asks Nielsen for his inside view on how quantum computing became a real field. Why wasn't it invented in the 1950s by someone like John von Neumann, who was both a pioneer of computing and deeply knowledgeable about quantum mechanics?

Nielsen points to two historically contingent developments that converged around 1980. First, computation became far more salient—you could buy an Apple II or a Commodore 64, and people became excited about these powerful new devices. Second, experimental techniques like the Paul trap made it possible to manipulate single quantum states for the first time. Richard Feynman, who got one of the first PCs around 1980-81, was so excited that he tripped and hurt himself carrying it. Having someone deeply knowledgeable about quantum mechanics who was also thrilled about these new machines created the conditions for his famous 1982 paper on quantum simulation.

Nielsen's own entry into the field came in 1992 when he took a quantum mechanics class from Gerard Milburn. After the fifth lecture, he asked Milburn for papers to read and received a stack that included the Feynman and Deutsch papers. "As soon as I read the papers, these are exciting papers. They're asking very fundamental questions," Nielsen recalls. "You're sort of like, oh, I can make progress here." He emphasizes that this was a very unusual choice at the time—very few people were working on quantum computing in 1992. The decision was driven by a sense that he was "in contact with something which is deeply important and which we as a civilization don't have."

1:35:29The Political Economy of Science

The conversation shifts to open science and the social construction of scientific credit. Nielsen notes that the modern system of publishing papers with attribution is only about three centuries old. Before that, scientists like Galileo and Kepler sometimes published their results as anagrams—scrambled sentences that could be unscrambled later to establish priority if someone else made the same discovery. This was "not an ideal foundation for a discovery system."

The transition to the modern system required building a reputation economy around published papers. But this system was designed for the printing press, not for the internet. There is no direct credit associated with sharing code, data, or in-progress ideas. "That's all constructed socially," Nielsen says. "Making it a live issue is a very important thing to have done."

He illustrates the arbitrariness of the current system with a story about preprint cultures. Physicists would say they upload to the preprint archive immediately because physics is so competitive that they need to establish priority. Biologists would say they don't upload preprints because biology is so competitive that they need to protect their results. "This emphasizes the extent to which this kind of attribution economy is just something we construct," Nielsen observes.

1:43:57Prolificness Versus Depth

Patel raises the question of whether scientists should prioritize publishing many papers or spending decades on a single deep idea. He cites the "equal odds rule" from Dean Keith Simonton, which suggests that any given paper from a researcher has roughly equal probability of being important—so the most productive researchers tend to have the most important work.

Nielsen admits he struggles with this question. He distinguishes between two types of work in creative projects: routine stuff where you want to avoid procrastination and do it as quickly as possible, and high-variance stuff where you need to be willing to take a lot of time and explore uncertain directions. "Balancing those two things is hard," he says. "It's almost like a personality trait, which one you prefer."

He shares a personal practice from his time in San Francisco: instead of taking the 15-minute walk to work, he would take the more beautiful 30-minute walk. "Partially just because it was beautiful, but partially also as a reminder that there are real benefits to not being efficient." He also notes that many brilliant people he has met never produce anything significant because they are obsessed with working on the great project that will make them famous. "It's a type of aversiveness," he suggests. "Very often they just don't want public judgment."

1:49:17How to Actually Internalize What You Learn

The conversation concludes with Patel's personal struggle: how to ensure that interviewing experts leads to genuine understanding rather than superficial familiarity that depreciates over time. He worries that he is building up knowledge that fades quickly because he never goes deep enough to understand the underlying mechanisms.

Nielsen's advice centers on raising the stakes and creating demanding contexts. He describes a technique he used with students: when someone said they would try to solve a problem in a week and failed, he would ask, "If a million dollars had been at stake, would you have put the same effort in?" The answer was invariably no. "Often you could do a lot more if you had just the right sort of demanding taskmaster standing by you."

Patel notes that for some subjects, there are clear forcing functions—implementing a transformer for AI, doing roofline analysis for chip design. For other fields like history, there is no equivalent. Nielsen suggests that the key is to change the structure of the work output itself. He draws a contrast between essays he wrote in two days and essays that took three months: "I feel like I didn't learn very much from the ones that only took a couple of days. Whereas some of the ones that took three months, 15 years later I'll still remember." The common element in his deepest learning has been "some creative artifact"—a class, an engagement with a group, an essay or book. "Spending time stuck is incredibly important," he says. "That used to just be annoying. Now it seems like this is actually maybe the most important part of the whole process."

Conclusion

This episode matters because it forces a reckoning with how little we understand about our own most successful knowledge-building enterprise. Nielsen's historical examples—from the 2,000-year verification loop for heliocentrism to the 85-year hostile verification loop for isotopes—demonstrate that science does not proceed by any simple method. The heuristics that guide progress are often invisible, historically contingent, and resistant to codification. This has urgent practical implications for AI-driven scientific discovery: if we cannot articulate how human science works, we cannot simply automate it. The conversation leaves the listener with a sense of both humility and wonder—humility about our ability to reduce science to a process, and wonder at the sheer fecundity of the universe's hidden principles, which may be far more numerous and varied than we currently imagine.

Key takeaways

• The standard narrative of the Michelson-Morley experiment as a clean falsification of the ether that motivated special relativity is largely a myth; Michelson believed in the ether until his death, and Lorentz's competing interpretation was experimentally indistinguishable from Einstein's for decades.
• Great scientists can remain wrong long after the community has moved on, and there is no centralized authority or standard procedure for resolving scientific disputes—expertise itself can become a prison.
• Natural selection took so long to discover not because the core idea was difficult, but because it required the prior establishment of deep time, paleontology, and biogeography as supporting frameworks.
• Deep learning models like AlphaFold represent a new type of scientific object that may require new "verbs" for how we interact with them—they are not explanations in the classic sense but may contain extractable explanations or serve as manipulable tools.
• Alien civilizations would likely develop fundamentally different science and technology stacks than humans, implying that the "tech tree" is vastly larger than any one civilization can explore and that there could be enormous gains from interstellar trade.
• The diminishing returns argument for scientific progress may be misleading because new fields keep opening up, creating fresh opportunities for breakthroughs by young researchers.
• The political economy of science—how credit and reputation are assigned—is socially constructed and could be redesigned to better incentivize sharing code, data, and in-progress ideas.
• Deep learning requires spending significant time stuck on problems and creating demanding contexts for oneself, as the most lasting learning comes from prolonged engagement rather than efficient production.