4. Arches and Scaffolds

Bridging Continuity and Discontinuity in Theory Change

Michel Janssen

I. Synopsis

In principle, new theoretical structures in physics, unlike arches and other architectural structures, could be erected without the use of any scaffolds. After all, that is essentially how the four-dimensional formalism of special relativity, the curved space-times of general relativity, and the Hilbert-space formalism of quantum mechanics are introduced in modern textbooks. Historically, however, such structures, like arches, were originally erected on top of elaborate scaffolds provided by the structures they eventually either partially or completely replaced. The metaphor of arches and scaffolds highlights the remarkable degree of continuity in instances of theory change that, at first sight, look strikingly discontinuous. After putting to rest some historiographical worries about the metaphor and presupposing as little knowledge of the relevant physics and mathematics as possible, I describe how some key steps in the development of relativity and quantum theory in the early decades of the twentieth century can be captured quite naturally in terms of arches and scaffolds. Given how easy it is to find examples of this kind, I argue that it may be worthwhile to further analyze this pattern of theory change with the help of some of Stephen Jay Gould’s ideas about evolutionary biology, especially his notion of constraints. In honor of Gould, I have tried to write this paper as a Gouldian pastiche.1

II. Metaphors for Theory Change

In the section “Plans for Research” of a 1953 application for a Guggenheim Fellowship, Thomas S. Kuhn outlined two book projects that would eventually result in The Copernican Revolution (Kuhn [1957] 1999) and The Structure of Scientific Revolutions (Kuhn [1962] 2012). He already had the title of Structure but not the terms paradigm and paradigm shift. Comparing science to architectural structures, he wrote:

Science, then, does not progress by adding stones to an initially incomplete structure, but by tearing down one habitable structure and rebuilding to a new plan with the old materials and, perhaps, new ones besides. (Hufbauer 2012, 459)2

The “adding stones” metaphor with which Kuhn contrasts the “tearing down” metaphor can be found, for instance, in the preface of Rudolf Carnap’s Aufbau, one of the central texts of logical positivism, the philosophical program that Kuhn was reacting against. In philosophy, Carnap wrote, one ought to proceed as in the natural sciences, where “one stone gets added to another, and thus is gradually constructed a stable edifice, which can be further extended by each following generation” (Carnap 1928; quoted in Sigmund 2017, 137).3

Neither of these building metaphors for how old theories get to be replaced by new ones does justice to all or even most instances of theory change. When building a new theory, one tends to neither simply add to nor simply tear down an old theory. The old cumulative picture may be wrong but so is the alternative picture of a new theory or paradigm built on the burning embers of the old one, a picture conjured up and reinforced by the way in which Kuhn exploited the political connotations of his revolution metaphor in Structure.4 It is good to remind ourselves right at the outset of this paper that “the price of metaphor is . . . eternal vigilance” (Lewontin 1963, 230).5

It has widely been accepted that neither the transition from geocentric to heliocentric astronomy nor the transition from nineteenth-century ether theory to special relativity fits the mold of a Kuhnian paradigm shift in the sense of tearing down one structure and replacing it by another (Swerdlow and Neugebauer 1984; Janssen 2002). In The Copernican Revolution, Kuhn ([1957] 1999, 182) himself used the completely different metaphor of a “bend in an otherwise straight road” to characterize the “shift in . . . direction in . . . astronomical thought” marked by Copernicus’s De Revolutionibus. Both this “bend in the road” metaphor and another metaphor Kuhn was fond of using, that of a gestalt switch (Kuhn [1962] 2012, 85), are incompatible with the metaphor of tearing down some old structure and erecting a new one.

Does the metaphor of Kuhn’s Guggenheim application at least capture the major theoretical upheaval of the mid-1920s known as the quantum revolution? In his book on the Bohr model of atomic structure, historian of science Helge Kragh suggests it does. He writes that matrix mechanics, the earliest incarnation of the new quantum theory, “grew out of what little was left” of the old quantum theory of Niels Bohr and Arnold Sommerfeld—“its ruins” (Kragh 2012, 368). The preface of a popular undergraduate physics textbook gives a similar impression:

Quantum mechanics is not, in my view, something that flows smoothly and naturally from earlier theories. On the contrary, it represents an abrupt and revolutionary departure from classical ideas. (Griffiths 2005, viii)

The Dutch physicist Hendrik B. G. Casimir, who studied with some of the quantum revolutionaries of the mid-1920s, likewise emphasized the disruptive nature of the quantum revolution. “Between 1924 and 1928,” he wrote in his autobiography, based in part on six lectures at the University of Minnesota in 1980, the development of a new quantum mechanics “swept physics like an enormous wave, tearing down provisional structures, stripping classical edifices of illegitimate extensions, and clearing a most fertile soil” (Casimir 1983, 51). Casimir’s mixed metaphor, however, still leaves room for continuity in the quantum cataclysm. His tidal wave did not level the classical building in its entirety but only washed away parts of it.

One will search Kuhn’s writings in vain for an account of the quantum revolution in which the new paradigm was erected on the ruins of the old one. And this is not just because, confounding some of his commentators,6 Kuhn avoided the terminology of Structure in his historical writings.7 In “Reflections on my Critics,” his contribution to the proceedings of the 1965 conference in London that pitted him against Karl Popper, Imre Lakatos, Paul Feyerabend, and others, Kuhn (1970, 256–59) sketched how he saw the transition from the old quantum theory to matrix mechanics. He put great emphasis on what he saw, with considerable justification, as the crisis of the old quantum (calling it a “case book example” of this key concept of Structure8) but characterized the way out of this crisis as “a series of connected steps too complex to be outlined here” and criticized Lakatos for introducing, in his account of the same episode, “the crisis-resolving innovation . . . like a magician pulling a rabbit from a hat” (Kuhn 1970, 256–57). So, for Kuhn, a paradigm shift following a crisis did not necessarily have to be a wholesale and abrupt break with the past.

In fact, many elements of continuity in the transition from classical to quantum physics are on display in Max Jammer’s (1966) classic, The Conceptual Development of Quantum Mechanics, the closest thing we have to a canonical account of this transition. As Jammer says in the preface, one of his main objectives was to show

how in the process of constructing the conceptual edifice of quantum mechanics each stage depended on those preceding it without necessarily following from them as a logical consequence. (Jammer 1966, vii)

Olivier Darrigol more explicitly focused on continuities rather than discontinuities in his book From c-Numbers to q-Numbers. In the introduction, he expressed his conviction that “to obtain new theories” modern physicists “extend, combine, or transpose available pieces of theory” (Darrigol 1992, xxii). Jürgen Renn (2006) has introduced the notion of “Copernicus processes” to make a similar point.9 One of the mottos Darrigol chose for his book has quantum architect Paul Dirac, in unpublished lecture notes of 1927, directly contradicting the assessment of the modern textbook writer quoted above:

The new quantum theory requires very few changes from the classical theory . . . so that many of the features of the classical theory to which it owes its attractiveness can be taken over unchanged into the quantum theory. (Darrigol 1992, xiii)

The clause I left out—“these changes being of a fundamental nature”—gives Dirac’s statement a paradoxical flavor. The key to the resolution of this paradox will be given in section IV (see note 47). Following Darrigol’s lead, more recent work on the early history of quantum physics has highlighted a variety of continuities.10

To sum up: as long as the concept of a paradigm shift includes the “tearing down” element emphasized in Kuhn’s Guggenheim application, neither the Copernican revolution, nor the relativity revolution, nor the quantum revolution fits the bill. Some of the revolutionaries in these cases can fairly be labeled iconoclasts, but none of them simply smashed the icons of the old guard.

I therefore propose—with some trepidation—a different building metaphor for theory change, one that involves both adding to and tearing down old structures and one that captures both continuities and discontinuities. I will present five examples from the early history of relativity and quantum theory, my area of expertise as a historian of science, in which a new theoretical structure can be seen, or so I will argue, as an arch built on top of a scaffold provided by an older theoretical structure discarded (at least in part) once the arch was finished.11

In four of these examples, the scaffold was discarded in its entirety once physicists recognized that the arch could support itself. In my last example, the second one from the history of quantum theory, only part of the scaffold was dismantled while other parts stayed in place long after the arch was finished. In this case, both arch and scaffold became part of the edifice of quantum theory as physicists use and teach it today. In this example, the relation between arch and scaffold is considerably more complicated than in the first four. However, to the extent that the arch-and-scaffold metaphor can profitably be used to characterize other instances of theory change at all, I expect the messy complicated cases to be more typical than the clean, simple ones.

As my two examples from the history of quantum theory will illustrate, the theoretical structure that plays the role of the arch in one instance of theory change can play the role of the scaffold in the next. This observation helps explain why scaffolds are sometimes only partially discarded after they have served their purpose in the building of an arch. What is merely a scaffold for some theorists may have been an arch for some of their predecessors and continue to be seen and treated as such by part of the relevant community.

The arch-and-scaffold metaphor can be broken down into specific elements with the help of Figure 4.1. This figure shows the construction of one of a total of nine arches, each 120 feet wide, of a bridge over the Thames in London in the 1810s. Originally called the Strand Bridge, it was renamed in 1817 to commemorate the Battle of Waterloo. It was demolished and replaced by a new one in the 1940s.

Figure 4.1 shows various components of the arch-and-scaffold metaphor that I will be using in my examples, mindful of the old adage that nothing kills a metaphor faster than the attempt to formalize it. The foundation on which both scaffold and arch are built is called the tas-de-charge. Then there is the scaffold or centering frame. The stones placed on this scaffold to make up the arch are called the voussoirs. Those at the ends are called the springers; the one in the center is called the keystone. The distinction between springers and keystones can meaningfully be made for the theoretical structures playing the role of an arch in my metaphor as well. For instance, the initial conception of a new theoretical structure can be seen as springers; the finishing touches as the keystone. Or, combining the two examples I will discuss from the history of special relativity, one can think of the new space-time structure as springers and of the new formalism for the physics of systems in that new space-time as the keystone. In the first of my two examples from the history of quantum theory (the fourth case study in section IV), we will even encounter an element (Niels Bohr’s correspondence principle as it was used by several physicists in 1924–1925) corresponding to the final element labeled in Figure 4.1, the windlass, the instrument used to lift both the scaffold and the arch stones.

A nice feature of the example of the scaffolding of a stone-arch bridge is that both scaffold and arch can serve as a bridge, though the latter will make for a sturdier one than the former. In the case of a bridge, the relation between arch and scaffold is thus similar in this respect to that between a scaffolded and a scaffolding theory. In fact, the scaffolding shown in Figure 4.1 has basically the same structure as the wooden footbridge in Cambridge known as the Mathematical Bridge shown in Figure 4.2. The same relation is illustrated by two incarnations of the arched Walton Bridge across the Thames: a wooden bridge was completed in 1750 (and painted by Canaletto in 1754); a stone bridge opened in 1788 (and was painted by J. M. W. Turner in 1805).12

The basic idea behind the arch-and-scaffold metaphor is hardly new. For instance, in his contribution to the proceedings of the 1965 conference in London mentioned above, Lakatos noted that

some of the most important research programs in the history of science were grafted on to older programs with which they were blatantly inconsistent. For instance, Copernican astronomy was “grafted” on to Aristotelian physics, Bohr’s programme on to Maxwell’s . . . As the young grafted program strengthens, the peaceful co-existence comes to an end, the symbiosis becomes competitive and the champions of the new programme try to replace the old programme altogether. (Lakatos 1970, 142)

Whereas Lakatos used the term grafting, Werner Israel, in a paper on the prehistory of black holes, actually uses the term scaffolding (“The old and discarded is often scaffolding for the new”) and explicitly offers the mechanism for theory change his metaphor is supposed to capture as an alternative to “the Kuhnian cycle of paradigm and revolution” (Israel 1987, 200). He gives two examples, both from nineteenth-century physics: “Faraday’s concept of field grew out of the aether, Carnot’s thermodynamics from the notion of a caloric fluid” (Israel 1987, 200). Unlike Lakatos and Israel, I will work out my examples in considerable detail.

There is some family resemblance between the arch-and-scaffold metaphor and two other well-known metaphors for the construction of knowledge, Sextus Empiricus’s and Wittgenstein’s ladders and Neurath’s ship. In the penultimate proposition of the Tractatus, Wittgenstein (1921, 89, Proposition 6.54) noted that his reader “must, so to speak, throw away the ladder after he has climbed up it.” This ladder metaphor has a rich history going back at least to Plato (Agassi 1975, 456–57; Gakis 2010). To mention just one precedent, Francis Bacon (1620, 14), in his Novum Organum, talked about “The Ladder of the Intellect.” He compared the way in which authors suppress some of the evidence for their claims to “what men do in building, namely after completion of the building, remove the scaffolding and ladders from sight” (Bacon 1620, 97, Aphorism 125).

Neurath used his ship metaphor in several publications. In a booklet written in response to the first volume of Oswald Spengler’s Decline of the West, he wrote:

We are like sailors who on the open sea must reconstruct their ship but are never able to start afresh from the bottom. Where a beam is taken away a new one must at once be put there, and for this the rest of the ship is used as support. (Neurath 1921, 199 [emphasis added]; quoted and discussed in Sigmund 2017, 88)

Substituting scaffold for support in the italicized clause, we see that Neurath’s metaphor is closer to the arch-and-scaffold metaphor than the proverbial ladders of Wittgenstein and others.13 This was recognized by Wimsatt and Griesemer (2007, 300), who refer to Neurath’s metaphor approvingly in a paper on scaffolding. The italicized clause, however, does not return in a later version of the metaphor. As Neurath put it in Erkenntniss, the house journal of the Vienna Circle in the early 1930s (Sigmund 2017, 223): “We are like sailors, who have to rebuild their ship on the open sea, without ever being able to dismantle it in dry-dock and reconstruct it from the best components” (Neurath 1932–1933, 92).14

The metaphor of a scaffold has also been used by several mathematicians. Carl Friedrich Gauss, we read in reminiscences recorded by a friend shortly after his death, “never gave a piece of work to the public until he had given it the perfection of form he desired for it. A good building should not show its scaffolding when completed, he used to say” (Sartorius von Waltershausen 1856, 67). Some of Gauss’s contemporaries therefore compared him to a fox erasing its tracks in the sand with its tail.15 Historian of astronomy Curtis Wilson quotes a much earlier example of this use of the scaffolding metaphor. Wilson (2001, 168–69) is relating how Newton’s early devotion to algebra gave way to a strong preference for geometry. He recalls how Newton himself once claimed that he had constructed the proofs of most propositions in his Principia analytically but presented them in geometrical terms and thus, to use the Gaussian metaphor, removed the analytical scaffolding. This would have made for a nice illustration of the metaphor had not Tom Whiteside, editor of Newton’s mathematical papers, shown Newton’s claim to be false. In a footnote to this passage, however, Wilson quotes Thomas Hobbes actually using the metaphor in his attack on John Wallis’s algebra: “Symbols are poor, unhandsome, though necessary, scaffolds of demonstration; and ought no more to appear in public, than the most deformed necessary business which you do in your chambers” (Hobbes 1656, 248; quoted in Wilson 2001, 185n36). This use of the scaffolding metaphor by Hobbes and Gauss is close to mine. The more common usage among mathematicians, however, appears to be quite different.

As David Hilbert liked to point out, the mathematician “erects living quarters before turning to the foundations” (Rowe 1997, 548). This is a paraphrase of the following passage in (unpublished) notes taken by Max Born, another quantum architect, of lectures Hilbert delivered in Göttingen in 1905:

The buildings of science are not erected the way a residential property is, where the retaining walls are put in place before one moves on to the construction and expansion of living quarters. Science prefers to get inhabitable spaces ready as quickly as possible to conduct its business. Only afterwards, when it turns out that the loosely and unevenly laid foundations cannot carry the weight of some additions to the living quarters, does science get around to support and secure those foundations. This is not a deficiency but rather the correct and healthy development. (Peckhaus 1990, 51; quoted in a different translation in Corry 1999, 163–64)

This attitude is clearly in evidence in Hilbert’s early work on general relativity. Whereas Einstein only threw in his fate with the elegant mathematics of Riemannian geometry once he had convinced himself that the blueprint it provided for a spacious new theory of gravity included a solid grounding in physics, Hilbert was happy to move into these new quarters without inspecting the foundations first (see note 36).

The same attitude can be found in the methodological reflections of other mathematicians and mathematical physicists. My first example comes from a 1900 textbook by Paul Volkmann, professor of physics in Königsberg and an associate of Hilbert:

The conceptual system of physics should not be conceived as one which is produced bottom-up like a building. Rather it is a thorough system of cross-references, which is built like a vault or the arch of a bridge, and which demands that the most diverse references must be made in advance from the outset, and reciprocally, that as later constructions are performed the most diverse retrospections to earlier dispositions and determinations must hold. Physics, briefly said, is a conceptual system which is consolidated retroactively. (Volkmann 1900, 3–4; quoted in Corry 2004, 61)

My second example comes from an address to the London Mathematical Society, delivered in 1924 by its outgoing president, William H. Young, and published two years later. I am not aware of any direct connection between Young and Hilbert, but Young’s wife, Grace Chisholm Young, a mathematician in her own right, earned her doctorate in Göttingen working with Hilbert’s most famous colleague, Felix Klein. Echoing the observation attributed to Gauss in the passage from a friend’s reminiscences quoted above, Young began his presidential address by reminding his audience of the common view that “all scaffolding is futile, because no scaffolding is to appear on the finished edifice” (Young 1926, 421). Young took exception to this view and argued that it had been harmful to “The Progress of Mathematical Analysis in the Twentieth Century,” the topic of his presidential address. “It is essential,” he insisted, “to go to a higher discipline [in this case: set theory] in order to master a lower one [in this case: analysis]” (Young 1926, 427). More advanced mathematics—to paraphrase Young’s point—may be needed to prove results in more basic mathematics, which lesser mortals can then use without having to worry about the higher mathematics ever again. So it is the higher rather than the lower mathematics that plays the role of the scaffolding for Young. Hilbert expressed the same idea in a 1917 lecture. Hilbert, to paraphrase again, “described the axiomatic method as a process analogous to constructing ever deeper foundations to support a building still under construction” (Rowe 1997, 548). In my final case study in section IV, I will present Jordan’s transformation theory as the scaffold on which von Neumann built the arch of his Hilbert space formalism. This case study, however, can alternatively be cast in terms of von Neumann deepening the foundation of Jordan’s theory, in which case von Neumann’s theory would provide the scaffold (in Young’s sense) built to secure Jordan’s mathematically unsound arch.

In his 1917 book, The Electron, American physicist Robert A. Millikan used a metaphor that combines my arch-and-scaffold metaphor and Hilbert’s building-and-foundation metaphor to describe the relation between Einstein’s formula for the photoelectric effect, which Millikan had experimentally verified the year before, and the controversial light-quantum hypothesis from which Einstein had derived this formula in 1905. Millikan wrote:

Despite . . . the apparently complete success of the Einstein equation, the physical theory of which it was designed to be the symbolic expression is found so untenable that Einstein himself, I believe, no longer holds to it, and we are in the position of having built a very perfect structure and then knocked out entirely the underpinning without causing the building to fall. It stands complete and apparently well tested, but without any visible means of support. These supports must obviously exist, and the most fascinating problem of modern physics is to find them. (Millikan 1917, 230; quoted and discussed by Stuewer 2014, 156)

Millikan, in other words, saw Einstein’s formula for the photoelectric effect as an arch in danger of collapsing and thus in need of a scaffold to support it. As we will see in section IV, we can likewise think of von Neumann bringing in Hilbert space as a scaffold to prevent Jordan’s arch from collapsing (see note 37 for another, more clear-cut, example).

On the face of it, Heinrich Hertz’s attitude toward Maxwell’s theory, expressed in his famous line that “Maxwell’s theory is Maxwell’s system of equations” (Hertz 1893, 21), may look similar to Millikan’s attitude toward the theory from which Einstein derived the formula for the photoelectric effect. Hertz, on this reading, took Maxwell’s “perfect structure and then knocked out entirely the underpinning without causing the building to fall.” Hertz himself, however, saw his contribution to Maxwell’s theory quite differently. In a lecture at the 1889 Naturforscherversammlung (the annual meeting of the German Society of Natural Scientists and Physicians) held that year in Heidelberg, Hertz used the metaphor of a bridge, similar to the one used by Volkmann in the passage quoted above, to explain the importance of his experimental demonstration of electromagnetic waves for the further development of Maxwell’s theory. In his commemorative speech at the German Physical Society in Berlin, shortly after Hertz’s early death, Max Planck referred to this lecture and further elaborated on Hertz’s metaphor. In his 1889 lecture, Planck recalled, Hertz

compared Maxwell’s theory to a bridge that, with a bold arch, spans across the wide ravine between the regions of optical and electromagnetic phenomena, characterized by molecular and cosmic wavelengths, respectively. Because of these fast electrical vibrations, he explained at the time, new land had been gained in the middle of this ravine and a firmly planted pillar now stood on it providing additional support for the bridge. Since that time, various kinds of expert craftsmanship have made this pillar taller and broader, ensuring that today the bridge stands stronger and prouder than ever. It no longer just serves, as it did in the past, the occasional forays of the odd bold adventurer. No, it can already carry the heavy trucks of research in the exact sciences, constantly shipping the land’s treasures from one region to another, thereby enriching both. (Planck 1894, 283)

Planck’s elaboration of this metaphor reminds us that bridges are not built to be admired but used. The same is true for the metaphorical arches and bridges to be examined in this paper.

III. Historiographical Scruples

Whiggishness: Three Lines of Defense

Before I present some concrete examples of theory change that, I argue, fit my metaphor of arches and scaffolds, I need to address the obvious worry that any historical narrative deploying this metaphor is intrinsically Whiggish.16 After all, we can only see in hindsight that one scientist’s arch was the next scientist’s scaffold. I have three lines of defense against this charge of Whiggishness, and I am prepared to make my stand on the third.

My first line of defense is that scientists sometimes do realize that they are building a scaffold and not an arch. Throughout the reign of the old quantum theory, for instance, Bohr was keenly aware of its provisional character. In a letter to Sommerfeld of April 30, 1922, he described his work on the theory as a “sincere effort to obtain an inner connection such that one can hope to create a valid fundament for further construction” (Sommerfeld 2004, doc. 55; translation from Eckert 2013, 126). In the early 1920s, Göttingen emerged as another center for work on the old quantum theory, alongside Bohr’s Copenhagen and Sommerfeld’s Munich. Born, the leader of this third center, clearly saw the limitations of the theory too. In the preface of his book, Atomic Mechanics, he cautioned:

The work is deliberately conceived as an attempt . . . to ascertain the limits within which the present principles of atomic and quantum theory are valid and . . . to explore the ways by which we may hope to proceed . . . To make this program clear in the title, I have called the present book “Vol. I;” the second volume is to contain a closer approximation to the “final” atomic mechanics . . . The second volume may, in consequence, remain for many years unwritten. In the meantime let its virtual existence serve to make clear the aim and spirit of this book. (Born 1925, v)17

The sequel would be written much sooner than Born anticipated. It was published only five years later (Born and Jordan 1930). First, however, in 1927, an English translation of Born’s 1925 book appeared. Given the rapid developments since 1925, the decision to republish his treatise in translation without any substantive changes or additions could be called into question. Born tries to preempt criticism on this score in the preface. The first argument he gives in his defense is that “it seems to me that the time is not yet arrived when the new mechanics can be built up on its own foundations, without any connection with classical theory” (Born 1927, xi). Without denying the self-serving purpose of this preface, we can say that Born clearly recognized the role of the old theory in building up the new one.

Einstein used a building metaphor, suggestive at least of an arch and a scaffold, to describe the sense in which he considered general relativity, his theory of gravity, to be preliminary. In the fall of 1915, Einstein and Hilbert found themselves in a race to complete the theory (Janssen and Renn 2015). They eventually arrived at the same field equation. This equation determines what gravitational field a given matter distribution will produce. The two men agreed that the left-hand side of this equation describes the curvature of space-time, reflecting the central idea of the theory that gravity is part of the fabric of space-time. They did not agree, however, on the interpretation of the right-hand side, describing the matter responsible for this space-time curvature. Hilbert endorsed the view of Gustav Mie, a late representative of the so-called electromagnetic worldview (see the second case study in section IV), that all matter ultimately consists of electromagnetic fields satisfying some nonlinear generalization of Maxwell’s equations. Einstein briefly flirted with this idea but rejected it. For him the right-hand side of his field equation was just a placeholder for whatever would be supplied later by a more satisfactory theory of matter. Two decades after he had introduced general relativity, Einstein was still waiting for such a theory. General relativity, he wrote in 1936,

is similar to a building, one wing of which is made of fine marble (left part of the equation), but the other wing of which is built of low grade wood (right side of equation). (Einstein 1936, 312; quoted and discussed in van Dongen 2010, 62)

The wing made out of wood can be thought of as a scaffold awaiting the arrival of the marble for the completion of this part of the building.

Robert Hooke is an early example of a natural philosopher with the humility to recognize that he was building a scaffold rather than an arch. Hooke even used the arch-and-scaffold metaphor, although he cast it in somewhat different terms.18 In the preface of his Micrographia, he wrote: “If I have contributed the meanest foundations whereon others may raise nobler Superstructures, I am abundantly satisfied” (Hooke 1665, xii–xiii). In a similar vein, Einstein (1917b, 91) wrote in his popular book on relativity that “the most beautiful fate of a physical theory is to point the way to the establishment of a more inclusive theory, in which it lives on as a limiting case” (translation from Holton 1981, 101).19

One would expect it to be easier to find examples of scientists recognizing the preliminary character of theories put forward by their contemporaries and calling the arches erected by them mere scaffolds. Despite some crowdsourcing, however, I have only been able to find two such examples so far. Only in one of those is the term scaffold used explicitly.

The first example comes from Arthur Stanley Eddington’s Gifford Lectures in Edinburgh in 1927. Eddington prefaced his discussion of wave mechanics with the following warning:

Schrödinger’s theory is now enjoying the full tide of popularity . . . Rather against my better judgment I will try to give a rough impression of the theory. It would probably be wiser to nail up over the door of the new quantum theory a notice, “Structural alterations in progress—No admittance except on business,” and particularly to warn the doorkeeper to keep out prying philosophers. I will, however, content myself with the protest that, whilst Schrödinger’s theory is guiding us to sound and rapid progress in many of the mathematical problems confronting us and is indispensable in its practical utility, I do not see the least likelihood that his ideas will survive long in their present form. (Eddington 1928, 210–11)

The second example comes from an article in The Electrician in 1893, in which John Henry Poynting criticized the mechanical model of the ether that Oliver Lodge (1889) had promoted in his bestseller Modern Views of Electricity. This model is known as the cogwheel ether (see Figure 4.3).20 Pierre Duhem famously and disapprovingly compared Lodge’s book to a factory:

Here is a book intended to expound the modern theories of electricity and to expound a new theory. In it there are nothing but strings which move around pulleys, which roll around drums, which go through pearl beads, which carry weights; and tubes which pump water while others swell and contract; toothed wheels which are geared to one another and engage hooks. We thought we were entering the tranquil and neatly ordered abode of reason, but we find ourselves in a factory. (Duhem 1914, 70–71; quoted in Hunt 1991, 87)

Poynting had more sympathy for Lodge’s project. “We are all looking forward,” he wrote, “to the time when by mechanical explanation of electromagnetism, light shall once more become mechanical” (Poynting 1893, 635). Yet, he cautioned, such explanations

are solely of value as a scaffolding enabling us to build up a permanent structure of facts, i.e., of phenomena affecting our senses. And inasmuch as we may at any time have to replace the old scaffolding by new, more suitable for new parts of the building, it is a mistake to make the scaffolding too solid, and to regard it as permanent and of equal value with the building itself. (Poynting 1893, 635)

The problem with Lodge’s work, according to Poynting (1893), was that he made the “scaffolding . . . as important as the building” (635).

As striking as these passages from the writings of Bohr, Born, Einstein, Eddington, Hooke, and Poynting are, I do not expect to find too many scientists characterizing the theories of their contemporaries, let alone their own, as mere scaffolds for future theories. These examples thus only go so far in deflecting the charge of Whiggishness against the arch-and-scaffold metaphor.

My second line of defense is that structures often inadvertently serve as scaffolds for other structures. So the use of the arch-and-scaffold metaphor does not automatically imply the kind of teleology responsible for the odium of Whig history. A good example—and the one that originally inspired my exploration of the metaphor—is the proposal by Alexander Graham Cairns-Smith (1985) in Seven Clues to the Origin of Life that the complex nucleotides of RNA and DNA were first assembled on a scaffold of minute clay crystals. Cairns-Smith (1985) asks: “How can a complex collaboration between components evolve in small steps?” (58). The answer, he suggests, is “that every so often an older way of doing things will be displaced by a newer way that depends on a new set of subsystems. It is then that seemingly paradoxical collaborations may come about” (Cairns-Smith 1985, 59). He uses the arch-and-scaffold metaphor, complete with two simple diagrams (see Figure 4.4), to illustrate how this can happen:

Consider this very simplified model—an arch of stones. This might seem to be a paradoxical structure if you had been told that it arose from a succession of small modifications, that it had been built one stone at a time. How can you build any kind of arch gradually? The answer is with a supporting scaffolding. In this case you might have used a scaffolding of stones. First you would build a wall, one stone at a time. Then you would remove stones to leave the “paradoxical” structure. Is there any other way than with scaffolding of some sort? Is there any other way to explain the kind of complex leaning together of subsystems that one finds in organisms, when each of several things depends on each other, than that there had been earlier pieces, now missing? (Cairns-Smith 1985, 59–60)

John Norton (2014, 685–87) used essentially the same analogy (including a drawing of an arch) that Cairns-Smith used for the origin of life to explain how the long sequence of inductive inferences that eventually got us to today’s scientific knowledge could ever have gotten off the ground.

Cairn-Smith’s example—and, for that matter, Norton’s—clearly shows that the arch-and-scaffold metaphor can be used without implying that the earlier structure was intentionally built as a scaffold. Yet it still does not quite put the worry about Whiggishness to rest. Unlike Norton, I am starting in medias res and not at the mythical dawn of humanity. In that case, a historical narrative in which a structure is called a scaffold before the construction of the arch built on top of it has even begun at least strongly suggests an element of teleology. Such a narrative is bound to lend an air of linearity and inevitability to the transition from the scaffolding to the scaffolded theoretical structure. Fortunately, spelling out the danger of the metaphor in this way points to an obvious way to avoid it: tell the story backward! In other words—as the order of the terms already happens to suggest—have narratives using the arch-and-scaffold metaphor introduce the arches before the scaffolds.

This third line of defense against the charge of Whiggishness is both more effective than the other two and more natural than it may initially sound. A narrative that moves from arches to scaffolds neither implies nor suggests teleology. Instead it answers a question one naturally asks when standing in awe in front of an arch or some other architectural marvel: How did they build that? A satisfactory answer to such questions will often involve the identification of an earlier structure that served as the scaffold for the architectural structure and was then partially or completely taken down. Similar questions can be asked about theoretical structures in physics, such as the four-dimensional formalism of special relativity, the curved space-times of general relativity, and the Hilbert-space formalism of quantum mechanics. How did physicists come up with these formalisms to describe and explore nature? In at least some instances, as the examples discussed below will illustrate, such questions can be answered by the identification of some earlier formalism that scaffolded the new formalism and was discarded either in whole or in part once it was recognized that the new formalism no longer needed extraneous support.

Following Hilbert’s lead (see section II), mathematicians might want to put the metaphor on its head and have the new formalism play the role of the scaffold built retroactively to support earlier formalism playing the role of the arch, built hastily and in danger of collapsing. Two of the examples I will be discussing—both, unsurprisingly, involving mathematicians (Minkowski and von Neumann)—can be used to illustrate this “inverted” use of the metaphor. When the metaphor is used in this way, we do not run afoul of the charge of Whiggishness because the scaffold is built after the arch.

When I use the metaphor without this kind of inversion (i.e., when the scaffold is built before the arch), I hope to steer clear of any pernicious Whiggishness by making the kind of question I am asking explicit in the way indicated above. The arch-and-scaffold narratives offered in response to such questions neither imply nor suggest that the progression from the earlier to the later theoretical structure was linear or inevitable. This line of defense, however, does place an important constraint on the construction of such narratives. To bring out and reinforce the analogy with the “How did they build that?” question we ask when we happen upon a puzzling architectural structure, such narratives should all start with at least a preliminary characterization of the theoretical structure that plays the role of the arch in the story. Ideally, one then proceeds to tell the story backward. This kind of time reversal has been attempted in many movies and TV shows—with limited success.21 Given this checkered track record, it may be wiser to stick to the tried-and-true strategy of presenting a new historical account as an alternative to some canonical received view. I will at least try to combine this safe standard approach with the challenging one of telling the story backward, which is clearly the more elegant way of constructing a historical narrative based on the arch-and-scaffold metaphor.

Some historians may still not be satisfied. Even if they grant that I am not giving Whiggish answers, they may still object that I am asking Whiggish questions. Asking how something came about, after all, obviously presupposes that it was there to begin with. This residual charge of Whiggishness does not bother me. I make no excuse for asking questions informed by present-day knowledge and for making decisions about what source material to look at based on what I expect to shed light on the development of currently accepted theories. Which is not to say that there are no dangers and pitfalls to this approach (see, e.g., note 43). Still, this kind of Whiggishness strikes me as relatively benign. In fact, it is implied by the commonly accepted names for the subfields I take myself to be working in—history of relativity theory and history of quantum mechanics. My examples of arches and scaffolds are all based on my earlier research in these two areas. The papers in which I published this research implicitly answer questions similar to those I explicitly raise when I recast parts of their narratives in terms of arches and scaffolds. Hence, far from compounding my Whiggish sins, I am actually atoning for them by adopting the arch-and-scaffold metaphor!

No Actors’ Categories: Two Lines of Defense

A related objection to the arch-and-scaffold metaphor is that it is not an actors’ category. I can offer two rejoinders to defuse that charge. The first is that, even though there may not have been many, several historical actors did use the language of arches (or, at least, buildings) and their scaffolds. We have already seen passages from (or attributed to) several scientists (Poynting, Millikan), philosophers (Bacon, Hobbes, Wittgenstein), and mathematicians (Gauss, Hilbert), in which they use the term scaffold or terms like it (ladder, support, foundations).

We saw Poynting, in his 1893 critique of Lodge’s theory, warn his readers not to mistake the scaffold for the building in science. Six years later, in his 1899 presidential address to the Section of Mathematical and Physical Science of the British Association for the Advancement of Science, he issued a more general version of this warning. “To give the hypotheses equal validity with facts,” he said on that occasion, “is to confuse the temporary scaffolding with the building itself” (Poynting 1899, 620). The paragraph concluding with this sentence suggests that “the building itself” refers to nature itself. This paragraph also yields two more occurrences of the term scaffolding and one of ladder (cf. the passage from Bacon’s Novum Organum quoted in section II):

While the building of Nature is growing spontaneously from within, the model of it, which we seek to construct in our descriptive science, can only be constructed by means of scaffolding from without, a scaffolding of hypotheses. While in the real building all is continuous, in our model there are detached parts which must be connected with the rest by temporary ladders and passages, or which must be supported till we can see how to fill in the understructure. (Poynting 1899, 620; quoted in Freund [1904] 1968, 227)

On the next page, Poynting alerted his readers to the danger of mistaking a preliminary theory for a definitive one: “It is necessary to bear in mind what part is scaffolding, and what is the building itself, already firm and complete” (Poynting 1899, 621; quoted in Freund [1904] 1968, 227). Here the contrast between scaffolding and building does not refer to the contrast between our scientific models of nature and nature itself but to that between preliminary partial theories (the “detached parts”) and more comprehensive and secure theories (the “understructure”).

A few years before Poynting, the American paleontologist Edward Drinker Cope had issued a similar warning. In a letter to the editor of Science, Cope used an analogy to buildings and scaffolds to defend the value of hypotheses as long as judgments about their truth or falsity are suspended:

Builders generally know the difference between the scaffolding and the building. And a builder will value the indication of faults in his scaffolding rather than general disquisitions on the uselessness of scaffolds in general. (Cope 1895, 522)

Cope and Poynting were by no means the first to use the analogy to buildings and scaffolds to make these points. In 1820, Humphry Davy had already voiced some of the same concerns they raised in the 1890s, using remarkably similar language. In his address to the Royal Society upon taking up its presidency, Davy had cautioned his audience to “attach no importance to hypotheses” and to treat “them rather as part of the scaffolding of the building of science, than as belonging either to its foundations, materials, or ornaments” (Davy 1820, 14).

On the European continent, Johann Wolfgang von Goethe made essentially the same point in a passage that only seems to have been published posthumously:

Hypotheses are scaffoldings which one puts up before building and which one tears down once the building is complete. They are indispensable for the worker: only one should not take the scaffolding for the building. (Beutler 1949, 9:653; translation from Agassi 1975, 457; see also Agassi 2013, 118)

The sentiment expressed here by Davy, Goethe, Cope, and Poynting can also be found in Preliminary Discourse on the Study of Natural Philosophy by the British astronomer and philosopher of science avant la lettre John Herschel: “To lay any great stress on hypotheses . . . except in as much as they serve as a scaffold for the erection of general laws, is to ‘quite mistake the scaffold for the pile’” (Herschel [1830] 1966, 204; see Agassi 1975, 457).

In an essay in Quarterly Reviews in 1840, Herschel provided a more elaborate and more eloquent statement of Poynting’s observation, quoted above, that “while the building of Nature is growing spontaneously [our] model of it . . . can only be constructed by means of scaffolding.” This is how Herschel put it:

In erecting the pinnacles of this temple [of science], the intellect of man seems quite as incapable of proceeding without a scaffolding or circumstructure foreign to their design, and destined only for temporary duration, as in the rearing of his material edifices. A philosophical theory does not shoot up like the tall and spiry pine in graceful and unencumbered natural growth, but, like a column built by men, ascends amid extraneous apparatus and shapeless masses of materials. (Herschel 1857, 67)

As the quotations from Planck in section II and Poynting in this section suggest, the history of electromagnetism may make for good hunting grounds for scaffolding metaphors. Here is one from the preserves of James Clerk Maxwell’s Treatise on Electricity and Magnetism:

We can scarcely believe that Ampère really discovered the law of action by means of the experiments which he describes. We are led to suspect, what, indeed, he tells us himself, that he discovered the law by some process which he has not shewn us, and that when he had afterwards built up a perfect demonstration he removed all traces of the scaffolding by which he has raised it. (Maxwell 1873, 162–63)

Maxwell’s observation about Ampère is reminiscent of Abel’s (or Jacobi’s) comparison of Gauss to a fox erasing its tracks (see note 15).

In his History of the Theories of Aether and Electricity, Sir Edmund Whittaker used a scaffolding metaphor to describe some of Maxwell’s own work. After discussing the paper in which Maxwell (1861–1862) first published what in hindsight we recognize as Maxwell’s equations of electrodynamics (see Figure 4.6 in section V for the mechanical model Maxwell used to derive those equations), he wrote:

Maxwell’s views were presented in a more developed form in a memoir . . . read to the Royal Society in 1864 [Maxwell 1865]; in this the architecture of the system was displayed, stripped of the scaffolding by aid of which it had been first erected. (Whittaker [1951–1954] 1987, 1:255)

At least one later commentator on Maxwell (1865) used the same metaphor: “The scaffolding could now be kicked away from the edifice” (Kargon 1969, 434).22 To give another example of a historian of science using the metaphor, Owen Gingerich (1989, 69), in an article about Kepler, wrote that “both Ptolemy in the Almagest and Copernicus in De Revolutionibus had carefully concealed the scaffolding by which they had erected their mathematical models.”

For my final example of scientists themselves using the metaphor, I return to Einstein. In 1953, the year before he died, Einstein compared the extraction of concepts from experience to the construction of houses and bridges with the help of a scaffold. In a letter to Maurice Solovine, his old friend and fellow member of the mock Olympia Academy of his halcyon days as a patent clerk in Berne, he wrote:

Concepts can never be derived logically from experience and be above criticism. But for didactic and also heuristic purposes such a procedure is inevitable. Moral: Unless one sins against logic, one generally gets nowhere; or, one cannot build a house or construct a bridge without using a scaffold which is really not one of its basic parts. (Einstein 1987, 147; cf. Agassi 1975, 456–59)

The point for which Einstein used this metaphor is rather different from the one pursued in this paper. It is closer, actually, to the point Bacon, Wittgenstein, and others wanted to make with their ladder metaphor (Agassi 1975, 456–58; cf. section II).

Despite these intriguing passages obtained largely through crowdsourcing (see the acknowledgments), I must admit that I have only found a handful of examples so far of historical actors using scaffolding metaphors for the development of science. Moreover, I am not aware of any of the actors in my five examples using this kind of language. Let me use another analogy to explain why I ultimately see this not as a weakness but as a strength of my project.23

Historians are in the business of selecting parts of invariably incomplete source material and carefully arranging it in historical narratives to make their audience look at it from their point of view. In this respect, historical narratives are not unlike museum exhibits of dinosaurs (cf. Figure 4.5).

When mounting a dinosaur for exhibit, curators use several devices (often in combination) to create the illusion of a complete animal. First, they restore missing or damaged pieces of fossil bone with plaster. Next, they combine complementary specimens of the same species to form a composite skeleton. Finally, they make sculptures or casts of any missing elements to create a whole animal. To hold the fossil bones in lifelike positions, they always use some kind of metal armature, or scaffold. Normally, the armature is designed to be as unobtrusive as possible to give the museum visitor the impression that the specimen is self-supporting. Eugene S. Richardson, a curator at the Field Museum in Chicago, wrote a laudatory poem about an exhibit of Gorgosaurus, which ends with the following lines: “Here he stands without a scaffold! Gorgosaur is self-supporting!” (Brinkman 2013, 223–25).

There is some controversy about the practice of disguising these curatorial devices to enhance the illusion. At some museums, restored pieces of fossil bone, or plaster casts of fossils, have been carefully painted to match the original fossil material and to hide its artificial origins. Other museums, by contrast, have made their restorations in a different color so that the visitor can easily distinguish them from original fossils. Museums will sometimes exhibit only original fossil material, restoring the complete animal in a background mural or a drawing.24

The latter approach, where the curator’s role in reconstructing the animal is explicitly acknowledged, corresponds to the approach to writing history of science that I am adopting in this project. (The approach in the papers that I draw on for this project was closer to the former.) Using a metaphor that is clearly my own and not an actors’ category to present my source material to my readers, I am like the curator whose use of elements that will not be mistaken for parts of the original specimen forcefully reminds visitors of her own role in the reconstruction of the specimen.

IV. Five Examples from the Development of Relativity and Quantum Theory

I have identified five instances of major theory change in the history of relativity and quantum theory that fit the arch-and-scaffold metaphor. In these examples, we will encounter two kinds of relations between arch and scaffold, which will be examined more generally in sections V–VI. In this section I present my five case studies, explain how they fit the metaphor, and indicate what we gain, in terms of our historical understanding of these episodes, by recasting their narratives in terms of it.25 To do so, I need to cover these examples in some detail. At sufficiently low resolution just about any episode in the history of science can be made to fit just about any metaphor. It is thus imperative to show that the arch-and-scaffold metaphor captures such episodes at a much more fine-grained level.

I have tried to write this section without presupposing any knowledge of the relevant physics. Even so, readers who are familiar with (the history of) relativity and quantum theory will undoubtedly find this section much easier to read than readers who are not. Those without a background in physics are encouraged to read as much as they can stomach of the first, the third, and the fourth case study and skim or skip only the mathematically more demanding (parts of) the second and the fifth. A detailed understanding of section IV is not required to appreciate the more general points about the use of the arch-and-scaffold metaphor in the history of science in sections V and VI.

How Minkowski Space-Time Was Scaffolded by Lorentz’s Theorem of Corresponding States26

The natural starting point for a history of special relativity in terms of arches and scaffolds is the lecture “Space and Time” by the Göttingen mathematician Hermann Minkowski (1909). Minkowski gave this lecture at the 1908 Naturforscherversammlung held that year in Cologne. It was published posthumously the following year. Minkowski began his lecture by proclaiming that

henceforth space by itself, and time by itself, are doomed to fade away into mere shadows, and only a kind of union of the two will preserve an independent reality. (Minkowski 1909, 75)

He proceeded to develop the now familiar geometry of what has come to be known as Minkowski space-time, the space-time structure of the special theory of relativity. Although it only got its name later, this is the theory introduced in the most famous paper of Einstein’s annus mirabilis (Einstein 1905). Minkowski showed that the transformations that relate the space-time coordinates of one reference frame in Minkowski space-time to the space-time coordinates of another reference frame in uniform motion with respect to the first are completely analogous to the transformations that relate the Cartesian coordinates with respect to one set of orthogonal axes in ordinary Euclidean space to the Cartesian coordinates with respect to another set of orthogonal axes rotated with respect to the first. In fact, such rotations in three-dimensional Euclidean space can be subsumed under Lorentz transformations in four-dimensional Minkowski space-time.

How did Minkowski build this magnificent arch? As he made clear in his lecture, he used a scaffold provided by recent work in electrodynamics. However, he also imagined a scenario in which the arch would have been built without a scaffold or, better perhaps, a different scaffold, provided by Newtonian mechanics rather than Maxwellian electrodynamics.27 The equations of Newtonian mechanics, he noted, “exhibit a two-fold invariance” (Minkowski 1909, 75). They do not change when we rotate the axes of our spatial coordinate system or when we set that spatial coordinate system in motion with a constant velocity. The latter invariance expresses the principle of relativity in Newtonian mechanics (Einstein extended this principle from mechanics to all of physics, especially electrodynamics). In Newtonian theory, Minkowski noted, these two operations

lead their lives entirely apart. Their utterly heterogeneous character may have discouraged any attempt to compound them. But it is precisely when they are compounded that the complete group, as a whole, gives us to think. (Minkowski 1909, 76)

“The thought might have struck some mathematician,” Minkowski mused, that maybe Newton’s theory, invariant under rotations and under transformations from one inertial frame to another (now called Galilean transformations to distinguish them from Lorentz transformations), ought to be replaced by a theory based on invariance under Lorentz transformations. In this way mathematicians might have anticipated special relativity. “Such a premonition,” Minkowski continued, “would have been an extraordinary triumph of pure mathematics.” Alas, this possibility had not occurred to any mathematician, including Minkowski himself, before physicists had recognized the importance of Lorentz transformations in electrodynamics. He consoled himself with the thought that “mathematics, though it now can display only staircase-wit,28 has the satisfaction of being wise after the event” (Minkowski 1909, 79).

In the opening sentence of his lecture, Minkowski had already identified the actual source of his insight into the importance of Lorentz transformations: “The views of space and time which I wish to lay before you have sprung from the soil of experimental physics, and therein lies their strength.” “They are radical,” he continued, before warning his audience that the old concepts of space and time were “doomed to fade away into mere shadows.” Minkowski thus emphasized the discontinuity in the transition from the old to the new views of space and time. At the same time, a certain continuity is suggested by his acknowledgment that these new views sprang “from the soil of experimental physics.”

At the end of the lecture, he returned to this point, locating the germ of his new views in the theoretical tools for cultivating the “soil of experimental physics” developed by Einstein and the Dutch physicist Hendrik Antoon Lorentz. In the course of his lecture, Minkowski had introduced what he called the “world postulate,” which basically says that we live in a four-dimensional world that can be described in infinitely many equivalent space-time coordinate systems all related to each other via Lorentz transformations. In the conclusion, he wrote:

The validity without exception of the world-postulate, I like to think, is the true nucleus of an electromagnetic image of the world, [a nucleus29] which, discovered by Lorentz and further revealed by Einstein, now lies open in the full light of day. (Minkowski 1909, 91)

Lorentz had been the first to show that Maxwell’s equations for electric and magnetic fields are invariant under Lorentz transformations (Janssen 2017). Initially, he could only do this to a good approximation and for a restricted class of charge distributions acting as sources of the fields (Lorentz 1895). Eventually, he could do it exactly and for arbitrary charge distributions (Lorentz 1904, 1916). Lorentz referred to this result as his “theorem of corresponding states.” A pair of corresponding states consists of two physical systems in the ether, the nineteenth-century medium serving as the carrier of light waves and electric and magnetic fields, one at rest, the other in uniform motion. The quantities pertaining to the system in motion are related to those pertaining to the system at rest by a Lorentz transformation, the name given to these transformations by the French mathematician Henri Poincaré.

Poincaré and Einstein recognized that quantities for the system in motion in Lorentz’s theorem of corresponding states are the space-time coordinates and the components of the electric and magnetic fields measured by an observer moving with the system. Before the advent of special relativity, Lorentz himself saw these quantities as nothing but auxiliary variables in terms of which the system in motion could be described in the same way as the corresponding system at rest in terms of the real quantities. Lorentz could show that the same measurement performed on two systems forming a pair of corresponding states would give the same result in a broad class of optical experiments. On the further assumption that, when set in motion with respect to the ether, the system at rest turns into the corresponding moving system, Lorentz could use his theorem to explain the negative results of many experiments aimed at detecting the earth’s motion through the ether, including the famous 1887 Michelson-Morley experiment.

This additional assumption, however, is not as innocuous as it looks. It boils down to the assumption that the laws governing the material objects with which light waves interact in optical experiments (mirrors, lenses, screens, etc.) are all invariant under Lorentz transformations, just as Maxwell’s equations governing the electric and magnetic fields making up the light waves themselves. As long as Lorentz invariance was restricted to the laws governing the fields, one could think of it as a special property of Maxwell’s equations. This, of course, was precisely how Lorentz had discovered Lorentz invariance in the first place. The negative results of many ether-drift experiments suggested that it was a much more general property, common to all laws of physics. One way to avoid this conclusion was to assume that all laws of physics could eventually be reduced to the laws of electrodynamics. This view was promoted by German physicists such as Wilhelm Wien and Max Abraham in the early years of the twentieth century (Janssen and Mecklenburg 2007). For several years this so-called electromagnetic view of nature was seen as the cutting edge in theoretical physics research. This is what Minkowski is referring to as the “electromagnetic image of the world” in the passage from his 1908 Cologne lecture quoted above. Minkowski had been an early supporter of the electromagnetic worldview but by 1908, as this same passage shows, he had distanced himself from it, supporting Einstein’s special theory of relativity instead (Janssen 2009, 39).

Although the title of his paper, “On the Electrodynamics of Moving Bodies,” suggests otherwise, Einstein, like Minkowski, recognized that Lorentz invariance has nothing to do with the particulars of electromagnetism but reflects a new space-time structure. In a letter of February 19, 1955, to Carl Seelig, one of his early biographers, he succinctly described the main novelty of special relativity as his “realization that the Lorentz transformation transcends its connection with Maxwell’s equations and has to do with the nature of space and time in general” (Janssen 2009, 41). Minkowski, as we saw, reached the same conclusion and rephrased it in geometrical terms, identifying his “world-postulate” as “the true nucleus” of the electromagnetic world picture.

The relation between arch and scaffold is pretty straightforward in this case. Lorentz invariance is the key structural element shared by arch and scaffold. In the scaffold, Lorentz invariance is tied to electromagnetism. We get from scaffold to arch in this case by recognizing, as Einstein and Minkowski did, that Lorentz invariance has nothing to do with electromagnetism per se but is a property of all laws governing systems in Minkowski space-time, the new space-time structure of special relativity. The Lorentz invariance of the laws reflects the symmetry of this space-time structure. It took a few years for physicists to distinguish between the Lorentz and the Einstein-Minkowski interpretation of Lorentz invariance (the two interpretations are empirically equivalent), but eventually, the latter prevailed. For physicists using Minkowski space-time today, the only thing left to remind them that it was scaffolded by Lorentz’s theorem of corresponding states is that the transformations between different perspectives on the arch are named after the man responsible for the scaffold.

Note how easy it was to tell this story backward in time, starting with Minkowski’s geometrical interpretation of Lorentz invariance and ending with Lorentz’s original interpretation of it in the context of electrodynamics. Also note that Minkowski’s contribution might alternatively be characterized in terms of Hilbert’s “build first and worry about the foundations later” metaphor (see section II). In that version of the story, Minkowski would be the mathematician who provided more secure foundations for the electromagnetic worldview of the physicist Abraham.

How Laue’s Relativistic Continuum Mechanics Was Scaffolded by Abraham’s Electromagnetic Mechanics30

Work in the history of special relativity typically focuses on how the theory changed our concepts of space and time. Special relativity, however, involved much more than the introduction of a new space-time structure and some minor adjustments to the laws of Newtonian mechanics to make them Lorentz invariant (such as the insertion of factors of in various equations, where ν is the relative velocity of two inertial frames, and c is the velocity of light). For one thing, the theory required the general relation between energy and mass (or inertia) expressed in its most famous equation, E = mc². It also required a “mechanics”—in the sense of a general framework for doing physics (cf. the term quantum mechanics)—of fields rather than particles.

In the years just prior to the arrival of special relativity, Abraham and others had developed such a mechanics for the special case of electromagnetic fields (Abraham 1903). It provided the foundation of the electromagnetic worldview, the attempt to reduce all of physics to electrodynamics. Insofar as the electromagnetic worldview is covered at all in histories of special relativity, it is typically presented, implicitly or explicitly, as a research program that was briefly considered cutting edge at the beginning of the twentieth century but was then vanquished by special relativity.31

However, as Einstein, for one, clearly recognized, it is more accurate to say that it was co-opted by special relativity. Within a few years of the introduction of special relativity, the electromagnetic mechanics of Abraham had morphed into the relativistic continuum mechanics presented in the first textbook on relativity (Laue 1911). Max Laue basically obtained his relativistic mechanics for fields by taking Abraham’s electromagnetic mechanics, rewriting it in terms of the four-dimensional formalism developed by Minkowski and Sommerfeld, and stripping it of its electromagnetic particulars.

The arch-and-scaffold metaphor captures this development in a natural way. It underscores the importance of the formulation of relativistic continuum mechanics in the development of special relativity by making it—to use the terms introduced in Figure 4.1—the keystone of the arch for which Minkowski had provided the springers.

The backward-looking perspective is especially important in this case. If the developments are covered forward in time, it is hard to bring out those features of the electromagnetic program that proved most relevant for the formulation of relativistic continuum mechanics without the account becoming blatantly Whiggish. Put differently, and as illustrated by the historiographically impeccable coverage of this episode by Richard Staley (2008, chapters 6–7), Whiggishness can be avoided only at the price of obscuring what in hindsight were the most salient elements of the electromagnetic program and how they were incorporated into special relativity.

The clearest way to bring out these elements is to rewrite some of the equations of the electromagnetic program in terms of the formalism of Minkowski, Sommerfeld, and Laue. The proponents of the electromagnetic program wrote their equations in terms of vectors, with the usual three (spatial) components and tensors one can think of as three-by-three matrices, and they handled spatial and temporal derivatives separately. The relativists grouped quantities into vectors with four space-time components and tensors one can think of as four-by-four matrices, and they put spatial and temporal derivatives on equal footing.

It will be instructive to look at this in a little more detail. Abraham replaced the energy and momentum of particles of Newtonian mechanics, with definite positions in space, by the energy density and momentum density of electromagnetic fields, spread out all over space. The x-, y-, and z-components of the electromagnetic momentum density he introduced are proportional to the electromagnetic energy flow density in the x-, y-, and z-direction. The electromagnetic energy flow density in turn is given by the Poynting vector, the cross product of the electric and the magnetic field and the main claim to fame of the physicist we already encountered in section III.

Central to Abraham’s efforts to reduce mechanics to electrodynamics was his attempt to eliminate the Newtonian concept of mass by replacing the inertial force on a massive particle by the force exerted on a massless charge distribution by the electromagnetic field generated by that charge distribution itself. The interaction of massless charges with these self-fields thus mimics inertial mass (those with limited tolerance for equations can skip ahead to the quotation following Equation (5)).

Abraham showed that the components (with i = 1, 2, 3 labeling x-, y-, and z-components) of the density of the electromagnetic force exerted by a charge distribution’s self-field can be written as minus the time derivative of the components of its electromagnetic momentum density and the divergence (a sum of spatial derivatives) of its stress-energy density, given by the so-called Maxwell stress tensor. This tensor can be thought of as a three-by-three matrix. Its nine components represent the flow of the x-, y-, and z-components of the electromagnetic momentum density in the x-, y-, and z-direction. Like the electromagnetic momentum density, the components of the Maxwell stress tensor are functions of the components of the electric and magnetic fields.

In modern notation, can be written as

(1)

where is the i-component of the electromagnetic momentum of the charge distribution’s self-field, and is the ij-component of its Maxwell stress tensor.32

The energy density, the three components of the momentum density, the three components of the energy flow density, and the nine components of the momentum flow density make for a total of sixteen components. In special relativity, they are combined (with an extra factor of c here and there) to give the sixteen components of the energy-momentum tensor Tμν (with μ, ν = 0, 1, 2, 3 corresponding to one time and three spatial components).33 This tensor can be thought of as a four-by-four matrix, with the first index labeling the rows and the second index labeling the columns. Its components are:

(2)

The energy-momentum tensor is symmetric in its indices—that is, Tμν = Tνμ In other words, the matrix representing Tμν stays the same if we switch rows and columns. Focusing on the first row and the first column, we have Tⁱ⁰ = T⁰ⁱ, from which it follows that

As was first realized by Planck (1906), this is an elegant way of expressing E = mc² in the four-dimensional formalism.

Equation (1) for the components of the electromagnetic force density, the rate of change of the momentum density of the self-field, can be combined with a similar equation for the rate of change of the energy density of the self-field. This combination can be written compactly in terms of the components of the energy-momentum tensor, for the charge distribution’s self-field:

(3)

where the four components of the electromagnetic four-force density are and (i = 1, 2, 3) and where xμ ≡ (ct, x, y, z). The final expression in Equation (3) is called the four-divergence of the energy-momentum tensor

In relativistic continuum mechanics, this equation is generalized from the electromagnetic field to arbitrary spatially extended systems. The subscript “EM” in Equation (3) can thus be dropped. Using the so-called Einstein summation convention, which says that any index occurring twice in the same expression (once “upstairs” and once “downstairs”) is summed over, we can also drop the summation sign. Finally, we introduce the abbreviation ∂μ ≡ ∂/∂xμ and arrive at:

(4)

In general there will be an external four-force density with components acting on the system as well as the four-force density with components that the system exerts on itself. The basic equation of motion for the system says that the sum of these force densities vanishes everywhere—that is, Using Equation (4) for we can write this equation as

(5)

This is the fundamental law of relativistic continuum mechanics.

As Einstein wrote in an unpublished review article on special relativity in 1912:

The general validity of the conservation laws [of energy and momentum] and the law of the inertia of energy [E = mc²] suggests that [the energy-momentum tensor (2) and the force equation (5)] are to be ascribed a general significance, even though they were obtained in a very special case [i.e., electrodynamics]. We owe this generalization, which is the most important new advance in the theory of relativity, to the investigations of Minkowski, Abraham, Planck, and Laue. (Einstein 1987–2018, vol. 4:92; cf. Janssen and Mecklenburg 2007, 110)

Note that Abraham, the undisputed leader of the electromagnetic program and a staunch opponent of special relativity, is mentioned here in the same breath as Minkowski, Planck, and Laue. Abraham richly deserved to be mentioned alongside this trio of enthusiastic supporters of special relativity. His electromagnetic mechanics provided the scaffold on which Laue built the arch of relativistic continuum mechanics.

The relation between scaffold and arch in this case is the same as in my first example. The scaffold exhibits the structure of the arch for the special case of electromagnetism. Once again, the arch is thus obtained by stripping the scaffold of its electromagnetic particulars. The comparison of the equations of the electromagnetic view of nature and relativistic continuum mechanics also highlights a different aspect of the relation between arch and scaffold in both these examples. The step from scaffold to arch in both cases also involved grouping various quantities defined in three-dimensional space (scalars, vectors, and tensors) into quantities defined in four-dimensional space-time.

Another passage from Einstein’s 1912 review article provides the natural starting point for telling the story about the transition from Abraham’s electromagnetic mechanics to Laue’s relativistic continuum mechanics backward in time. Einstein gave the following concise characterization of how one applies the latter:

To every kind of material process we want to study, we have to assign a symmetric tensor (Tμν), whose components have the physical meaning given in [Equation (2)]. [Equation (5)] must always be satisfied. The problem to be solved always consists in finding out how (Tμν) is to be formed from the variables characterizing the processes under consideration. If several processes take place in the same region that can be isolated in the energy-momentum balance, we have to assign to each individual process its own stress-energy tensor (), etc., and set (Tμν) equal to the sum of these individual tensors. (Einstein 1987–2018, vol. 4:92)

The question to which the arch-and-scaffold narrative then provides the answer is: How did physicists go from Newtonian particle mechanics to relativistic continuum mechanics as a general framework for doing physics. In other words, how did they go from writing down the forces acting on some collection of bodies and using Newton’s second law, F = ma, to solve for the motion of these bodies to writing down the energy-momentum tensors for various processes occurring in some region of space-time and setting the four-divergence of their sum, equal to the external four-force density,

One final question that needs to be answered about this episode is why the new relativistic mechanics, despite being identified by Einstein as “the most important new advance in the theory of relativity,” did not catch on in the physics community of the early 1910s. The short answer: because of quantum theory. Only two years after Laue published his relativity textbook, in which relativistic continuum mechanics takes center stage, Niels Bohr (1913) proposed his model of the hydrogen atom based on nonrelativistic Newtonian particle mechanics. This model was further developed in the old quantum theory of Bohr and Sommerfeld (Kragh 2012; Eckert 2014). The old quantum theory gave Newtonian mechanics, especially Newtonian celestial mechanics, a new lease on life. Sommerfeld used relativistic particle mechanics to explain the fine structure of spectral lines, but the old quantum theory had no use for relativistic continuum mechanics. The latter only played a role in the development of general relativity, the topic of my next example.

How the Field Equations of General Relativity Were Scaffolded by the Field Equations of the Earlier Entwurf Theory34

In 1907, only two years after he published the special theory of relativity, Einstein, still a patent clerk in Berne, started thinking about a generalization of the theory that would incorporate gravity. By 1911, when he was appointed full professor in Prague, he had arrived at the basic idea of his general theory of relativity. Contrary to what its name suggests, this theory does not extend the principle of relativity from uniform to nonuniform motion (Janssen 2014), but it does weave gravity into the fabric of space-time. In general relativity, gravity is represented by space-time curvature. Free-falling bodies (i.e., particles subject to no other forces than gravity) will follow the straightest possible paths in these curved space-times. Such paths are called geodesics and satisfy the geodesic equation.35

This much was already becoming clear to Einstein when he returned from Prague to Zurich in 1912 and began working in earnest on his new theory of gravity with the help of his new colleague at the Federal Institute of Technology (ETH), Marcel Grossmann. Grossmann introduced Einstein to the elegant mathematics of Bernard Riemann, Elwin Bruno Christoffel, Gregorio Ricci-Curbastro, and others needed for the formulation of the kind of theory Einstein was after. The two of them had been classmates at what had then still been called the Federal Polytechnic back in the late 1890s. Grossmann had become professor of mathematics at their alma mater in 1907. Their collaboration, recorded in Einstein’s famous Zurich notebook (Renn 2007, vols. 1–2), resulted in a joint paper published in the spring of 1913 (Einstein and Grossmann 1913). Its title modestly announced an “outline” (Entwurf) of a new theory of gravity and a generalized theory of relativity. Historians refer to it as the Entwurf theory.

The Entwurf theory already put in place most of the formalism of the general theory of relativity, which Einstein completed two and a half years later, in November 1915, in four short communications to the Prussian Academy in Berlin (Einstein 1915a, 1915b, 1915c, 1915d). In March 1914, he had left Zurich to take up a prestigious appointment in the German capital. In his papers of November 1915, Einstein basically changed only one important element of the Entwurf theory: its field equations.

The field equations, as the reader may recall from section III, determine how matter curves space-time. (It is customary to use the plural equations even though they can be written as one equation because this one equation has several components.) It was clear that matter had to be represented by its energy-momentum tensor (cf. the preceding case study). This tensor appears on the right-hand side of the field equations. The difference between the field equations of 1913 and 1915 was the left-hand side. In his first November paper, Einstein claimed that he and Grossmann had already considered the new candidate for the left-hand side three years earlier. The Zurich notebook confirms this. The notebook shows how mathematical considerations, supplied by Grossmann, had led Einstein to this elegant candidate and how he had abandoned it because it looked as if the resulting field equations did not reduce to the equations of Newtonian theory in the appropriate limit and were incompatible with energy-momentum conservation. Einstein thereupon switched from a mathematical to a physical strategy. Exploiting the analogy with the Maxwell-Lorentz theory for the electromagnetic field (rewritten in the four-dimensional formalism of Minkowski, Sommerfeld, and Laue), he constructed field equations for the gravitational field guaranteed to satisfy energy-momentum conservation and to have the correct Newtonian limit. These are the field equations published in the Entwurf paper.

In the introduction of his first paper of November 1915, Einstein made it sound as if he had suddenly turned his back on the physical strategy that had led him to the Entwurf field equations and gone back to the mathematical strategy that had originally led him to the field equations with which he now proposed to replace them. Subsequent sections of the paper make it clear that this was, at best, an exaggeration. Einstein relied heavily on lessons learned pursuing the physical strategy over the preceding two and a half years to show that these resurrected field equations passed muster on the counts of energy-momentum conservation and the Newtonian limit on which they had failed earlier. To put it in terms of the arch-and-scaffold metaphor: Einstein may already have envisioned the arch in 1913, but the confidence to put weight on it only came in November 1915.36

Closer examination of both the first November paper and Einstein’s correspondence at the time makes it doubtful that there was an eleventh-hour return to the mathematical strategy and strongly suggests that, instead, it was his dogged pursuit of the physical strategy that led Einstein back to the field equations to which the mathematical strategy had already led him in the Zurich notebook. As Jürgen Renn and I have argued in detail, Einstein used the Entwurf field equations as a scaffold to construct the field equations with which he replaced them in his first paper of November 1915 (Janssen and Renn 2015, 2019).

By late 1914, Einstein had perfected the analogy between the Maxwell-Lorentz theory for the electromagnetic field and the Entwurf theory for the gravitational field. He convinced himself that the formalism he had developed relying on this analogy uniquely determined the field equations and that these were the Entwurf field equations. Satisfied that his arch was now complete, he published a lengthy review article on his theory. The title no longer talks about an “outline” of a “generalized” theory of relativity but promises nothing less than the “formal foundation of the general theory of relativity” (Einstein 1914).

In October 1915, Einstein discovered, to his dismay, that his uniqueness argument for the Entwurf field equations was fallacious. Rather than abandoning his general formalism—or tearing down his house, to use the metaphor of Kuhn’s Guggenheim application quoted in section II—Einstein merely replaced the definition of the gravitational field in the Entwurf theory by what seemed to be the only other physically plausible candidate, the so-called Christoffel symbols. Inserting this new definition into his general formalism, he ended up with the same equations that he had rejected in the Zurich notebook, thereby reestablishing the connection between his theory and the elegant mathematics that, as Einstein (1915a, 778) noted in his first November 1915 paper, he and Grossmann had abandoned “with a heavy heart” in 1913. The general formalism developed for the Entwurf theory provided a number of relations, the counterparts of similar relations for the electromagnetic field, that the gravitational field had to satisfy to be acceptable from a physics point of view. These relations continued to hold when the old definition of the gravitational field was replaced by the new one. This, then, is how Einstein got from scaffold to arch. By changing the definition of the gravitational field, he swapped out one building block for another, confident that the structure he had erected with the old building blocks would remain stable upon this substitution.37

Einstein himself identified this as the crucial step in the transition from the Entwurf field equations to the field equations of his first November paper. In the paper, he called the new definition of the gravitational field “the key to the solution.” In the letter to Sommerfeld from which I already quoted in note 36, he called the old definition “a fateful prejudice” (Janssen and Renn 2007, 859, 875–79).

Worried that Hilbert might beat him to the punch, Einstein rushed his new field equations into print. Over the next three weeks, he continued to tweak them. Throughout this period Einstein was laboring under the misconception that the extent to which his theory generalized the principle of relativity from uniform to nonuniform motion is directly related to the degree of covariance of its equations—that is, to the size of the class of coordinate transformations under which the equations retain their form. The covariance of the field equations of his first November paper was much broader than that of the Entwurf field equations, but they were still not generally covariant—that is, they do not retain their form under arbitrary coordinate transformations. Einstein quickly realized, however, that with relatively minor modifications these new field equations could be turned into generally covariant field equations that just happened to be written in a special form in which their general covariance is not immediately apparent.

In his second November paper, he proposed one such modification, only to replace it with another, more satisfactory, one in the fourth (Einstein 1915b, 1915d). The latter modification was to add a term with the so-called trace of the energy-momentum tensor (the sum of the terms on the diagonal in Equation (2)) on the right-hand side of the field equations. This trace term is necessary to ensure that the quantity representing the energy-momentum density of the gravitational field enters the field equations in the exact same way as the energy-momentum tensor for matter. Since this was a requirement that had been one of Einstein’s guiding principles, he was now confident that no further corrections would be needed. The amended equations of this fourth and final communication to the Berlin Academy of November 1915 are the Einstein field equations used to this day.38 The trace term formed the keystone of what is widely admired as the most marvelous arch Einstein left us.

Einstein still had to write his field equations in a form in which they only retain their form under a restricted class of coordinate transformations, as this was the only way in which he could connect them to the general formalism for the Entwurf theory and show that they were compatible with energy-momentum conservation. As a result, the arch unveiled in Einstein’s papers of November 1915 still showed clear traces of the scaffold used to build it. The same is true for the section on the field equations in the review article that Einstein (1916a) published in May the following year to replace the premature review article of late 1914. It was only in a short paper published in November that year that Einstein (1916b) finally removed all traces of the Entwurf scaffold.

A natural starting point for telling this story backward in time is to begin with Einstein’s later recollections of how he found the field equations of general relativity. This is the approach Jürgen Renn and I took in a talk we have given in various places based on our article in Physics Today (Janssen and Renn 2015).39 The older Einstein routinely claimed that he had found the Einstein field equations following the mathematical strategy. Some commentators, notably John Norton (2000), have taken him at his word. Renn and I concur with the conclusion of Jeroen van Dongen’s (2010) study of Einstein’s unified field theory program that these statements should be seen, first and foremost, as propaganda for this program. In his ultimately fruitless pursuit of a classical field theory unifying general relativity and electromagnetism, Einstein relied on a purely mathematical strategy. It served his purposes to suggest that this approach could boast of at least one spectacular success, the discovery of the Einstein field equations.

An arch-and-scaffold narrative working backward from these later pronouncements by Einstein reveals that, while supported by several passages in his writings and correspondence from the gestation period of general relativity, they do not square with the full range of textual evidence available. The arch-and-scaffold metaphor can be used to put the physical strategy, suppressed in Einstein’s later recollections, in sharp relief, which makes it easier to compare competing accounts of how Einstein found his field equations in the fall of 1915, the account of Janssen and Renn (2007), in which Einstein stuck to the physical strategy, and the classic account of Norton (1984), in which he switched to the mathematical strategy.40

The clarification of the difference between the two strategies Einstein used in his search for the field equations of general relativity may shed light on at least two other issues in modern history and philosophy of physics. The first concerns the interpretation of general relativity (Lehmkuhl 2014). Should one think of it in analogy with electrodynamics as the theory of a particular field, as the physical strategy suggests, or should one think of it as a theory about geometry, as the mathematical strategy suggests?41 The second issue is about methodology in physics. On the face of it, Einstein’s own later account of how he found the field equations of general relativity in 1913–1915 is strong testimony in support of a purely mathematical approach to theory construction. This approach remains popular to this day in certain quarters of the physics community. An account of this episode that emphasizes the importance of the physical strategy, conveyed concisely with the help of the arch-and-scaffold metaphor, can likewise serve as powerful countertestimony.

How the Matrix Mechanics of Heisenberg, Born, and Jordan Was Scaffolded by the Kramers Dispersion Formula42

The paper known as the Umdeutung (Reinterpretation) paper, with which Werner Heisenberg (1925) laid the foundation for matrix mechanics, draws on an earlier paper he wrote as the junior coauthor of Bohr’s right-hand man in Copenhagen at the time, the Dutch physicist Hans Kramers. This paper by Kramers and Heisenberg (1925) provides a detailed derivation and further exploration of a new quantum formula for optical dispersion that Kramers (1924a, 1924b) had proposed in two short notes in Nature the year before. Max Dresden (1987, 275), Kramers’s biographer, goes so far as calling this paper “the direct, immediate, and exclusive precursor to the Heisenberg paper on matrix mechanics.” Recent work by Alex Blum, Martin Jähnert, Christoph Lehner, and Jürgen Renn (2017) at the Max Planck Institute for the History of Science in Berlin suggests that the Umdeutung paper owes as much, if not more, to work on intensities in multiplet spectra.43 The work on dispersion theory, however, remains an important strand in the genealogy of the Umdeutung paper. This strand is nicely captured by the arch-and-scaffold metaphor. Blum et al. (2017, 4n3) find this to be true for the multiplet strand as well.

Optical dispersion is the phenomenon, familiar from rainbows and prisms, that the refraction of light in an optical medium depends on its color. Although it already occupied the minds of Descartes and Newton, it was not until two centuries later that a halfway satisfactory theory of the phenomenon was formulated.44 Particularly challenging was a puzzling feature discovered by early pioneers of photography in the 1840s. Normally, the index of refraction increases with the frequency of the refracted light. Blue light is refracted more strongly than red light. However, in narrow frequency bands around the absorption frequencies of an optical medium, the index of refraction decreases with increasing frequency in some materials. This is called anomalous dispersion. In the 1870s, Wolfgang Sellmeier and others introduced a new generation of dispersion theories that could account for this phenomenon. The characteristic feature of this new class of theories is that optical media contain small oscillators with resonance frequencies at the absorption frequencies of the material. These theories correctly predict that dispersion becomes anomalous in the vicinity of these resonance frequencies. In the 1890s, Hermann von Helmholtz, Lorentz, and Paul Drude reformulated these originally purely mechanical theories in terms of electromagnetic waves interacting with electrically charged oscillators, soon to be identified as electrons. Such harmonically bound electrons were sometimes called dispersion electrons.

These classical dispersion theories were incompatible with Bohr’s (1913) atomic model. In this model, electrons orbit the nucleus the way planets orbit the sun, except that in the atom only a discrete set of orbits are allowed, labeled by integer-valued quantum numbers. A straightforward adaptation of the classical dispersion formula to Bohr’s new atomic model would have been to replace the oscillation frequencies of the harmonically bound dispersion electrons in the former by the orbital frequencies of the planetary electrons in the latter. This was not an option. The problem is that, in general, these orbital frequencies differ sharply from the atom’s absorption frequencies. Light is absorbed or emitted in a Bohr atom when an electron jumps from one orbit to another. The frequency of the absorbed or emitted radiation is determined not by the orbital frequencies of either of these orbits but by the energy difference between them. Only in the limit in which the quantum numbers labeling the orbits get very large do the radiation frequencies coincide with the orbital frequencies. Simply replacing oscillation frequencies by orbital frequencies would thus lead to a theory predicting anomalous dispersion at the wrong frequencies.

While this posed a serious problem for Bohr’s atomic model and the old quantum theory that grew out of it, the classical dispersion theory also faced a serious problem, which could actually be solved with some of the resources provided by the old quantum theory. Experimentalists had found puzzling values for an important set of free parameters of the classical theory, the so-called oscillator strengths. In the classical theory, the oscillator strength for a particular resonance frequency is the number of dispersion electrons per atom with that resonance frequency. Intuitively, one would expect these numbers to be in the single digits—a few dispersion electrons with the same resonance frequency per atom—but the values giving the best fit with the data tended to be much lower. It was not uncommon to find values as low as one dispersion electron with a particular resonance frequency per hundreds or even tens of thousands of atoms.

The German experimentalist Rudolf Ladenburg (1921) reinterpreted these parameters in a way that such low values were only to be expected. The oscillator strength does not, Ladenburg suggested, represent the number of electrons with a particular resonance frequency but the number of electron jumps between two orbits associated with the absorption of radiation at that frequency. Ladenburg set the number of jumps equal to the product of the occupation number of the initial orbit (the fraction of the total number of electrons in that orbit) and the probability that an electron in that initial orbit would jump to the final orbit. For these probabilities he used the probability coefficients for transitions between different quantum states introduced by Einstein (1917a). Replacing numbers of electrons by products of occupation numbers and transition probabilities, Ladenburg turned the classical dispersion formula into a new quantum formula.

The formula still had two limitations. First, it was restricted to situations in which electrons would jump to and from their ground state, the orbit of lowest energy. Second, Ladenburg could still not explain why dispersion is anomalous at the absorption frequencies. He just retained this feature of the classical formula, as it was clearly borne out by the experimental data. In a follow-up paper, Ladenburg and Fritz Reiche, a theoretical physicist, introduced the notion of substitute oscillators (Ersatzoszillatoren) operating between two orbits and with resonance frequencies equal to the absorption frequencies associated with transitions between them (Ladenburg and Reiche 1923). If one thought of these substitute oscillators as the conduits of dispersion, one at least had some way of understanding why dispersion is anomalous at these transition frequencies.

This is where matters stood when Kramers entered the fray. Most likely at the instigation of Bohr (whose doctoral adviser at the University of Copenhagen, Christian Christiansen, had done important work on optical dispersion), Kramers tried to derive a dispersion formula in the old quantum theory modeled on the one given by Ladenburg. The central tool he used for this derivation was Bohr’s correspondence principle. In the hands of Kramers (1924a, 1924b), Born (1924), and the American theoretical physicist John H. Van Vleck (1924a, 1924b), this principle turned into a powerful instrument for generating quantum formulae designed to merge with their classical counterparts in the limit of high quantum numbers.45

Using canonical perturbation theory in special momentum and position variables known as action-angle variables, a technique originating in celestial mechanics, Kramers first derived a formula for dispersion in classical mechanics. He then made three substitutions to turn this classical formula into a quantum formula. First, he expanded the orbital motion into a Fourier series and replaced the squares of the amplitudes of the various Fourier components by the Einstein coefficients for transition probabilities. Second, he replaced orbital frequencies by radiation frequencies corresponding to transitions between orbits. Third, he replaced derivatives with respect to action variables by difference quotients. The basic quantization conditions of the old quantum theory, which select the allowed electron orbits in an atom, set such action variables equal to an integral multiple of Planck’s constant h. In the limit of high quantum numbers N, where the allowed orbits are getting closer and closer together, one can thus approximate a derivative of a quantity with respect to an action variable by subtracting that quantity’s value at the N^th orbit from its value at the (N + 1)^th orbit and dividing the result by h.

With these three substitutions, the classical dispersion formula Kramers had derived turned into a quantum formula. Because of the third substitution, this formula is the difference of two terms. Both have the same structure as Ladenburg’s formula. As long as electrons only jump to and from their ground state, Kramers’s formula reduces to Ladenburg’s. Kramers’s formula, however, applies to all possible transitions between orbits. Its construction guarantees that it merges with the well-tested classical formula in the limit of high quantum numbers. It still required a leap of faith that the formula would continue to hold all the way down to the smallest quantum numbers, but its agreement with Ladenburg’s formula for the ground state was reassuring on that score. In hindsight, Kramers’s faith in his formula was well placed. It carries over completely intact to modern quantum mechanics and has been fully confirmed experimentally.

The Kramers dispersion formula was incorporated into a short-lived but influential quantum theory of radiation proposed by Bohr, Kramers, and Slater (1924) and known as the BKS theory. John C. Slater was an American postdoc visiting Copenhagen at the time. The substitute oscillators introduced by Ladenburg and Reiche (1923) return in the BKS theory under the name virtual oscillators. The BKS theory thus introduces a dual representation of atoms. To the set of orbits of Bohr’s original theory, the BKS theory adds—to use a term introduced by another early quantum theorist, Alfred Landé (1926, 456)—an orchestra of virtual oscillators associated with every possible transition between those orbits. All information about observable quantities—that is, frequencies and intensities of spectral lines, is contained in the latter. The Kramers dispersion formula nicely illustrates this: it only contains quantities referring to transitions between orbits and makes no reference whatsoever anymore to individual orbits.

After the examples given in the first three case studies in this section, I trust that the reader will have no trouble seeing in the sequence of dispersion theories outlined above (Sellmeier, Helmholtz-Lorentz-Drude, Ladenburg-Reiche, Kramers) how the later theory was scaffolded by the earlier one. But how did the Kramers dispersion formula (partly) scaffold Heisenberg’s Umdeutung paper? As sketched above, Kramers derived his quantum dispersion formula in two steps. First, he derived a formula in classical mechanics. Then he used the correspondence principle to translate the result into a quantum formula. The fundamental idea of Umdeutung is to use the correspondence principle to translate the input rather than the output of such classical derivations and do the entire derivation in terms of the new quantum language. This strategy is not limited to the derivation of a formula for dispersion. Heisenberg realized that it could serve as a new framework for all of physics. A little more concretely, the basic idea is to take positions and momenta, the fundamental variables of classical mechanics, in terms of which Kramers had derived his classical dispersion formula, translate them according to “the scheme of the dispersion theory,” as Heisenberg himself put it in an interview for the Archive for the History of Quantum Physics (AHQP) in the early 1960s (cf. note 8),46 into quantum variables and calculate with those new variables on the assumption that they satisfy the same laws as their classical counterparts. Hence, the term Umdeutung: rather than repealing the laws of classical mechanics, Heisenberg sought to reinterpret them.47

In Heisenberg’s Umdeutung or reinterpretation scheme, quantities associated with a single orbit get replaced by quantities associated with a transition between two orbits. Electron orbits are eliminated altogether. Heisenberg formulated his theory entirely in terms of transition quantities without answering the obvious question “transitions between what?” These transition quantities have two indices, referring to an initial and a final state, but Heisenberg had nothing whatsoever to say about the nature of those states. Multiplication of his two-index objects, Heisenberg found, is noncommutative: A × B ≠ B × A. In their elaboration of Heisenberg’s Umdeutung paper, Born and his former student Pascual Jordan identified these two-index objects as matrices, their rows and columns labeled by Heisenberg’s two indices (Born and Jordan 1925). This showed that Heisenberg’s strange noncommutative multiplication rule is nothing but the standard multiplication rule for matrices.48

The relation between arch and scaffold in this example is a combination of those encountered in the relativity examples. First, the way in which Heisenberg, with help from Born and Jordan, generalized Kramers’s theory for a specific phenomenon (dispersion) to a new framework for all of physics (matrix mechanics) is reminiscent of the way in which Laue generalized Abraham’s electromagnetic mechanics to a new framework for all of physics (relativistic continuum mechanics). Second, the way in which Heisenberg replaced classical quantities by two-index objects soon to be recognized as matrices while keeping the structure of classical mechanics intact is reminiscent of the way in which Einstein replaced the definition of the gravitational field in the Entwurf theory by a new definition while keeping the formalism developed for the Entwurf field equations intact.

The example also illustrates an element of the arch-and-scaffold metaphor identified in Figure 4.1 that we did not encounter in the relativity examples. Kramers built his quantum formula and Heisenberg built his quantum theory on the foundation—the tas-de-charge in terms of Figure 4.1—of classical mechanics. The instrument they used to erect their quantum constructions, the windlass in terms of Figure 4.1, was the correspondence principle, in the specific guise of the three substitutions listed above.

As part of his Umdeutung project, Heisenberg also had to bring the quantization conditions of the old quantum theory, formulated in terms of individual orbits, into his new framework. Translating these conditions according to the “scheme of the dispersion theory” (as he put it in his AHQP interview), Heisenberg arrived at a corollary of the Kramers dispersion formula found independently by Werner Kuhn (1925) and Willy Thomas (1925; see also Reiche and Thomas 1925). This Thomas-Kuhn sum rule served as the quantization condition in the Umdeutung paper. It thus has nothing to do with Thomas S. Kuhn, who is the only one I know who refers to it as the Kuhn-Thomas sum rule (Duncan and Janssen 2007, 594). Born and Jordan (1925) showed that this sum rule can be rewritten as

(6)

(j = 1, 2, 3), where and are the components of position and momentum (with the “hats” to indicate that these quantities are not numbers but matrices), i is the imaginary unit, and is Planck’s constant divided by 2π (Duncan and Janssen 2007, 659–60). Equation (6) gives the diagonal elements (j = k) of

(7)

(j, k = 1, 2, 3; δ_jk = 1 for j = k, and δ_jk = 0 for j ≠ k), the familiar commutation relations for position and momentum that serve as the basic quantization conditions in matrix mechanics.

As in the case of the November 1915 papers in which Einstein first presented the Einstein field equations, the Umdeutung paper, the harbinger of matrix mechanics, still showed clear traces of the scaffold on which it was built. Heisenberg’s two-index objects satisfying a peculiar noncommutative multiplication rule are still somewhere in between the transition amplitudes and transition frequencies of the Kramers dispersion formula and the matrices introduced by Born and Jordan. The Thomas-Kuhn sum rule, the quantization condition of the Umdeutung paper, comes straight out of dispersion theory.

Most importantly, perhaps, the notion of a virtual oscillator that Bohr, Kramers, and Slater (1924) had taken over from Ladenburg and Reiche (1923) served as a placeholder until a more satisfactory way had been found to represent the states that the systems studied in matrix mechanics were transitioning between. New and better representations would soon be provided, be it Schrödinger’s wave functions or von Neumann’s vectors in Hilbert space (see the next case study). Virtual oscillators could now be identified either as Fourier components of a wave function (Duncan and Janssen 2007, 617) or as matrix elements of position (Casimir 1973, 492). However, in their follow-up to the Umdeutung paper, written before these contributions by Schrödinger and von Neumann, Born and Jordan (1925, 884) still talked about “substitute oscillators,” Ladenburg and Reiche’s original term for virtual oscillators. Although today it is used in connection with the BKS theory, Landé (1926) actually introduced his “orchestra of virtual oscillators” to describe matrix mechanics. At least one popular book continued to use closely related imagery—a “band” (Kapelle) of “assistant musicians” (Hilfsmusiker)—to explain matrix mechanics to a lay audience long after the concept of a quantum state had been incorporated into it (Zimmer 1934, 161–62; quoted in Duncan and Janssen 2007, 616).

In Duncan and Janssen (2007), we already indicated how to tell this story backward in time. Our starting point was exactly the kind of wonder one experiences upon first seeing an improbable architectural structure. One of those left wondering how Heisenberg built his arch is particle physicist and Nobel laureate Steven Weinberg. Talking about the Umdeutung paper in Dreams of a Final Theory, he wrote:

If the reader is mystified at what Heisenberg was doing, he or she is not alone. I have tried several times to read the paper that Heisenberg wrote on returning from Helgoland [where he had gone to seek relief from an attack of hay fever], and, although I think I understand quantum mechanics, I have never understood Heisenberg’s motivations for the mathematical steps in his paper. (Weinberg [1992] 1994, 67; cf. Duncan and Janssen 2007, 559)

This same quote is used to motivate at least two other studies of the Umdeutung paper (Aitchison, MacManus, and Snyder 2004; Blum et al. 2017). This underscores the point I made in section III that the arch-and-scaffold metaphor, far from compounding the historiographical sin of Whiggishness, can be seen as an attempt to legitimize a common and benign form of it, even if one has to remain vigilant (see note 43).

How von Neumann’s Hilbert-Space Formalism Was Scaffolded by the Dirac-Jordan Statistical Transformation Theory49

Quantum theory developed rapidly in the years 1925–1927. By the middle of 1926, four different versions were in circulation: the Göttingen matrix mechanics of Heisenberg, Born, and Jordan; the wave mechanics of Austria’s Erwin Schrödinger; the q-number theory of Cambridge’s Paul Dirac; and, though more problematic and less influential than the other three, the operator calculus of Born and the American mathematician Norbert Wiener. Schrödinger (1926) had shown that wave mechanics and matrix mechanics always give the same empirical predictions. Born (1926a, 1926b) had shown that Schrödinger’s wave functions call for a probabilistic interpretation. A general formalism tying the four different versions together, however, had yet to be found. Then, in late 1926, independently of one another, Jordan (1927a) and Dirac (1927) submitted papers proposing essentially the same overarching formalism along with its probabilistic interpretation. It became known as the Dirac-Jordan statistical transformation theory, or transformation theory for short. I focus here on Jordan’s formulation, though I will borrow some of Dirac’s vastly superior notation. For a comparison of Jordan’s approach to Dirac’s—widely disseminated through his influential textbook on quantum mechanics (Dirac 1930)—see Duncan and Janssen (2013, 185–90).

Statistical transformation theory can be seen as an arch built on a scaffold constructed out of the four related yet different theories it unified. The arch that Heisenberg (1925) had built on the scaffold of the Kramers dispersion formula (see the preceding case study) thus became part of the scaffold on which Jordan (1927a, 1927b) and Dirac (1927) erected their arch. Within a few months, the Hungarian polymath John von Neumann (1927a, 1927b, 1927c) would use Jordan and Dirac’s arch as a scaffold to build an arch of his own, his Hilbert-space formalism for quantum mechanics, although one could also say, in the spirit of Hilbert and Young (see section II), that von Neumann produced a scaffold to prop up Jordan’s arch. Like Dirac’s paper, von Neumann’s papers were later expanded into a book (von Neumann 1932).

I will not even attempt to characterize the relation between arch and scaffold in the transition from the four early versions of quantum theory to transformation theory, other than to say that it is considerably more complicated than in the examples analyzed so far. It will be difficult enough to decide which formalism played the role of the arch and which that of the scaffold in the transition from Jordan’s version of transformation theory to von Neumann’s Hilbert-space formalism. Either way, it is a challenge to precisely characterize the relation between these two formalisms.

As mentioned in the introduction, another important difference between this case study and the other four in this section is that in the transition from matrix and wave mechanics to transformation theory as well as in the subsequent transition from transformation theory to Hilbert space, the scaffold was not dismantled once the arch had been built. Elements of all four of these formalisms continue to be used to this day. While many philosophers of quantum mechanics use vectors in Hilbert space, quantum chemists for the most part get by with Schrödinger wave functions. This is true both in research and in teaching. In introductory physics courses, quantum mechanics is typically presented in the guise of wave mechanics, while for some problems techniques from matrix mechanics are used (e.g., raising and lowering operators to find the energy spectrum of a simple harmonic oscillator). More advanced courses typically present a blend of von Neumann’s Hilbert-space formalism and Dirac’s version of transformation theory. As we will see, this blend depends, for its mathematically cogent formulation, on advances made long after the period under consideration here, the late 1920s, to which I now return.

Jordan’s new foundation for quantum theory. The new foundation (Neue Begründung) of quantum theory that Jordan (1927a) announced in the title of his paper is based on two fundamental ideas. First, quantum mechanics is ultimately a theory about conditional probabilities that some physical (i.e., observable or measurable) quantity has the value a given that another physical quantity has the value b (the tildes indicate that these physical quantities are quantum variables; q-numbers in Dirac’s terminology). Second, such conditional probabilities are given by the absolute square of corresponding complex probability amplitudes, ϕ (a, b). I use the notation of Duncan and Janssen (2013), which follows Dirac rather than Jordan, whose notation is a veritable nightmare.50

Examples of probability amplitudes are the familiar energy eigenfunctions ψ_n (x) of Schrödinger’s wave mechanics, where n refers to the eigenvalue E_n and where, for convenience, we restrict ourselves to one-dimensional problems. The absolute square of this function, | ψ_n (x) |² = ψ_n (x)* ψ_n (x) (where the star denotes complex conjugation), multiplied by the infinitesimal distance dx, gives the probability that the position of the system is somewhere in the narrow interval (x, x + dx) given that its energy is equal to E_n:

(8)

Though eventually named after Born (1926a, 1926b), the probabilistic interpretation of ψ_n (x) in this particular form is due to Wolfgang Pauli, a quantum theorist of the same generation as Heisenberg and Jordan, who was in close contact with all three founders of matrix mechanics and made several key contributions himself (Duncan and Janssen 2013, 182–83). Jordan (and Dirac) generalized Equation (8) for position and energy to arbitrary quantities and :

(9)

In Jordan’s formalism, the energy eigenfunction ψ_n (x) thus becomes the probability amplitude ϕ (x, E_n).

In Jordan’s first paper on his new formalism, only quantities with continuous spectra are considered. When, in a second paper, Jordan (1927b) tried to generalize his formalism to quantities with wholly or partly discrete spectra (such as, typically, the energy), he ran into serious difficulties, which mercilessly exposed the limitations of his approach.

Jordan’s approach, reflecting his mathematical training in Göttingen, was axiomatic (Lacki 2000). He started from a set of postulates for his probability amplitudes and then looked for a realization of these postulates. As Hilbert, von Neumann, and Lothar Nordheim, one of Hilbert’s assistants at the time, put it in a joint paper on Jordan’s new formalism:

One imposes certain physical requirements on these probabilities, which are suggested by earlier experience and developments, and the satisfaction of which calls for certain relations between the probabilities. Secondly, one searches for a simple analytical apparatus in which quantities occur that satisfy these relations exactly. (Hilbert, von Neumann, and Nordheim 1928, 2–3; cf. Lacki 2000, 296)

The number of Jordan’s postulates in various expositions of his formalism fluctuates between two and six (Duncan and Janssen 2013, 199). I will use a version here based on three postulates.

Jordan’s postulates. The first postulate gives the probability amplitude for the basic variables, a generalized coordinate and its conjugate momentum (again, we will restrict ourselves to one-dimensional problems):

(10)

This postulate takes the place of the commutation relations in Equation (7) for position and momentum as the basic quantization condition in Jordan’s formalism. Since | ϕ (p, q) |² = 1, he concluded (ignoring the issue of how to normalize his probabilities) that “for a given value of [] all possible values of [] are equiprobable” (Jordan 1927a, 814). Jordan’s formalism thus contains the kernel of the uncertainty principle, which Heisenberg (1927), drawing on Jordan’s work, would publish later that year.

The second postulate says that the amplitude ϕ (b, a) is the complex conjugate of the amplitude ϕ (a, b):

(11)

For example, from which it follows that for a given value of all values of are equiprobable.

The basic amplitude in Equation (10) trivially satisfies the following pair of differential equations:

(12)

Jordan thought that Equations (10)–(12) sufficed to find the probability amplitudes for any pair of quantities and related to and by a so-called canonical transformation.

Canonical transformations belong to the bag of tricks the old quantum theory had borrowed from celestial mechanics. Closely related techniques were central to the derivation of the Kramers dispersion formula (see the preceding case study). Born, Heisenberg, and Jordan (1926) had imported canonical transformations into matrix mechanics in their famous Dreimännerarbeit. Before he worked out his new foundation for quantum mechanics, Jordan (1926a, 1926b) had published two papers on how to implement canonical transformations in matrix mechanics (Lacki 2004; Duncan and Janssen 2009). Asked about the use of canonical transformations in the Dreimännerarbeit in an interview for the AHQP (cf. note 8) in the early 1960s, Jordan said:

Canonical transformations in the sense of Hamilton-Jacobi [theory in celestial mechanics] were . . . our daily bread in the preceding years, so to tie in the new results with those as closely as possible—that was something very natural for us to try. (Duncan and Janssen 2009, 355)

Canonical transformations, however, proved ill-suited to the task Jordan assigned to them in his new formalism. They are both too restrictive and too permissive for his purposes. They are too restrictive because quantities related by a canonical transformation always have the same spectrum (Duncan and Janssen 2013, 216). A canonical transformation can thus never take us from a quantity with a continuous spectrum to a quantity with a (partly) discrete spectrum. As Jordan (1927b) eventually had to concede, this means that there is no canonical transformation that takes us from the basic amplitude ϕ (p, q) satisfying the pair of differential equations (12) to the new amplitude ϕ (x, E_n) = ψ_n (x) satisfying a transformed version of this pair of differential equations, one of which would have to be equivalent to the time-independent Schrödinger equation.

Canonical transformations are also too permissive. Jordan’s realization of his postulates turned on identifying probability amplitudes with quantities characterizing associated canonical transformations. Unfortunately, as we will see, there are many canonical transformations giving probability amplitudes that do not satisfy Equation (11), Jordan’s second postulate. Jordan thus had to artificially restrict the class of allowed canonical transformations.51 In hindsight, we can see that Jordan was stretching the classical formalism beyond its breaking point in trying to make it work for his new quantum formalism (Duncan and Janssen 2013, 188–91, 253–54).

Jordan’s third postulate, to which we now turn, also has its share of problems, though these are not fatal to his project. This postulate is about how to combine probability amplitudes for different pairs of quantities. It states that in quantum mechanics the usual rules of probability theory, the addition rule for the disjunction and the multiplication rule for the conjunction of two outcomes, apply to the probability amplitudes rather than to the probabilities themselves. Following Born and Pauli, Jordan (1927a, 812) called this the “interference of probabilities.”

The famous double-slit experiment illustrates that this is a sensible name. Let ϕ₁ be the amplitude for the conditional probability that an electron strikes a screen at position x given that it went through the first slit. Let ϕ₂ be the amplitude that the electron strikes at x given that it went through the second slit. According to Jordan’s addition rule for probability amplitudes, the probability of the electron striking at x if it could have gone through either slit is then given by

(13)

The first two terms in the final expression give the probability that the electron strikes the screen at x if it went through one of the slits. The last two terms give the interference effects if the electron could have gone through both.

In the paper introducing the uncertainty principle, Heisenberg (1927) took Jordan to task for his third postulate, arguing that the laws of probability are what they are independently of the laws of physics. Even quantum mechanics cannot change them. While most modern commentators would agree with this criticism, it does not affect Jordan’s formalism. Jordan only used his dubious new quantum probability laws to derive two conditions, which in the further elaboration of the formalism took over the role of those new probability laws as the third postulate. These two conditions are eminently reasonable whether or not one accepts Jordan’s derivation of them. They both continue to hold in modern quantum mechanics.

The first of these two conditions says that the probability amplitudes ϕ (a, b), ϕ (b, c), and ϕ (a, c) involving the physical quantities and should satisfy the relation

(14)

In the example of the double-slit experiment, is the position where the electrons hit the screen (with a continuum of values a), is the position of the source of the electrons (with some fixed value c), and is the position of the slits (with two possible values b₁ and b₂). The integral in Equation (14) then reduces to a sum of two terms,

These two terms are more explicit expressions for the amplitudes ϕ₁ and ϕ₂ in Equation (13) (Duncan and Janssen 2013, 186–87n38).

The second of the two conditions effectively serving as Jordan’s third postulate says that if in Equation (14), it should be the case that

(15)

where δ (x) is defined as “vanishing everywhere except at x = 0 where it is infinite.” I put this definition in scare quotes to flag its gross mathematical sloppiness. Dirac (1927) introduced this notorious delta function in his version of transformation theory. Jordan used it implicitly. Equation (15) expresses the obvious requirement that the probability of finding the value a for a quantity given the value a′ for that same quantity should be zero unless those two values are the same.

If has a fully discrete spectrum, its possible values can be labeled with a discrete index, and the requirement (15) can be formulated in mathematically unobjectionable fashion as:

If has a fully continuous spectrum, the Kronecker delta δ_ij (see Equation (7) for its definition) has to be replaced by the Dirac delta function.

A realization of Jordan’s postulates using canonical transformations. Jordan’s three postulates boil down to the requirement that his probability amplitudes satisfy the relations (10), (11), (14), and (15). All that is left to do at this point is to find a mathematical representation of these probability amplitudes such that these four relations are guaranteed to hold (see the description of Jordan’s approach by Hilbert, von Neumann, and Nordheim above). Jordan does not tell us how he arrived at this representation. He just states his choice and shows that with that choice his postulates are satisfied. Jordan’s choice, however, is a natural one.

Consider the familiar result that an energy eigenfunction ψ_n (p) in momentum space is the Fourier transform of that energy eigenfunction ψ_n (q) in position space:

(16)

Using notation introduced by Dirac (1927), we can write this transformation of ψ_n from q-space to p-space as

(17)

If p and q were discrete indices, the integral would turn into a sum, and the equation would express that a vector with components ψ_n (p) is equal to the product of a matrix with components (p/q), where p labels rows and q labels columns, and a vector with components ψ_n (q). Equation (17) can be seen as the generalization of this relation to a situation in which p and q are continuous variables. Neither Jordan nor Dirac was overly concerned with the mathematical niceties of this generalization.

Comparison between Equations (16)–(17) and Equation (10) suggests that the basic probability amplitude for momentum and position be identified with the “matrix” (more accurately: the integral kernel) for the transformation from position space to momentum space:

(18)

This in turn suggests that the probability amplitude for an arbitrary pair of physical quantities and be identified with the “matrix” for the transformation from b-space to a-space,

(19)

This, of course, is why the Dirac-Jordan formalism is called statistical transformation theory.

Equation (18) shows that the first postulate (i.e., Equation (10)) is satisfied. As long as the transformation “matrix” (a/b) is unitary—which means that its inverse (a/b)−¹ = (b/a) is given by its complex conjugate (a/b)*—the second postulate (i.e., Equation (11)) is also satisfied. Alas, not all canonical transformations are unitary, which is why Jordan somewhat artificially had to restrict the class of allowed transformations (see note 51).

Substituting ψ_n (p) = ϕ (p, E_n) = (p/E_n), ψ_n (q) = ϕ (q, E_n) = (q/E_n) and into Equation (16), we arrive at

(20)

This shows that Equation (14), one of the two conditions effectively playing the role of Jordan’s third postulate, is satisfied in the special case that the quantities and are and respectively. To show that this is true for any triplet of quantities, consider some eigenfunction ψ of the energy or some other quantity. Its transformation from c-space to a-space is given by

(21)

Its transformation from c-space to a-space via b-space is given by

(22)

Comparison of these two transformations shows that (a/c) in Equation (21) is equal to the expression in large parentheses in the last line of Equation (22). This is just as it should be according to Equation (14) (Duncan and Janssen 2013, 185).

To verify that Equation (15), the other half of Jordan’s third postulate, is also satisfied, compare the final expression for ψ (a) in Equation (22) for c = a′ to

(23)

which holds on the basis of the defining equation for the delta function (i.e., for any function f (x), ). This comparison shows that

(24)

in accordance with Equation (15).

A realization of Jordan’s postulates using Hilbert space. In the first installment of a trilogy of papers that would provide the backbone of his famous 1932 book, von Neumann (1927a) introduced the Hilbert-space formalism of quantum mechanics. With the help of a modern version of this formalism, a new realization of Jordan’s postulates can be given. In this new realization, integral kernels of canonical transformations, which Jordan used to represent his probability amplitudes, are replaced by “inner products” of “vectors” in Hilbert space. I use scare quotes to indicate that the justification for treating the relevant quantities as vectors and inner products of vectors turns on results in mathematics only found much later, in particular the theory of distributions and the theory of rigged Hilbert space, both developed in the 1950s. These developments are beyond the level of this paper—and beyond my command of mathematics. They nicely illustrate the point William Young (1926) made in his presidential address to the London Mathematical Society (see section II). Sometimes more sophisticated mathematics can be used to shore up more basic mathematics. Once that has been done, one can use the latter without worrying about the former. A modern student of quantum mechanics will hardly ever go wrong envisioning elements in Hilbert space as vectors in a finite-dimensional vector space. This section is written in that spirit. From now on, I will talk about vectors and inner products without using scare quotes, even though I will remind the reader at several junctures of the mathematical difficulties lurking just below the surface. With that preamble, let me introduce the Hilbert-space formalism and sketch how it can be used to construct a realization of Jordan’s postulates.

In the Hilbert-space formalism, physical quantities, are represented by certain linear operators mapping vectors onto other vectors in a complex, infinite-dimensional vector space known as Hilbert space: (a “hat” denotes an operator; | 〉 is the standard modern notation, due to Dirac, for a vector in Hilbert space). That is linear means that

(25)

for any vectors |f₁〉 and |f₂〉 and any complex numbers λ and μ. If a vector |a〉 satisfies

(26)

it is called an eigenvector of and the (in general, complex) number a is called an eigenvalue of Physical quantities are represented by so-called self-adjoint (or Hermitian) operators. Their eigenvalues are always real numbers. The (infinite) set of all eigenvectors of any self-adjoint operator forms an orthogonal basis for Hilbert space.

The standard notation, again due to Dirac, for the inner product of two arbitrary vectors, |f〉 and |g〉, in Hilbert space is 〈f | g〉. Since this will in general be a complex number, the order matters:

(27)

It thus makes a difference whether | 〉 enters an inner product 〈 | 〉 on the right, as a vector | 〉, or on the left, as a dual vector 〈 |. The dual vector of | 〉 is 〈 | where is called the adjoint of For self-adjoint operators,

The energy is represented by a self-adjoint operator with normalized eigenvectors |E_n〉 and eigenvalues E_n. The position is likewise represented by a self-adjoint operator with normalized eigenvectors |x〉 and eigenvalues x. The normalization is mathematically more problematic in the case of continuous spectra than in the case of discrete spectra. For systems with a fully discrete energy spectrum, for instance, we can simply use the Kronecker delta: . For quantities such as position with fully continuous spectra, we need the Dirac delta function: 〈x | x′〉 = δ (x − x′). The inner products 〈x | E_n〉 of these normalized eigenvectors give the familiar energy eigenfunctions ψ_n (x) of wave mechanics. As we saw above (cf. Equations (8)–(9)), these are also the probability amplitudes ϕ (x, E_n).

This is true in general. Jordan’s three postulates are satisfied if the probability amplitude ϕ (a, b) for any pair of physical quantities and is set equal to the inner product 〈a | b〉 of the normalized eigenvectors |a〉 and |b〉 of the corresponding self-adjoint operators and .

It will be instructive to explicitly verify this for Jordan’s second and third postulates. The second postulate (i.e., Equation (11)) follows directly from the definition of the inner product in Hilbert space: ϕ (b, a) = ϕ (a, b)* because 〈b | a〉 = 〈a | b〉* (see Equation (27)). There is no need for the kind of restrictions on 〈a | b〉 that Jordan had to impose on (a/b).

The third postulate (i.e., Equations (14)–(15)) holds by virtue of a key result of von Neumann’s Hilbert-space formalism, his famous spectral theorem for self-adjoint operators. We need not worry about the proof of this theorem, but we do need to understand at least roughly what it says.

Consider a discrete orthonormal basis {|e_i〉} (with 〈e_i | e_j〉 = δ_ij) in a finite dimensional complex vector space. Any vector |f〉 in that space can be written in terms of its components with respect to this basis:

(28)

The complex number 〈e_i | f〉 gives the component of |f〉 in the direction of |e_i〉. Equation (28) can also be parsed in a different way. We can identify the expression |e_i〉 〈e_i| as a projection operator,

(29)

that maps any vector |f〉 onto the part of |f〉 in the direction of |e_i〉 ( is a self-adjoint operator). Equation (28) then expresses that the sum of these projection operators is the identity operator

(30)

which maps any vector |f〉 back onto itself. Equation (30) is called the resolution of unity corresponding to the orthonormal basis {|e_i〉}.

Von Neumann’s spectral theorem sanctions the generalization of Equations (28)–(30) from finite-dimensional complex vector spaces to infinite-dimensional complex Hilbert space with both discrete and continuous orthonormal bases. The analogue of Equation (28) in Hilbert space with an orthonormal basis consisting of normalized eigenvectors |a〉 of the self-adjoint operator is

(31)

where the integral is to be taken over all eigenvalues of Using the decomposition of |f〉 in Equation (31), the definition of the eigenvectors |a〉 in Equation (26), and the linearity of the operator we can write the action of on |f〉 as

(32)

It follows that can be written as

(33)

where, in analogy to in Equation (29), the projection operator is given by

(34)

This operator maps any vector |f〉 in Hilbert space onto the part of |f〉 in the direction of |a〉. In analogy to Equation (30), the integral of over all eigenvalues of is the identity operator,

(35)

Equation (33) gives the spectral decomposition of the self-adjoint operator . Equation (35) gives the corresponding resolution of unity.

Once the hard work of proving the spectral theorem is done, it is easy to show that Equations (14)–(15) (and thereby Jordan’s third postulate) are satisfied if probability amplitudes ϕ (a, b) are identified with inner products 〈a | b〉. Equation (14) requires that

(36)

where |a〉, |b〉, and |c〉 are the normalized eigenvectors of the self-adjoint operators and representing the quantities and This relation holds by virtue of the resolution of unity corresponding to the spectral decomposition of which allows us to rewrite the right-hand side as Equation (15) requires that

(37)

This relation holds by virtue of the spectral decomposition of and the normalization 〈a | a′〉 = δ (a − a′) of the eigenvectors of

How von Neumann did not build his arch on Jordan’s scaffold and why not. The preceding two subsections suggest that we have found another picture-perfect example of my arch-and-scaffold metaphor in the history of early twentieth-century physics. The relation between arch and scaffold in this case is reminiscent of the general-relativity example discussed earlier. In both cases, swapping out one building block for another while leaving the structure built with them intact resulted in a new building exhibiting the splendor of a magnificent mathematical formalism that had been waiting in the wings. In the case of general relativity, the building blocks were two different definitions of the gravitational field, and the mathematical formalism was the differential geometry of Riemann and others. In this case, the building blocks are two different realizations of Jordan’s probability amplitudes—ϕ (a, b) = (a/b) and ϕ (a, b) = 〈a | b〉—and the mathematical formalism is Hilbert’s spectral theory of operators as generalized by von Neumann.

Historically, however, this is not how von Neumann got from the Jordan-Dirac transformation theory to his own Hilbert-space formalism. Even in the historical literature, von Neumann’s formalism is not always clearly distinguished from Dirac’s. In the classic book on the conceptual development of quantum mechanics mentioned in section II, for instance, Jammer (1966, 307–22) gave the section dealing with von Neumann (1927a, 1927b, 1932) the misleading title “The Statistical Transformation Theory in Hilbert Space” (Duncan and Janssen 2013, 193n51). This is what von Neumann had to say about Dirac in the preface of the book that grew out of his 1927 papers:

Dirac’s method does not meet the demands of mathematical rigor in any way—not even when it is reduced in the natural and cheap way to the level that is common in theoretical physics . . . the correct formulation is not just a matter of making Dirac’s method mathematically precise and explicit but right from the start calls for a different approach related to Hilbert’s spectral theory of operators. (von Neumann 1932, 2)

Rather than using Hilbert space to provide a new realization of probability amplitudes, von Neumann wanted to avoid probability amplitudes altogether. One of his major objections against the Dirac-Jordan formalism was its reliance on the Dirac delta function. This is not a well-defined function, and von Neumann (1927a, 2) dismissed it as simply “absurd.” He also objected to the basic probability amplitude for position and momentum introduced by both Dirac and Jordan (with the latter even elevating it to the status of a postulate; see Equation (10)). Although this is at least a well-defined function, the integral of its absolute square diverges. That means that it is not an element of the space of square-integrable functions, which is one instantiation of Hilbert space.

As mentioned above, the mathematics needed to solve these problems (the theory of distributions and the theory of rigged Hilbert space) was not developed until the 1950s. Using these new tools, we can replace transformation “matrices” (a/b) by “inner products” 〈a|b〉 in a mathematically rigorous way. So, contrary to what von Neumann believed in 1927 and 1932, it is possible to make “Dirac’s method mathematically precise and explicit.” I already alluded to the continued use of the resulting formalism, blending elements of Dirac and von Neumann, in more advanced courses on quantum mechanics, although textbook writers and instructors typically (and understandably!) only gesture at the mathematics needed for its rigorous formulation.

Given the familiarity of this formalism, modern readers may be tempted to read it back into Dirac’s original paper of 1927 on transformation theory—that is, to read his “brackets” (a/b) as inner products 〈a|b〉 and then break those up into “bra”-s 〈a| and “ket”-s |b〉, the now familiar names, due to Dirac, for vectors (kets) and their duals (bras) in Hilbert space. Although Dirac (1930) made use of the Hilbert-space formalism in his book, it was only in 1939 that he himself first split “brackets” into “bras” and “kets” (Borrelli 2010).

How von Neumann did introduce his formalism in response to Jordan’s. There is no doubt that von Neumann introduced his Hilbert-space formalism in response especially to Jordan’s version of the Dirac-Jordan statistical transformation theory. What is not clear, as I mentioned at the beginning of this section, is whether von Neumann’s formalism is best understood as an arch built on top of the scaffold provided by Jordan’s formalism or as a scaffold built to support Jordan’s mathematically unsound arch. I will return to this ambiguity at the end of this section, after I have gone over the steps actually taken by von Neumann in 1927.

I already mentioned the paper on Jordan’s formalism by Hilbert, von Neumann, and Nordheim (1928), submitted in April 1927 but published only the following year. The authors emphasized the mathematical problems with Jordan’s formalism and referred to a forthcoming paper by von Neumann that would address them. Rather than confronting these problems head-on, however, von Neumann (1927a) avoided them by deviating from Jordan’s approach almost from the start. He only took over the two basic ideas on which Jordan had built his formalism: first, that quantum mechanics is a theory about conditional probabilities; second, that these probabilities satisfy some peculiar rules.

As Hilbert and his coauthors had written approvingly about Jordan’s third postulate, “axiom IV” in their exposition:

This requirement is obviously analogous to the addition and multiplication theorems of ordinary probability calculus, except that in this case they hold for the amplitudes rather than for the probabilities themselves. (Hilbert, von Neumann, and Nordheim 1928, 5)

In his own paper, von Neumann reiterated that, in Jordan’s formalism,

the multiplication law of probabilities does not hold in general (what does hold is a weaker law corresponding to Jordan’s “combining of probability amplitudes”). (von Neumann 1927a, 46)

Instead of introducing Jordan’s probability amplitudes, however, von Neumann constructed a formula for conditional probabilities out of projection operators in Hilbert space—Einzeloperatoren, as he called them, or E.op.s for short. Using the notation for projection operators introduced in Equation (34), we can write von Neumann’s formula as (Duncan and Janssen 2013, 242–44)

(38)

where the trace of any operator is defined with the help of an arbitrary discrete orthonormal basis {|e_i〉} of Hilbert space:

(39)

It is easily shown that the result does not depend on which orthonormal basis we use to evaluate the trace.

Using this definition and using Equation (34) for the projection operators, we verify that Equation (38) reduces to Equation (9), Jordan’s formula for the same conditional probability:

(40)

In the last step we used that is the identity operator and that 〈b | a〉 〈a | b〉 = 〈a | b〉* 〈a | b〉 = |〈a | b〉|². It is important to note, however, that the projection operators were the fundamental quantities for von Neumann. Expressing them in terms of “bras” and “kets” reintroduces some of the mathematical objections that he got around by using projection operators instead of probability amplitudes.

Using the resolution of unity the same way as in Equation (40), one readily verifies that for arbitrary operators and Von Neumann’s formalism thus reproduces the relation that follows directly from Jordan’s second postulate (see Equation (11)).

It is also easy to verify that the relation

(41)

in von Neumann’s formalism is the equivalent of Equation (14), which expresses the “interference of probabilities” in Jordan’s formalism. If the projection operators are written in terms of bras and kets, the left-hand side of Equation (41) reduces to 〈c | a〉 〈a | c〉 (cf. Equation (40)). The right-hand side can similarly be written as

(42)

It follows that

(43)

which is Equation (14) for Jordan’s probability amplitudes if these amplitudes are identified with inner products in Hilbert space (see Equation (36)). It is probably no coincidence that Equation (41) is nowhere to be found in von Neumann (1927a). Von Neumann was interested in the outcome of an actual measurement of one quantity given the outcome of a prior measurement of another quantity. In the type of situation involving three quantities considered by Jordan in Equation (14), it is critical, modern quantum mechanics tells us, that the quantity is not actually measured.

Getting from the formalism of Jordan (1927a) to the formalism of von Neumann (1927a) is clearly not as straightforward as replacing one building block by another. It is true that projection operators replaced probability amplitudes as the basic elements but, unlike the substitution of inner products for transformation “matrices,” this replacement was accompanied by invasive structural changes in the edifice built out of these elements.

We can distinguish three layers in Jordan’s formalism: basic ideas, postulates expressing those ideas, and a realization of those postulates. Von Neumann only took over the first of these layers. His paper with Hilbert and Nordheim, however, shows that he had carefully examined Jordan’s entire building. While this inspection had revealed it to be a rickety mathematical structure from top to bottom, it at least had given him a good idea as to what a general formalism for quantum mechanics would have to deliver to be viable as a new framework for doing physics.

Von Neumann recognized that a generalization of Hilbert’s spectral theory of operators was much more appropriate for the purposes of Jordan and Dirac than the theory of canonical transformations that they themselves had pressed into service. The Hilbert-space formalism thus freed quantum mechanics from some of the vestiges of classical mechanics that can clearly be recognized in transformation theory (in the original form in which Jordan and Dirac presented it) and its progenitors, matrix mechanics, and q-number theory.

We already saw that Jordan’s strong reliance on canonical transformations created a number of serious problems. Many of these are specific to Jordan’s axiomatic formulation of transformation theory and do not affect Dirac’s formulation. In other respects, however, both Jordan and Dirac were handicapped by their commitment to canonical transformations. A feature of canonical transformations that I did not emphasize so far is that it is always a transformation from a pair of variables, some generalized coordinate q and its conjugate momentum p, to another such pair. As long as probabilities are defined in terms of canonical transformations, all physical quantities thus need to be sorted in terms of such conjugate variables. Part of von Neumann’s new way of defining these same probabilities in terms of projection operators was the recognition that physical quantities can be represented by self-adjoint operators acting in Hilbert space. With that recognition, the need to group quantities in pairs of conjugate variables evaporated: one no longer had to mind one’s p’s and q’s (Duncan and Janssen 2013)

Von Neumann’s Hilbert-space formalism also brought the definitive clarification of the relation between matrix mechanics and wave mechanics. The two theories correspond to different instantiations of Hilbert space. Wave mechanics works in the space of square-integrable complex functions; matrix mechanics in the space of square-summable complex sequences. Von Neumann (1927a) referred to theorems by Parseval, Fischer, and Riess—theorems mathematicians had known about for at least two decades—proving the isomorphism of these two infinite-dimensional complex vector spaces (Duncan and Janssen 2013, 238–39).

Von Neumann (1927a) submitted the paper in which he introduced his Hilbert-space formalism for quantum mechanics in May 1927. Six months later, he submitted another paper in which he distanced himself even further from Jordan’s approach than he had in May. In Duncan and Janssen (2013, 187n39), we conjectured that this was in response to Heisenberg’s uncertainty paper published in late March. In that paper, Heisenberg (1927) criticized Jordan’s idea that quantum mechanics called for a modification of the basic rules of probability theory. In April (in his paper with Hilbert and Nordheim) and in May, von Neumann had endorsed Jordan’s position (see the quotations above). In November, however, he unequivocally rejected it. One of the shortcomings of his earlier paper, he wrote, was that

the relation to the ordinary probability calculus was not sufficiently clarified: the validity of its basic rules (addition and multiplication law of the probability calculus) was not sufficiently stressed. (von Neumann 1927b, 246)

The title of von Neumann’s new paper accordingly promised a new “probability-theoretical construction” (Wahrscheinlichkeitstheoretischer Aufbau) of quantum mechanics. Von Neumann was familiar with the work on probability theory that Richard von Mises (1928) would publish in book form the following year. To define the probability that a particular property of a system has a particular value, von Neumann, following von Mises, imagined an ensemble of a large number of copies of the system and asked about the relative frequency with which a copy randomly drawn from this ensemble would have that value for that property. He introduced the as yet unknown function (…) for the expectation value of a property in such ensembles. Assuming that properties are represented by self-adjoint operators acting in Hilbert space and imposing some seemingly innocuous conditions on the function (…), von Neumann was able to derive a unique expression for it (Duncan and Janssen 2013, 247–50; Bub 2010; Dieks 2017).

In modern terms, von Neumann’s formula for the expectation value of a property of a system, a property represented by some self-adjoint operator in an ensemble of a great many copies of this system, an ensemble characterized by a density operator can be written as

(44)

For a uniform ensemble consisting of identical copies of the system, the density operator, von Neumann showed, is just the projection operator onto the unit vector |ψ〉 representing the state of all members of the ensemble. Inserting

(45)

for in Equation (44), we recover the more familiar expression for the expectation value of the property represented by in a system in the state |ψ〉,

(46)

For a nonuniform ensemble, the density operator is a weighted sum of projection operators onto unit vectors in the set {|ψ_k〉} representing the various states of the members of the ensemble:

(47)

where the α_k’s are real numbers such that Inserting Equation (47) for in Equation (44), we find that the expectation value in this nonuniform ensemble is given by

(48)

Von Neumann’s dissatisfaction with Jordan’s uncritical introduction of probabilities thus led him to the important distinction between uniform and nonuniform ensembles or, in modern terms, between pure states and mixed states. Thermal states are represented by mixed states in quantum mechanics. Before the end of the year, von Neumann (1927c), using his density operators to describe various ensembles, published yet another paper, the final installment of his 1927 trilogy, that helped lay the foundations for quantum-statistical mechanics.

Jordan and von Neumann: arch or scaffold? If we look at the sequence of general formalisms for quantum mechanics in Neue Begründung (Jordan 1927a, 1927b), Mathematische Begründung (von Neumann 1927a), and Wahrscheinlichkeitstheoretischer Aufbau (von Neumann 1927b), we can clearly see how important elements of the earlier formalisms were retained in the later ones while others were dropped (such as, for instance, the need to group quantities in pairs of conjugate variables). However, if we try to characterize this progression in terms of arches and scaffolds, it is not clear which version of the metaphor we should use. Was the earlier formalism used as a scaffold to facilitate the construction of the arch of the later formalism, or was the later formalism used as a scaffold to prevent the earlier arch from collapsing? We can tell the story using either version of the metaphor. The best way to tell it may be by mixing the two. In any event, this case calls for a loosening of the metaphor. No matter which formalism played the role of the scaffold and which one that of the arch, the fact remains that the scaffold was never taken down. Instead we are left with a composite of arch and scaffold.

V. The Arch-and-Scaffold Metaphor and Evolutionary Biology

In the introduction to his magnum opus, The Structure of Evolutionary Theory, Stephen Jay Gould (2002, 1–6) used an architectural metaphor to describe the development of evolutionary theory that fits nicely with the arch-and-scaffold metaphor, even though they work on different scales. Gould compared a sequence of closely related theories to a cathedral; I compare pairs of adjacent terms in such a sequence to arches and scaffolds.52

Gould took his metaphor from the Scottish geologist, botanist, and paleontologist Hugh Falconer. In an 1863 paper on Darwin’s theory of descent with modification from a common ancestor through natural selection, Falconer suggested that the further development of the theory would end up resembling the building of the Duomo in Milan. This cathedral was built over several centuries and combines conflicting Gothic and baroque styles. Gould contrasted Falconer’s view that Darwin had laid the foundations for a building bound to be built according to plans very different from Darwin’s original ones with Darwin’s own view, expressed in his response to Falconer that the latter would continue to govern the construction of the entire building (or, in Darwin’s own terms, that the whole “framework will stand,” not just the foundations).

In Gould’s view, the actual development of evolutionary theory has been much closer to what Falconer than to what Darwin expected (Grantham 2004, 30). In other words, the way Gould saw it, contemporary evolutionary theory resembles the Duomo in just the way Falconer envisioned. Whittaker made a similar observation about the development of Maxwell’s theory of electromagnetism. After discussing the elaboration of Maxwell’s theory by J. J. Thomson, George Francis FitzGerald, Oliver Heaviside, Poynting, and others, he noted: “Maxwell’s theory was now being developed in ways which could scarcely have been anticipated by its author. But although every year added something to the superstructure, the foundations remained much as Maxwell had laid them” (Whittaker [1951–1954] 1987, 1:318). My first stab at an analysis of the development of quantum theory in terms of arches and scaffolds in section IV suggests that similar observations can be made about quantum theory.

Talking about an early stage in the development of quantum theory, physicist and philosopher Henry Margenau used the same building metaphor as Falconer: “Bohr’s atom sat like a baroque tower upon the Gothic base of classical electrodynamics” (Margenau 1950, 311; quoted in Lakatos 1970, 142). Unlike Falconer, however, he considered this “a malformation in the theory’s architecture” (Margenau 1950, 311; quoted in Lakatos 1970, 142). In a lecture at Keio University in 1989, condensed-matter icon Philip W. Anderson also compared science to a cathedral but did so to emphasize science’s beauty. After a brief sketch of various important contributions to physics that build on the 1957 paper by John Bardeen, Leon Cooper, and John Robert Schrieffer introducing the BCS theory of superconductivity named after them, Anderson (1994, 239) asked: “Where does the beauty reside?” The best answer he could come up with is that it resides in the network of citations connecting the relevant papers. He then added:

Science has the almost unique property of collectively building a beautiful edifice: perhaps the best analogue is a medieval cathedral like Ely or Chartres . . . where many dedicated artists working with reference to each other’s work jointly created a complex of beauty. (Anderson 1994, 239)

The same metaphor has been used to describe technological developments. Discussing the question “who deserves the most credit for inventing the internet” in his bestseller The Innovators, Walter Isaacson (2014, 260)53 quotes pioneer Paul Baran “using a beautiful image that applies to all innovation”:

The process of technological development is like building a cathedral. Over the course of several hundred years new people come along and each lays down a block on top of the old foundations. (Hafner and Lyon, 1996, 79)

Another comparison of science to a cathedral can be found in the preface of a book on thermodynamics by Gilbert Lewis and Merle Randall (1923). Their image of the cathedral of science under construction is reminiscent of the factory that Duhem saw in Lodge’s Modern Views of Electricity (see section III). Unlike Duhem, however, Lewis and Randall saw this as a good thing. The awe inspired by science’s cathedrals should not get in the way of its day-to-day business.

There are ancient cathedrals which, apart from their consecrated purpose, inspire solemnity and awe . . . The labor of generations of architects and artisans has been forgotten, the scaffolding erected for their toil has long since been removed, their mistakes have been erased, or have become hidden by the dust of centuries. Seeing only the perfection of the completed whole, we are impressed as by some superhuman agency. But sometimes we enter such an edifice that is still under construction; then the sound of hammers, the reek of tobacco, the trivial jests bandied from workman to workman, enable us to realize that these great structures are but the result of giving to ordinary human effort a direction and a purpose.

Science has its cathedrals, built by the efforts of a few architects and of many workers. In these loftier monuments of scientific thought, a tradition has arisen whereby the friendly usages of colloquial speech give way to a certain severity and formality. While this may sometimes promote precise thinking, it more often results in the intimidation of the neophyte. Therefore, we have attempted, while conducting the reader through the classic edifice of thermodynamics into the workshops where construction is now in progress, to temper the customary severity of the science insofar as is compatible with clarity of thought. (Lewis and Randall 1923, vii)

In the preface of his textbook on special relativity, J. L. Synge similarly wrote: “My ambition has been to make [Minkowski] space-time a real workshop for physicists, and not a museum visited occasionally with a feeling of awe” (Synge [1955] 1972, vii).

After these examples of physicists comparing the development of their field to the building of cathedrals, I return to Gould’s discussion of Falconer’s use of the metaphor. After contrasting the different ways in which Darwin and Falconer expected the cathedral of evolutionary theory to be built, he noted parenthetically that

no one has suggested the third alternative, often the fate of cathedrals—destruction, either total or partial, followed by a new building of contrary or oppositional form, erected over a different foundation. (Gould 2002, 6)

As I pointed out in section II, the original Waterloo Bridge did suffer the fate of Gould’s third alternative, which corresponds to the metaphor Kuhn used in his Guggenheim application of “tearing down one habitable structure and rebuilding to a new plan.” Neither the development of evolutionary theory nor the development of quantum and relativity theory fits this metaphor.

Since I brought up evolutionary theory, the question naturally arises how these architectural metaphors for theory change (arches and scaffolds, building a cathedral) relate to accounts of theory change modeled on biological evolution. Toward the end of Structure, Kuhn ([1962] 2012) used an analogy “that relates the evolution of organisms to the evolution of scientific ideas,” albeit with the caveat that the analogy “can easily be pushed too far” (171). The evolutionary biology Kuhn had in mind was almost certainly the population genetics of the Modern Synthesis, which reigned supreme in the early 1960s (Bowler 2003, chapter 9).

The analogy between population genetics and cultural evolution is best known through the last chapter of Richard Dawkins’s The Selfish Gene, in which selection of memes, units of culture, takes the place of selection of genes (Dawkins 1976, chapter 11). Dawkins does not apply his model for cultural evolution to science but gives no indication that it could not be applied there as well.

One key difference between the evolution of theories and the evolution of species, however, is that modifications of theories, unlike variations in species that form the input for natural selection, are anything but generated at random. Kuhn has little to say about where new theories come from,54 and some of what he does say might give comfort to those tempted to push the analogy beyond its breaking point. Consider, for instance, the following passage in Structure: “The new paradigm, or a sufficient hint to permit later articulation, emerges all at once, sometimes in the middle of the night, in the mind of a man deeply immersed in crisis” (Kuhn [1962] 2012, 90). Combining statements such as these with Kuhn’s emphasis on the proliferation of different articulations of a paradigm in a period of crisis—both in general (Kuhn [1962] 2012, chapter 7) and more specifically (Kuhn 1970, 257; “more and wilder versions of the old quantum theory than before”)—one may come away with the impression that modifications of theories, not unlike variations in species, are typically generated in great profusion and in no particular direction and that the way in which modifications of theories compete for acceptance by a given scientific community is not dissimilar to the way variations in species compete for a given ecological niche.

Incidentally, in his critique of Lodge mentioned in section III, Poynting used the biological metaphor suggested by Kuhn toward the end of Structure to characterize the tradition of constructing mechanical models for the ether. Lodge, he wrote,

uses the main idea of Maxwell’s model [see Figure 4.6] but replaces Maxwell’s duality of magnetic wheels and electric “idle” wheels by a duality of electric wheels. It is, perhaps, an open question whether this is really a simplification, but the attempt was well worth making, for it is only by variation and natural selection that the mechanical model will be suited to its environment in the electric world. (Poynting 1893, 635)

The random proliferation of ether models in great profusion suggested by this analogy hardly does justice to the development of ether theory by late-Victorian Maxwellians. The arch-and-scaffold metaphor fits this development much better. Lodge’s ether was scaffolded by Maxwell’s, as a superficial comparison of Figures 4.3 and 4.6 already suggests.

Gould was among those leading the charge against the hard-line version of the Modern Synthesis.55 In his attack on this hardening orthodoxy, he used an architectural metaphor that has become more popular than the one by Falconer he unearthed. Gould’s own metaphor involves the Basilica di San Marco in Venice rather than the Duomo in Milan (see Figure 4.7 below). In “The Spandrels of San Marco and the Panglossian Paradigm,” Gould teamed up with Richard Lewontin—he who warned that the price of metaphor is eternal vigilance (see section II)—to offer, as they announced in the subtitle of their paper, “a critique of the adaptationist programme.” Panglossian refers to Dr. Pangloss, Voltaire’s caricature of Leibniz in Candide, who sees adaptation everywhere. In the abstract, the authors wrote:

An adaptationist programme has dominated evolutionary thought in England and the United States during the past forty years. It is based on faith in the power of natural selection as an optimizing agent. It proceeds by breaking an organism into unitary “traits” and proposing an adaptive story for each considered separately . . . We criticize this approach and attempt to reassert a competing notion (long popular in continental Europe) that organisms must be analyzed as integrated wholes, with baupläne so constrained by phyletic heritage, pathways of development, and general architecture that the constraints themselves become more interesting and more important in delimiting pathways of change than the selective force that may mediate change when it occurs. (Gould and Lewontin 1979, 581 [my emphasis]; see also 593–94 as well as Gould 1980, 39–40)

Gould and Lewontin thus championed an approach to biological evolution that de-emphasizes the agent of evolutionary change (natural selection) and puts the emphasis on the role of constraints instead. In his bestseller Your Inner Fish, Neil Shubin (2008) shows what an account of evolutionary change along these lines looks like. Taking the same backward-looking perspective that I argued we should strive for in arch-and-scaffold narratives (see section III), Shubin traces the evolution of various parts of the human body back along our branch of the evolutionary tree. Although he clearly acknowledges that natural selection is the mechanism that “mediate[s] change when it occurs,” I do not recall coming across the term natural selection even once when reading his book and a search in an electronic version did not return a single instance of the term. Shubin’s emphasis is on Gould and Lewontin’s constraints. He even uses the term scaffolding at one point: “The scaffolding of our entire body originated in a surprisingly ancient place: single-celled animals” (Shubin 2008, 123).

A full-blown version of the approach advocated by Gould and Lewontin, which goes by the acronym evo-devo (for evolution and development), has become popular in biology (Sansom and Brandon 2007). Evo-devo fits much better with the arch-and-scaffold metaphor for the evolution and development of scientific theories than the population genetics of the Modern Synthesis. An arch-and-scaffold narrative for an episode of theory change highlights how structures in a later theory can be traced to structures in an earlier one. It brackets the question of how the new theory displaced the old one and instead focuses on how the new theory grew out of the old one. The parallels to Gould and Lewontin’s “Spandrels of San Marco” or Shubin’s Your Inner Fish should be obvious. They are interested in tracing structures in later species to structures in earlier species and less interested in spelling out the details of the selection process through which the former displaced the latter.

The concept of constraints can fruitfully be used both in evo devo–type accounts of biological evolution and in arch-and-scaffold-type accounts of theory change, even though the forces behind the constraints are different. With biological organisms, as with architectural structures, the constraints ultimately come from limitations of the malleability of (the arrangement of) the components out of which the organism or the architectural structure are made. These components and arrangements can only be tweaked so much before the creature ceases to be viable or the building collapses. Over time, natural selection can change one creature into a radically different creature, but it cannot get there in one fell swoop. In a mature science, it turns out, a theory that has proved its mettle by accounting for a wide array of empirical data can likewise only be tweaked so much before it ceases to be empirically viable. In this case, it is conceivable, of course, that an exceptionally imaginative scientist dreams up a radically different theory that accounts for an even more impressive array of data than the prevailing one. At first sight, it may even look as if that is essentially how relativity theory and quantum theory burst upon the scene. On closer examination (see section IV), the founding fathers of these theories arrived at them by tweaking existing theories under the tight constraints imposed by empirical viability. But whereas nature tweaks species at random, scientists tweak theories by design—or so one would hope! Because of this key difference, constraints play an even larger role in the evolution of theories than in the evolution of species. It is only natural then for modern historians of science looking for help from evolutionary biology in their studies of theory change to turn to Gould and evo-devo and to forget about Dawkins and population genetics. Of course, fifty years from now this new alliance might mainly show us that we were prisoners of our time, just as Kuhn was of his.

Be that as it may, it seems to me that the concept of constraints, as developed by Gould and others, has the potential to help us overcome the limitations of the arch-and-scaffold metaphor that we ran into at various junctures in section IV. In particular, it may help us articulate relations between the scaffolded and the scaffolding theory that do not naturally fit the basic architectural metaphor of arches and scaffolds. To illustrate this potential, I close this section with a first exploration of the possible applications of Gould’s ideas about constraints to the evolution and development of scientific theories.

In a paper titled “The Evolutionary Biology of Constraint,” written around the same time as “The Spandrels of San Marco,” Gould distinguishes two kinds of constraints that the adaptationist tends to neglect:

One is that the possible routes of selection are channeled by inherited morphology, building material, and the amount and nature of variation itself. Though selection moves organisms down the channels, the channels themselves . . . impose primary constraints on the direction of change. The second is that selection on one part of a structure may impose a set of correlated and nonadaptive modifications of other parts . . . Many features, even fundamental ones, may be nonadaptive (though not, to be sure, strongly unadaptive) either as developmental correlates of primary adaptations or as “unanticipated” structural consequences of primary adaptations themselves. (Gould 1980, 44)

The structural constraints most relevant to my arch-and-scaffold metaphor for theory change are of the first kind. The transition from the old quantum theory to matrix mechanics provides a nice example of this kind (cf. section IV). Consider perturbation theory in matrix mechanics developed in the famous Dreimännerarbeit (Born, Heisenberg, and Jordan 1926). Perfectly adapted to the task at hand, one might think that it was especially developed for matrix mechanics. It was not. The perturbation techniques of celestial mechanics had been transferred to atomic mechanics in the old quantum theory of Bohr and Sommerfeld. Heisenberg’s (1925) Umdeutung, or “reinterpretation,” of classical mechanics to what would become matrix mechanics essentially also dictated how these perturbation techniques had to be “reinterpreted.” Recognition of this state of affairs helps us pinpoint what accounts for the continuity in the transition from the old quantum theory to modern quantum mechanics (cf. section II). Suman Seth (2010, 266) quotes Sommerfeld emphasizing that continuity in 1929: “The new development does not signify a revolution, but a joyful advancement of what was already in existence, with many fundamental clarifications and sharpenings.” Seth focuses on a continuity of scientific practice. What made this continuity of practice possible, however, was a continuity of mathematical structure and technique (Midwinter and Janssen 2013, 146–47, 198).

The spandrels of San Marco from the title of Gould and Lewontin’s article are constraints of the second kind—more specifically, “‘unanticipated’ structural consequences of primary adaptations.” As they explain in the introduction of their paper,

[s]pandrels—the tapering triangular spaces formed by the intersection of two rounded arches at right angles [see Figure 4.7]—are necessary architectural by-products of mounting a dome on rounded arches. Each spandrel contains a design admirably fitted into its tapering space. An evangelist sits in the upper part flanked by the heavenly cities . . . The design is so elaborate, harmonious, and purposeful that we are tempted to view it as the starting point of any analysis, as the cause in some sense of the surrounding architecture. But this would invert the proper path of analysis. The system begins with an architectural constraint: the necessary four spandrels and their tapering triangular form. (Gould and Lewontin 1979, 581–82, see also Gould 2002, 1249–53)

As a simple example of “spandrels” in biological evolution, Gould and Lewontin point to the tiny front legs of Tyrannosaurus rex. One could try to give an adaptationist account of this odd feature—maybe they developed “to help the animal rise from a lying position” (Gould and Lewontin 1979, 587)—but, given the homologies between T. rex and its ancestors (i.e., every bone in the skeleton of one corresponds to a bone in the skeleton of any other), it is more likely that it was “a developmental correlate of allometric fields for relative increase in head and hindlimb size” (Gould and Lewontin 1979, 587). Simply put: with a limited supply of bone material, for some parts to get bigger, other parts had to get smaller. The tiny front legs would then be an automatic by-product of evolution driven by constraints. As with the spandrels of San Marco, this seemingly useless feature was subsequently given some purpose.56

This simple example can be used to give a (rough) definition of the concept of a “spandrel” independently of its architectural origin. A spandrel is a feature that initially looks specifically designed for a particular purpose but that on closer examination is an inessential but inevitable by-product of a highly constrained development that was only subsequently given some purpose. Defined in this way, “spandrels” can also be recognized in instances of scientific theory change. I can think of at least one example in the history of quantum theory.

On first encountering Bohr’s theory of the atom and the old quantum theory of Bohr and Sommerfeld that grew out of it, one might think that the notion of electron orbits is perfectly adapted to the job at hand—that is, the explanation of atomic spectra. Electron orbits represent the different energy states of electrons in atoms, and jumps between those energy states are associated with the spectral lines that were the main object of study in the old quantum theory. Electron orbits are so central to the old quantum theory that they came to dominate the theory’s iconography (Schirrmacher 2009). Using Gould and Lewontin’s metaphor, however, one can say that electron orbits were nothing but “spandrels” arising as by-products of the use of mathematical techniques borrowed from celestial mechanics in atomic physics. These techniques were very effective in determining the energy levels of an electron in an atom. It was therefore only natural that elements associated with these techniques got transferred along with them. Planets orbiting the sun in the solar system thus became electrons orbiting the nucleus in an atom. This element, it seemed, could be put to good use. Energy states of electrons were represented in the old quantum theory by definite electron orbits. This representation, however, turned out to be highly problematic and was abandoned in the transition from the old to the new quantum theory (see the fourth case study in section IV; Duncan and Janssen 2007, 2014, 2015).57

VI. The Arch-and-Scaffold Metaphor and Scientific Innovation

In this chapter, I showed that a metaphor of arches and scaffolds can be used to capture both continuities and discontinuities in various episodes in the early history of special relativity, general relativity, and quantum theory. The metaphor thus helps stake out a middle ground between the traditional cumulative picture of theory change and the discontinuous picture of paradigm shifts made popular by Kuhn’s Structure (though its author, as we saw in section II, vacillated between different and not necessarily compatible metaphors for theory change). In four of my five examples, I indicated how the narrative could be constructed backward in time, which, I argued, is the most effective defense against the obvious charge of Whiggishness against the metaphor (section III).

I identified two specific ways in which a scientist can get from the theory playing the role of the scaffold in the metaphor to the theory playing the role of the arch and gave concrete examples of each in the five case studies in section IV.

1. 1. Generalization. A scientist recognizes that a structure exhibited by the scaffold for a special case has broader significance.
   1. 1a. Einstein and Minkowski realized that the Lorentz invariance of Lorentz’s theory of electromagnetism transcends its connection with electromagnetism and reflects a symmetry of a new space-time structure (first case study).
   2. 1b. Laue developed relativistic continuum mechanics by stripping Abraham’s electromagnetic mechanics of its electromagnetic particulars (second case study).
   3. 1c. Heisenberg recognized that the way in which Kramers had used Bohr’s correspondence principle to construct a new formula for optical dispersion could be generalized to construct a new framework for all of physics (fourth case study).
   4. 1d. Von Neumann unified wave mechanics and matrix mechanics by showing that their mathematical formalisms are different instantiations of a more general formalism that he called Hilbert space (fifth case study).
2. 2. Substitution. A scientist replaces the basic building blocks of the scaffold with new ones while leaving the structure built out of them intact. An example of this in the evolution of technology would be the replacement of vacuum tubes by transistors in a logic board.
   1. 2a. Einstein arrived at equations within hailing distance of the Einstein field equations of general relativity by changing the definition of the gravitational field in the formalism he had developed around the older Entwurf field equations (third case study).58
   2. 2b. The central idea of Heisenberg’s Umdeutung paper was to replace classical quantities by two-index quantum objects soon to be recognized as matrices without changing the relations between those quantities given by the laws of classical mechanics (fourth case study).
   3. 2c. One obtains the Hilbert-space incarnation of the Dirac-Jordan transformation theory by replacing transformation matrices by inner products of vectors in Hilbert space (fifth case study).

In this last example, however, I also noted that this is not how von Neumann introduced Hilbert space. Like John Stachel’s (2007) Newstein fable (see note 35), this arch-and-scaffold story provided a counterfactual history that could be used as a foil for the actual history. The actual history in this case can also be captured in terms of the arch-and-scaffold metaphor. It is not clear, however, whether the best way to do so is in terms of a scaffold built before the arch, discarded when the arch could support itself, or in terms of a scaffold built after the arch, left in place to prevent the arch from collapsing. The former use of the metaphor is mine; the latter is a tweaked version of Hilbert’s metaphor of building a house before laying its foundations (see section II). In my version of the metaphor, Jordan’s transformation theory is the scaffold, and von Neumann’s Hilbert space-formalism is the arch. In the Hilbert-inspired version, it is just the other way around. In this final case study, I ended up mixing these two metaphors in my attempt to characterize the relation between these two general frameworks for quantum mechanics.

This should serve as a reminder that the arch-and-scaffold metaphor is an expository device—a gimmick, some might say—not an analytical tool. As an expository device, it does useful work, as is perhaps best illustrated by the general-relativity example (Janssen and Renn 2015). Both Einstein himself and later commentators have suggested that he found the Einstein field equations in November 1915 by switching from physics to mathematics at the eleventh hour. The arch-and-scaffold metaphor helped counter this dramatic but highly misleading account by putting the alternative account, with Einstein doggedly pursuing his physics, in sharper relief.

At a more basic level, the arch-and-scaffold metaphor served to bring out common patterns in different instances of theory change that would have been much harder to spot without it. In the introduction, I broke down the metaphor into specific elements using a picture of the construction of the Waterloo Bridge (Figure 4.1). These elements worked well to draw special attention to certain features of the examples I presented in section IV. The metaphor of Minkowski providing the springers and Laue providing the keystone of the arch of special relativity helped underscore the importance of relativistic continuum mechanics. The metaphor of Kramers and Heisenberg using the correspondence principle as their windlass nicely brought out the way in which several physicists used this principle in the period right around Umdeutung. But breaking down the metaphor in this way should not be mistaken for turning it into a philosophical tool for further analysis, either of the general pattern or of the individual examples. It remains an expository device similar to the curatorial devices used in museum exhibits of dinosaurs (see section III).

For analytical tools we need to look elsewhere. As I suggested in section V, they may be found in evolutionary biology, not in the population genetics of the Modern Synthesis but in the more recent tradition known as evo-devo. The concept of constraints looks especially promising, but additional concepts will undoubtedly be needed. In the development of special relativity, for instance, we saw that the transition from scaffold to arch involved grouping various quantities defined in three-dimensional space into new quantities defined in four-dimensional space-time. Can analogous processes be identified in evolutionary biology? If so, can the concepts developed to deal with those processes be customized to deal with their possible analogues in the evolution of theories? Such concepts could then be used to bring features glimpsed through the lens of the arch-and-scaffold metaphor into sharper focus. In this way, my project could support broader efforts, already underway, to develop a new framework for cultural evolution, including the evolution of science, that draws on advances made in evolutionary biology over the past few decades (Caporael, Griesemer, and Wimsatt 2014; Laubichler and Renn 2015; Renn 2019). In the spirit of Hooke (see section III), I would be satisfied if the arch-and-scaffold metaphor were to help scaffold this new framework and were then thrown away like Wittgenstein’s ladder.59

Notes

References