3 Number Sense in Large Language Models

What Is a Prime Number?

The use of LLMs like ChatGPT is increasingly common as they have demonstrated impressive capabilities of language regurgitation and production of novel outputs. Initially trained via unsupervised learning on a diverse range of text data, the model subsequently undergoes a reinforcement learning from human feedback process that “corrects” for the model’s lack of “preference” (or recognition) of truthful, useful, and inoffensive content (Millière and Buckner 2024). First, “crowdworkers” are asked to rank model responses according to quality, then these data are used to train a “reward model” that guides fine-tuning (Millière and Buckner 2024). Finally, reward model outputs are used as feedback signals in a reinforcement process (Millière and Buckner 2024). Further, the three-step training process can occur multiple times to amplify effectiveness, but once the model is deployed parameters remain fixed (Millière and Buckner 2024). LLM programs continuously redeploy and fine-tune models toward maintaining effectiveness and responsiveness to user interaction.

The article “How Is ChatGPT’s Behavior Changing over Time?” (Chen et al. 2023), sought to understand model updates over time. Lingjiao Chen et al. (2023) asked questions of two different versions, GPT-3.5 and GPT-4,1 evaluating change in effectiveness to various prompts. Of particular interest is the deterioration observed in identifying prime versus composite numbers, where GPT-4 from March to June had its accuracy decrease from 84 percent to 51 percent while GPT 3.5 demonstrated increased effectiveness, from 49.6 percent to 76.2 percent. Findings suggest LLM drift could change whether and how instructions were followed, and demonstrated a strong bias toward viewing all integers, including primes, as composite numbers.

From a computational perspective, we can speculate as to why this happened. The LLM models that ChatGPT uses, transformer models, “tokenize” inputs, defined as a chunking of words into smaller sections to be used as the model’s basic linguistic unit (Kaushal and Mahowald 2022; Millière and Buckner 2024). These are processed in parallel rather than sequentially, through “self-attention,” which allows parts of a sequence to be differentially weighted to attend to interrelationships among words in a sequence (Millière and Buckner 2024). These transformer-based models boost training efficiency and increase the scale and complexity of tasks that can be performed by encoding information (tokens) with real numbers, facilitating both regurgitation and production of novel output. Of relevance here is that numbers do not need to be tokenized into meaningful units (Millière and Buckner 2024); they are treated in the same way as text tokens,2 embedded as distributed vectors (Wallace et al. 2019). GPT-3 might process “940” as a single token and “941” as two tokens, “9” and “41,” leading Raphaël Millière and Cameron Buckner (2024) to speculate that this may be why LLMs struggle with multiple-digit arithmetic.

Cybernetics and Flawed Number Sense

I believe that language models have a cybernetic character and history, inheriting early paradigms pioneered by Warren Weaver and influenced by Claude Shannon, canonical figures in the cybernetic turn. Cybernetics is a theory and practice of communication networks that connects human and nonhuman agents toward the task of handling and transmitting information (Galloway 2014, 2021). In cybernetic systems, the information carried, rather than the semantic meaning, is privileged, with the aim of maintaining, regulating, and controlling the overall network. Alexander R. Galloway (2014) labels these cybernetic systems algorithmic due to their operational and executable nature, requiring a high level of organization and management to maintain stability, determinism, and homeostasis.

This is what we might call ChatGPT’s descriptive statement (Bateson 2000a); the process of “learning” is defined by the algorithmic network’s attempts at maintaining a homeostasis defined and evaluated through a human process. ChatGPT’s perceived ability to learn is determined by its output’s accuracy and flexibility, just as the recursive composition of cybernetic networks renders actions nondeterministic but teleological, open-ended but not predetermined (Routhier 2023). This lack of predetermination is what generates Chen et al.’s (2023) research topic and paper title; LLM models’ “behavior,” or outputs, can be predictable but are subject to any number of variables and changes within the network. The perspective and doing of ChatGPT’s cybernetic, closed-loop paradigm of learning, its descriptive statement, is conserved in its pursuit of more unknown unknowns to correct. But the arrival of resolution is always already deferred, as in its privileging of message over meaning; it lacks the capacity to ever definitively calculate quantitatively. For instance, if the model has never encountered a number in a text (pick a very large and weird number) then it is less likely to know how to do arithmetic with it. These faulty number facts are just one part of the limited reasoning capacity of LLM (Marcus 2022). This makes ChatGPT’s algorithm, in an uncanny way, not quantitative, as its descriptive statement infuses doubt into any and all of its mathematical calculations, since these calculations are, in actuality, inductive predictions.

There can be productivity to sitting with the doubt produced by this “error” (Amoore 2020). Gregory Bateson (2000b) states, as we see here, that cybernetic explanation is always negative, involving a consideration of alternative possibilities and speculation as to why those possibilities were not followed and why the event that could occur, occurred. We know 17,077 is a prime number. Numerical signifiers, when deployed in mathematical contexts, carry an unreasonable effectiveness that, despite being a human cultural invention (Rotman 2000), elevate its grammar to law. ChatGPT treats numbers as text and thus in the same semiotic register as language, but in so doing it fails to recognize that number itself has a different semantic flexibility compared to letters. This flexible semantic life of number, whereby numbers are multiply decomposable through various functions in different mathematical milieus, cannot be properly learned from near infinite sets of texts, even when the text training the algorithm states and demonstrates these number facts.

Number sense is as much embodied and ecological as any other mode of sign making. The programmers’ desire to create, implement, and conserve the recursive, recalculating paradigm of learning of ChatGPT leads to a constant state of quantitative instability and indeterminism. These errors prompt us toward challenging the wonder and innovation ascribed to these models and lead us to state that the model is mistaking the map for the territory in its privileging of mediation and code, or what Millière and Buckner (2024) refer to as the grounding problem.

(Hu)man Feedback

My concern is that the mathematical error emerging from within the machine is indicative of a larger, structural limitation to any argument that ChatGPT can be a robust, cognating machine. I see this error as twofold: algorithmic and epistemological. The algorithmic error is where I have spent my time thus far. The epistemological error and “what is to be done” solution (Wynter 2015) pursued by programmers will be to reprogram the cybernetic loop indefinitely. The possibility of the unknown unknown will always lurk, each mathematical prediction carrying an air of doubt, demanding a recalculation to ease uncertainty (paradoxically producing more labor). ChatGPT’s posthuman closed-loop paradigm will seek to conserve itself, altering code, protocols, and testing procedures, claiming to invent the new but simply producing a better map that just reaches the next yet-to-be-named prime number. As Antonia Majaca and Luciana Parisi (2016) argue, the cybernetic paradigm stretches data to its limits, and opens up to a tame unpredictability of results, where relentless recalculations of data guarantee another known unknown. Millière and Buckner (2024) state that the “critical question” for philosophy and computation is whether LLMs “can implement the same processes or algorithms as the mind, when described at an appropriate level of computational abstraction?” (10). They ultimately turn away from the terminology of “understanding” and toward “semantic competence” that emphasizes the legibility of the output (externalized within human linguistic communities) rather than a presumed mental process via computational means.

What emerges here are algorithms trained toward becoming a prosthetic for Man to (re)produce results coherent with our genres of being—yet, always already in a state demanding maintenance. As asserted by Édouard Glissant, “to maintain equilibrium, is, in fact, not to develop” (1989, 41). Changes thus far to ChatGPT, meant to improve its number sense, increase the iterative cybernetic layers, but cannot engender any creative “divergence games”3 instead of imitation games, without intervention and reappropriation from the human. I would argue as well that they cannot script refusals and the kind of radical (Black) imagination that opens onto the new (Keeling 2019). ChatGPT’s ability to open toward a radical future is restricted because it is being evaluated through outputs externalized and read through our current genre of being.

This issue of poor number sense presents a contradiction. Luciana Parisi and Ezekiel Dixon-Román (2020) argue that AI models inherit sociogenic principles of and on the flesh.4 The expression “of and on” is crucial because societies of control, in practice, are oscillations of both discipline and control, operating by computational means but also enacting and operating on the body and with/through bodies (Dixon-Román 2023; Puar 2023). Despite being a computing machine, ChatGPT, in its current construction, can never be a mathematical, calculating machine. Though the problematic aspects of its narrative sociogenic inheritance are said to be suppressed through a human feedback process that prioritizes truth, utility, and the inoffensive, its inheritance comes roaring back by that which it is blind to—the concept of “ecology” as feedback loop. Both the material construction of computers, servers, and hardware (Ferreira da Silva 2017) and their operation enact ecological violence via differential destruction to people and nature—people who, not coincidentally, are prescribed as the “nature” to the West’s “culture,” where this epistemological disavowal of nature threatens species extinction. This is the epistemological error; survival is always cast as a loop between organism and environment (Bateson 2000d; Bateson 2000).

The ecological violence is only worsening with generative AI. A 2019 estimate (Dhar 2020; Strubell et al. 2019) found that training a single LLM produced the equivalent of a hundred twenty-five round-trip flights between Beijing and New York in carbon dioxide emissions. Analysis from The Guardian found that 2020–22 emissions from data centers for Google, Microsoft, Meta, and Apple are likely 662 percent higher than reported (O’Brien 2024). Working with researchers at the University of California, Riverside, the Washington Post found that GPT-4 consumes anywhere from one to three bottles of water per hundred-word email generated (Verma and Tan 2024). Our ethical indifference to this violence, and the disavowal of LLMs’ material base, continues to “[extend] crisis . . . as an unequal differential that law likely follows the Western world-system’s concomitantly institutionalized Color cum Developed/Underdeveloped Lines or Divides” (Wynter 2015, 236).

Because ecology resides outside of ChatGPT’s descriptive statement, the “what is to be done” solution, in triggering another recursive feedback loop, will be blind to the violence the algorithm produces. Bateson, on epistemological errors: “When you have an effective enough technology so that you can really act upon your epistemological errors and can create havoc in the world in which you live, then the error is lethal . . . you create around yourself a universe in which that error becomes immanent in monstrous changes of the universe that you have created and now try to live in” (2000c, 493). And all this consumption and destruction is achieved not through the quantitative or the application of number sense, but instead through the constructed cybernetic feedback loop that mutes its ecological foundation. The question of whether automating, extending, and accelerating our material-discursive genre of Man is desirable never enters the frame. The full extent of Western onto-epistemological violence is muted because it’s a violence done to differentiated subjects whose victimization can be justified through logic that posits that those people and spaces are outside of modernity. In the wake of World War II, cybernetics “present[ed] itself as the appropriate response to the Great Fear of the destruction of the world and of the human species” (Tiqqun 2020, 44). And yet now the ecological world is turning against the feedback loops of cybernetic “civilization” in its own recursive, self-preservative move (Routhier 2023), at a more rapid pace due to the increasing demand of data centers and infrastructures.

Notes

1. 1. Versions March 2023 and June 2023.

2. 2. Text tokens are not all treated the same. Token length can differ across languages, impacting cost of services, processing times, and discrepancies in quality (Petrov et al. 2023). Further, the direction by which strings are tokenized has a significant impact on performance (i.e., left to right or right to left; Singh and Strouse 2024).

3. 3. See Arkady Plotnitsky, this volume.

4. 4. Hortense J. Spillers (1987) defines the flesh as the first level of the body, whereas everybody starts as an instance of the flesh and then comes into the body as they gain sociality (Ferreira da Silva 2024). With the concept of the flesh, Spillers was naming “the scarring, the tactile properties of enslavement, the handling of bodies” (22), and identifying a contrast between precariousness and a less vulnerable position, where this position would be granted through law, philosophy, and ideology (Ferreira da Silva 2024).

References