7 Who Controls the Curriculum for AI?

Participatory AI as Attempts to Control AI Development

To enable the values and preferences of stakeholders to shape the design of AI, some have proposed “participatory” approaches to AI development (Birhane et al. 2022; Delgado et al. 2023), including for educational AI (e.g., Frauenberger et al. 2015; Holstein et al. 2019). Participatory design (PD) grew out of traditions of labor organizing and workers’ rights, with PD researchers emphasizing that technology design decisions were fundamentally political (Muller and Kuhn 1993). As such, PD theories and methods attempt to grapple with questions of power dynamics between and within stakeholder groups.

For AI systems, the power dynamics between AI developers and impacted communities are exacerbated by the current design paradigm of large-scale pre-trained models, where a small set of “pre-trained” models—sometimes called “foundation models” (Bommasani et al. 2021)—are deployed at scale across many geographic contexts, thus limiting the opportunity for any given community to shape the design of the pretrained model or its training data.

Political theorists have long grappled with how to understand and reconcile the values and desires of various groups in society, which may provide lessons for PD of educational AI. For instance, Laclau and Mouffe (1985) argue that, despite a popular imaginary of a singular notion of society or community, we must attend to the plurality of desires, identities, and antagonisms that are often lumped under a single rubric such as “workers’ struggles”—or families’ preferences for educational AI.

The scale at which AI systems are deployed may pose challenges for the use of PD for AI, in ways that may privilege hegemonic forces such as technology companies and majoritarian user groups over the desires of less powerful or more marginalized communities. For instance, PD methods (e.g., co-design of prototypes, focus groups) were originally intended to be used to design specific technologies for specific workplace contexts (Muller and Kuhn 1993).

To address this, some researchers and technology companies have proposed computational methods for extracting, aggregating, and synthesizing the values and preferences of communities at scale—sometimes referred to as “value alignment” (Gabriel 2020). Such computational methods to elicit and aggregate preferences for AI output at scale draw on traditions of preference aggregation from social choice theory, which have been critiqued by political theorists, as they eliminate any “encounter with others whose preferences may differ” (Young 2002). Moreover, these computational methods narrowly constrain participation to predetermined areas of focus (e.g., the output of models, rather than their training data or their purpose or design), in ways that depoliticize inherently political design decisions and manufacture consensus on algorithmic terms (Delgado et al. 2023).

A complementary approach to responding to technology is refusal or resistance (Hirschman 1970; Zong 2020; Slot and Opree 2021). Indeed, the history of education technology can be seen as a history of technology refusal, as students (Lubar 1992) and teachers (Hodas 1996; Woods 1996) have resisted education technologies they see as emblematic of dehumanizing or deprofessionalizing forces, tactics that may be useful for educational AI.

Training Data as the “Curriculum” of Educational AI

If we take seriously the metaphor of machine learning (ML) as learning, then one can think of the training data as the curriculum for AI—indeed, a popular method for training models is known as “curriculum learning.” When AI is used in education, this “curriculum” shapes both the curricular content that AI models generate (e.g., for personalized tutoring) and the implicit values of their hidden curriculum (cf. Rosiek and Kinslow 2015).

In contrast with previous paradigms of AI, where custom datasets were used to train individual ML models, such as for intelligent tutoring systems (Kulik and Fletcher 2016), the current paradigm of foundation models involves collecting massive amounts of data from the internet writ large (Dodge et al. 2021) to “pre-train” (i.e., the “P” in GPT) a single large model.

However, models trained via data from one context (e.g., a school district or state) may be deployed in a different context despite the unsuitability of the training data or model for that context. In education, there are key differences across schools, districts, and states in educational standards and policies, learners’ needs, and ways of measuring learning outcomes (Baker and Hawn 2022), which suggests that foundation models are likely to homogenize educational curricula across contexts.

As a result, given the likelihood for LLMs to reproduce both the specific content and the values implicit in their training data, the question of what, precisely is in that training data is of crucial importance for educators, learners, their families, education researchers, and education policymakers. Unfortunately, despite calls for increased transparency of AI training data (Liao and Vaughan 2023), the training data for large language models and other large-scale generative AI systems are a closely guarded trade secret, making it difficult for the public to provide input.

Some have argued that so-called foundation models should be used for education by training them on data from massive open online courses and programming feedback forums like StackOverflow (Bommasani et al. 2021). However, although such content may be more appropriate for education than, for instance, training on Reddit data (Dodge et al. 2021), those choices about training data are likely to smuggle with them the pedagogical values implicit in the data sources (Blodgett and Madaio 2021). More recently, the UK Department of Education has proposed £3 million to develop a “content store” for private sector companies to train AI models on a centralized repository of curriculum guidance, lesson plans, and assessments, but with apparently no means for public input into the content that will comprise this repository (Cumiskey 2024).

If training data is the curriculum that shapes the output of a trained AI model—and the explicit and hidden curriculum for learners who learn through interactions with an educational AI application (e.g., a chatbot “tutor”)—what might we learn about the current calls for participation in AI design from the history of struggles to shape educational curricula?

Control over AI Design as Political Contestation

Educational historian Campbell Scribner has written about the history of contests over local control of educational curricula throughout the twentieth century—conflicts between local communities and district, state, and national governments (Scribner 2016). In his telling, “one cannot understand U.S. public education apart from local control,” a movement that he argues was animated by nostalgia for a lost memory of the one-room schoolhouse where parents could allegedly observe and control their children’s education. For Scribner, previous book ban movements were driven by a “desire to reassert control over children’s learning” away from teachers and school boards (Scribner 2016).

This desire for control over education was often a rhetorical pretext for racial biases: as the “romanticized view of local wisdom” and the “allure of individual choice” were often “a mask for racial and class bias” (Scribner 2016). Similarly, the resegregation of public schools in the 1990s was motivated by “rhetoric about returning governance of schools to local control” (i.e., lifting desegregation orders from the Department of Justice) (Rosiek and Kinslow 2015).

This history of contests over local control of educational curricula thus prefigures the struggle for control over AI design that participatory design of AI is running headlong into. As generative AI becomes increasingly centralized in a small number of pretrained foundation models, the vision for the personalized education of the future resurrects the one-room schoolhouse in the form of a single foundation model—allegedly infinitely fine-tuneable and customizable for downstream applications but that instead enacts hegemony by design.

In education, the school board meeting and town hall are the modes of participation de jure in educational policy decisions, and much has been written about their susceptibility to co-optation by political groups (Kerr 1964; Tracy and Durfy 2007; Sampson and Bertrand 2022), including how they privilege people with more free time to attend or power within their communities, such that their voices are heard over others—that is, “representation by loudmouths” (Young 2002). Political theorists have argued that such forums for deliberative democracy rely on shared premises and goals, which may not be shared across different groups within the same school district, and which privilege those who are able to deftly articulate their desires in arguments (Young 2002). As a result, educational inequities may compound, exacerbated by the resegregation of educational systems (Rosiek and Kinslow 2015), “limit(ing) equal participation in civil society” (Young 2002).

If, as political theorists argue, the “public sphere (is a) site of contentious struggle” (Young 2002), how might modes of participation for designing educational AI make space for such contestation, while grappling with the limits of inclusion as a solution to the challenges of pluralistic democracy? The modes of participation for AI design are still being developed—now is the time to treat AI as a “public problem” (Ananny 2024) and work toward more democratic control over the curriculum of educational AI, including both its training data and model output.

References