The Center: A Case Study of Data Science in Education

The Center was established in 2012 and has three main responsibilities:

1. provide data analysis, information and evaluation that improve effectiveness, efficiency and accountability of education programs and strategies;
2. collect essential education data and provide a one-stop shop for information—a single access point to education data that has appropriate safeguards to protect data confidentiality and integrity; and
3. build capacity across the whole education sector so that everyone can make better use of data and evidence.6

Within the state education department that hosts it, the Center is the primary unit for data management and analysis, with most of the statistical and analytical functions from other units in the department now falling under the Center’s remit. When we conducted the fieldwork on which this chapter is based (in 2016–2017), the Center was increasingly providing policy direction to the state education minister, as well as providing analysis and evaluation for stakeholders within the system, such as principals and teachers.7 In the analysis that follows, we draw directly on interviews with five senior policy makers, technical staff, and data scientists involved with developing and implementing the Center’s BI strategy, in addition to analyzing publicly available documents, including the BI strategy, from the Center’s website. To protect the anonymity and confidentiality of the Center and its staff, we have not provided the source for these documents and have used pseudonyms for the organization and our participants.

Since 2013, the Center has been implementing a BI strategy to support strategic decision-making and the improvement of student outcomes (Figure 5). The strategy outlines the Center’s aim to become “information centric” through “significant growth both in . . . [the Center’s] usage of information and understanding of what we can do with it.” Moving from its previous “data centric” approach to become “information centric” involves moving from accessing and analyzing data for specific purposes (e.g., generating reports on student assessment outcomes) to embedding data analytics in all organizational decision-making, with a particular focus upon improving learning outcomes. An “information maturity roadmap,” published in the strategy document, outlines the Center’s plan to develop predictive analytics and modeling capabilities by 2019, before moving toward optimization, at which point data analytics will ideally enable maximally efficient and effective education (Figure 5).

To support the analytic and policy functions of the roadmap, the Center was in the process of introducing new BI capabilities: a data warehouse, cloud computing platforms, and user interfaces, or dashboards, that enable easy analysis and data visualization. These BI capabilities were designed to support analyses of big data sets and embed near real-time data analytics in the day-to-day operations of the department. At the time of our fieldwork, the Center had hired two data scientists to undertake exploratory data analyses, along with other staff with information science backgrounds. While the Center was still in the early stages of developing these capacities, its ambition and progress arguably have positioned it as a leader in this area among Australian education departments and potentially among education systems globally. It is, thus, an ideal site for an exploratory case study.

When we first visited the Center in 2016, we were surprised by the ambition of the BI strategy represented in Figure 5. The strategy document explains that

the implementation of the phase 1 strategy moved us up the maturity curve. Our usage of information has started to have a greater impact, and we are increasingly focused on what information we need. We are now almost on the border between tactical use and focused delivery, still well within the reactive usage of information. The phase 1 strategy was data centric, and put in place the foundational blocks that allowed us to mature. This strategy however, is information centric. It is intended to support us in moving further up the maturity curve, to where we begin to look to the future.8

In 2015, the Center commenced phase 2, with a new focus on moving “from being data centric to being information centric,” premised on “the recognition that Business Intelligence is a tool to assist us in making complex decisions, and not just a data access point.” Phase 2 aims to change decisions and actions to improve learning outcomes. The strategy describes thirty initiatives that were supported by BI from 2016 to 2020, and in what follows, we focus on two of the most “mature” projects at the time. These projects demonstrate the emergence of new data-driven modes of reasoning in the Center’s work, and we are interested in theorizing the potential implications of the strategic developments that were planned for 2019 onward: predictive analytics, predictive modeling, and the goal of optimization.

Data Dashboards: Connecting Analytics with Action

The outcomes of the analyses performed in the two projects above need to be made accessible to policy makers, educational leaders, and other stakeholders to shape decision-making. Translating analysis into real impacts on learning is a key feature of the “information maturity” that the Center is pursuing. Data dashboards are thus a key point of mediation between the work of the Center and its use in policy, schools, and classrooms.

Dashboards enable “the creation of visual analytics to direct organizational behavior” by translating information into formats that enable decision-making and action.10 As Kitchen, Lauriault, and McArdle suggest,

Dashboards act as cognitive tools that improve the user’s “span of control” over a large repository of voluminous, varied and quickly transitioning data . . . enabling domains to be explored and interpreted in an easily digestible and intuitive way without the need for specialist analytics skills (the systems are point and click and require no knowledge of how to produce such graphics or coding).11

Dashboards can acclimatize users to representations and uses of data, and the uncertainty of the analysis underpinning these representations can be hidden or “black-boxed.” As a representation of new forms of knowledge generation, including but not exclusively associated with data science approaches, dashboards are also the outcome of political decisions—for example, policies that require school leaders to access and utilize forms of data. Williamson posits that dashboards “structure the user’s interaction with . . . data to facilitate social action.”12 In education, these dashboards vary in sophistication, from basic tabular representations with minimal real-time updates, to the capacity to bring together multiple data sources to produce reports powered by machine learning, as in the case of the Center.

For those managing the Center’s information strategy, dashboards are a way of representing the large volumes and diverse varieties of data in the data warehouse to an increasing user base. As the policy lead for the Center explained,

The idea being that we could improve the capability of our people and give them access to data in ways that enable them to become more analytical about what they know and understand, and can do. . . . So it made data available through these tools, but we want to expand the access, so we will make it outward facing. . . . Where it’s been school leaders, corporate staff, to date, it will become available for teachers, nongovernment-sector users, and others. And we are building the capability of some highly expert users to start to think about how to mine the information that is accessible there, so that they are making the best use, joining up the data sets available to tell you stories about what we can know.

The data scientist explained how capability is connected to the user-friendliness of dashboards, to encourage engagement with the Center’s data:

I think most people get it. Maybe because it’s presented to them in more of a BI context. . . . That’s all basically built using stuff from the data store. It’s something that people can immediately see as being an advantage to them, because they have a nice dashboard, and they can see how many schools have this, that, or the other, you know, or what proportion they have. They can ask quick and simple questions, and they can sort of see—they have some sense of what the underlying data is. I think there’s a general feeling that, you know, data is a good thing, and a useful thing.

The new data management systems established by the Center are connecting users through dashboards and reports, potentially increasing the desire for, and trust in, the data and analyses offered by the Center. In the next section, we consider how new algorithmic rationalities, imported from the world of data science, are shaping the Center’s analytics work and the information presented to users.

The Experimental Nature of Data Science

The Center aims to use data science to provide diagnostic and formative feedback that differs from school-level capabilities. As outlined in chapter 1, data science is “the study of the generalizable extraction of knowledge from data” with a focus on “the computational aspects of pragmatically carrying out data analysis, including acquisition, management, and analysis of a wide variety of data.”13 There is overlap between AI and data science in the areas of data modeling, which includes computational statistics and high-performance algorithms, and data science also involves exploration and indeterminacy. In this section, we discuss the uncertainty inherent in the algorithmic rationalities underpinning data science, and begin to exemplify, empirically, theoretical perspectives on the exploratory nature of machine learning.

Education data science differs from previous statistical approaches due to the speed and scope of analysis made possible by new computing capabilities and the development of fields like machine learning. The introduction of data science approaches in the Center’s work marks a shift in the types of expertise brought to bear in education policy and governance in this context. The move to data science and computational approaches thus reflects a shift in the professional knowledge that shapes education governance, from statistics and the professional knowledge of educators to algorithmic rationalities. Williamson contends that data science expertise and practices “reflect a particular data scientific style of thinking that views learning in scientific terms as quantifiable, measurable, actionable and therefore optimizable.”14 Units like the Center employ analysts that have come from fields such as bioinformatics who then constitute new knowledge communities made up of “data scientists” and “learning analysts” who shape data production and management.15

Developments in the Center illustrate the early stages of this process through which the data science and its speculative rationalities are beginning to gain authority. During our second visit to the Center, in 2017, it was clear that data science approaches are becoming increasingly central. The data scientist explained the nature of this work as follows:

And so they say, “Here’s some data.” Sometimes you have to go off and get it yourself from the data store; other times someone will supply a file, a CSV file that they’ve made. Then you sit down with it and you kind of explore it, so you understand how it’s structured and what the contents are, and how you’re going to aggregate it, and summarise it, and then you spend a bit of time thinking about the explorations you’re going to do, or the statistical methods you’re going to do.

The BI manager also conveys the exploratory nature of this work, describing how data science is being applied to analyze nonstudent factors that may affect student outcomes. This work is beginning to drive policy:

We started doing some . . . data science investigation work on the impact of [school] principals on student outcomes, without really knowing what we were going to find, or what to expect. So, OK, here’s the data warehouse, here’s all this information, here’s a data scientist, have a great time. . . . So data science is kind of like walking around a room in the dark, bumping into furniture. Oh, look! There’s some gold! And oh, look! There’s the mud. You never know what you’re going to find. So we have two data scientists doing that at the moment. They’re about to start a piece of work on the usage of . . . ICT in schools, and again, we have terabytes of data and no idea where we’re going to go or what we’re going to find. . . . That kind of thing will drive policy changes.

This approach of “walking around a room in the dark” is common to some areas of data science and machine learning, where substantive domain knowledge is not necessarily a prerequisite for analysis. As the examples of the school population simulations and NAPLAN prediction projects show, these techniques can be applied to a range of areas, from planning to performance data, and their perceived value (“gold” and “mud”) is also premised on their inherent uncertainty (e.g., “no idea where we’re going to go or what we’re going to find”).

The uncertainty of data science explorations challenges claims evidence-based policy-making is based on causal accounts that inform decision-making.16 But we suggest that this lack of certainty does not hinder its possible usefulness. Parisi provides another way of thinking about the introduction of data science and AI in education, based on exploring data using abductive logic:

Abductive reasoning does not involve a probability calculation of the best explanation to the best hypotheses because, differently from inductive thinking, abduction does not rely on already-established hypotheses and observed discrete facts. . . . Abductive reasoning is ignorance preserving, which means that it entails the emergence of a new order, that is, new hypotheses and beliefs, or conceptual changes.17

Indeterminacy is inherent in the description of the Center’s data science work (e.g., “walking around in the dark, bumping into furniture”). As Parisi observes, “Indeterminacy is . . . intrinsic to the algorithmic generation of hypothesis and as such the technoscientific articulation of truths and facts can no longer be confined to recurring functions and executions of the already known.”18 Rather than a causal claim, this perspective on data science has an affinity with critiques of policy science and the identification of policy-making and implementation as unavoidably ad hoc, contingent, and uncertain.19

The data science work involving simulations, explorations, and predictive analytics is beginning to affect the Center’s “cognitive infrastructure” of policy advice and educational decision-making. For example, the NAPLAN project exemplifies an embryonic form of automation and constitutes an early example of “the emergence of levels and meta-levels of inference that have radically changed our methods of reasoning.”20 NAPLAN is often criticized for only providing a snapshot of student performance, and it may be an ineffective predictor of the future academic performance of individuals. From this perspective, the project can be critiqued on grounds that the predictions may well be incorrect, but such critiques focus on the validity of the analysis rather than its exploratory aspect. While analyses conducted in the Center can be interpreted as a pursuit of optimization, and this goal is obviously identified explicitly in the roadmap (Figure 5), the data scientist and BI manager give emphasis to another, more uncertain side of their work. In these accounts, we catch a glimpse of how creative inferences can be made from multiple data sources, as part of a broader “‘operating platform’ for human thinking,” potentially opening new possibilities for decision-making and understanding policy problems.21

Becoming Information-Centric: The Uncertainty of Prediction

The new types of knowledge and rationalities data science introduces into education reflect desires to create certainty through accurate prediction and anticipation. Adams, Murphy, and Clarke posit that

tied to optimization, anticipatory regimes create spaces for “ratcheting up” our technologies, economies and politics in response to our urgent need to be prepared. . . . These leverage new spaces of opportunity and reconfigure our sense of “the possible.”22

The capacity of the Center’s BI system to bring together new volumes and varieties of data makes this promise of prediction seem eminently possible, if not achievable, as it is formulated as a technical problem. However, the Center demonstrates both the potential and the problems with this ideal when it is pursued in practice. For example, the BI manager for the Center highlights the limitations with current predictive capacities and expresses skepticism about the potential to totally replace other governing rationalities:

You can never predict it that well. Your models will never be perfectly aligned . . . so data science can do some things. It can’t do a lot of other things, but invariably you’re always trying to do it to change something, so you say, “Well, this is what’s happening right now.” . . . It’s always about changing, or trying to influence the prediction.

As the BI manager notes, prediction is always uncertain and to be augmented by other knowledge when making decisions and acting. Limitations on the predictive capabilities of the Center can be attributed to the field of enquiry; education is a notoriously complex practice, and there is a risk of predictions creating self-fulfilling prophecies with adverse effects. As Zeide contends,

Long term predictions are particularly problematic in education spaces, which are explicitly environments dedicated to student development. Learning, and life trajectories, are rarely linear. Predictive analytics cannot “literally predict [student] life outcomes” because they cannot incorporate the impact of outside circumstances and student agency.23

To reiterate an axiomatic point, predictions are probabilistic; they are correlative, not causal. And yet correlation easily becomes conflated with causation, in a political if not technical sense, when education systems rely heavily on measuring student outcomes and attempting to link these outcomes with causal factors.24

Overconfidence in predictive analytics arises when policy makers demand too much from data and do not, or choose not to, hear data scientists’ explanations of the limitations on their methods and expertise. For example, one of the Center’s data scientists explained,

Sometimes you start, say, trying to predict something and you discover that you can make predictions, but they’re not great. They have sort of huge—the confidence intervals are really wide, or there’s a lot of errors, or something. Then you’ll take that back to . . . [a decision maker] and say, “This is the best I can do with this data.” And there will be a bit of back and forth, because sometimes people hope things will be done which can’t really be done! So I guess a lot of the job [is] trying to explain. . . . I have to put into fairly plain words, “This is what I’ve done, and this is as good as it gets.” That’s all I can do.

Therefore, the limits of prediction, which are clear to data scientists, can run up against political desires for prediction. There is a risk these desires will encourage automation bias in education, according to which “we value automated information more highly than our own experiences.”25

The limits of prediction are also clear in the goal of optimization on the Center’s roadmap. The data scientist explained that the Center’s current work on “predictive analytics . . . is ultimately going towards optimization.” Optimization is the goal located at the top end of the maturity curve in Figure 5, and the BI manager defines this as being the point at which data analysis becomes wholly future-oriented, with a direct impact on the improvement of outcomes. The BI manager describes it as an aspirational goal, essentially capturing the desire to use data to maximize outputs relative to inputs. When asked to elaborate on the organizational understanding of optimization, the BI manager stressed the obstacles to reaching this point:

We have this ongoing argument with State Treasury . . . because they try to have a cost benefit model, and we’re like, OK, fine. How are you going to do it? . . . They’ve gotten pretty close to saying, if you complete years 11 and 12, what that means is that society will be better off because you can then do the following things, which means you’ll earn more in your career, which means you can pay more tax, which means more tunnels, or whatever. Alright? But there’s so many things that feed into it that they can’t control. It’s like trying to be an economist, they have this great thing called ceteris paribus that, holding everything else constant, if I move this lever, that lever moves. Yes, but what you’re holding constant is this enormous mountain of stuff that you have no control over at all. So you just try and have the best teachers and staff you can, in the right schools. And identify where it’s not working, and fix it.

The Center is using large volumes of data, and increasing the speed and impact of analysis, while acknowledging these analyses are embedded in contexts that cannot be controlled and are thus inherently unpredictable. The BI manager emphasized the nature of education as an unruly enterprise. As such, the abstract optimizations suggested by data science and associated techniques like machine learning, if implemented as policy, will interact in complex and unpredictable ways with the multiplicity of forces at work in any policy context.

What is determined to be optimal in simulations and predictive analyses will never be optimal in practice given the messiness of the contexts in which data-driven policy is implemented. Moreover, the operational definition of “optimal” depends on values, or what is considered to be “good,” as the BI manager explained:

I once asked someone who’d been here for a long time how you know if you’re getting better at education, and they said, “Well, first you have to define what ‘good education’ is,” and no one seems able to do that, because it’s different for everybody.

The BI manager expressed doubt about algorithmic promises “that the future is resolvable through the optimized output of algorithmic decision engines.”26 This skepticism creates another point where the possibility of things unfolding differently is kept open. Introducing data science techniques and early experiments in automation into the Center, thus, appears to be epistemologically indeterminate, insofar as they involve abductive logics as Parisi argues, and due to the limitations of data science and the impossibility of determining what “optimal” education would be for all learners. Nevertheless, as the Center tries to move toward its goal of optimization, the growing use of data science and AI—in conjunction with other rationalities and policy knowledge—may create new conditions for conceiving of and responding to policy problems.

Toward Synthetic Governance

Our exploratory case study of the Center shows how synthetic thinking is only beginning to emerge in its work, and that AI is yet to be deployed outside of capabilities embedded in its proprietary BI platform. However, the Center’s early forays into data science illustrate an emergent “coevolutionary cognitive infrastructure” informed by algorithmic rationality and a degree of indeterminacy that Parisi argues is also evident in machine learning algorithms. Algorithmic rationalities are likely to be added to other rationalities in education policy, intensifying some and subjugating others, rather than simply replacing them. As such, we construct the problem of which types of knowledge and expertise should inform education governance in terms of both/and, rather than either/or. We want to remain open to the possibility that new modes of data analytics can potentially open new perspectives on governing. There seems little point in the Center’s efforts to develop educational data science and pursue information maturity if it will only ever match what educators already know. If it always matches what was already known, then “data science” and “machine learning” become rhetorical devices only. There must also be a desire for data science to tell us something different from, or more than, what we could find out through other means. The desire at the heart of these developments is the same as that which has driven the development of AlphaGo and led to the originality of move 37. While we have not yet seen deep neural network technologies being integrated into education governance, the roadmaps toward this point are clearly laid out.

As Deleuze emphasizes, understanding and engaging with the mechanisms of control societies requires the creation of new concepts and strategies, and thus new values and desires.27 The role of desire and creative uncertainty in synthetic thought, which emerges when algorithmic rationalities and machine learning are integrated with other modes of thinking, is particularly important to consider when studying data science approaches in education policy. Rizvi and Lingard remind us that

in education, policy processes have to juggle a range of values . . . often simultaneously, against a calculation about the conditions of possibility. . . . This requires privileging some values ahead of others. In the process, the meaning and significance assigned to each value is re-articulated.28

The synthetic thinking we argue is emerging in the Center will shape the juggling, privileging, and rearticulation of values. The developments contributing to this emergence can be critiqued based on present values by, for example, suggesting that the Center’s work delegitimizes school- and classroom-based professional knowledge. Such critiques can highlight the partiality and instrumentality of performance data. Additionally, just as we have seen with the use of AI in search engines such as Google, biases in existing data sets and algorithms can produce biased and perverse outcomes (see chapter 5).29 There is a critical literature that raises concerns about the negative effects of datafication and associated technological change on education governance, as well as teaching and learning in classrooms. We acknowledge the importance of these critiques; however, we contend that focusing solely upon the instrumentality of machine cognition can occlude other potentialities inherent in these developments. Education is a field in dire need of new possibilities and new values, as extensive research on the pernicious and inequitable consequences of education policies and institutions for different groups has shown. Indeed, there may be a good argument, as Ball suggests, “for being against rather than for education.”30

We cannot predict what new policy desires and values might emerge from the introduction of AI into education governance, just as it was not possible to anticipate AlphaGo’s original moves. However, we can predict that we will learn to make policy differently and therefore policy analysis will change, in the sense that our values, desires, and expectations will change, because of integrating AI into policy analysis and creating new modes of synthetic thought and governance. Synthetic thinking may create necessary new conditions for contesting how education policy should be made and for what purposes. Rather than only conceiving of data science and machine learning as an instrument for determining the truth, through inductive analyses of large data sets that are more or less valid, we can also conceive of this work as a speculative method for creating new values, desires, and conditions for thought. As Parisi argues,

If the antagonism between automation and philosophy is predicated on the instrumental use of thinking, techno-philosophy should instead suggest not an opposition, but a parallel articulation of philosophies of machines contributing to the reinvention of worlds, truths, and facts that exist and can change.31

The syncopation of machine cognition and human thought will reconfigure meaning and significance. This chapter has only begun to discern some of the possibilities of synthetic thinking. The optimizations pursued by the Center will always find their way into connection with the messy and contested worlds in which policy is implemented. All forms of analysis create arbitrary neatness from uncertainty, and this is their purpose, as Gregory Bateson’s famous line about maps and territories conveys so clearly.32 Policy analysis and policy texts are no different; both are forms of abstract representation that collapse into “ad hocery” when implemented.

We see a need for methodological developments and new research designs that enable more pragmatic and collaborative explorations of what new cognitive infrastructures can do. This experimentation must of course be accompanied by critical questions about the values that are being replaced or changed, the interests driving the development and use of new platforms and data science methodologies, and the risks of these approaches for teachers, students, and societies. What we have tried to do across this book is offer conceptual tools that can help us to see how new data analytics can both provide reductive abstractions and actively add to the creative uncertainty of governance. This creativity is due not only to developments in data science and deep learning, but also in the coevolutionary cognitive infrastructure in which these algorithms are embedded. As Parisi argues, “Today it is crucial to develop a philosophy of computation accounting for the transformations of logic, and the emergence of a social artificial intelligence, in which the nonconscious function of automated algorithms is just one dimension.”33 The point is not simply that synthetic thinking and governance involves uncertainty, but that this uncertainty may increasingly enable education policy makers to create new values and possibilities for, and desires about, acting upon the world.