23 Digital Technologies and Perceptual Curation

The Embodiment Shift

Our understanding of learning as an embodied phenomenon has developed rapidly in recent years. After many decades of psychological research that treated knowledge as a symbolic transformation, confined by the computational limitations of processing speed and working memory, scholarship has shifted attention back to the corporeal. Theories of embodied learning ask how we make meaning through our body-based systems of perception and our ability to move and act in our physical environment (Abrahamson and Lindgren 2022; Kontra et al. 2012; Nathan 2021; Nemirovsky et al. 2020; Stolz 2015). This perspective sees embodied learning in the ways that children’s gestures and their natural interactions with physical objects shape their understanding of mathematics or other knowledge (e.g., Congdon and Goldin-Meadow 2021). More generally, the focus on embodied learning allows us to study how both learners and expert practitioners use imagistic and motor-driven ways to make sense of texts and tasks (e.g., Glenberg et al. 2004; Lakoff and Nunez 2000; Sadoski 2018). But most notably in this era of technology-enhanced learning, we see this approach used in the design of environments that shape the sensory experiences of learners, making certain relationships salient or guiding someone down a path of discovery.

This new paradigm of embodied learning is more characteristic of a design science (Simon 2019) than a traditional psychological science; with every new digital platform designed for bodily engagement, we learn more about how the physical environment can shape the development of new understandings in specific knowledge domains. This learning occurs not by what the designed environment explicitly tells us, but rather by what it leads us to see, to hear, and to feel. Embodied learning is a field that seeks to understand how the selection and ordering of sensory experiences will lead us to make productive inferences and to make predictions about future events. Rather than a narrow focus on improving the efficacy of declarative knowledge acquisition, designing for embodied learning seeks to cultivate inferences and insights through curated activity and perceptual engagement.

The notion of curating perceptual experiences has weighty implications for what it means to learn under algorithmic conditions, because it directly implicates what these designed algorithms are meant to do. Are they meant to supply and bolster our cognitive system with robust and persistent knowledge structures that we can carry with us from context to context? Or might the algorithms instead be tasked with managing the intake of perceptual experiences such that they create the formative conditions of knowledge construction? Although the latter may seem more consistent with the Deweyan constructivism that most educational designers today purport to adhere to, the mainstream design approach to learning technologies seems to favor the former. My interest in this memo is in how we might start thinking concretely about using sensors and machine learning technologies differently for designing, modifying, and inventing environments that shape our experiences and guide our processes of learning.

Origins of Perceptual Curation

While it has not typically guided our pedagogies or our educational designs, the idea of learning as maturation of our perceptual systems is not entirely new. Gibson (1966) describes perceptual learning as the “increase in specificity of discrimination of the stimulus input” and she describes how this refinement occurs through experience, particularly experiences designed to make certain features of the problem space salient. For example, she described how a mathematics teacher might draw a grid within a rectangle and divide it into unit squares to show how its area can be calculated, and to illustrate why the same approach applies to a parallelogram. In this approach, understanding how to calculate the area of a rectangle means being led to “see” a rectangle in terms of its composite unit squares. Goodwin (1994) applies this idea of perceptual learning to describe how people learn in the context of their professional training, such as the student archaeologist being guided through the visual landscape of an excavation site by an expert archaeologist, helping them see the important distinctions and patterns in the dirt. Importantly, it is also a process of learning what not to see, what to ignore so as not to obscure the professional significance of the scene.

The notion of being guided through a complex visual scene as a form of instruction may be a sufficiently palatable proposition, but the idea of being led through a tactile “scene” might be harder to conceptualize. And yet, when we consider how practitioners ranging from auto mechanics to dental hygienists likely learn their trades through carefully configured physical contexts that invite specific actions (and implicitly discourage others), it becomes clear that perceptual curation occurs across all forms of perceptual learning. Embodied cueing (Lindgren 2015), for instance, is a powerful form of perceptual curation that elicits actions with a high probability of cultivating critical knowledge. And this is true not only for learning physical trades; there is also evidence that cueing learners to perform select actions such as gestures can give rise to conceptual insights (e.g., Novack and Goldin-Meadow 2015; Walkington et al. 2022).

To effectively and ethically curate perceptual experiences that lead to embodied learning, however, will require a mix of pedagogical precision and openness to diverse forms of embodiment. The challenge is particularly pronounced when teaching formal physical “laws” in science that may be true for all bodies but are often not clearly perceptible through direct interaction with objects. For example, one could imagine presenting secondary school children with opportunities to interact with moving objects of different sizes and velocities such that they could observe, and develop an intuitive sense of, Newton’s second law (F=ma). But given the peculiarities of the human perceptual system (e.g., nonlinearity, attentional biases, etc.) it would be challenging, if not impossible, for an instructor to deliver the adequate number and sufficient variation in sensory experiences for students to make these robust inferences. This is where algorithms and computational environments, including AI, have the potential to bring great value. A haptic device that is configured with a sensitivity to human perception could potentially deliver a sequence of simulated interactions with moving objects that allow students to “feel” the relationship captured in the equation F=ma. Further, these technologies would have the ability to strategically layer in other factors such as friction or air resistance, which tend to impede our ability to sense these core relationships in the real world.

Example Case: Perceptual Curation for Understanding Dynamic Equilibrium

One of the many types of scientific phenomena that are difficult to fully comprehend based on everyday experience are those phenomena that exist in a state of dynamic equilibrium, where the forward and reverse processes are occurring at the same rate such that the macro phenomena appear to be in a state of inactivity. How does one learn that both stasis and mobility are at play and codependent in such situations? In dynamic equilibrium there is activity, and forces at play, and in order to effectively explain many physical and biological phenomena, one has to acknowledge the multiple forces present. One such system of dynamic equilibrium is the resting membrane potential caused by the movement of ions across the permeable membrane of a biological cell. Movement of the ions is governed by two distinct forces: a chemical concentration gradient and an electrical gradient. The ways that these two forces interact leading to a stable state in a resting cell is described by a formalism called the Nernst equation. But understanding this formalism is difficult if students do not have any intuitions about what exactly is happening at the membrane, and specifically what forces are acting on an individual ion. Lira (2020) describes moderately successful attempts to guide students’ attention within a visual simulation of ion movement between the intracellular and extracellular fluid and across the membrane. However, it is not easy to “see” both forces in a visual simulation, and so students still struggle to distinguish the different factors and map them to the Nernst equation. One might interpret this difficulty in terms of perceptual distraction, and the fact that the perceptual field is still too noisy for students to parse the simulation and explain the phenomenon.

I am part of a project team that is attempting to improve comprehension of the simulation by offloading perceptual information pertaining to one of the forces (the electrical gradient) to the haptic channel and creating dedicated and focused feedback for each factor. We created an updated version of the cell membrane potential simulation in Unity (Figure 23.1, top) so that we could easily connect it to peripheral devices such as a haptic glove. In this version of the simulation a user plays the role of a single ion that starts out in the extracellular fluid but can traverse the cell membrane. The user’s hand embodies the ion and a tracking device (LeapMotion) communicates to the simulation where the user’s hand is located in space in front of the simulation. We built a custom haptic glove (Figure 23.1, bottom) that can deliver precise haptic feedback, a vibration with variable amplitude, corresponding to the electrical force that an ion would receive based on its position within the system. This allows the user to feel and potentially understand, through the haptic feedback, which forces are acting on the ion, even if those forces are not visually apparent. While empirical validation of the glove’s impact on student reasoning is still in progress, the design of the glove demonstrates a novel use for haptic feedback, cueing students to move in ways that are consistent with scientific principles such as dynamic equilibrium. Rather than having all visual and motion pathways open to them, students are guided to see and feel the activity from the perspective of the simulation components. Their perceptual experience is curated in ways that potentially lead to noticing key relationships with deeper understanding.

Perceptual Curation and the Cybernetic Image of Learning

The use of a haptic glove to distribute and focus the feedback from an interactive simulation is just one example of how emerging technologies can shape the perceptual experiences of students in ways that potentially benefit learning; this example is part of an expansive possibility space for customizing perceptual experiences to specific topics and learning contexts. Numerous instances of stimuli that vary in precise ways are sometimes needed to convey complex and nuanced relationships, and AI can potentially be configured to create these stimuli for learners. The current availability of AI image generators such as DALL·E or Midjourney suggest that the ability to create rich and vivid stimuli meeting very specific requirements is a current reality, and these capabilities may be transferable to tactile experiences in the near future. The potential of combining AI tactile and AI visual feedback in a single augmented reality or virtual reality experience is particularly compelling, given the potential for creating immersive environments designed to convey specific ideas and elicit understanding.

However, the potential for curating high fidelity perceptual experiences for learners also creates ethical concerns that should not be taken lightly. First, the use of AI to create vivid sensory experiences that meet the requirements of precision and depth in augmented and virtual simulations of reality raises the question about “whose” reality is being simulated. We have long accepted simulated realities (e.g., television) as part of our media consumption, but with haptic technologies this consumption moves off of screens and onto our bodies, which are personal, political, and contested sites (Platoni 2015). Second, with the expanding possibility space of algorithmic intervention, it has been argued that young people in particular are under a siege of digital “ubiquitous sensation” (de Freitas and Rousell 2021). Young people are growing up within a vastly expanded sensory ecology, as sensor technologies are increasingly embedded in environments and students traffic in various kinds of wearable technology. These digital sensors and AI technologies operate beneath the timescale and sensory threshold of the human, meaning there is the potential for them to destabilize individual embodiment and agentic learning. As we move forward in exploring the potential of perceptual curation, we must do so with caution (see Pedersen and Illiadis [2020] for a robust exploration of pertinent ethical issues). While there may be profound benefits for steering attention and reducing noise, it is critical that we maintain awareness of how our perceptions are being guided. Just as we voluntarily pick up a work of fiction or walk into a theater ready to have our senses co-opted for our own entertainment, we must also consent to having our senses orchestrated, even for the virtuous purpose of learning.

References