The experiments documented in some of the NEST videos build on an augmented interface. This interfaces enmeshes the emergent hallucination mechanics embedded in Stable Diffusion models with real-time open world gameplay. This combination is aimed to unpack correlations between game worlds and how machine learning models interpret, augment and hallucinate based on the perception of virtual game worlds in real-time. This perception-hallucination loop is the basis of an experimental game-interface that was developed for NEST. It completely overwrites the visuals that is fed into, yet allows the real-time control of the game world. The effect of hallucination oscillates between the original input (i.e. the 3D graphics of the game world) and a complete hallucination. On one hand this oscillation allows for the shift in perception between a consistent, self-contained virtual world and the complete delamination of control and perception, and on the other the system converts the contents, figures and essentially every pixel of the origin world into a input data that can be transfigured by machine learning models, a sort of prompt for mining the black-box mechanics of the neural network. Through these hallucinations the patterns and glitches embedded in the models and the datasets surface, producing a recombination of fictional and real data, unfolding in real-time as zombie media that stitches together virtual worlds, datasets, machine pareidolia and text inputs.

Game Engines like Unity or Unreal Engine often serve the backend of contemporary industrial automation processes. They offer a virtual and dynamic environment that can produce be rendered photorealistic, providing a fertile ground for (re)creating infinite reproductions of products, objects and everything that can be reproduced virtually. Current object detection and recognition machine learning algorithms require a large amount of visual data in order to have a precise understanding of target objects, and to recognize or detect that object in any given environmental circumstances that can be found in the physical world, where these algorithms are often deployed as parts of automation and surveillance systems. The automation of navigation and driving in contemporary self-driving vehicles use similar principles to prepare the vehicle’s navigation and operation system for real-world scenarios. In order to cover most cases of driving on real roads the models these systems use are trained on millions of hours of hybrid data. This hybrid data combines real world driving footage scraped from the surface web, vehicle data captured during long driving sessions of hired and often underpaid workers using vehicles specifically designed for on-site training and virtual simulations. This combination in theory offers a robust database of how driving looks like, how the underlying principles of operating a vehicle can be generalized and translated for a self-driving system to be processed and inferred real-time. Once trained, these self-driving systems work on the principles of machine-perception of real-time visual and spatial data provided by on-board sensors, like multiple cameras and laser based range-sensors (eg.: LiDAR); and a combination of simulation and machine prediction. The perception, simulation and prediction goes hand in hand: the system is fed a constant spatio-visual data that is being processed simultaneously with a real-time simulation, that tries to predict future scenarios, potential situations, threats and accidents before they happen, then the system compares the corrections by the operator (driver) and events unfolding to the simulation and prediction, creating a constant verification feedback loop somewhere between the past, present and future. In NEST this spatio-temporal constellation is called as xenopresence, the presence of a synthetic timeline embedded in physical reality by a self-operating machine. Nora Khan recalls Aimee Roundtree’s reflection on simulation, that contextualizes synthetic data and its relation to evidence and real-world applications: “As Aimee Roundtree notes in her excellent book on simulations and rhetorical imagination, even though "simulation" and "model" are often used interchangeably, simulations are more than just models. The model—as of the body described above—has static features, but the simulation puts the model to work, running it through different hypothetical scenarios, with different driving variables and conditions. They apply and capture a models behaviors in motion. Further, the simulation has the appearance of evidence.…Simulations are the real-world activations of data to calculate and predict future action and movement: how a star will explode, how a hurricane will move. They are both literal products of equations that describe what actions can happen inside of a virtual world, and potent metaphors for future-casting. We simulate when we imagine ourselves and others in the future, and we base our current actions on that mental simulation. Simulations have extremely useful qualities which also make them very difficult to "critique" in the way we might critique a camera or a piece of software: they are not static. They have no end state. And they arent solely dependent on mathematical principles, but on a speculative theory about how people or things work or behave or think, which is applied to data.”

Pareidolia is a psychological phenomenon affecting perception, when a person see meaningful things in random patterns, visual or sonic. A simple example for this is seeing shapes, like animals in cloud formations in the sky. This phenomenon can occur in non-human in nonhuman perception and psychology as well. You can think of a smart phone’s camera app, that tries to figure out faces in the input images, and sometimes recognizes facial features in random patterns. One of the early, widely known examples of a pareidolic hallucination of computer vision algorithms was DeepDream, an algorithm created by Google engineer Alexander Mordvintsev that used a convolutional neural network to find, enhance and over exaggerate patterns in images, producing face like fractal patterns. Contemporary generative machine learning models tend to hallucinate data that is not part of their original training set or the training set’s quality is lacking in some way. They fill in the missing gaps with “false” or inaccurate data in order create consistency and novel outputs that match the prompt they are fed with. The hallucination and pareidolia of machine learning is not specific to generative models only, they are integral parts of any machine learning algorithm, from predictive-recognition driven to generative applications. They can falsely identify cancer cells in images of tissue samples, they hallucinate criminal activity before it even happens, they generate facts that are untethered from reality. NEST builds on this pareidolic perception and hallucinatory effect to unpack correlations between game worlds and how machine learning models interpret, augment and hallucinate based on the perception of these worlds in real-time. This perception-hallucination loop is the basis of an experimental game-interface that was developed for NEST. It completely overwrites the visuals that is fed into, yet allows the real-time control of the game world. The effect of hallucination oscillates between the original input (i.e. the 3D graphics of the game world) and a complete hallucination. On one hand this oscillation allows for the shift in perception between a consistent, self-contained virtual world and the complete delamination of control and perception, and on the other the system converts the contents, figures and essentially every pixel of the origin world into a input data that can be transfigured by machine learning models, a sort of prompt for mining the black-box mechanics of the neural network. Through these hallucinations the patterns and glitches embedded in the models and the datasets surface, producing a recombination of fictional and real data, unfolding in real-time as zombie media that stitches together virtual worlds, datasets, machine pareidolia and text inputs.

Games and play carry within a worldbreaking potential. Open world games for example present themselves through immersive environments, engaging action and stories, yet all their building blocks are part of their interface, they are gameworld interfaces as Kristine Jorgensen conceptualizes them in her 2013 book with the same name, "Gameworld Interfaces". They call for actions within their logic, yet their core control mechanics (i.e. navigating the world) allow for the exploration of the edges of their micro-reality. NEST centers these edges, the glitches and failures within game design and play. This is a mode of gameplay that shifts the attention from carrying out missions or following progression to exploring how the world works. This gameplay is built on extended periods of non-interaction, hijacking the first or third-person controllers for their static breathing loops that enframes the environment, durational non-movement and extended free-move camera scene recordings among others. These actions or non-actions often result in gameplay failures, in-game deaths or penalties by the core loop that creates a periphery of play. But on this periphery the worldmaking systems reveal themselves: the state machines of ecosystems, the clock of the core loop that drives the weather, the looping choreography of interface notifications and the emergent constellations of secondary and tertiary game-loops among others. Through non-interaction the game plays itself as a world in a loop of emergence, a controlled unbecoming of its own, the game-reality starts to deconstruct itself. This mode of gameplay perhaps, could be understand as Jack Halberstam puts it in an interview between him and Jesper Juul: “I think that what’s interesting when you enter the territory of unbecoming, you lose vocabulary, because the vocabulary available to us is much more an ac- tive doing/playing vocabulary, which makes sense given our political trajectories. So, when our political trajectory is to not become, or to not complete or to not gain, we only have these negative models. We don’t have the full suite of words that we might need to explain these other forms of human experience. That’s because they are counterintuitive. What I really appreciate about Jesper’s book and the conversation that it opens up is the idea of a counterintuitive mode of play- ing that takes us to a completely different level, where we’re not simply thinking about moving through the game, acquir- ing points, building strength, getting our health up, and com- pleting, we’re actually repeating, spinning, falling, failing, disappearing—all of these other things that offer another kind of pleasure.”

<- ->