The Game Loop
Part 1
Mental Pipelines
Mental Pipelines
Google Doc version (easier to follow footnotes)
Part 1 Mental Pipelines
Part 2 Interleaved IO Flows I
Part 3 Interleaved IO Flows II
Part 4 In Search of the Fundamental Inputs I
Part 5 In Search of the Fundamental Inputs II
Computers were invented in an attempt to automate certain repetitive mathematical tasks. They do not need to be electronic in nature. The first calculating machines were, in fact, mechanical – for simplicity, think of a mechanized device that uses levers to automatically move the beads of an abacus. The simplest computer can be thought of as a fixed pipeline.
Here, every stage of the computation performs a specific simple operation and passes the result to the next stage. And again, we can look at this very broadly. We can imagine a farm pipeline where workers pass crates of potatoes.
Every worker in the pipeline performs some very simple operation, for example, removing one potato from the crate and passing it on. The workers are not aware of the bigger picture, but whoever designed the pipeline made it such that when the potatoes pass through, at the end we get a number of potatoes that represents the result of an intended mathematical operation. For example, if x potatoes are fed at the beginning of the pipeline, at the end we may get 2x potatoes1. The same thing can be achieved through the mentioned mechanized abacus or an electronic computer, where, in the latter, instead of potatoes, we have a flow of charged particles (electrons), and instead of simple workers, we have voltage gates performing simple logical operations. The reason we give so wildly different examples is to sharpen our attention to the fact that there’s nothing magical in electronic computers. It’s all about transforming the quantity of some phenomena that we can count or measure – potatoes, beads, electric charges, etc. The whole operation looks like a meaningful computation only for the one who understands the pipeline and sees the input and output potatoes as carrying symbolic meaning. Why symbolic? Because we are not interested in the raw potatoes as such (nor in voltage levels), but only in their abstracted patterns and quantities. In many cases, however, a simple linear pipeline is not sufficient. Sometimes, a certain decision must be made based on the result coming from the previous stage.
For example, an egg processing pipeline may have such a decision stage realized as holes with different sizes that effectively sort and dispatch differently sized eggs into separate branches of the pipeline, providing different treatment according to their size. Yet, for the pipeline to be truly versatile, one more thing is needed – not only taking different branches, but also that such branches could lead back to prior stages of the pipeline.
When such branching is intended to be performed repeatedly, it is usually called a loop. Without loops, certain behaviors are very cumbersome or impossible to achieve. For example, consider a stage in an industrial pipeline where an object is polished. After that, there’s a check stage that verifies whether the part is sufficiently polished.
Notice how the loop case simply repeats the activity until the condition is met. On the other hand, if we use only linear (feed-forward only) pipelines, we need to explicitly prescribe any additional polishing as additional stages.
In modern computer programs (‘program’ is a synonym for the pipeline’s designed structure), we can have simple commands that take some input, pass it through a pipeline, and then finish by presenting the output result. However, any long-running program, such as a web browser, resorts to a pattern known as the main loop, or core loop, or event loop, etc. This is especially understandable in computer games, where it can be called the game loop.
Practically any video game you have ever played resorts to some form of the above pattern. Input events are detected by the hardware and relayed to the game logic. Then the game state is updated. For example, if the right arrow key is currently pressed, a variable in the game state containing the x coordinate of the player is modified – for example, xnew = xcurrent + 1. If the left arrow key is pressed, the update operation would be something like xnew = xcurrent - 1. Of course, depending on the game, things would be much more complicated. For example, before incrementing the x coordinate, a check could be needed to see whether there’s an obstacle on the game map at xcurrent + 1, in which case the incrementing should be prevented. Besides updates that are directly driven by player inputs, there could be many changes in the game state that depend only on the previous state. For example, a non-player character (NPC) may have its own x coordinate which needs to be updated based on some AI logic, even if it is presently not interacting with the player (maybe the NPC is in a faraway part of the game map). Then, finally, an output representation of the game state is built, such as activating the pixels of a display. This process repeats until eventually some exit condition is met. Without such a condition, the loop is said to be infinite – that is, we cannot exit the game (except by forcefully powering down the computer).
Once again, everything described can, in principle, be implemented through a potato farm pipeline. The inputs would be passed as crates of potatoes, where their quantity encodes the kind of command we perform. For example, a crate of 752 potatoes could mean ‘left’, while 77 – ‘right’. No one within the pipeline knows or needs to understand this. They have their fixed instructions and only perform simple operations (If I’m given a crate of 75 potatoes, I remove one potato from crate X…). Many of the game state variables are never exposed directly as outputs but are only passed internally within the pipeline loop. One such variable is the player’s x coordinate, for example. It does influence the way the output pixels are rendered, but usually we do not have a specific output that says, “x is 57”. Instead, we only have certain pixel configurations in the output, and we may infer that there’s probably a variable in the game state that stores the player’s x coordinate. Some variables could be even less exposed. Such would be the case if an NPC’s coordinates correspond to an as-of-yet unexplored part of the game map. These variables are present within the pipeline and take part in every game state update, but do not yet contribute significantly to the output stage.
The output display of our computers is also nothing special. We can implement a potato display by forming a rectangular grid of crates that are filled with varying amounts of potatoes. If we have different varieties of potatoes with different colors, we may use them to give some form of chromaticity to the display. Then the factory manager observes the configuration from his mezzanine office and interprets its symbolic meaning, even though none of the workers understands it in this way – they only follow simple instructions for adding and removing potatoes.
The elegance of this input–update–output loop has long been recognized as a potential metaphor for our human existence. Through our inner activity – thinking, feeling, willing – we continuously provide inputs that augment the World state in some ways, and aspects of this modified state are fed back to us (output) in the form of conscious phenomena – color, sound, tactile sensations, emotions, memory and mental images, etc3. Just like in a video game, only some portion of the rendered output seems to have anything to do with our inputs. There are many aspects of the output that feel completely uncorrelated with our activity, such as the terrain in the game map, the movements of another game character, etc. Thus, we conclude that the game state is something much vaster – our inputs sink into the game loop, and we usually recognize only some limited aspects of the output as having something to do with our input activity. This leads us to philosophize about the game-state-in-itself as something greater and mostly independent of our inputs.
At first glance, such a metaphor seems very fitting. It is readily understandable by any modern human because it seems to artistically depict very aptly what our existence feels like. Things, however, become more complicated when philosophers and scientists go further and try to understand more about the nature of the game loop – what variables it consists of and what logic updates it. The first stumbling block is that such a model implies a kind of Cartesian mind-matter dualism. It appears that we are a conscious something, distinct from the game loop, that provides inputs and perceives the outputs in a kind of phenomenal soul space. Modern thinkers are not very happy with this view and justifiably so. Thus, the intellect basically says, “Well, even if there are two such distinct realms, they obviously interact with each other and thus are necessarily a part of a greater unified system. So I’ll try to conceive of a greater game state that encompasses both realms. In this greater state, there should be variables for the mind stuff and the rest of the state.”
This seems like an elegant solution, because now the whole World pipeline can be encompassed in our mind, but this comes with its own set of challenges. It is trivial to follow the transformations along the pipeline, but it is not at all clear why a certain part of it should feel like a subjective experience of inputs and phenomenal outputs. Furthermore, if pipeline processing is all it takes for conscious experience to arise, then that should exist even if it is implemented with a potato pipeline. Some part of the potato pipeline should have an inner experience of being fed in outputs and providing inputs. This is a major unsolved problem for any philosophy that imagines that consciousness is what computation processing feels like. On the other hand, if we decide to leave that question in a shroud of mystery, we easily end up with some version of the homunculus paradox.
We simply shift the problem, which leads to a regression. Now, the same question applies to the small person in our head – why should that part of the pipeline feel like something? This problem, known as the hard problem of consciousness, is usually simply dismissed as secondary. It is left to philosophers to bang their heads against it, while true scientists try to figure out the structure of the pipeline, which is seen as the primary way to understand reality. The structure of the pipeline is seen as a mental model of objective reality, while the fact that this reality also feels like something subjective is considered a peculiar but non-essential fact.
When the intellect attempts to include everything within a unified pipeline, because of computational equivalence, it can all be simplified to a monolithic game state that is recursively updated by a universal game function4.
This approach of modelling reality does not accept anything outside the game loop. Even if there was such a thing, the intellect could simply inflate its mental model further to encompass it. In this way, every next game state depends only on the variables within the present state and the rules of the game logic.
When things are seen in this way, it becomes necessary that everything in existence – including the inner sense of inputs and outputs – is fully determined by some variables in the game state, continuously updated by the game logic. This leads us to the age-old debate about freedom. In our context, we can say that the above view necessitates the illusion of input. In other words, it may consciously feel that we are innerly active, that we are somehow ‘doing’ something, yet this is only a sense of confusion fueled by incomplete knowledge of the game state and its update pipeline.
But what does it even mean to understand the game loop of reality? From within our conscious experience and in the above scientific context, to understand the World means to transform our mental flow into a mental pipeline that mimics the workings of the sought-after World process. In other words, just like small children pretend to be animals and roar like lions, so we pretend in our imagination to be a potato processing pipeline and transform our mental states from stage to stage according to the rules5. Since we are mentally simulating (calculating or philosophizing about) the universal game loop, which has no external inputs and outputs, we also need to account for the mental crates and potatoes that represent what should feel like the illusions of input and output – including the experience of the mental flow that pictures the potato pipeline in the first place. This naturally leads to the conclusion that the consciously experienced mental process that performs the simulation is only a process within the true universal game-loop-in-itself. In other words, an aspect of the true universal game loop ‘feels like’ a mental pipeline flow mimicking the supposed workings of the greater game-loop-in-itself within which it has its existence. Maybe we can make this more pictorially clear through another metaphor.
The popular game Minecraft allows one to construct structures that pass signals using Redstone dust, similar to copper’s ability to conduct electricity. While initially this was intended for simple tasks, such as activating an in-game lever, which passes the impulse through Redstone dust for a door to be opened at some distance, more creative players quickly realized that it is possible to construct any logical gate using these in-game assets. It was only a question of time until the first Minecraft computers (MC) were created, and then even something as spectacular as an MC running a simplified version of Minecraft within Minecraft!
This Minecraft running on an MC can be seen as a metaphor for what our mental modeling is doing. One difference is that the MC above is still intended to be played on (by jumping with our in-game character on the buttons of the gigantic gamepad made of blocks). Yet, we can always remove the inputs and devise a different version of the simplified Minecraft that includes auto-play logic (AI), which makes it more like the universal pipeline without external inputs. In this case, the display output has only an informative role because acting upon its output is impossible. The output is now more like an overseer dashboard that helps inspect the MC state – it may present lots of additional information, split-screen presentations of different AI players, values of hidden variables, and so on. We can also receive the same information by directly observing the in-game blocks of which the MC is made, although it would be far more inconvenient.
Our sought-after Theory of Everything (ToE) is a mental MC in whose state transformations we seek to recognize symbolic correlations with the greater output of our conscious experience. For example, in the MC above, we conclude that we are getting a grasp on the primary game’s workings if the output of the simulated Minecraft screen mimics the output of the primary game. We say ‘symbolic correlations’ because, obviously, a block depicted on the simulated screen is not identical to the perception of a block in the primary output. As a matter of fact, if we are being completely literal about it, we can say, “What simulated block? In what you call ‘screen’, I only see a huge wall of (primary) blocks that occasionally light up.” Conversely, we may also be completely absorbed in the symbolic meaning of the simulated play, while the fact that we are observing nothing but the blocks of the primary game remains somewhat peripheral to our awareness. Similarly, we may be absorbed in the story of a movie while forgetting that the raw experience is only one of lighting up pixels on the TV screen.
It is obvious that the MC is only a representation of the supposed game-loop-in-itself. No matter how well the simulated behavior seems to correlate with the dynamics of the primary blocks, the former can never ‘become’ the latter. This is a modern-day way to convey the same thing that Kant conveyed in the philosophical language of his time – that in our intellect, we live only in representations (mental images) of the thing-in-itself, whose true nature remains transcendental. Today, more and more scientists realize (something that should have been clear all along since the time of Kant) that in our scientific theories we do not approach the essence of reality but only a mental pipeline for mimicking its appearances as they are available as output to our conscious experience. As an analogy, the fact that the simulated MC may mimic the primary game better and better does not guarantee that this is how the latter is implemented. As a matter of fact, even the in-game MC can be implemented in countless different ways, using different architectures and pipelines that can equally produce the same display dynamics. Analogously, it is also not possible to determine the hardware on which the primary game works by investigating its output – it could be running on a PC, Macbook, phone, virtual machine, etc. This is the ‘transcendental’ reality that Kant speaks of. Furthermore, we cannot even be sure if that reality can at all be represented as a pipeline6.
In the face of the above fundamental limitations, in our day, more and more thinkers come to see our mental pipeline modeling as a kind of obsession. It’s like constructing an MC and then trying to see reality through its prism. In this way it is felt that we lose sight of the primary game output with its richness and color, and reduce everything to the low-resolution monochromatic pixels of the simulated display. Thus, today, we can see in many areas that interest grows into what is called the pure experience of (primary) existence, untainted by mental modeling. However, when this tendency is pursued to its extreme, one becomes convinced that to contemplate primary existence, one needs to become a pure observer. Any sense of being causally active is seen only as a mistake where the pure observer feels as if they have something to do with the metamorphoses of phenomena. We would commit such a mistake if we contemplate a busy street and become convinced that we somehow cause the movements of people. Similarly, the described extreme position aims to detach from the busy movements of mental images and experience of will, and feel like a pure observer of the flow of existence that happens on its own. Thus, we once again reach the illusion of input, even though from another direction.
These two tendencies – the scientific and what we can call mystical – are in a way polar opposites. Nevertheless, what they share in common is that the sense of being inwardly active is seen only as a persistent illusion. They try to eliminate that illusion in different ways. In science, one strives to reach a more and more complete mental model of reality, such that any ambiguity is extinguished. In other words, the intellect can only feel completely satisfied when every next state of its existence can be seen as lawfully proceeding from everything that is already known; that is, every future state must develop from a past state according to well-established rules. If we feel that we are providing certain inputs to the universal game loop, this could only be because we do not fully perceive the past output that has transformed into our present. So the ideal of the so-inclined intellect is to have every transformation, every action and thought, as prescribed by a rulebook applied over the past output.
The opposite approach aims to directly eliminate the sense of input. What is left then is the pure experience of universal output that continuously updates itself in mysterious ways. In both cases one strives for freedom from input. In the first case, one strives for this freedom through knowledge that allows every input to be securely pre-calculated, thus eliminating all human imperfections like ignorance, doubt, uncertainty, regret, etc. It is as if one strives for the dreamed-of MC that will continuously display prescriptions on what to do next in the primary game – that is, the output of the MC should be directly translated into inputs in the primary game. As the state of the latter updates, so does the one of the MC, which prescribes the next input, and so on. In the second approach, one surrenders to the primary experience of the self-updating universe and assumes that in this way, when we stop interfering with it through our tumorous-like illusion of thinking and doing, everything will finally flow in a mysteriously ordained, wise and harmonious way.
Yet, in both cases, these sought-after ideals keep evading us, just like the carrot on a stick in front of the donkey. We have no choice but to keep resorting to the illusion of input in our day-to-day activities. The scientist awaits the future ToE MC, the mystic awaits death as the final liberation, but in the meantime, none of them can go by without the illusion of inputs. To be sure, what we have described are only the extreme positions on both ends. Many accept that the illusion of input can never go away. It is even imagined that life would lose its flavor if there’s no thrill in making a mistake, feeling spontaneity or uncertainty, and so on. Nevertheless, in all cases one thing remains a fact – in no way are the inputs questioned. One wants to build an MC pipeline whose output acts like an oracle, being able to unerringly predict the next state of the containing primary game loop and thus eliminate the need for error-prone and pain-inflicting human input. The other wants to lift their hands off the keyboard altogether and surrender to the flow of output patterns. But neither question whether our knowledge of how to interact with the primary game may be incomplete. Could it be that we are drawn to one of the two extremes described above only because we see no other alternative? Could it be that a deeper investigation of the way we interact with the game may reveal unsuspected ‘buttons’ and ‘sticks’, which fundamentally change our comprehension of what the primary game is about? This is what we’ll investigate in the following parts of this essay.
-----
1 Obviously, since this operation should return more potatoes than those fed in, an additional hidden source of potatoes within the pipeline would be needed.
2 These numbers happen to be the scancodes of the left and right arrow keys on the PC keyboard. The numbers are transmitted over the keyboard’s cable in the form of electric pulses (high or low voltage) representing the binary forms of the numbers (0100 1011 and 0100 1101), which are then received by the PC hardware and decoded by the operating system.
3 To be sure, the terms ‘input’ and ‘output’ can equally be used in the opposite way if interpreted from within our inner experience. It can then be said that we provide outputs to the World state, while the latter feeds in input phenomena to our consciousness. To avoid ambiguity, we’ll stick to the usage of the terms as customary in computer interfaces.
4 To gain intuition why such equivalence is possible, consider simple cellular automata, like Conway’s Game of Life. This is a perfect example of a state (the grid of dead and alive pixels) and a function that recursively updates the state (the application of the four basic rules of the game). It is a fact that the rules of the game are broad enough that any computational system can be modelled within it (example). In this way, no matter how complicated the theoretical pipeline of the universe is, how many compartments and aspects it may have, in the end, it could all be simulated inside such a cellular automata, where everything is encoded in its state (grid of dead or alive pixels) and the simple four-rule update function.
5 One may object that it is not possible to simulate reality entirely in our mind because we can hold only seven or so mental images at a time. This, however, doesn’t change things in principle. We can always use pen and paper to note and later regenerate mental images that happen to fade from our consciousness. Electronic computers also do not add anything fundamentally different – they only serve to automate and speed up our simple mental operations. All in all, our mental process is at the core, while paper and computers are only its support and accelerators.
6 Probably the most well-known proponent of this idea is Roger Penrose, who argues that there are aspects of the true workings of reality that are non-computable, that is, they cannot in principle be reduced to a pipeline process.
The soul seeks its essential nature in feeling life. It lives in imaginative pictures of art, Heaven, God, after life, etc. The soul dreams of its reality." (from your first essay)
I'm not entirely ready. I wanted to finish everything and then begin to post, but it takes forever. So I finally decided to start posting and refine the parts as we go. For this reason, I think there might be some changes in already posted things. Even for the first part I already want to change some things 