The Game Loop
Part 3
Interleaved IO flows II
Part 3
Interleaved IO flows II
Google Doc version (easier to follow footnotes)
Part 1 Mental Pipelines
Part 2 Interleaved IO Flows I
Part 3 Interleaved IO Flows II
Part 4 In Search of the Fundamental Inputs I
Part 5 In Search of the Fundamental Inputs II
Previously, we saw how our inner life can be seen as metamorphosing through distinguishable flows of intuitive IO patterns. Now we’ll look into additional aspects of these observations while attempting to gain even more intimate experience and comprehension. Imagine a blank screen with a cursor displayed. You move the mouse, and the cursor moves. This is a ‘game’ where the output is almost entirely a reflection of our inputs (we say ‘almost’ because we do not control the shape of the cursor, the color of the background, etc.). Now, imagine a normal game. As we move the cursor we pass over output elements that are actionable or not. Hovering over some elements may even affect the way they are displayed. For example, hovering over a button may make it glow. These are common techniques in user interface design that give feedback to the user, helping them to recognize actionable elements. Clicking on the non-actionable elements does not produce any effects – our input sinks and disappears into the game loop, never to be reflected back as output. On the other hand, clicking on the actionable elements leads to certain changes in the output or even a whole series of changes. For example, in a real-time strategy game, we may drag to select certain units and then click to issue a command.
Mastering a game requires us to learn what kinds of inputs we can provide and what kinds of outputs usually follow in response. In the very beginning, we are mostly in the position of a baby that learns to move its body parts – we click haphazardly, distinguishing clicks that produce outputs from those that do not. Then, we gradually become used to expecting certain outputs in response to specific inputs. For example, we learn that moving the cursor while the button is held transforms the output such that a white rectangle is displayed, and any blob of pixels (representing a unit) that falls within it becomes encircled (signifying ‘selected’). Then we can click somewhere on the map, and the green markers and lines are displayed in response. Then the unit pixels-blobs – but only those that are encircled – change positions, closing the distance to the markers. This IO flow is already a part of the gameplay IO flow. It is very closely related to the physical controller IO flow, yet it is distinguishably independent – for example, we can perform these gameplay operations with either a mouse or the laptop’s touchpad. Our physical interaction with the controller is different, but the basic game action ‘selecting units’ is the same. We can call this level of gameplay interaction the operational IO flow.
We discover a higher level of the gameplay IO when we no longer focus on these operational actions and feedback, but when we shift to intuitive vocabulary like ‘move these units over there’. We are no longer focused on the fact that we can drag-select and produce certain changes in pixels, but now on the fact that we’re dealing with units that have certain positions, can be relocated, can perform certain actions, and so on. This aspect of the gameplay can be called the tactical IO flow. Notice how the temporal nature at this level changes. At the operational IO flow, we are concerned mainly with momentary clicks, mouse movements, and their immediate output feedback. At the tactical IO flow, our intuitive input feels to persist through time. When we command the units above to move, our intuitive intent continues to ‘hover’ in our experiential context and makes sense of the changes in the output. If the units stop because of an obstacle or some other reason, it is as if we feel “This output doesn’t resonate with my overarching intent for the units to move from A to B.” Then we usually add additional inputs to keep the output consistent with our temporally extended, tactical intuitive intent.
We can surely move units from A to B, but for what ends? As we master the intuitive language of the tactical IO flow, we can begin shaping the strategic IO flow. Now, an even higher-order intuitive language begins to take shape, consisting of IO gestures that persist through even greater time spans. At that level, we strive to shape the IO pipeline into gestures like ‘gather resources’, ‘produce units’, ‘reveal the enemy position’, ‘assess the weakest point for attack’, ‘devise unit relocation path’, and so on. All of these are contextualized in the even more encompassing overarching intent of winning the game.
Even though we have analyzed the gameplay IO flow into discrete operational, tactical, and strategic flows, our goal is not to postulate these as some fundamental axioms. Rather, we use them only as anchor points that can help us get a sense of this particular gradient of temporal intuitive integration. In a similar way, we refer to children, adults, and elderly people as approximate regions along the age gradient. Thus, we should always remember that we are speaking of a gradient of intuitive integration, even though our intellect needs to delineate discrete concepts to use as handles.
In addition to developing intuition about how our inputs ripple through the output, we also learn to recognize patterns in outputs that feel related to other outputs. For example, the movements of non-player units feel independent of our inputs, but we still learn that certain outputs always appear together with or are likely to be followed by others. For example, when we see the enemy units building up, we recognize that the output may soon transform into one where we are being attacked.
In this gameplay example, we once again have the superimposed flows of different-scale intuitive IO patterns. Surely, not too many players reflect consciously on this contextual IO depth. The gameplay usually unfolds in a more instinctive way, where we live through erratic context switching. Yet, if we try to, it is possible to bring the contextual IO flows into better alignment and experience their music-like composition – clicks and mouse moves fit within the overarching tactical intuition, the latter fits in the even more encompassing strategic intuitive flow. These correlations are not merely passively informational, but they continuously work into each other both in the direction from the strategic toward the operational and vice versa. For example, our most general intent to win the game serves as a kind of temporal envelope that contextualizes the total gameplay IO flow from the beginning to the end. It is as if the total game flow is like a movie script for which we say, “I don’t know any details about this movie, but I know how it starts, and I intend to experience the scene of winning at the end.” So, at that level, we do not have any clear intuition of any of the details, but it can nevertheless serve as an intuitive tuning fork telling us whether our more detailed flow is aligned with the general goal or not. Then we have the strategic flow, which often has distinct characters in the beginning, middle, and end stages. Without the overarching goal of winning, the different strategic gameplays can be randomly switched, as if we write different chapters of a book without much concern about how they cohere in an overarching plot. It is the continuous intuitive cross-referencing with the overall goal that helps us filter those strategies and their order, which can fulfill the plot. The same goes for the tactical and the operational IO flows.
While when writing a book or a movie script, it might be possible to proceed mostly in a top-down manner, by starting from the most general plot-arc and work out all the details by bifurcating into the chapters, scenes, all the way to individual words or movie frames, in actual gameplay such completely top-down intents would continuously go out of phase with the outputs. We may have decided to employ a certain strategy, but then we see that the opponent has undertaken actions that make it unusable. Now we receive bottom-up feedback, and we are forced to transform our strategic flow so that it is still musically coherent with the overarching goal of winning. In some cases, however, it might turn out that all remaining strategies that we can conceive turn out to be dissonant with the plot-arc, and we end up losing. Additionally, it is not only the outputs related to the opponent that interfere with our intents. Our own skills are also decisive. It is not enough to simply hold on to the general idea of winning if we have never spent time developing the lower IO flow patterns. All these flows must be mastered and harmoniously interrelated. Some players may have great strategic vision, but lack the technical skills (they continuously misclick, they are slow, etc.). Others may have outstanding, machine-like technical performance, but struggle to sequence these actions in persistent unfoldment, coherent with overarching intents.
It should be emphasized that everything described still pertains entirely to our phenomenological experience in the here and now. Even though we artistically symbolize the IO flows as waves extending in time, we are not at all implying that such waves exist in some speculative metaphysical dimensions of reality. All IO flows are known in the present, in their momentary intersection. Their extension through time corresponds to our persistent effort of maintaining the musical unfoldment of the IO flow.
We can translate such examples to the game of reality, and things will hold in pretty much the same way. If we assume the first-person perspective of the emperor, we are once again provided with the World outputs – those that we gather directly from our perspective or through the intermediary of our advisors who summarize outputs for us. Our understanding of the World gameplay once again depends on learning what inputs are usually followed by what outputs and what outputs go together with other outputs. It depends largely on our past experience and the thus-developed intuition to issue certain inputs and have anticipation for corresponding outputs. If we provide the input ‘raise taxes’ and expect the output ‘cheering citizens’, we do not yet have a very well-developed intuition about the IO relations in the flow.
What we have described can be seen as a general picture of what it means to have intuitive orientation within the flow of existence. We basically become accustomed to certain patterns of outputs and the way they are related to inputs or other outputs. In this way, we can follow such flow patterns in what we call cause-and-effect chains. We should be very careful when we say that one thing causes another. For example, when we see on screen that an enemy rocket hits our unit and its health bar indicator shortens, we may say, “That rocket caused damage to my unit.” Yet, in the case of video games, we know that there’s no ‘rocket’ that ‘causes’ damage. It is all a part of a computational potato pipeline, parts of which lead out to the display crates. With this, it is not suggested that we should look upon all phenomena as illusions having no relation whatsoever to what happens under the hood of reality. What we point out is that as long as we build intuition about the IO patterns without abstractly speculating about how exactly one input or output causes another output, we are on safe phenomenological ground. Furthermore, we can easily see that such an attempt to find the true cause-and-effect relations is not as straightforward as often imagined. One may say, “But when I see two billiard balls hitting, I can plainly see that one causes the movement of the other.” But what about when we dream of playing billiards? Does one dream-ball cause the movement of the other? It can be further objected, “But in waking life, we can investigate the structure of matter and see that it is the atoms of the first ball that electrically repel the atoms of the other and thus set them into motion.” Yet this only displaces the problem. Now atom-balls take the place of the billiard balls, and we can again ask how exactly they interact to repel each other. It would be answered that they exchange photons. We can further ask what happens when a photon hits an atom, and so on. This turns into something like the Achilles and the tortoise paradox1. Thus, at some point, we simply stop breaking things down. For example, quantum mechanics doesn’t say what exactly happens when an electron and a positron annihilate and produce two photons. Rather, for example, represented with a Feynman diagram2, it says: “Here’s the situation before – we have the electron and positron – here’s the situation after – there are now two photons.” We do not observe the particles ‘melting’ and gradually ‘evaporating’ as photons, but we can mathematically describe the probability of finding one state after the other. The theory is quiet about how exactly this metamorphosis happens. Thus, we see that even in our modern science, at the end, everything boils down to building intuition of what outputs are likely to be followed by others. Ultimately, we do not know whether the collision output (registered in the particle accelerator) causes the output of the photons any more than we know that the display output of the rocket causes the output of the damage. Again, this is not to suggest that it is useless to build intuition about the flow of IO patterns, but only that we should avoid fantasizing tortoises in the gaps. We can still use the term ‘cause’, but we should take that only as an indicator of the order in which we are intuitively accustomed to behold the outputs.
When we investigate the chains of IO patterns, we may behold the output of the empire and trace the ripples back to an order of the emperor. But what is the nature of that order? Well, it can also be found in the output – it is a spoken word or a written order. Can we backtrace even further? Sure, we can say that the emperor was thinking and planning, and finally, he allowed his inner life to manifest in bodily gestures. If we are the emperor, the thinking is also part of the output of existence. We hear our planning, thinking voice, and picture the future of our empire. For a citizen, however, this mental output is not part of their direct experience. Nevertheless, our scientific thinking would say, “What is called thinking and planning are the processes in the brain, and these can clearly be beheld in the output if we attach the appropriate probes.”
The vast majority of today’s scientific intuition concerns such output-to-output relations. When we study history, politics, economics, geology, biology, physics, and so on, in general, we are confronted with and develop intuition primarily about output-to-output patterns. Even when describing the actions of human beings, for example when history depicts the deeds of ancient rulers, we mainly study the output behavior as it ripples through World events, not so much the inner life. There’s a certain convenience in studying relations of phenomena that are comfortably laid out in the output field. This is why, when it comes to our inputs, we feel tempted to cast them down into outputs too. Why deal with two different categories if we can get away with just one? It is simply easier to observe the brain and try to see everything as rippling outputs, then imagine that the illusion of input is merely “what this process feels like.” In our age, it is almost a universal fact that as soon as one tries to point attention to the inwardly experienced real-time input process, we are immediately thrown back into the experience of a mental flow outputting images of neurons, energy, information, souls, and so on. When we philosophize about the input, we are no longer clearly aware of the actual philosophizing flow, but instead, we are immersed in the meaning implied in the outputted mental images. Now, ‘input’ is simply a name for the arrangement of metaphysical mental images in the output, while the true input doing the arrangement remains instinctive and hidden in the blind spot of our cognitive activity. If we indeed try to grasp the input process, if we try to observe how our input activity is reflected in mental images, but at the same time we fight to preserve our habit of seeing all reality as faithfully laid out within the output, we feel like a dog chasing its tail, it is like trying to draw a picture that also includes our hands painting the picture.
Faced with such facts of experience, one possibility is to conclude that our human input process is not even in principle fit to be known directly. That’s why all efforts are directed toward building a Minecraft computer that can be beheld entirely within the output field and then draw parallels between its outputs and the primary ones. But are things really so hopeless? If we examine things more carefully, we can realize that the problem stems from trying to apply to the inputs, the skills and habits of studying that we are used to applying to output-to-output patterns. But what if it is not needed to make an output picture of the input process and study it in isolation, detached from the living inner process from which it precipitates? What if we could study the real-time experience of input, its degrees of freedom, and constraints directly?
To get a hint of the way our scientific process should transform, let’s consider some simple examples. We can take two arbitrary aspects of our phenomenal experience – the sense of warmth and the sense of loudness of auditory perceptions. The nature of these phenomena is such that we think of them as continuous. We can conceive a whole gradient of warmth, from the feeling of cold to being hot. Note that we’re not speaking of heat in the physical sense of how vigorously the particles of matter oscillate and collide. We’re focused entirely on the inner experience of warmth (the sensation that causes us to say “I’m cold” or “I’m hot”), without trying to relate it to anything else or explain it. We can conceive of a similar gradient also for loudness. Next, we can recognize that the two sensations are mostly independent of each other. This allows us to symbolize them with orthogonal axes:
Now every point in that plane corresponds to some combination of the two sensations. We can feel cold and loud, cold and quiet, warm and loud, and so on. With this we’re not aiming to build some theoretical model of inner experience but only strive to focus on two sensations and the ways they can manifest in the output flow. As the loudness or warmth changes, we can imagine that the point traces a certain path in the plane. For example, if we enter a quiet, warm room from the outside’s winter night, the transformation of our experiential flow can be represented as a point that moves along a horizontal line somewhere from quadrant III toward IV, as we gradually get warmer. The plane and the point are only an abstract map symbolizing our intuition for the way these sensations can transform and in what combinations they can be experienced. As the actual sensations change, we do not feel as moving spatially in one direction or another – we are always at the center of experience, and the sensations change in our phenomenal volume.
While this simple example addresses sensory phenomena that enter our experience mostly as outputs, we can also consider axes that are more tightly related to our inputs. One axis can be described through the question, “Am I currently active with my input? Do I willfully strive to augment the outputs of the flow, or am I surrendered to the receptivity of outputs that are mostly independent of my intents?” Another axis-question could be “Am I focused on the output sensations related to my bodily and sensory organs, or am I living more in my imagination (memory and mental images)?” And finally, we can ask a question in line with what we talked about in the beginning: “Does my present intuition of the flow elucidate the momentary phenomenal metamorphoses or the more overarching strategic unfoldment?” These axes can be depicted thus:
Once again, with this we do not imply the existence of some metaphysical axes, nor that the chosen three aspects are somehow fundamental and the whole complexity of our phenomenal existence can be represented through them. It is all only an artistic depiction of our intuition about these concrete aspects of the existential flow. Now, just as with the first graph, at any time we can inwardly assess where on this map we are. If I’m trying to add two numbers in my mind, I’m probably in octant VIII – I’m active, I live in my imagination, I’m focused on a short-term operational intuitive IO flow. If I’m watching a movie, I’ll spend time mostly in octants II and VI – I’m receptive, I’m absorbed in the audio-visual perceptions, and my intuition expands and contracts between the meaning of momentary happenings and the overarching plot. If I’m somewhere in III or VII, I’m probably daydreaming, surrendered to the imaginative flow and the intuitions it might evoke. Most of my daily activities would be found in octants I and V, where I am active, I’m absorbed in the sensory and bodily flow, and I imply operational, tactical, or strategic meaning in my actions.
In the case of the simple sensations, it can be said that our experience is always at some concrete point on the map. For example, it is difficult to conceive of loud and quiet simultaneously. In the second example, however, we should imagine that there could be many different points that are active in varying degrees at the same time, especially those along the z-axis of temporal intuition. It needs to be repeated that when we think of these points, we shouldn’t remain with the abstract geometric images but feel how every point corresponds to a certain flavor of our full first-person phenomenal experience of the flow of becoming. For example, if I’m watching a movie but at some point I drift into daydreaming, it’s not that I experience movement through some abstract space. I’m still centered in the here and now, but the nature of my inner flow transforms. The movie sensations are still there, but my gaze is relaxed and I look and listen ‘through’ them, while my inner attention is now surrendered to the flow of the daydream's imaginative scenes. It is in this sense that we say the different flows are still present and superimposed. When I return to the point representing the movie-flow experience, I don’t move spatially, but my daydream images dissipate, while the sensory sensations come into focus again. So, alternatively, instead of picturing a single point that moves within that space along a thread-like path, we can imagine that many points could be simultaneously present. Then, our inner flow can be thought of as the change of these intensities.
We are not ‘here’ or ‘there’ in that space. In fact, we should not expect that we can ever find such space as part of experience, except for in our picture-in-picture imaginative sub-flow. Nevertheless, the map can still artistically represent valid intuitions and lawfulness surveyable from our living experience. The points abstractly correspond to first-person experiential flows that are all concentric and superimposed (like the daydream and the visual flows, or the mouse and the strategic IO flows), but those more intensely ‘illuminated’ dominate and thus feel as the central experience. The others feel more ‘dimmed down’, as if in the periphery (not necessarily in a spatial sense) of our experience. So movement of our central experience from one octant to another is like dimming down the currently intense flows and lighting up others that have been hitherto in the background, yet concentric and superimposed with our present3. Seen in the map, this looks as if a cloud of points is moving, yet this movement is of the same nature as the ‘movement’ of the pixels on the display on which you see the animation – the pixels do not move but only change their intensity and color. With all of this, we do not imply some speculative quantum theory of existence, but only try to highlight facts available to our direct experience.
Here, we should make an important distinction which should be obvious in the face of what we said above, but we’ll state it explicitly anyway. Picturing the cube and imagining a point-cloud moving in it is not the same as actually transforming our full experiential flow (popularly referred to as “the map is not the territory”). While I’m picturing the cube, I live mostly in octant VIII. Just because I picture in my imagination the redistribution of the points’ intensity, say, toward octant I, it doesn’t mean that I in fact become active in bodily sensations, and seek the unfoldment of a strategic idea. It is only by trying to make the true inner transformations that we can really grasp what the map represents. We need to do experiments. For example, I can strenuously imagine the rotation of a triangle (octant VIII), then completely surrender to the sensation of the color of my desk (octant VI). I may be kneading dough for bread and focus on the momentary sensations (octant V) or try to feel the actions within the intuitive context of the preparation of the meal for the guests I expect (octant I). Just like what we said in the previous part, we cannot perform these transformations and simultaneously philosophize about them. Instead, we need to immerse ourselves in them, and later we can enter a philosophical reflection flow (somewhere in octants IV and VIII) about the memory images of the experience. This, however, doesn’t mean that we should lack lucid intuitive awareness while traversing the space through the non-philosophical octants. It is only that we do not immediately cast the experiences into memory images that we then try to conceptualize and arrange into pipelines.
It is not difficult to grasp these things intellectually, but while experimenting, we may have found that it is not so easy to navigate that space intentionally. For many, the attempt to transition into a given direction feels like the tip of a magnetic pendulum that is continuously nudged by the most varied forces.
Nevertheless, even if we find this navigation difficult, or even impossible, the mere attempt to do it already helps us realize that there could be kinds of inner inputs that we do not normally utilize consciously. On one hand, these navigating movements feel very familiar. This is because we experience such transitions between the most varied IO flows all the time. To an extent, we do know what it feels like to move within the octants. On the other hand, the attempt to consciously and intentionally navigate them may feel like something new; something that has been slumbering as potential waiting to be awakened. To make an analogy, imagine a person who does not know that they can control their breathing (like today we generally lack control over our heartbeat). They have been breathing all their life, they know how it feels, the rhythmic traversal of the axis between inhalation and exhalation is experienced, yet one day they discover that they can navigate that axis consciously and intentionally. They can now change the pace of breathing, reverse it any time, hold the breath, and so on.
This is the main idea that these examples aim to convey. We seek novel inputs to the universal game flow, not by merely trying new combinations of, say, hand movements, but by awakening to aspects of the flow that we have been hitherto falling through completely instinctively. On one hand, we say, “But this has always been happening in the background!” On the other hand, however, by awakening to the possibility of consciously navigating these contextual aspects of the flow, we also attain new kinds of intuition about certain unsuspected lawfulness of the flow. For example, if we quickly glance at our inner movie, it may feel like the chaotic bubbling in the above animation – we are tossed around from octant to octant. However, once we begin to discover new degrees of freedom through which we can navigate these spaces, we also start to intuit, from within the experience, some regularities and relations. The magnetic pushes and pulls of our magnetic pendulum tip are not completely arbitrary.
The examples we give are elementary, but they should serve as a sneak preview of the direction in which we are headed. The most important thing is to realize that we do not merely seek ‘new buttons’ that can give us certain advantage in the gameplay of existence, but we continuously awake within aspects of the flow that have hitherto been unfolding completely instinctively and dreamily, in a state of free-fall, so to speak. As such, the ability to steer our inner flow in novel ways is not merely the unlocking of a technical capability for our present strategic and tactical intuition to capitalize on, but leads to a higher-order consciousness of what we have been traversing so far without clear awareness. We should be open to the possibility that such new awareness and navigating inputs may expand our intuition into strategies and even overarching life-goals that may be presently unsuspected.
At this point, one may still say, “But why be so interested in the experiential peculiarities of the input process? Maybe the input-trying-to-grasp-itself is only a quirk, a form of self-recursion that may feel weird, but we may be only mistaking this weirdness for something of value. On the other hand, when we investigate the relations of outputs, we gain valuable intuition of the game process, we can build technology, and so on. We are surrounded by the output recognized as the Cosmos, the Sun, the Earth, and our inner human nature. This is what the human condition is. We feel certain needs and desires, we have hopes and dreams, so the best thing we can do is investigate the outputs, understand the lawfulness by developing intellectual pipelines that can symbolically replicate their appearances, recognize where we go wrong, and then use our hands to augment the output such that peace, harmony, freedom, and happiness may prevail.” This is a valid point, and it is not difficult to suppose that dabbling at the experiential interface between input and output may make one preoccupied with their own psyche and become disinterested in the wider output. However, the objection once again disregards the possibility that what we presently know as inputs may not be the full palette. Furthermore, we’ll see that when this route is followed seriously and responsibly, it leads right back into the wider output, yet in quite an unexpected way. Then we may even realize how peace, harmony, freedom, and happiness, keep evading us not because we have not yet discovered the true mental potato pipeline that is supposed to perfectly reflect the workings of the game-loop-in-itself (and thus turn into the perfect oracle), but maybe because by not knowing the nature of our inputs, their degrees of freedom and constraints within the octants we dreamily traverse, and by keep pushing instinctively, compelled by age-old desires and needs, we keep agitating elastic-like processes that keep us oscillating like the magnetic pendulum, dragging against friction and pain, never coming to grips of what we try to reach.
Keynotes:
The IO flows can be understood as being contextualized through a gradient of temporal intuition, roughly delineated as operational, tactical, and strategic flows.The temporal IO flows exist in continuous top-bottom and bottom-up interplay. A higher flow serves as an intuitive tuning fork or envelope to which the lower flows can resonate or dissonate. The higher flow leads us to seek the lower flows that can be musically aligned with it, but in other cases, the lower flows provoke us to transform the higher (changing strategies and goals).Our focus is on the lawfulness through which the flows manifest, and not on postulating metaphysical elements that we imagine to hold the causes for the unfoldment.Contemporary thinking feels most comfortable in encompassing output-to-output relations. This tempts us to disregard our inner experience of inputs and seek the latter as part of the cause-and-effect chains surveyable from the output.If we investigate our inputs in a living experiential way, we can discover that we can gain intuition and input degrees of freedom for navigation within aspects of the flow, through which we ordinarily free-fall completely instinctively and dreamily.-----
1 https://en.wikipedia.org/wiki/Zeno%27s_paradoxes
2 https://en.wikipedia.org/wiki/Feynman_diagram
3 It would be less confusing if all points in the map were drawn on top of each other, sharing the same center. Visualized like that, we wouldn’t be misled by the spatial spread-out representation, but it would also be quite impossible to see what octant-flavor points are lighting up or dimming down.
