Memory, Visualization, and Simulation: Benefits of Closing the Fidelity Gap



Photo Credit: The Matrix © 1999 Warner Bros.

The Fidelity Gap

Today the distinction between authentic sensory experiences and the synthesized experiences produced by memory, visualization (i.e. our mind’s eye), and computer simulations is clear. The fidelity of real-world sensory stimuli is incomparably vivid.

But this will not be true for much longer. Sometime later this century computing technologies will almost certainly close the fidelity gap, meaning that they will be capable of rendering totally compelling simulated realities like The Matrix as well as allow us to capture and recall perfect memories of any experiences we have – whether real, imagined, or simulated.

Science and science fiction alike have examined some of the problems this might create. But there a number of potential benefits as well, and they have so far received much less attention than they are due.

Mo VR Mo Problems?

One potential downside of closing the fidelity gap stems from the fact that we human beings are, at least in part, motivated by the differences between “authentic” experiences based on sensory stimuli and “synthetic” experiences that are either remembered, visualized, or artificially simulated. In other words, one reason we get out of bed in the morning is because nothing in our heads or on a screen compares to the real world. Yet.

But what happens when technology makes remembered, visualized, and simulated experiences just as vivid – or more vivid – than the real thing? In the relatively near future computing technology is almost certainly going to eliminate the fidelity gap for memory (via artificial total recall) as well as for both visualization and simulation (via fully immersive virtual reality).

In some scenarios, individuals become lost, trapped, or simply addicted to virtual reality. In other scenarios, entire civilizations retreat into virtual reality.  The underlying cautionary theme is to beware the consequences of allowing virtual reality to become so captivating and rich that it offers more utility (variety, pleasure, control, etc.) than the real world.

Another potential downside is that the same technologies that enable us to close the fidelity gap will necessarily enable mind-reading as well, and so threats of abuse and oppression stemming from invasions of privacy are cause for serious concern.

This is well-trodden ground, so I won’t rehash the details here.

Virtual Reality, Real Opportunities

The potential benefits of closing the fidelity gap have received less attention, so let me highlight them in several broad categories.


Though many may be loathe to admit it, synthetic experiences of equal or greater fidelity than authentic experiences may simply be better experiences. Even though I count myself among those who are intuitively repelled by that idea, and who feel that authenticity is something of a trump card, there is nevertheless a reason why we go to the movies and play computer games. Whether a person lives in a dangerous neighborhood, or in poverty, or whether it is just a cold rainy day outside, there are times when what the real world has to offer just isn’t that great.

It seems to me we must admit that in some ways and under some circumstances, synthetic experiences are, on balance, superior to authentic ones. And if this is even marginally true with today’s technology – whether it is for entertainment, education, therapy, or any other specific application – then it can only become more true in the future.

Moreover, we are so far only talking about experiences that are directly comparable. There are almost certainly additional synthetic experiences for which there are simply no authentic counterparts. Who knows what mind-bending possibilities await us in virtual reality, unbound by the physical and logical limits of the real world?

Variety and Creativity

Although these need no elaboration, we should nonetheless acknowledge that the space of possible synthetic experiences is vastly larger than that of authentic ones, and so opportunities to create and explore will eventually be much greater in virtual reality than they are in the real world.


So long as the user retains control, synthetic experiences are almost certain to be safer than authentic ones. If something goes wrong while base jumping or swashbuckling inside virtual reality, there are unlikely to be lasting physical consequences.

This is not to say that there are no real risks of psychological injury in virtual reality. At the modest end of the spectrum, it is quite possible to damage real relationships and suffer emotional trauma in virtual reality – especially in environments that multiple users are sharing. And at the more severe end, the possibility of using virtual reality as a means of torture is already a looming ethical concern.

But with those important caveats in mind, it seems clear that virtual reality promises us opportunities to explore experiences that would otherwise be too dangerous to contemplate in the real world. Gaming today already suggests that, on balance, having opportunities to engage in simulated risk and violence actually decreases the likelihood that gamers will take risks and act violently in the real world, not vice versa. Of course it remains to be seen whether this holds true for all potentially dangerous activity and all individuals. Certainly we can imagine circumstances under which a young or mentally unhealthy individual’s proclivities toward physical, sexual, and emotional violence should not be indulged. But where is the harm in a healthy adult indulging in fantasies – however violent or debauched – if there are no real-world victims? At the very least, it would seem to beat the alternatives.


The real-world has a multitude of wondrous authentic experience to offer, if you have the means to avail yourself of them. If you are a billionaire, the world is your oyster. But what about those of us who aren’t? Skiing on fresh powder in the Swiss Alps, scuba diving on the Great Barrier Reef, tasting wine in Napa Valley, roaring around a private race course in a Bugatti Chiron, relaxing on a private beach in Maui, taking a trip to the International Space Station – surely these all make for good times. But what if you work a minimum wage job at a fast food restaurant? What if you’re a subsistence farmer in a less-developed country? What if you’re a wage slave in a sweatshop? Or setting financial hardship aside, what if you suffer from a severe physical disability? It hardly seems noble to vaunt the value of authentic experiences if only a privileged minority of people have access to them.

This is not to say authentic experiences can only have value to the person who enjoys them directly. Many of us take great vicarious pleasure in the authentic experiences our children enjoy, for example. And more abstractly, most of us take a degree of comfort simply from the knowledge that extraordinary authentic experiences are available to someone, even if that someone isn’t us – a form of what ecologists and economists call existence value.

Nevertheless, at least some of the value of authentic experiences must lie in the extent to which individuals have access to them. And many extraordinary experiences are inaccessible to the majority of us. Synthetic experiences, by contrast, do not suffer from this limitation.

Consider music by analogy. Prior to the advent of recording technology, only the relatively rich had ready access to high-quality live performances. Once analog recordings appeared, access to music grew explosively but was nonetheless still limited to those who could afford to acquire a collection of records and the means to play them – limitations that long excluded the majority of the world’s population. But by the early 2000s, digital music recordings were no longer bound to a particular physical media, and so today virtually all of the world’s music is readily available for any person with Internet access to enjoy.

There is of course still great value in the authentic experience of live musical performances. But the synthetic experience inside a good pair of headphones is wonderful now too. Is the world not a better place for being filled with music?

We can imagine all experiences, not just listening to music, following a similar pattern.


As synthetic experiences become more widely available, we may become more selective in our choice of which authentic experiences to have, and this may help make those authentic experiences more special.

Music again provides an instructive example. The widespread availability of high-quality digital recordings, speakers, and headphones means that most of us only rarely indulge in a live musical performance. The authentic musical experience is therefore a special occasion for many of us today, perhaps in a way that live performances were not special in prior ages when every instance of music was a live performance.

Food is perhaps another example. Once upon a time, virtually all meals were home-cooked from scratch. Today, an authentic home-cooked meal is a special occasion for many of us. In the future when we can eat anything in any quantity we please in virtual reality, grandma’s real Thanksgiving dinner may be an even more extraordinary culinary occasion than it already is now.

So although we can easily imagine a future in which authentic experiences are vastly outnumbered – and likely outclassed – by synthetic ones, this may actually serve to elevate rather than diminish authentic experiences. Even a “low quality” live musical performance is a delight today, now that they are so rare. And where grandma’s home cooking might once have been mundane, today it is for many of us a rare treat to be relished. And so it may go with many other real-world experiences as well.

The upshot is that we may become much more discerning in our choice of real-world experiences – and that may not be a bad thing at all.

Ecological Efficiency

Now we come into my wheelhouse: the environment.

Take the example of cheeseburgers. As I’ve pointed out before, humans eat cheeseburgers in titanic quantities – some 50 billion per year in the United States alone. But cheeseburger production carries an enormous ecological footprint, and they aren’t particularly healthy for us either. So then why do we eat such huge numbers of them? Because they taste great.

But what if you could eat your weight in beef in a single sitting within fully-immersive virtual reality?

Or take the example of travel. Today more than 100 million Americans travel by car to get to work. But what if you could telecommute to work? Research has already begun to show that even today’s primitive telepresence technologies like Skype have a substantial impact on travel, with a corresponding reduction in environmental footprint.

Sustainability scientists like me often talk about the efficiency with which a population meets its needs. But what do these terms actually mean? Since human needs are so complex, we typically just use production and consumption as a proxy, which we often measure via GDP. Efficiency is therefore the per-dollar ecological footprint of human activity. Technology comes into the picture by affecting efficiency: better technology allows us to meet our needs (i.e. produce and consume a given amount) with a smaller ecological footprint (e.g. energy use, land use, carbon emissions, pollution, and so on).

But, quite obviously, what we are really after with much of consumption is an experience. In the case of cheeseburgers, we seek a pleasurable eating experience. And today there is no substituting authentic eating experiences with synthetic ones.

Fully immersive virtually reality is therefore a killer app for ecological efficiency. If synthetic experiences of eating and other earthly pleasures were  totally compelling, the ecological efficiency with which we meet our needs would skyrocket. Eating cheeseburgers and other environmentally impactful activities of that sort could become something we do almost exclusively in virtual reality, with their real-world counterparts becoming a rare and perhaps even obscene indulgence.

For the sake of the environment, we really cannot invent The Matrix soon enough.