Toward Compelling and effecient Manipulation
Virtual Reality has the potential to offer a world of new approaches to story-telling. Unfortunately, it’s difficult to explore this potential without inducing motion sickness. In an effort to ensure the comfort of people as they experience virtual reality, publishers, such as the Oculus Store, provide guidelines that recommend taking extra care with locomotion. The current guidelines suggest that acceleration should be based on real-world movement, that either slow or unnaturally fast speed is more comfortable, that sideways, strafing, and backward movements should be avoided in general, and that enclosed spaces should be avoided. The guidelines go so far as to suggest that the best approach to ensure comfort is to simply forgo locomotion all together.
While comfort is always a primary concern, following the guidelines exactly limits story-telling potential for Minimum Mass. A number of our scenes feature dioramas, which each contain a different slice of the story world.
Snapshots showing example dioramas. Each diorama is a “slice” of the story world. The viewer can “manipulate” the diorama to explore it.
As part-researcher and part-engineer, my goal is to design and develop a scheme that let viewers navigate our scenes, having the freedom to explore the dioramas, in a way that not only feels comfortable but also in a way that is compelling and exciting. Taking inspiration from control schemes in Quill and Medium, we immediately started working toward what Richard Stoakley coined the world-in-miniature approach: this is the idea that you see an a miniature copy of the an otherwise to-scale virtual space.
Richard Stoakley proposed the world-in-miniature scheme in 1995. The scheme describes how to realize control over a full-scale virtual environment using a hand-held miniature. The miniature exists inside another that full-scale virtual world, which corresponds to the viewer’s real world at a 1:1 scale.
While we haven’t decided on the exact details yet, we’re hoping to use the world-in-miniature approach to let viewers explore ours dioramas. Different from Stoakley’s idea, the miniature becomes our story-telling platform and the full-scale world is reduced (perhaps to static skybox or even an infinite empty space).
At least for now, our design places the dioramas in floating in front of the viewer. You can reach out and use your hands to move the diorama in any directions, turn it around, and even change its size. Offering these modes of control, a viewer can choose their own framing over the story: they can look up from underneath the floor, peer through the window from outside, turn the diorama around to see back-stage, and perhaps even change its size to view the scene from an “ants” or “giants” perspective — or some other creature of size in-between. Each framing may afford a unique experience: perhaps a way to see part of the diorama that is otherwise hidden. Ultimately, an unhinged world-in-miniature approach offers the sensation of theatre, but one where you can choose any seat in the house.
While the premise of developing a miniature representation of the virtual world and allowing control over its position, orientation, and size is a simple concept, I encountered two important problems when putting developing our prototype of the scheme in the Unreal Engine.
The first problem is about how manipulation sometimes appears instead as navigation, which causes users to become disoriented.
The second problem is that transforming the world as a hand-held diorama leads to unintended side-effects.
In the rest of this post I’ll describe these two problems in more detail, talk about the progress we’ve made toward them, and, interestingly, consider how the first problem can be used to resolve the second.
The first problem
The first problem is that, depending on how the viewer thinks the controls are working, they can get the sensation that they are orbiting around the diorama, rather than moving it in their hands. This effect is most prevalent when manipulating an object the consumes the entire view. To understand why let’s run through a quick thought experiment. Imagine you are in an empty space. The space is infinite and black. There is nothing in the space except for a single blue ball that is floating in front of you. Now imagine taking one step to your left. After taking that step, the ball appears to be a half-meter-or-so away on your right. Now, let’s restart - the ball is back in the center. This time around, reach and move the ball a half-meter-or-so to your right. After doing this, the ball again appears a half-meter-or-so away on your right. Here’s the interesting thing: in both cases, the outcome is the same (from your point of view).
An illustration of the problem that the movement of an object is an otherwise empty space can be interpreted either as movement on behalf of the object or the viewer. The illustration shows the rendered visuals (top) against two scene configurations. In both cases - one where the ball moves right and another where the headset moves left - the rendered visuals are the same.
This problem has occurred in our testing, particularly when the dioramas consumes the full rendered view (in which case the user cannot see the background that corresponds to the real-life environment). Things become even more problematic when the diorama moves by itself, such as a subtle undulating to bolster the feeling that it is floating in the space.
To address the problem, we currently limit the freedom that the viewer has. In particular, we find that limiting control over rotation to only the vertical axis helps reduce disorientation, even when you cannot see the background. We also find that limiting the range of available sizes is useful, which helps keep at least some of the background in view during the experience. And finally, we add a friction effect to ease-out movements; allowing you to move or rotate the scene quickly, let go of it, and then watch it slowly settle into place. While the scheme paired with our constraints and physics effect is reasonably compelling and comfortable to use, it ultimately limits some of the freedom of point of view. As we continue further into development, we will how much of these constraints can be elided while user comfort is preserved.
The second problem
Once I’d worked out how to map the viewer’s gestures into a diorama movement the Unreal Engine makes it almost trivial to get things up and running. I’ll give the details behind the calculation and the implementation in another post; but, for now, check out this demo of our manipulation scheme applied to a testing diorama (with our constraints turned off):
Notice how the movement of the diorama (the grey container) has an impact on physics objects (the cube and the cylinder)? That’s the second problem: whenever we move the diorama around, we either need to apply the same movement to every object in the virtual world. That might not sound so bad, but on the one hand it involves of lot of computation. However, to really “transform” the entire world, we also need to update properties used in dynamic elements of the world: the direction of gravity needs to change, the world’s notion of how large a metre is (used in physics calculations), and more. In other words, we need not only manipulate geometry but also an entire engine worth of properties.
To solve this first problem, we can actually borrow from our leg work on the first problem. Remember that, in our case of the blue ball in an otherwise empty space, navigation and manipulation are the same with respect to the viewer’s point of view? We might say navigation and manipulation are oppositely the same, or to borrow a word from graph theory, that they are isomorphic. This observation, surprisingly, becomes a solution to our second problem. Instead of moving the entire virtual world and reconfiguring all the properties that go with it, we can simply move the viewer oppositely: if the diorama should take a step left, we make the viewer take a step right; if the diorama should turn clockwise, we rotate the viewer anti-clockwise, and if the diorama should be smaller, we instead make the viewer larger.
There’s a bit more to cover on the side of calculations (coming in a future post, or otherwise take a look at Jimmy Alamparambil’s and Tim Mowrer’s talk from Unite Austin 2017); but for now, check out the demonstration below, where instead of the diorama moving, I’ve changed the prototype to move the viewer oppositely. The result is that it looks like diorama is moving. Can you see any difference? I’d be willing to bet you can’t. And, now that the diorama is static there are no side-effects to handle and we’ve solved the second problem.
Thanks to the isomorphism, we are able to offer manipulation without the side-effects.