layout for interactive virtual reality

In 2D animated films, layout has always been the stage in production where an artist takes hand-drawn backgrounds and figures out how characters and the camera move through an environment. According to the VFX Artist Adam Schnitzer, “3-D layout is the art of staging the action of an animated scene in relationship to the camera.” So while CG changed the nature of the role and the skills of the layout artist, both traditional 2D and contemporary 3D and VFX layout artists rely on their knowledge of image composition and how camera movement and editing will support characters as they move through an environment. This brought me to my first challenge. In additional to never having professionally worked as a layout artist, I had no “camera” or frame to work with—how do I start?

Modeling, texturing, lighting, and effects each have unique limitations that present challenges in VR, but once we move past issues of optimization, frame rate, and pipeline, the conceptual framework remains very similar to what we know from offline visual effects production or even real-time gave development. With layout however, our inability to predict the location of camera/frame—or rather the participant’s attention—represents a fundamental shift in the conceptual framework of layout as a discipline.

We decided as a team, to start with the design of our User Interface. We knew the experience would be presented in diorama mode. Co-Director Areito Echevarria and our Level Designer Chris Garnier figured out that the Direct Manipulation UI should only allow the participant to rotate around a center pivot and translate along the Y-axis of each environment. From this, two design elements became clear: verticality and floating worlds.

For verticality, I looked at Chris Ware’s BUILDING STORIES. His use of a three-story brownstone through which the lives of multiple characters play out out across time really resonated with me. Also useful was the similarity in themes with MINIMUM MASS—specifically how loss, love, and place can encourage participants to move through a storyworld.

Hayao Miyazaki’s design for the floating worlds in CASTLE IN THE SKY was also an inspiration. Our UI allows the participant to lift up an environment and see beneath it. This gesture needed to be both meaningful and artfully designed. From Miyazaki’s floating worlds concept, Sunny Teich, our Effects Artist, built terrains and “ripped earth” that extended downward from each built environment. Jimi Wilson, our Sound Designer, then built spatialized soundscapes into these spaces. Like Chris Ware’s BUILDING STORIES, the environments began to reflect the emotional landscape of their inhabitants. At times they are self-contained and mysterious, and at other times they decay and fall apart.

Below is a mood board bringing together images of BUILDING STORIES, floating worlds, miniatures, and decaying buildings (Loic Vendrame) to see how these design elements might work together in a single environment.

The various stages of moving from rough layout to final map. Below left, a very simple layout of three stacked rooms that make up a medical center set. In the middle, blocking in lighting to emphasize separation of space between rooms. On the right, FX Artist Sunny Teich’s destruction and Texture Artist Sean Pickersgill’s material work which add layers of improved geometry and fine detail.

Below, a cinematic rendering of the current environment. This is still WIP, but layout now supports character, performance, and UI. It’s clear to me that the iterative nature of this layout process—moving between concept, FX, surfacing, and performance is similar to how layout has always functioned in live-action and animation. However, in VR, user interface and interaction design replace the need to compose for camera, and this necessitates a different design methodology.