Best practices for mixed reality AR/VR experiences

Vuforia supports the development of mixed reality apps for video see-through digital eyewear. You can combine AR and VR to develop immersive experiences and build apps for digital eyewear devices.
  • Design mixed reality applications that combine AR and VR for a new level of creative freedom
  • Provide more intuitive VR experiences by enabling users to interact with ordinary objects
  • Transition gracefully between AR and VR modes using a variety of user experience techniques

This article provides recommendations for designing mixed reality AR/VR user experience, and tips on how to implement them.

Starting the AR Experience

Staging the environment

It's important to provide users with some guidelines on how to stage and execute your AR experience. They should clearly understand how to prepare their environment, and how to initiate the app experience. Once they've initiated the app session, you should guide them through the modes and phases of the experience so that it can be executed successfully.
 
Users should:

  • Ensure that there is adequate lighting.

  • Sit down when executing the experience, for their comfort and physical safety.

  • Place the targets at an appropriate distance for your app's user experience. They should do this before they put on the viewer.

Indicating what to do and what to look for

Your app experience should provide clear indications and instructions for users, to help them initiate the experience and easily transition between its AR and VR modes. You should also consider what they will see when then enter the alternate mode .
 
When the user starts the app session, you should provide them with a clear indication of where they should focus their attention. If the app experience uses a target to select and/or initiate the VR mode, provide a graphic of the target and an indication that they are in a preliminary state of the experience. This can be done using a rendering effect, or a messaging within the app's UI.
 
Examples:
  • Display an image of the target orthographically in the center of the user's field of view, which directs the user to find the target image that they see on screen.
  • Present the representation of the target at a scale that corresponds to an ideal distance from the actual target.
  • Apply a transparent shaded backdrop over the video background to indicate that the experience has not yet begun.
  • Once it's found, make the backdrop fully transparent to indicate that they have successfully initiated the AR mode of the experience.

Distance and motion considerations for Digital Eyewear

AR experiences that rely on digital eyewear differ from AR experiences that use handheld devices in two important ways. 

  1. Greater target distances
The device camera(s) on a digital eyewear device is on the user's head. This means that targets will need to be detectable and trackable from greater distances than those typical of AR applications where the user is holding a device out in front of themself. This greater distance can be accommodated by increasing the size of the target, and also bringing the user nearer the target. 
 
Another implication of increased target distances is that the far clipping plane of the 3D camera in your scene may need to be adjusted to render content at farther distances. You can adjust the position of the far clipping plane using the Vuforia APIs. In Unity, you can set the clipping plane distances in the Inspector for each camera instance in the Hierarchy.
  1. Reduced range of motion

Users can manipulate a handheld device with much more flexibility than they can a digital eyewear device. You should be aware of the range of movements and positions that will be comfortable for your users and design your app experience accordingly. 

Triggering the AR to VR transition

There are many ways to trigger AR/VR transitions, the following are techniques that we have evaluated and found to be effective.
 
Gaze and time-based triggers
 
You can cause a transition in the AR/VR experience to occur when the user looks at a specific 3D object or surface in the scene. This technique uses ray casting and is demonstrated in the AR/VR sample. An advantage of using gaze is that its implementation is portable across different viewer contexts because it doesn't rely on hardware inputs. 
 
When using gaze detection, it is recommended to render a reticle or pointer image in the center of the user's field of view to indicate where their gaze is currently directed. The use of a reticle is demonstrated in the AR/VR sample. When triggering an event based on a user's gaze, you should require that the gaze is maintained for 2 or more seconds before allowing the event to be executed. This will ensure that the gaze event is fired intentionally.
 
It is also recommended that you implement a countdown indicator to communicate to the user that their gaze has been recognized and that the associated event will be executed at the end of the countdown. 
 
Virtual buttons
 
Virtual Buttons are a feature of the Vuforia platform that allow you to detect when a specific area of an Image Target is concealed in the camera view. They can be used to make target surfaces interactive, so that an event is fired when the user touches the target image. The advantages of Virtual Buttons is that, like gaze detection, they are portable across viewers and can be configured with varying sensitivities to ensure that users deliberately intend to trigger them. 
 
See:
How To Implement Virtual Buttons
 
Target Detection and Tracking
 
AR to VR transitions can also be triggered when the user looks at a specific image or object in their environment. This can be accomplished using one of the Vuforia target types and by initiating the transition after the target has been detected and has been tracked for a few seconds. The transition can also be initiated by a change in the orientation or position of the target, which can be determined by monitoring its pose.
 
As with the other triggering techniques, it is advisable to implement a minimum timer for target based events so that the user doesn't accidentally trigger a transition by glancing at the object or image. A two second timer is recommended.
 
Target based transition event handling can be implemented using the following target types:
  • Images
    • Image Targets
    • Multi Targets
    • Cylinder Targets
    • User Defined Targets
  • Objects
    • Object Targets
  • Text
    • Text Recognition

Transitioning from AR to VR

The transition between the AR and VR modes of a mixed reality experience can be accomplished in several ways. Transitions should be relatively short and provide an intuitive segue into the next phase of the experience. 
 
Fades
 
A fade is a gradual transition between two images, colors, or transparency states. Fades can be used to indicate a change between one scene, or perspective, and the next. This technique is demonstrated in the VR sample app.
 
Fade-to-black transition: 
  • A fade-to/from-black effect can be implemented by rendering a black quad in front of the camera, aligned to the near clipping plane, and then animating its alpha value between 0 and 1 to indicate a change in the scene mode.
  • The AR/VR sample demonstrated this technique in the BlackMaskBehaviour.cs script.
Cross-fade between AR and VR:
  • Fade the AR video background to transparent while you fade the VR scene to opaque, and vice versa when transitioning from VR to AR.
  • When fading the AR video background, it is recommended that you disable the MeshRenderer on the Backgroundplane object after the background becomes invisible. Re-enable this Renderer when transitioning back to AR mode.
Scaling
  • Scale the VR scene with an animation.
  • You can bring the user into and out of the VR scene by animating the scene to a scale appropriate for each mode.  For example, when the user is in a VR mode, the scene may be expanded to a real-world scale. When they are in an AR mode, the scene can be reduced to a smaller scale so that the user is looking into the scene from above.

Orienting the user in VR

Remember that the user’s orientation while in the AR experience, will be carried over to the VR experience.  For example, if the user is looking down at a target on a table, once they enter the VR experience they would be viewing the portion of the VR scene that corresponds to that head position. Care should be taken such that user’s opening view of the VR scene is a meaningful one. For example, if the user is entering a VR scene containing the solar system, orient their perspective so that they can see the planets. Don't have them looking out into empty space. You may also want to provide visual cues that help guide the user to look in the direction of the most meaningful starting point of the VR experience.
 
If the user is transitioning into an AR mode from VR, first direct them to orient their gaze to a perspective in VR that will resolve to a view that contains the AR targets that your app is using. Then when the transition has completed, they will be looking at the target and immediately ready to execute the next phase of the experience. 

Performance Optimizations

You can ensure the highest possible frame rates for your apps by applying the following recommendations for handling the display parameters and rendering behavior in the AR and VR modes of your experience.
  • Disable the Video Background Behaviour on both the left and right stereo cameras when in a VR mode to free resources for scene rendering. This behaviour is only required for rendering the camera image onto the video background when in an AR mode.
  • Disable the Hide Excess Area Behaviours when in an VR mode and re-enable them when in AR. This will establish the correct viewport dimensions for each mode based on the Fields of View and aspect ratios of their respective cameras. Note that this behavior is only required to support transitions to an AR mode, it's not necessary to mask excess viewport areas in VR 
  • Disable the Mesh Renderer component on the BackgroundPlane GameObject under the left camera when switching to a VR mode. This component is only required to render the video background in AR
AR/VR Sample optimizations - these can be applied to the AR/VR sample to improve performance
  • Replace the 2 Point Lights in the scene with Directional Lights
  • Enable the Static flag for the Terrain model that is used in the sample's VR mode
  • Replace the Diffuse shader used in the materials of the FBX models with a Vertex Lit shader
See:  Advanced Tips for Optimizing Stereo AR/VR App Performance

Also see Unity's rendering and lighting recommendations, and performance optimization tips from Oculus:
http://docs.unity3d.com/Manual/OptimizingGraphicsPerformance.html
http://docs.unity3d.com/Manual/LightPerformance.html
https://developer.oculus.com/blog/squeezing-performance-out-of-your-unity-gear-vr-game/