This article presents the concept of continued experiences for paused AR applications built with Vuforia Engine. Continued AR experiences are applications that can resume the AR experience after pausing or interruption. You can now provide a convincing and coherent story to your application where the AR content is sustained from where the users left off.
Continued AR experiences work on all fusion providers and device platforms by maintaining anchors and Vuforia targets between pauses and application interrupts. The Device Pose Observer must be enabled to have a continued AR experience. Additionally, familiarize yourself with Vuforia’s Device Pose Status and Status Info, as these states are essential for developing and specifying continued AR sessions.
Requirements for continuity to function optimally
- Stable tracking – feature-rich area.
- Prior mapping of the surroundings.
- Stable lighting settings.
Definition of levels of interruptions of an AR experience
Limited Tracking – While it is not a temporal interruption of the application itself, tracking loss can be abrupt and easily break the immersion of an AR experience. It is one of the trickiest elements of working with AR; the users will not adopt an application with poor tracking stability. To solve this, Vuforia offers Extended Tracking via Device Tracking at runtime, which allows you to track a target even when the detection is limited or the target is no longer in the camera frame.
Application interrupt – Pausing, push notifications, or briefly changing applications are all examples of an application interrupt. The disruption from other applications would often require users to restart the AR session and repeat the progress. To maintain the AR session, Vuforia records the position of targets, anchors, and virtual content in the environment, which will be ready once the users resume the AR session. For continuity to work optimally, it is recommended that users resume the AR session in roughly the same position that it was paused.
Application Relaunch – There will be cases where the application is shut down and resumed later. Vuforia Engine currently does not support the continuation and longer-term persistence of a closed AR session.
Here’s how to create continuity in your app after resume
As mentioned above, the management of states allows you to create continued AR experiences after the application resumes, and getting familiar with the statuses is highly recommended. In addition, it also depends on the platform you are using. Below is a description of platform-specific approaches for realizing the continued behavior:
Important: AR experience continuity will not work if you reset the Device Pose Observer or deinitialize the Vuforia Engine when the app is paused. In this case, you will lose all the information about previously tracked targets. The correct approach to get continued behavior is to stop the Observers on pause and then start the Observers on resume. For an overview of the lifecycle of a Vuforia app, please refer to Engine Lifecycle.
ARCore, VISLAM and HoloLens
The behaviors are alike on these platforms. Upon recovering an AR session, the Device Pose Observer will report as TRACKED-NORMAL. It is possible to start the Extended Tracking of new targets before recovering the targets tracked before the interruption. Also, the new targets will not be affected when the tracking of the preceding targets is restored.
NOTE: In rare cases where Extended Tracking fails to resume as expected, the user should be directed to return to the original target to recover tracking.
ARKit
Upon resuming an interrupted AR session, the Device Pose Observer will report a LIMITED-INITIALIZING status for 1-2 frames before it switches to LIMITED-RELOCALIZING status info. There will be no extended tracked targets or anchor reports during this time. To return to a TRACKED-NORMAL status, developers should inform users to return to the position where targets were previously tracked, and the Device Pose Observer will relocate itself.
NOTE: ARKit cannot start extended tracking of new targets until it has recovered from the LIMITED-RELOCALIZING status and is tracking normally. For this reason, you may wish to provide users with a way to reset the device tracking manually or automatically if this status continues for some time. Consult the example for handling tracking status or this article about the Vuforia sample app setup.
NOTE: Resetting the Device Pose Observer will lose all the previously detected targets, and they will have to be re-detected again. However, the reset will enable the system to start extended tracking of new targets. This applies only when you are using ARKit and Vision-only SLAM.
Vision-only SLAM
Devices that track only on visual cues will return a NO_POSE-RELOCALIZING status as the Vuforia Engine attempts to restore tracking of targets between interrupts. To restore tracking to TRACKED-NORMAL, users should return to a position where tracking worked with previously identified targets.
NOTE: The vision-only SLAM system will automatically reset itself if it detects a new target while in the NO_POSE-RELOCALIZING mode.
Re-establish Tracking
It should now be evident that, depending on the platform, certain statuses can be used to inform users how to restore tracking. It is best practice to indicate what users should do when they resume the AR session from a pause or interruption. These indicators can be messages or an on-screen overlay asking users to:
- Return to a position from the previous session.
- Map the area by moving the device left-right or front-back until relocalization is successful.
You may provide a message to indicate a time interval until which relocalization can be attempted, and if that time interval is exceeded, the session will be restarted. Another option is to simply offer a reset button to the user to restart the experience after some time upon resuming the app activity if relocalization does not succeed.
Disabling the Continuation Behavior
It is possible to return to the application behavior before SDK 9.0 by resetting the Device Pose Observer whenever the application is resumed. This will break the continuity whenever the app is paused or interrupted. Similarly, all mapping stored by the Device Pose Observer will be lost if you deinitialize Vuforia during a pause.
To learn more about continuity using ARKit, please visit the external documentation on the subject of Managing Session Life Cycle and Tracking Quality.