Extended Tracking

Extended Tracking utilizes features of the environment to improve tracking performance and sustain tracking even when the target is no longer in view. 

Extended Tracking allows the tracking a degree of persistence once a target has been detected. As the target goes out of view, Vuforia uses other information from the environment to infer the target position by visually tracking the environment. Vuforia builds a map around the target specifically for this purpose and assumes that both the environment and target are largely static.

Extended Tracking can significantly enhance two kinds of user experience:

  • Game or game-like experiences with a large amount of dynamic content that requires the user to point the device away from the target as the user follows the content
  • Visualizations of large objects like furniture, appliances, large home furnishings and even architectural models at the right scale and perspective

Use this feature to facilitate creating more robust applications, because any augmentations attached to these Targets will persist. In practice this means that after you point your device away from the initial Target, any augmentations maintain their positions with respect to the real world and are consistent with the initial reference frame defined by the target. The more detailed and feature-­rich the environment, the better Extended Tracking works.

Let’s see how this works with regard to pictures and scenarios that use the new Vuforia Samples, which contain an Extended Tracking toggle in the options. The following example depicts the effects when running the Image Targets application with this toggle selected.

Figure A shows a device capturing a Stones Target in its field of view (FOV). The result is Figure B, which shows the Stones Target in view, along with augmentations.

User-added image
                                Figure A -left                                                                  Figure B - right


As Figure B shows, the augmentations of buildings on top of a wooden table are displayed on the screen of the device. This encourages the user to pan to these items by moving the angle of the camera to point upwards, thereby making the Stones Target disappear from view.

In previous versions of Vuforia, this led to all augmentations disappearing, because the Target reference frame is lost. With the Extended Tracking feature, the buildings remain visible even when the Stones Target is not in the camera view. Figures C and D illustrate this:

User-added image
                                   Figure C - left                                                              Figure D - right


Note that after the Stones Target goes out of view it is not required for further display of the augmentations if this feature is enabled. That means that the augmentations persist without the target, and this capability supports more robust and continuous AR experiences.

Note that after the Stones Target goes out of view it is not required for further display of the augmentations if this feature is enabled, i.e., the augmentations persist without the target, and this capability supports more robust and continuous AR experiences.

Extended Tracking

What if the environment or target is not static?

By default, Extended Tracking assumes that the environment the camera sees is mostly static. The Extended Tracking map will continually expand, so new objects added to the scene will become part of the map. This does imply some extra background processing to managing and optimizing the map which could be perceived as slow-down in the app if there's a lot of other processing going on. If a large part of the scene moves, it risks pulling the map with it and negatively impacting the offset accuracy between the target and the map. This offset may be recomputed when the target is tracked again. If only a small part of the scene moves, then it's possible that a negative impact to the offset may be minor and go unnoticed thus introducing small error in accuracy of extended tracked pose estimates. In this way, larger scene changes can sometimes be less problematic than more subtle ones.

If scene changes or changes of the target location are expected, then the application should NOT use Persistent Extended Tracking (PET). Not using PET is the safer approach here, as the map will be reset whenever tracking stops for a few seconds. PET should only be used if:

  • the application does NOT need to afford scene or target location changes (and the 5 s auto reset is therefore only harmful to the experience)
  • the application lets the advanced user reset the map manually when a scene change has happened.

Persistent Extended Tracking

With persistent mode turned on, Vuforia never resets or rebuilds the map during extended tracking. Instead, there is a dedicated resetExtendedTracking API (see below) that puts this in the hands of the developer. The application can keep the map for much longer, providing a seamless experience if the device is put down or the camera is stopped temporarily. Furthermore, if an additional target is detected and tracked, this target is appended to all existing targets that are being tracked.

Trackables that support Extended Tracking

These targets support Extended Tracking:

  • Object Targets
  • Image Targets
  • Multi-Targets
  • Cylinders
  • User-Defined Targets
  • Cloud Recognition
  • VuMarks

Text and Frame Markers do not support Extended Tracking.

Behaviour of Extended Tracking with multiple simultaneous targets

Under certain conditions, Extended Tracking can work with multiple targets, but be aware that all targets need to be in the right scale with respect to each other.

When Vuforia is configured to track one target at a time:

A target may still be extended tracked when it is out of view. Only when that target has lost tracking can a new target be tracked.

When Vuforia is configured to track more than one target simultaneously:

Extended Tracking should work fine with multiple simultaneous targets. If the targets are in physically disjoint locations, only one of the targets will be extended tracked. When tracking multiple simultaneous targets, it is recommended to NOT use Persistent Extended Tracking so that the Extended Tracking map will be able to reset 5 seconds after tracking is lost and be ready to use extended tracking on the next tracked target.

How To Use The Extended Tracking API

The following is a description of the API with examples of how to implement Extended Tracking in native code and Unity.

Native API

The following code shows the updated Trackable base class definition, along with the updated API for startExtendedTracking() and stopExtendedTracking(), which control this feature.
 

class Vuforia_API Trackable : private NonCopyable
{
    //...

    /// Starts extended tracking for this Trackable
    /**
     *  This function will return true if extended tracking was 
     *  successfully started for this Trackable (or already started) 
     *  and false otherwise.
     */ 
    virtual bool startExtendedTracking() = 0;
 
    /// Stops extended tracking for the Trackable
    /**
     *  This function will return true if extended tracking was
     *  successfully stopped for this Trackable (or already stopped) 
     *  and false otherwise.
     */
    virtual bool stopExtendedTracking() = 0;
 
    virtual ~Trackable()  {}
};


Remember that this applies only to the following Trackable types, all of which are subclasses of ObjectTarget:

  • CylinderTarget
  • ImageTarget
  • MultiTarget
  • VuMarkTarget
  • VuMarkTemplate

Text and Framemarkers do not support Extended Tracking.

The TrackableResult class was updated in Vuforia SDK 2.8 so that getStatus() returns an EXTENDED_TRACKED status if the Natural Feature Tracker is no longer generating poses for the Trackable. When the Natural Feature Tracker is generating poses for the Trackable, getStatus() will return a TRACKED status.

In addition, the Tracker class was updated to enable and disable Persistent Extended Tracking and to reset the Extended Tracking, as described earlier.

class Vuforia_API Tracker : private NonCopyable
{
    //...

    /// Enables/disables persistent extended tracking
    /**
     *  In persistent extended tracking mode, the environment map will only
     *  ever be reset when the developer calls resetExtendedTracking().
     *  This function will return true if persistent extended tracking
     *  was set successfully (or was already set to the specified value)
     *  and false otherwise.
     */
    virtual bool persistExtendedTracking(bool on) = 0;
 
    /// Resets environment map for extended tracking
    /**
     *  Environment map can only be reset by the developer if persistent
     *  extended tracking is enabled.
     *  This function will return true if environment map was reset
     *  successfully and false otherwise.
     */
    virtual bool resetExtendedTracking() = 0;
};
 

The resetExtendedTracking() API works only if persistExtendedTracking() has already been called to turn it on.

Unity API

In Unity, implement this feature by ticking the Extended Tracking option in the Inspector, so that using an API is not mandatory. It can, however, be useful to understand how to control it using scripting for use in advanced scenarios.

First, all supported targets implement a new interface, ExtendedTrackable, which contains two methods that mirror the native counterparts.

public interface ExtendedTrackable : Trackable
{
    // Start extended tracking. The Target can be tracked although it is not visible.
    bool StartExtendedTracking();

    // Stop extended tracking. Target will only be tracked when it is visible.
    bool StopExtendedTracking();
}
 

The TrackableBehaviour.status () contains the values TRACKED and EXTENDED_TRACKED. When the Natural Feature Tracker is tracking the object, this value will be TRACKED, even if the Extended Tracking option is enabled for the Trackable. When the Natural Feature Tracker loses the Trackable and Extended Tracking takes over (if Extended Tracking is enabled for the Trackable), the status will change to EXTENDED_TRACKED. The OnTrackableStateChanged() method in the DefaultTrackableEventHandler.cs file illustrates this.


 
public void OnTrackableStateChanged(
                                TrackableBehaviour.Status previousStatus,
                                TrackableBehaviour.Status newStatus)
    {
    if (newStatus == TrackableBehaviour.Status.DETECTED ||
        newStatus == TrackableBehaviour.Status.TRACKED ||
        newStatus == TrackableBehaviour.Status.EXTENDED_TRACKED)
    {
        OnTrackingFound();
    }
    else
    {
        OnTrackingLost();
    }
 

Finally, the ObjectTracker class was updated to enable and disable Persistent Extended Tracking, and also to reset the Extended Tracking to mirror the native counterpart.

 
ObjectTracker tracker = TrackerManager.Instance.GetTracker<ObjectTracker>();
   
bool success = tracker.PersistExtendedTracking(!mPersistExtendedTracking);

if (success)
{
    Debug.Log("PersistentExtendedTrackingEnabled");
}

// To reset Extended Tracking

tracker.ResetExtendedTracking();
 

How To Implement Extended Tracking

The following code groups show how to add extended tracking to the Chips Image Target within a standard Image Target application.

The code for Android and iOS shows a modified loadTracker() function (with the updated lines in bold) that goes through the dataset on startup and calls startExtendedTracking() when it finds the chips target.  In this instance the function is implemented in the startup functionality for the sake of convenience, but this function can be called at any time.

Android implementation

The following C++ and JAVA code is used to initialize this feature on Android.

C++ Implementation:

 
// Find “chips” target in dataset
for (int i = 0; i < myDateSet ->getNumTrackables(); i++)
{
    Vuforia::Trackable* trackable = aDataSet.dataSet->getTrackable(i);
    if (strcmp(trackable->getName(), "chips") == 0)
    {
        // Start extended tracking on “chips” target
        if (!trackable->startExtendedTracking())
        {
            LOGD ("Failed to start extended tracking on chips target");
        }
    }
 

Java Implementation:

 
// Find “chips” target in dataset
for (int i = 0; i < myDateSet.getNumTrackables(); i++)
{
    Trackable trackable = myDataSet.getTrackable(i);
    if (trackable.getName().equals( "chips" ))
    {
        // Start extended tracking on “chips” target
        if (!trackable.startExtendedTracking())
        {
            DebugLog.LOGD ("Failed to start extended tracking on chips target");
        }
    }
 

iOS implementation;

You can find this original code in the sampleutils.mm file in the Image Targets sample. This code snippet shows the context in which Extended Tracking is enabled.

 
// Load the tracker data [performed on a background thread]
- (void)loadTracker
{
    // Background thread must have its own autorelease pool
    NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
    BOOL haveLoadedOneDataSet = NO;
   
    if (targetType != TYPE_FRAMEMARKERS)
    {
        // Load all the requested datasets
        for (DataSetItem *aDataSet in targetsList)
        {
            if (aDataSet.path != nil)
            {
                aDataSet.dataSet = [self loadDataSet:aDataSet.path];
                if (haveLoadedOneDataSet == NO)
                {
                    if (aDataSet.dataSet != nil)
                    {
                        // activate the first one in the list
                        [self activateDataSet:aDataSet.dataSet];
                        haveLoadedOneDataSet = YES;
                       
                       
                    // Set extended tracking mode
                    //  start extended tracking mode code
                       
                        // Find “chips” target in dataset
                        for (int i = 0; i < aDataSet.dataSet->getNumTrackables(); i++)
                        {
                            Vuforia::Trackable* trackable = aDataSet.dataSet->getTrackable(i);
                            if (strcmp(trackable->getName(), "chips") == 0)
                            {
                                // Start extended tracking on “chips” target
                                if (!trackable->startExtendedTracking())
                                {
                                    NSLog(@"Failed to start extended tracking on chips target");
                                }
                            }
                        }
                    //  end of extended tracking mode code
 
                    }
                }
               
            }
        }
       
       
        // Check that we've loaded at least one target
        if (!haveLoadedOneDataSet)
        {
            NSLog(@"sampleutils: Failed to load any target");
            appStatus = APPSTATUS_ERROR;
            errorCode = QCAR_ERRCODE_LOAD_TARGET;
        }
    }
 
    // Continue execution on the main thread
    if (appStatus != APPSTATUS_ERROR)
        [self performSelectorOnMainThread:@selector(bumpAppStatus) withObject:nil waitUntilDone:NO];
   
    [pool release];
}
 

Unity implementation

To enable Extended Tracking for an Image Target, you must check the corresponding checkbox in the Inspector when it is highlighted. As the following figure shows, you can find Extended Tracking under the target behaviour component in the target's inspector:


object target with extended tracking

If the application must dynamically enable or disable the extended tracking at run time, you can use the Extended Tracking API's startExtendedTracking and stopExtendedTracking methods as illustrated in the following code example:

 
// Starts/stops extended tracking for a given trackable
public void UseExtendedTracking(string trackableName, bool enabled) 
{
    IEnumerable<TrackableBehaviour> tbs = TrackerManager.Instance.GetStateManager().GetTrackableBehaviours();
 
    foreach (TrackableBehaviour tb in tbs) {
        // check trackable name
        if (tb.TrackableName.Equals( trackableName )) {
            if (enabled) 
                tb.StartExtendedTracking();
            else
                tb.StopExtendedTracking();
            return;
        }
    }
}