How To Use Object Recognition in Unity

Object Recognition enables you to create apps that can recognize and track objects, such as toys. This article will show you how to add Object Recognition and Object Targets to a Unity project, and how to customize the behaviours exposed through the Object Recognition API and also implement custom event handling.

Before you proceed, make sure to first read Getting Started with the Unity Extension

You will also need to get a license key from the License Manager and add it to your project.

See: 
How To Create an App License
How To add a License Key to your Vuforia App

Adding Object Targets to your Project

The workflow for adding Object Targets to a Vuforia Unity project is very similar to that of the Image Target types.
 
The first step is to import the Vuforia Unity Extension or the Object Recognition Unity sample into your project. The sample will provide a pre-populated and configured Unity scene for Object recognition.
 
To import the extension either select Asset > Import Package in the Unity Editor or double click the extension file on your file system.
 
The prefabs used for Object Recognition can be found in /Assets/Vuforia/Prefabs within the Project panel. You’ll be using the ARCamera and ObjectTarget prefabs.

Adding a Device Database containing an Object Target

  1. Create a Device Database in the Target Manager that includes the Object Target that you want to use. See: Vuforia Object Scanner
  2. Download that database as a *.unitypackage.
  3. Either import the unitypackage from Asset > Import Package in the Unity Editor, or by double clicking the package file on your file system. The database will be added to Streaming Assets/QCAR
Note: with the introduction of ObjectTarget we are also introducing a new type of configuration file that will be copied into Editor/QCAR.  You absolutely need to get these files copied when you import your database.

Adding and Configuring the Object Target Prefab

  1. Add an ARCamera instance to your scene.
  2. Go to the Data Set Load Behaviour in the ARCamera’s Inspector and check the Load and Activate options associated with your database name.
  3. Add an ObjectTarget instance to your scene, both the ARCamera and Object target should be at the same level in the scene Hierarchy.
  4. Go to the Object Target Behaviour in the Object Target’s Inspector and select the Dataset name and Object Target name that you want to associated with the Object Target instance. There is a 1:1 relationship between Object Target prefab instances and Object Targets in a Device Database.
See the Customizing Event Handling section for instructions on how to implement custom event handling for an Object Target instance.

Workflow for Creating a new Object Recognition Scene

1. Import the Object Recognition Unity extension, or the Object Recognition Unity sample.
2. Import a Device Database containing your object target.
3 .Add an ARCamera prefab instance to your scene and configure it to load and activate this Device Database. Be sure to delete the default Main Camera that is present in newly created scenes.
4. Add a Directional Light ( Create > Directional Light ) to illuminate the scene.
5. Add a ObjectTarget prefab instance to your scene and configure it to use your Object Definition.

Object Target Prefab in Project View
 
Define the Data Set and Object Target to use within the ObjectTarget’s Inspector.

Configuring the Object Target Inspector
6. Set the Transform Position of the instance to ( 0,0,0 ) in its Inspector panel.
7. Add any content that you want to augment the object to the scene at the top level of the scene Hierarchy.
8. Use the bounding box to position and scale your content in relation to the target.
9. Now make the content a child of the ObjectTarget instance.

Your scene Hierarchy should look like this..
Resulting Scene Hierarchy
10. Extend the DefaultTrackableEventHandler on the ObjectTarget instance to implement custom event handling, or create your own event handler by implementing the ITrackableEventHandler interface and add that script to the ObjectTarget instance.
11. Test your scene in Play Mode by pressing the Play button at the top of the Editor.  

Using the Bounding Box to register augmentations

Bounding  Box in Scene View
 
A bounding box matching the grid area of the Object Scanning Target will be presented in the Scene view when Show Bounding Box is selected in the Object Target Behaviour component of an Object Target’s Inspector panel. This bounding box can be used to accurately position content that will augment your target object. The bounding box is only rendered in Scene view.
The origin of the coordinate system of the bounding box is in the forward lower left corner of the box, and the coordinate scale is in millimeters, which is the default scene unit.

Recommended augmentation workflow

The Object Target bounding box corresponds to the grid region of the Object Scanning target. You can use the Object Scanning target as a physical reference for registering media in relation to the physical object.
 
1. Place the Object Scanning target in your work area so that it is visible by a webcam attached to your development system. You’ll be using Vuforia Play Mode to prototype the scene.
2. Place the physical object on the grid region at the position where it was originally scanned.
3. Go to the ARCamera Web Cam Behaviour in the Inspector and select the web cam that you want to use.
4. Press the Play button at the top of the Unity Editor. This will activate the Play Mode simulation.
5. Bring the Object Scanning Target in view. You’ll see the bounding box augmented on the target when the physical object is detected. Move your object to align these if they are offset.
6. Toggle between the Game and Scene tabs to iteratively place and verify the position of your content. Use the target grid and bounding box grid to accurately adjust the position of your virtual objects.

Controlling the Object Recognition Tracker

See: RigidBodyUIEventHandler.cs in /Assets/Scripts
 

Obtaining the Object Recognition Tracker instance

ImageTracker mTracker = TrackerManager.Instance.GetTracker<ImageTracker>();

Methods

Starting and stopping the Object Recognition Tracker
Start()   starts the Object Recognition Tracker
Stop()    stops the Object Recognition Tracker
Reset()  resets the Object Recognition Tracker

Customizing event Handling

The ObjectTarget prefab implements the ITrackableEventHandler interface in DefaultTrackableEventHandler.cs to enable and disable the Renderers and Colliders of any objects that are children of the ObjectTarget instance. Extend this code to implement custom event handling.
private void OnTrackingFound()
        {
            Renderer[] rendererComponents = GetComponentsInChildren<Renderer>(true);
            Collider[] colliderComponents = GetComponentsInChildren<Collider>(true);

            // Enable rendering:
            foreach (Renderer component in rendererComponents)
            {
                component.enabled = true;
            }

            // Enable colliders:
            foreach (Collider component in colliderComponents)
            {
                component.enabled = true;
            }

            Debug.Log("Trackable " + mTrackableBehaviour.TrackableName + " found");
        }


        private void OnTrackingLost()
        {
            Renderer[] rendererComponents = GetComponentsInChildren<Renderer>(true);
            Collider[] colliderComponents = GetComponentsInChildren<Collider>(true);

            // Disable rendering:
            foreach (Renderer component in rendererComponents)
            {
                component.enabled = false;
            }

            // Disable colliders:
            foreach (Collider component in colliderComponents)
            {
                component.enabled = false;
            }

            Debug.Log("Trackable " + mTrackableBehaviour.TrackableName + " lost");
        }

Activating Loaded Datasets at runtime

See: ObjectTargetUIEventHandler.cs in /Assets/Scripts
/// <summary>
    /// Sample implementation of how to activate the datasets through code
    /// </summary>
    /// <param name="datasetPath"></param>
    private void ActivateDatasets(string datasetPath)
    {
        ObjectTracker objectTracker = TrackerManager.Instance.GetTracker<ObjectTracker>();
        IEnumerable<DataSet> datasets = objectTracker.GetDataSets();

        IEnumerable<DataSet> activeDataSets = objectTracker.GetActiveDataSets();
        
        List<DataSet> activeDataSetsToBeRemoved = activeDataSets.ToList();

        //1. Loop through all the active datasets and deactivate them.
        foreach (DataSet ads in activeDataSetsToBeRemoved)
        {
            objectTracker.DeactivateDataSet(ads);
        }

        //Swapping of the datasets should not be done while the ObjectTracker is working at the same time.
        //2. So, Stop the tracker first.
        objectTracker.Stop();

        //3. Then, look up the new dataset and if one exists, activate it.
        foreach (DataSet ds in datasets)
        {
            if (ds.Path.Contains(datasetPath))
            {
                objectTracker.ActivateDataSet(ds);
            }
        }

        //4. Finally, start the object tracker.
        objectTracker.Start();
    }

How To Support Multiple Object Targets in a Unity Scene

Object Recognition supports the detection and tracking of multiple Object Targets in an Object Recognition experience. Up to 2 Object Targets can be tracked simultaneously.

Use the following instructions to configure your Object Recognition scene to support multiple Object Targets in Unity. Set the Maximum Simultaneous Object Targets value in the VuforiaConfiguration asset which is accessible from the ARCamera's Inspector panel via the Open Vuforia Configuration button. You can also search for VuforiaConfiguration in the Project panel.

For Object Targets, the maximum is 2.



If you have enabled Extended Tracking on an Object Target, you must also enable it on any other Object Targets in the scene. Be aware that the use of Extended Tracking is not recommended for objects that may be moved during the Object Recognition experience.

How To Support Moving Objects

Object Recognition can track moving objects, provided the motion is slow and smooth. You'll need to configure your Object Targets to support motion using the following steps. To support movement, you’ll need to disable Extended Tracking on the Object Target.

For Unity apps, Extended Tracking can be disabled by unchecking  the Extended Tracking box in the Object Target Behaviour component in the Object Target's Inspector panel.



Extended Tracking can also be enabled and disabled programmatically using the startExtendedTracking() and stopExtendedTracking() methods exposed on the Object Trackable. Extended tracking is disabled by default. You won’t need to call stopExtendedTracking() unless you had previously started extended tracking for that Trackable. 

See: Extended Tracking

How To use Object Targets with other Target Types

Object Targets can be used in combination with Image Targets, Multi Targets, and Cylinder Targets by loading and activating multiple target datasets. When you download a Device Database that contains both Object Targets and alternate target types, the database archive will contain two distinct datasets, each with its own XML configuration file. You will need to load and activate both of these datasets to be able to detect and track Object Targets along with the other targets. The procedure for doing this depends on which development tools you are using.
These same steps can also be used to load and activate two separately downloaded Device Databases.

Considerations

  • A maximum of 20 Object Targets can be included in a single database
  • Object Targets cannot be used to initialize a Smart Terrain experience, though they can be used in conjunction with Smart Terrain if an alternate target type is used as the initialization target.
  • Object Targets support both persisted and non-persisted modes of Extended Tracking. Don't use Extended Tracking if you intend for the user to move the target.

Developing in Unity

For Unity development, download the Device Database as a *.unitypackage. You can then double click the package, or select Assets > Import Package > Custom Package in the Unity Editor, to automatically import the datasets into the correct directory in your project (/Streaming Assets/QCAR). Then set the Load and Activate checkboxes on the appropriate Databases in the VuforiaConfiguration asset which is accessible from the ARCamera's Inspector panel via the Open Vuforia Configuration button. You can also search for VuforiaConfiguration in the Project panel.

Data set load behaviour

Additional Articles:

How To Use Object Recognition in an Android App
How To Load and Activate Multiple Device Databases at Runtime
How To Add Targets to a Device Database
How To Detect and Track Multiple Targets Simultaneously

How to Unlock content in response to Object Recognition events

Object Targets can have multiple detection and tracking states as defined by the Status of their TrackableResult. This result enables you to develop apps that unlock content when the Object Target is detected and is being tracked.

You can use this information to determine when an Object Target has been recognized, when it is being actively tracked, and when tracking and detection have been lost on an object that was previously being tracked.

The handling of detection and tracking events is demonstrated in the Object Target samples for Android, iOS and Unity. 


See: How To Use the Trackable Base Class

How To Align Digital Content on an Object

Recommended Augmentation Workflow

The Object Target bounding box in your Object Recognition Unity scene corresponds to the grid region of the Object Scanning target. You can use the Object Scanning target as a reference for positioning content in relation to the physical object that you have scanned.

Note: You’ll be using Vuforia Play Mode in Unity.
 

Steps to placing content:

  1. Place the Scene and Game Views side by side in your Editor Window. See Fig. 2.

  2. Select the webcam that you want to use in the ARCamera’s Webcam Behaviour.

  3. Place the Object Scanning target in your work area so that it is visible by you webcam.  See Fig. 2.

  4. Place the physical object on the grid region at the position where it was originally scanned.

  5. Press the Play button at the top of the Unity Editor. This will activate the Vuforia Play Mode simulation.

  6. Bring the Object Scanning Target in view. You’ll see the bounding box augmented on the target when the physical object is detected. Move your object to align these if they are offset.

  7. Compare the Game and Scene views to iteratively position and verify the registration of your content onto or near the physical object.  See Fig. 3-5

    • View the object from multiple perspectives using your webcam. You can use the Scene Gizmo to achieve the same perspective in the Scene View.
    • Use the Scanning Target grid and Bounding Box grid to accurately adjust the position of your virtual objects.
    • Moving the content in Scene View will cause it to move to the same position in Game View.
    • Be aware that your content will move back to its original position when you turn off Play Mode!!
  8. When you’re happy with the position of your content, be sure to record its position and scale values in the Inspector.
  9. Exit Play Mode and re-enter the position and scale values for your content.                            

Fig. 1
Bounding Box in Scene View
The Object Target Bounding Box in the Scene View

Fig. 2
size by side scene and game views
Side by Side Scene and Game Views  in Play Mode

Fig. 3
Fitting a mesh to the front of the car
 
Fig. 4
Alt perspective on mesh fitting
Fig. 5
Fitting a particle emitter

How to Use an occlusion model

An occlusion model enables you to mask the surfaces of a physical object so that virtual objects are not rendered over them when they are behind the object in your scene. This effect relies on depth masking and provides much more realistic rendering for scenes that combine physical and virtual objects.

Depth masking using an occlusion model is accomplished by developing a 3D model of the regions of the object that you want to mask, and applying a depth mask shader to that model. The depth mask will prevent any objects behind the masked region from being rendered.  An example of a depth mask shader is provided with the Vuforia Unity Extension (Assets/Vufroria/Shaders or Assets/Vuforia/Materials).

An occlusion model can match the exact geometry of the physical object that it corresponds to, or it can be a simplified representation of the object, such as its bounding box. The latter technique is an easy way to occlude physical objects with simple geometries, like boxes, balls and cylinders. This approach is similar to use of Collider objects for physics simulation in Unity.


 volumetric occlusion model
A volumetric occlusion model
 
When using a geometrically accurate occlusion model, it’s recommended that you only use the exterior surfaces of the model, at the lowest polygon count needed to provide satisfactory masking of your physical object. You can use a 3D editor to simplify your model by reducing the polygon count and removing any duplicated or hidden (e.g. internal ) geometries.

Tip: If you use an occlusion model and are also augmenting the surface of the physical object with content, be sure to test the rendering results on the device platforms that you intend to deploy to. When 3D models are closely aligned, they can become confused in the renderer's depth buffer. This is an problem known as Z-Fighting ( Z is for Z Axis ) and can result in noisy rendering artifacts. To avoid Z-Fighting, use the MODE_OPTIMIZE_QUALITY setting on the ARCamera and offset your models so that they are not overlapping each other. 

geometrically accurate occlusion model
A geometrically accurate occlusion model.

How to configure a model to occlude for a physical object

See: "How To Align Digital Content on an Object" above to learn how to position virtual content on a physical object using Object Recognition in Unity.

1. Import your occlusion model to your Unity project, or create a simple bounding model by selecting a primitive mesh geometry from Game Object > Create Other in the Editor menu.
2. Drag the model onto your Object Target instance in the Hierarchy so that it is a child of the Object Target instance.

scene hierarchy

3. Follow the steps in "How To Align Digital Content on an Object" to accurately position your model in relation to its corresponding physical object.
geometrically accurate occlusion model

4. Add a Depth Mask Shader to your model.

depth shader selection in the Inspector

5. Add another model to your scene to verify that your occlusion model is masking the physical object accurately. Position it near the occlusion model.
6. Start Vuforia Play Mode to evaluate the occlusion effect in the scene.

testing the depth mask

7. Verify the accuracy of your occlusion model registration from multiple positions in the scene.

verifying from the front