How to Use Cloud Recognition

This article will introduce you to the steps as presented in the Vuforia Core Samples for creating a basic Vuforia Engine cloud recognition app in Unity and the native SDKs for Android, iOS, and UWP. Use cloud recognition in your application when you plan to track your many targets or if your targets frequently are updated.

Sections

To learn more about the benefits, licensing, and use cases of Cloud Recognition, please refer to the Cloud Recognition article.

Accessing Sample Code

The following sample code snippets presented are accessible in the Core Features section of the Vuforia Engine Developer Portal Samples page. Retrieve the latest sample project suitable for your development environment.

Platform  Description
Unity In the Books sample, refer to the CloudRecoEventHandler.cs script attached to the CloudRecognition prefab. The OnNewSearchResult method shows how to get a targetSearchResult object, from which you can then get the metadata.
Android In the Books sample project, refer to the code in the Books.java file.
iOS In the Books sample project, refer to the code in the BooksEAGLView.mm file.

Implementing in Unity

The following steps explain how to create a cloud recognition app. The app reacts to a cloud target detection event, retrieves the metadata associated with the target, and displays the metadata text.

Prerequisites 

  • Unity 2019.2.0+ or later with the latest Vuforia SDK Package
  • Your preferred code editor (Visual Studio, Mono Develop, XCode, etc.)V
  • uforia Engine Developer Account

Instructions

  1. Create a new project in Unity.
  2. Create a license key for your project.
  3. Add the license key to your Vuforia Engine app.
  4. In the Hierarchy window, delete the Main Camera object.
  5. In the GameObject menu, select and add Vuforia Engine -> AR Camera
  6. In the GameObject menu, select Vuforia Engine -> Cloud Image -> Cloud Recognition. A new Cloud Recognition object is created.
  7. In the Inspector window of your Cloud Recognition GameObject, enter your cloud database client keys in the Access Key and Secret Key fields. For information about creating a cloud database, refer to the Cloud Databases article. 
  8. Create a C# script named SimpleCloudRecoEventHandler.cs 
  9. Attach the SimpleCloudRecoEventHandler script to the Cloud Recognition GameObject.
  10. Implement the SimpleCloudRecoEventHandler script by referencing the following code samples:
    1. Implement the RegisterEventHandlers interface and register it with the CloudRecoBehaviour:
using UnityEngine;
using Vuforia;

public class SimpleCloudRecoEventHandler : MonoBehaviour
{
    private CloudRecoBehaviour mCloudRecoBehaviour;
    private bool mIsScanning = false;
    private string mTargetMetadata = "";
    
    // Register cloud reco callbacks
    void Awake()
    {
        mCloudRecoBehaviour = GetComponent<CloudRecoBehaviour>();
        mCloudRecoBehaviour.RegisterOnInitializedEventHandler(OnInitialized);
        mCloudRecoBehaviour.RegisterOnInitErrorEventHandler(OnInitError);
        mCloudRecoBehaviour.RegisterOnUpdateErrorEventHandler(OnUpdateError);
        mCloudRecoBehaviour.RegisterOnStateChangedEventHandler(OnStateChanged);
        mCloudRecoBehaviour.RegisterOnNewSearchResultEventHandler(OnNewSearchResult);
    }
    //Unregister cloud reco callbacks when the handler is destroyed
    void OnDestroy()
    {
        mCloudRecoBehaviour.UnregisterOnInitializedEventHandler(OnInitialized);
        mCloudRecoBehaviour.UnregisterOnInitErrorEventHandler(OnInitError);
        mCloudRecoBehaviour.UnregisterOnUpdateErrorEventHandler(OnUpdateError);
        mCloudRecoBehaviour.UnregisterOnStateChangedEventHandler(OnStateChanged);
        mCloudRecoBehaviour.UnregisterOnNewSearchResultEventHandler(OnNewSearchResult);
    }
}
  1. Implement the OnInitialized(), OnInitError() and OnUpdateError() methods for the interface:
  public void OnInitialized(TargetFinder targetFinder) {
      Debug.Log ("Cloud Reco initialized");
  }
  public void OnInitError(TargetFinder.InitState initError) {
      Debug.Log ("Cloud Reco init error " + initError.ToString());
  }
  public void OnUpdateError(TargetFinder.UpdateState updateError) {
      Debug.Log ("Cloud Reco update error " + updateError.ToString());
  }
  1. Implement the OnStateChanged() method of the interface to determine whether Vuforia Engine is scanning the cloud:
  public void OnStateChanged(bool scanning) {
  mIsScanning = scanning;
      if (scanning)
      {
      // clear all known trackables
      var tracker = TrackerManager.Instance.GetTracker<ObjectTracker>();
      tracker.GetTargetFinder<ImageTargetFinder>().ClearTrackables(false);
      }
  }
  1. Implement the OnNewSearchResult() method:
  // Here we handle a cloud target recognition event
  public void OnNewSearchResult(TargetFinder.TargetSearchResult targetSearchResult) {
  TargetFinder.CloudRecoSearchResult cloudRecoSearchResult = 
      (TargetFinder.CloudRecoSearchResult)targetSearchResult;
  // do something with the target metadata
  mTargetMetadata = cloudRecoSearchResult.MetaData;
  // stop the target finder (i.e. stop scanning the cloud)
  mCloudRecoBehaviour.CloudRecoEnabled = false;
 }
  1. Implement the OnGUI() method to display the current scanning state and the metadata of the last cloud target detected:
  void OnGUI() {
      // Display current 'scanning' status
      GUI.Box (new Rect(100,100,200,50), mIsScanning ? "Scanning" : "Not scanning");
      // Display metadata of latest detected cloud-target
      GUI.Box (new Rect(100,200,200,50), "Metadata: " + mTargetMetadata);
      // If not scanning, show button
      // so that user can restart cloud scanning
      if (!mIsScanning) {
          if (GUI.Button(new Rect(100,300,200,50), "Restart Scanning")) {
          // Restart TargetFinder
          mCloudRecoBehaviour.CloudRecoEnabled = true;
          }
      }
  }
  1. Save the current scene.
  2. Build and run the project.

Showing a 3D augmentation object

Perform the following steps to show a 3D augmentation object on top of your cloud target upon detection:

  1. In the GameObject menu, select Vuforia Engine > Cloud Image > Cloud Image Target to add a new Image Target object. 
  2. Right-click ImageTarget and select 3D Object > Cube to create a simple cube object. 
  3. Center and scale the cube object appropriately so that it fits nicely on top of the Image Target plane in the scene.
  4. Add directional lighting to the scene so that any augmentation appears shaded.
  5. Modify the SimpleCloudRecoEventHandler script by adding the following code to the OnNewSearchResult() method. This code programmatically instantiates an ImageTarget that corresponds to the one detected by the Cloud Recognition engine:
// Build augmentation based on target 
        if (ImageTargetTemplate) { 
            // enable the new result with the same ImageTargetBehaviour: 
            ObjectTracker tracker = TrackerManager.Instance.GetTracker<ObjectTracker>(); 
            tracker.GetTargetFinder<ImageTargetFinder>().EnableTracking(targetSearchResult, ImageTargetTemplate.gameObject); 
        }
  1. Add a public member variable named ImageTargetTemplate to the SimpleCloudRecoEventHandler script:
public ImageTargetBehaviour ImageTargetTemplate;
  1. Select the CloudRecognition object in the scene.
    The Inspector should display the Image Target Template field in the Simple Cloud Handler component.
  2. In the Hierarchy window, drag ImageTarget onto the Image Target Template in the Inspector window.
  3. Save the current scene.
  4. Build and run your project.

Implementing for native SDKs

The Cloud Recognition APIs is accessible from the ObjectTracker class as a service by calling getTargetFinder(). The Cloud Recognition Service is controlled with the TargetFinder class and detected recognition events are managed with the CloudRecoSearchResult class.
Differently from other tracking API in the SDK, where you get continuous tracking updates via the State, Cloud Recognition does not necessarily get continuous information; you need to explicitly make an update. This is done by calling updateQueryResults() on the TargetFinder.

Initializing the system

To use Cloud Recognition you first need to initialize the system and set the credentials for your image database. The following code initializes and de-initializes Cloud Reco:

C++:

  Vuforia::TargetFinder* finder;
  
  const char* const kAccessKey = "Insert access key here";
  const char* const kSecretKey = "Insert secret key here";
  
  onApplicationInit()
  {
   Vuforia::TargetFinder* targetFinder = objectTracker->getTargetFinder   (Vuforia::ObjectTracker::TargetFinderType::CLOUD_RECO);
      assert(targetFinder != NULL);
  
      // Initialize Cloud Reco with project credentials:
      if (targetFinder->startInit(kAccessKey, kSecretKey))
      {
          targetFinder->waitUntilInitFinished();
      }
  }
  
  onApplicationDeinit()
  {
      // Deinitialize Visual Search
      targetFinder->deinit());
  }

Java:

private static final String kAccessKey = "Insert access key here";
private static final String kSecretKey = "Insert secret key here";       
 onApplicationInit()
 {
 TargetFinder targetFinder = objectTracker.getTargetFinder(ObjectTracker.TargetFinderType.CLOUD_RECO)    ;
       
     // Initialize Cloud Reco with project credentials:
     if (targetFinder.startInit(kAccessKey, kSecretKey))
     {
         targetFinder.waitUntilInitFinished();
     }
 }
  
 onApplicationDeinit()
 {
     // Deinitialize Cloud Reco
     TargetFinder finder = objectTracker.getTargetFinder(ObjectTracker.TargetFinderType.CLOUD_RECO);
     targetFinder.deinit());
 }

Starting and Stopping a Cloud Recognition Search

The following sample code shows what a simple application needs to do to start Cloud Recognition when the application is started (or resumed) and to stop it when the application is paused:

C++:

  Vuforia::TargetFinder* finder;
  
  onApplicationResume()
  {
      // Start cloud based recognition
      finder->startRecognition();
  }
  
  onApplicationPause()
  {
      // Stop cloud based recognition
      finder->stop();
  }

Java:

onApplicationResume()
{
    // Start cloud based recognition
    TargetFinder finder = objectTracker.getTargetFinder(ObjectTracker.TargetFinderType.CLOUD_RECO);
    finder.startRecognition();
}

onApplicationPause()
{
    // Stop cloud based recognition
    TargetFinder finder = objectTracker.getTargetFinder(ObjectTracker.TargetFinderType.CLOUD_RECO);
    finder.stop();
}

Handling Search Results

During the execution, the developer must regularly check for the availability of new Cloud Recognition search results and, if desired, enable the target(s) for tracking.

This can be done in the Vuforia_onUpdate() method of the UpdateCallback class.

C++:

Vuforia::TargetFinder* finder;

void MyVuforia_OnUpdate()
{
    // Check if there are new results available:

    TargetFinderQueryResult queryResults = finder->updateQueryResults();
    if (queryResults.status == Vuforia::TargetFinder::UPDATE_RESULTS_AVAILABLE)
    {
        // Iterate through the new results:
        for (const Vuforia::TargetSearchResult* result: queryResults.results) {
        {
         const Vuforia::CloudRecoSearchResult* cloudRecoResult = static_cast<const    Vuforia::CloudRecoSearchResult*>(result);
            // Get the target metadata
            const char* metadata = cloudRecoResult->getMetaData();

            // Check if this target is suitable for tracking:
            if (cloudRecoResult->getTrackingRating() > 0)
            {
                // Enable this target for tracking:
                finder->enableTracking(cloudRecoResult);
            }
        }
    }
}

Java:

void MyVuforia_OnUpdate()
{
    TargetFinder finder = objectTracker.getTargetFinder(ObjectTracker.TargetFinderType.CLOUD_RECO);
    // Check if there are new results available:
    TargetFinderQueryResult queryResults = finder.updateQueryResults();
    if (queryResults.getStatus() == TargetFinder.UPDATE_RESULTS_AVAILABLE)
    {
        // Iterate through the new results:
        for (int i = 0; i < queryResults.getResults().size(); ++i)
        {
            CloudRecoSearchResult result = (CloudRecoSearchResult)queryResults.getResults().at(i);
            // Get the target metadata
            String metadata = result.getMetaData();

            // Check if this target is suitable for tracking:
            if (result.getTrackingRating() > 0)
            {
                // Enable this target for tracking:
                finder.enableTracking(result);
            }   
        }
    }
}

Cloud Recognition Filter Mode

You can modify the filter applied to Cloud Recognition search results. This functionality is intended for diagnostic purposes only, and it is not recommended for use in production apps.

The updateQueryResults() method accepts a filter setting value of either FILTER_NONE ( disable filtering ) or FILTER_CURRENTLY_TRACKED ( enable filtering for targets being tracked ). The latter is the default behavior of the TargetFinder.

By default, Cloud Recognition query results are not returned for targets that are currently being tracked. If the FILTER_NONE argument is used, these results will be returned, regardless of whether the same target has already been enabled for tracking. The purpose of removing this filter is to enable developers to determine if recognition events are unexpectedly occurring on the same target due to user behavior. Make sure that you do not re-enable the target for tracking as this will interrupt the tracking process.

See below for Filter Mode values:

 /// Filter modes to be passed into updateQueryResults() function
    enum
    {
        FILTER_NONE = 0,///< No results are filtered, all successful queries are returned
        FILTER_CURRENTLY_TRACKED = 1  ///< Filter out targets that are currently being tracked (Most Common)
    };

Update visual search results with a filter value

/**
     *  Clears and rebuilds the list of updateQueryResults with results found
     *  since the last call to updateQueryResults (). Returns the status code
     *  UPDATE_RESULTS_AVAILABLE if new search results have been found.
     *  By default, targets that are already enabled for tracking are not included
     *  in the list of updateQueryResults unless the target or its associated
     *  meta data has been updated since they were last enabled for tracking.
     */
    virtual int updateQueryResults (int filter = FILTER_CURRENTLY_TRACKED) = 0;