Area Target Capture API - BETA

The Vuforia Area Target Capture API enables your application to capture and generate Area Targets at runtime and use them instantly to augment environments

Please note, that the Area Target Capture API is still in a beta state and breaking changes may be introduced in future releases.

Jump to

Introduction

The Area Target Capture API is a controller that creates a capture handle with configuration options. The API directly enables you to implement the logic for capturing rooms and spaces and generate an Area Target from the captured data directly on the device. The Area Target can be immediately used afterwards for tracking the space.

Supported Platforms

  • iOS devices with LiDAR sensor

Signing Credentials

If you are interested in developing with the Capture API, please reach out to vuforia‑feedback@ptc.com stating your use case and your request of signing credentials.

Connectivity

The generation of an Area Target with the Area Target Capture API requires a connection to the Internet. The connection is used to verify the credentials, the remaining allowance (starting with 20 Area Target generations), and to enable the dataset generation. No captured data is being shared or uploaded to PTC services in the process.

Area Target Capture GameObject

In Unity, an Area Target Capture GameObject implements the Area Target Capture Behaviour. Add the Area Target capture to your build to enable capturing spaces. Acquire the Access and Secret Key from the Vuforia Support team, see Signing Credentials.

  1. The GameObject can be found in Vuforia Engine -> Area Target -> Area Target Capture.
  2. Add the AR Camera to your scene and set up Vuforia Configuration with an application license key and the Share Recordings in iTunes setting enabled.

The Area Target Capture Prefab contains a default UI to control the capturing and generation process. You may replace this with your own UI from the component Inspector window of the Area Target Behaviour.

Area Target Behaviour component

The Area Target Capture Behaviour requires an Access Key and a Secret Key as input.

You should also provide the capture with a Target Name with 1-64 characters without spaces and use only ASCII characters (numerals, literals, dash, and underscore).

Display a preview mesh during the capture process for user feedback if you have a dedicated capture phase in your application. When enabled, the Preview Mesh Material will be rendered on the captured surfaces. See Preview Mesh for details.

Additional Assets

Choose which artefacts you wish to generate from the capture. When enabled, the Authoring Structure to generate authoring mesh (glb), occlusion mesh (3dt), and Navigation Mesh files. A .unitypackage can also be generated from the capture; this package will contain the files needed for authoring Area Targets in Unity. See Area Targets in Unity for details. 

The Storage Directory is the persistent data folder in Unity, and when Share recording in iTunes is enabled, files will be easily transferable, See the Recording and Playback feature for details. Each capture receives the Target Name and a timestamp if multiple captures are created during a session. The Storage Directory is a read only cannot be changed.

Preview Mesh

An essential part of the Area Target Capture is the PreviewMesh GameObject. The PreviewMesh is used for visualizing the capturing process by rendering a layer on captured surfaces at runtime. Changing the Display Material will also affect the Preview Mesh Material in the Area Target Behaviour component.

Capture Workflow

Scanning an Environment provides general guidelines for capturing a smaller space with a handheld device. In addition, the Area Target Capture provides a status and status info on the progress, quality, and generation of the Area Target from the scan.

During capturing

There are a couple of things to have in mind besides the general Scanning an Environment practices, namely, the movement during capturing and reaching the size limit of the scan.

  • To initiate the capture after start, some movement of the device is required before the capture instance switches from PREPARING to CAPTURING.
  • Users need to find a balance between too little and too much movement during capturing: You can inform the user with visuals when either of the following two are reported from the status info.
    • The status info CAPTURE SIZE LIMIT REACHED reports when the maximum size of the captured data is reached and that the capture will start discarding earlier captured data to make space for the new one. It is recommended to stop capturing at this point.
    • The CAPTURE SHOULD STOP is a warning message that the capture cannot add more data and the users should stop the capturing immediately.
  • NOTE: When resuming from a paused capture session, users are required to relocalize in the space by pointing the device’s camera towards an already captured area.

Area Target Capture - Native

The Area Target Capture controller creates a capture handle based on a configuration setup. The configuration holds all fields required to successfully capture and generate an Area Target, such as the Device Pose Observer, the generated target's name, the target folder where artefacts will be placed into, the signing credentials, and a set of flags that control the automatic start of the capture and which additional artefacts to generate.

VuController* controller = nullptr;
vuEngineGetAreaTargetCaptureController(engine, &controller);

VuAreaTargetCaptureConfig config = vuAreaTargetCaptureConfigDefault();
config.userAuth = ””; // Signing credentials
config.secretAuth = ””; // Signing credentials
config.targetFolder = “/myFolderPath/”;
config.targetName = “myTargetName”;
config.devicePoseObserver = devicePoseObserver; // Provide a valid device pose observer

// Default configuration values
config.generateAuthoringFiles = VU_TRUE;
config.generatePackages = VU_FALSE;
config.start = VU_FALSE;

Use the provided credentials as authentication in the configuration. Provide a path to an existing directory that is writable. The path can be absolute or relative. All generated files will be placed into the target folder, and therefore, it must exist for the entirety of the capture’s lifetime.

The names of the generated files will be based on the given targetName. The targetName must be an ASCII string that is between 1-64 characters long and only consists of the following characters: numerals (0-9), literals (a-z, A-Z), dash (-), and underscore (_).

Set generateAuthoringFiles to VU_TRUE to generate additional artefacts, namely, the authoring mesh, the occlusion mesh, and the navigation mesh.
Set generatePackages to VU_TRUE to generate a .unitypackage. The generateAuthoringFiles must also be VU_TRUE in this case, otherwise the generation of the .unitypackage fails.

Create an Area Target capture handle for each target generation operation by using a configuration that contains the obligatory fields: authentication, target folder, target name, and the Device Pose Observer. The Device Pose Observer must be active to capture data.

Use the vuAreaTargetCaptureConfigDefault() function to create a config with default values; generatePackages(VU_FALSE), start (VU_FALSE), and generateAuthoringFiles (VU_TRUE).

VuAreaTargetCapture* capture = nullptr;
VuAreaTargetCaptureCreationError error = VU_AREA_TARGET_CAPTURE_CREATION_ERROR_NONE;
vuAreaTargetCaptureControllerCreateAreaTargetCapture(controller, &config, &capture, &error);

Only one capture can exist at a time, therefore, destroy the existing handle before creating a new one. The capture is created with status VU_AREA_TARGET_CAPTURE_STATUS_INITIALIZED, except if the configuration's start flag is VU_TRUE, then the capture is automatically started on creation with status VU_AREA_TARGET_CAPTURE_STATUS_PREPARING.

The create method returns VU_SUCCESS on successfully creating the capture object. On failure, the create method returns VU_FAILED and the error parameter will hold information about the cause.

Capture Handle

You can start, pause, and resume the capture. While the capture is running, it provides status and status info about its state; this includes if there is an error, if there is sufficient data for a target to be generated, and if the capture has reached a state where it cannot continue accumulating data. Generating an Area Target from the capture data can only be done when the capture is stopped. See Capture States overview for more information

vuAreaTargetCaptureStart(capture);

/* Walk around and capture */
vuAreaTargetCapturePause(capture);
vuAreaTargetCaptureResume(capture);
vuAreaTargetCaptureStop(capture);

After starting the capture, you may pause and resume data acquisition. The resume function returns the handle back into the PREPARING or CAPTURING state; which of the states you return to depends on what state it was in when pause was called.

Capture states overview

In the capture start process, the handle will switch from INITIALIZED into a PREPARING status; it starts accumulating data until there is enough information to generate a functional Area Target that is reported with the CAPTURING status. Data acquisition continues to accumulate, but you can at this point stop the capture and trigger the target generation.

vuAreaTargetCaptureStatus status;
vuAreaTargetCaptureGenerate(capture, &status);

Should the vuAreaTargetCaptureGenerate() fail and return VU_FALSE, the status will hold the respective error information. You can attempt a new call to vuAreaTargetCaptureGenerate() if the following errors are reported:

  • VU_AREA_TARGET_CAPTURE_STATUS_ERROR_NO_NETWORK_CONNECTION
  • VU_AREA_TARGET_CAPTURE_STATUS_ERROR_SERVICE_NOT_AVAILABLE
  • VU_AREA_TARGET_CAPTURE_STATUS_ERROR_CANCELED

Since vuAreaTargetCaptureGenerate() is a time-consuming blocking call, you should run it on a helper thread as a background process to avoid blocking the main thread.

You may cancel a concurrently running target generation by using the vuAreaTargetCaptureCancelGeneration() function. Upon successfully cancellation, vuAreaTargetCaptureGenerate() will fail with the error VU_AREA_TARGET_CAPTURE_STATUS_ERROR_CANCELED.

NOTE: On rare occasions, it may take multiple calls to vuAreaTargetCaptureCancelGeneration() to successfully cancel the running target generation.

vuAreaTargetCaptureCancelGeneration(capture);

While the capture is generating, you can query for the associated progress and remaining time estimate.

float progress = 0.0f;
uint32_t remainingTimeSeconds = 0;
vuAreaTargetCaptureGetGenerationProgress(capture, &progress);
vuAreaTargetCaptureGetGenerationTimeEstimate(capture, &remainingTimeSeconds);

NOTE: vuAreaTargetCaptureGetGenerationProgress() may not provide an estimate at all times during generation; it returns VU_FALSE on such occasions.  

As soon as the completed state is reached, all requested artefacts specified in the configuration can be found in the specified targetFolder. Destroy thereafter the handle to free up memory.

vuAreaTargetCaptureDestroy(capture);

NOTE: Calling destroy while the capture is running will cancel any data acquisition and destroy the capture handle. The destroy method cannot cancel the generation process.

Capture Status Info

Help your users during the capturing activity with the status info and on-screen messages.

VuAreaTargetCaptureStatusInfo statusInfo;
vuAreaTargetCaptureGetStatusInfo(capture, &statusInfo);

The following status info values are available:

VU_AREA_TARGET_CAPTURE_STATUS_INFO_NORMAL

The capture is running normally.

VU_AREA_TARGET_CAPTURE_STATUS_INFO_MOVEMENT_NEEDED

The capture is initializing or relocalizing, the user should move around or return towards a previously mapped area in order to resume normal capturing.

VU_AREA_TARGET_CAPTURE_STATUS_INFO_EXCESSIVE_MOVEMENT

The user is moving too quickly.

VU_AREA_TARGET_CAPTURE_STATUS_INFO_CAPTURE_SIZE_LIMIT_REACHED

The capture is removing old data to make room for new data and should be stopped soon.

VU_AREA_TARGET_CAPTURE_STATUS_INFO_CAPTURE_SHOULD_STOP

The capture is unable to add new data, the user should stop the capture.

VU_AREA_TARGET_CAPTURE_STATUS_INFO_GENERATING_PREPARING

The capture is processing the captured data.

VU_AREA_TARGET_CAPTURE_STATUS_INFO_GENERATING_AUTHORING_FILES

The capture is generating the authoring artifacts (3dt, glb).

VU_AREA_TARGET_CAPTURE_STATUS_INFO_GENERATING_DEVICE_DATABASE

The capture is generating the Vuforia device database (dat, xml).

VU_AREA_TARGET_CAPTURE_STATUS_INFO_GENERATING_PACKAGES

The capture is generating the package(s).

Mesh Observer

The Mesh Observer retrieves during capturing mesh updates provided by an Area Target Capture instance. Use the Observer to render a preview mesh of a capture giving users appropriate UX feedback on the covered area. Create and configure the Observer and then use its mesh observations to render meshes at runtime. The Mesh Observer configuration requires an Area Target Capture instance and the parameter activate that can be set to VU_FALSE or VU_TRUE.

VuMeshAreaTargetCaptureConfig config = vuMeshAreaTargetCaptureConfigDefault();

// Default configuration values
config.capture = capture; // Must be a valid VuAreaTargetCapture handle
config.activate = VU_TRUE; // Auto-activates on creation

The Area Target Capture instance is the exclusive source of the Mesh Observations reported by the Mesh Observer.

The Mesh Observer is dependent on the existence of the Area Target Capture instance from which it is drawing the Mesh observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Capture instance.

VuObserver* meshObserver = nullptr;
VuMeshAreaTargetCaptureCreationError error = VU_MESH_AREA_TARGET_CAPTURE_CREATION_ERROR_NONE;
vuEngineCreateMeshObserverFromAreaTargetCaptureConfig(engine, meshObserver, &config, &error);

Destroy the Mesh Observer.

vuObserverDestroy(meshObserver);

Status and status info

Retrieve the Mesh observations from the Engine State. First, create an observation list to hold the Mesh observations.

// Create observation list
VuObservationList* obsList = nullptr;
vuObservationListCreate(&obsList);

// Get the Mesh observation list
vuStateGetObservationsByObserver(state, meshObserver, obsList);
int numObservations = 0;
vuObservationListGetSize(obsList, &numObservations)
for (int i = 0; i < numObservations; i++)
{
   VuObservation* obs = nullptr;
   vuObservationListGetElement(obsList, i, &obs));
   // process observation
}

Each Mesh Observation holds an object of type VuMeshObservationInfo that contains a matrix to transform the meshes to world space, and a list of Mesh Blocks. The VuMeshObservationInfo always hold a complete list of the meshes from the Area Target Capture instance, including meshes that were unchanged, recently updated, or added to the list. Meshes removed from the list will not have their ID reported in the latest retrieved observation.

// Get observation info from a Mesh observation
VuMeshObservationInfo info;
vuMeshObservationGetInfo(obs, &info);

Mesh observation block

Each mesh block holds metadata and the actual mesh data. The metadata consists of a unique ID, a timestamp, and a version number. These fields enable the unique identification of each mesh block and the tracking of potential updates to the mesh block.  Both the ID and the version are positive numbers.

The IDs are unique within a Vuforia session, they are generated at runtime and are not persistent across Vuforia sessions. Moreover, the IDs are not reused, meaning that a newly added mesh block can never have the same ID as an already removed mesh block. Whenever a mesh block has been updated, its timestamp is adjusted to reflect the time of the change, and its version is increased. The actual data of a mesh block consists of an axis aligned bounding box of the mesh, and a pointer to a VuMesh structure that holds the mesh.

int numBlocks = 0;
vuMeshObservationBlockListGetSize(info.meshes, &numBlocks);
for (int b = 0; b < numBlocks; b++)
{
    VuMeshObservationBlock block;
    vuMeshObservationBlockListGetElement(info.meshes, b, &block);
    // Render VuMesh of Mesh Block
    render(block.mesh);
}

 

Learn More

Area Targets