Mesh Observer for Area Targets

The Mesh Observer API provides runtime meshes from Area Targets and Area Target Captures. Create the Mesh Observer to add occlusion for your Area Target environment or to provide a preview for your Area Target Capture.

Prerequisites

The Mesh Observer requires either an Area Target Observer or an Area Target Capture instance as its source. 

Create Mesh Observer from Area Target

Create the Mesh Observer with a VuMeshAreaTargetConfig that associates the Mesh Observer with an Area Target Observer and its occlusion mesh. Thereafter, retrieve the Mesh Observations to render the result. See the Render Mesh at Runtime for details and example snippet.

The Mesh Observer configuration VuMeshAreaTargetConfig requires an Area Target Observer, an external path to the Area Target’s occlusion .3dt file, and the parameter activate that can be set to VU_FALSE or VU_TRUE.

// Create the Mesh Observer configuration from Area Target source
VuMeshAreaTargetConfig meshObserverConfig = vuMeshAreaTargetConfigDefautlt();

// Must be a valid Area Target Observer
meshObserverConfig.areaTargetObserver = areaTargetObserver;

// Must be a valid path to an existing occlusion 3dt file
meshObserverConfig.occlusionMeshPath = "pathAsString";

// Auto-activation on creation is the default
meshObserverConfig.activate = VU_TRUE;

// Create the Mesh Observer 
VuObserver* meshObserver = nullptr;
VuMeshAreaTargetCreationError error = VU_MESH_AREA_TARGET_CREATION_ERROR_NONE;
vuEngineCreateMeshObserverFromAreaTargetConfig(engine, &meshObserver, &meshObserverConfig, &error);

The Area Target Observer is the exclusive source of the Mesh Observations reported by the Mesh Observer. The Mesh Observer aligns its Mesh Observations with respect to the areaTargetObserver.

The occlusionMeshPath is the external path to the occlusion mesh artifact (occlusion.3dt). The occlusion mesh must originate from the Area Target that is being tracked by areaTargetObserver.

Destroy the Mesh Observer.

vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Area Target Observer from which it is drawing the Mesh observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Observer.

Create Mesh Observer from Area Target Capture

Create the Mesh Observer and retrieve live mesh updates during capturing from an Area Target Capture instance and provide users helpful UX feedback on the scanned areas during capturing. The Mesh Observer configuration requires an Area Target Capture instance and the parameter activate that can be set to VU_FALSE or VU_TRUE.

// Create the Mesh Observer configuration from Area Target Capture source
VuMeshAreaTargetCaptureConfig config = vuMeshAreaTargetCaptureConfigDefault();

// Must be a valid VuAreaTargetCapture instance
config.capture = capture; 

// Auto-activation on creation is the default 
config.start= VU_TRUE;

// Create the Mesh Observer 
VuObserver* meshObserver = nullptr;
VuMeshAreaTargetCaptureCreationError error = VU_MESH_AREA_TARGET_CAPTURE_CREATION_ERROR_NONE;
vuEngineCreateMeshObserverFromAreaTargetCaptureConfig(engine, meshObserver, &config, &error);

The Area Target Capture instance is the exclusive source of the Mesh Observations reported by the Mesh Observer.

Destroy the Mesh Observer.

vuObserverDestroy(meshObserver);

The Mesh Observer is dependent on the existence of the Area Target Capture instance from which it is drawing the Mesh Observations from. Therefore, destroy the Mesh Observer before you destroy the Area Target Capture instance.

Mesh Observations

Retrieve the Mesh Observations from the Engine State. Create an observation list to hold the Mesh Observations. Retrieving the observation information from the state is the same for the Mesh Observer from an Area Target and from an Area Target Capture.

// Create observation list
VuObservationList* observationList = nullptr;
vuObservationListCreate(&observationList);

// Get mesh observation list
int numObservations = 0;
vuStateGetObservationsByObserver(state, meshObserver, observationList);

// Get mesh observation list size
vuObservationListGetSize(observationList, &numObservations);

Each Mesh Observation holds a pose of type VuPoseInfo, a respective status info of type VuMeshObservationStatusInfo, and an object of type VuMeshObservationInfo that holds a list of meshes of type VuMeshObservationBlock.

Mesh Observation Status and Status Info

Each Mesh Observation holds a VuPoseInfo object to align the content of the observation with the world coordinate system. Additionally, each Mesh Observation holds a respective VuMeshObservationStatusInfo to provide more detailed information about the pose.

The reported pose status and status info pairs are shown in the table below.

STATUS

STATUS_INFO

NO_POSE

VU_MESH_OBSERVATION_STATUS_INFO_NOT_OBSERVED

EXTENDED_TRACKED

VU_MESH_OBSERVATION_STATUS_INFO_NORMAL

LIMITED

VU_MESH_OBSERVATION_STATUS_INFO_RELOCALIZING

A Mesh Observer with an Area Target source reports a pose status that corresponds to the pose status of the Area Target Observer. A Mesh Observer with an Area Target Capture source reports either NO_POSE or a pose with status EXTENDED_TRACKED, but never a pose with status LIMITED.

Mesh Observation Info

Each Mesh Observation holds an object of type VuMeshObservationInfo that holds a list of meshes of type VuMeshObservationBlock.

// Get observation info from a mesh observation
VuMeshObservationInfo info;
vuMeshObservationGetInfo(meshObservation, &info);

The Mesh Observation info always holds a complete list of meshes. It includes meshes that have been updated, recently added, and meshes that have remained unchanged. Meshes removed from the list will not have their ID reported in the latest retrieved observation.

Mesh observation block

Each VuMeshObservationBlock holds metadata and the actual mesh data. The metadata consists of a unique ID, a timestamp, and a version number. These fields enable the unique identification of each mesh block and allow for tracking of updates to the mesh block.

The VuMeshObservationBlock ID and version are positive numbers. The IDs are unique within a Vuforia session. They are generated at runtime and are not persistent across Vuforia sessions. Moreover, the IDs are not reused, meaning that a newly added mesh block can never have the same ID as an already removed mesh block. Whenever a mesh block is updated, its timestamp is adjusted to reflect the time of the change, and its version is increased.

The actual data of a mesh block consists of a transformation matrix, the axis aligned bounding box of the mesh, and a pointer to a VuMesh structure that holds the mesh. The transform holds the transformation matrix that aligns the block with the observation coordinate system. The Mesh Observation, on the other hand, holds a pose to align the observation with the world coordinate system. The snippet below demonstrates how to assemble a matrix that transforms a mesh block to the world coordinate system.

VuPoseInfo poseInfo;
vuObservationGetPoseInfo(meshObservation, &poseInfo);
VuMatrix44F modelMatrix = vuMatrix44FMultiplyMatrix(poseInfo.pose, block.transform);

NOTE: A VuPoseInfo with status NO_POSE does not hold a pose to align the Mesh Observation with the world coordinate system

Render Mesh at Runtime

Below you find a simple code snippet that iterates over all Mesh Observations reported by a Mesh Observer and renders the mesh blocks of each Mesh Observation that has a valid pose.

// Create list that holds the mesh observations
VuObservationList *observationList = nullptr;
vuObservationListCreate(&observationList);

// Retrieve mesh observations from the Mesh Observer
vuStateGetObservationsByObserver(&state, meshObserver, observationList);

// Retrieve number of mesh observations held by the list
int numObservations = 0;
vuObservationListGetSize(observationList, &numObservations);

// Iterate over all mesh observations
for (int o = 0; o < numObservations; o++)
{
    // Retrieve the observation
    VuObservation *observation = nullptr;
    vuObservationListGetElement(observationList, o, &observation));

    // Retrieve the pose info of the observation
    VuPoseInfo poseInfo;
    vuObservationGetPoseInfo(observation, &poseInfo);

    // Skip if the observation does not have a pose
    if (poseInfo.poseStatus == VU_OBSERVATION_POSE_STATUS_NO_POSE)
    {
        continue;
    }

    // Retrieve mesh observation info that holds the list of mesh blocks
    VuMeshObservationInfo info;
    vuMeshObservationGetInfo(observation, &info));

    // Retrieve number of mesh blocks
    int32_t numBlocks = 0;
    vuMeshObservationBlockListGetSize(info.meshes, &numBlocks));

    // Iterate over all mesh blocks
    for (int b = 0; b < numBlocks; b++)
    {
        VuMeshObservationBlock block;
        vuMeshObservationBlockListGetElement(info.meshes, b, &block);

        // Assemble matrix to transform mesh block to world coordinates
        VuMatrix44F modelMatrix = vuMatrix44FMultiplyMatrix(poseInfo.pose, block.transform);

        // Render mesh block
        render(modelMatrix, block.mesh);
    }
}

// Destroy mesh observation list
vuObservationListDestroy(observationList);