Using the Vuforia Fusion Platform Handle

Vuforia Engine provides access to ARKit and ARCore specific functionalities. This can be used to combine Vuforia functionality with functionality that only exists in ARKit and ARCore. For instance, an application can access plane boundaries or additional lighting information through this mechanism.

Detailed information on ARKit can be found here, and information on ARCore can be found here.

NOTE: The Platform Handle mechanism is only exposed in the native Vuforia Engine API. For Unity, use Vuforia Engine’s integration with ARFoundation instead.

Platform Access

Retrieve the platform controller to get access to platform-specific functionality by calling vuEngineGetPlatformController(). Use then vuPlatformControllerGetFusionProviderPlatformType() to return a VuFusionProviderPlatformType struct with the platform provider info on the following supported types.

VU_FUSION_PROVIDER_PLATFORM_TYPE_ARKIT
VU_FUSION_PROVIDER_PLATFORM_TYPE_ARCORE

NOTE: Other platform fusion provider types exist but do not need handles to be provided.

The values contained within the struct are platform-specific and need to be used with platform-specific code. Please refer to the Vuforia Reference Documentation for more details on the enum types.
See also Vuforia Fusion for a more general introduction.

API Usage - Best Practices

Pointer ownership

Vuforia owns the pointers contained in this struct; developers must exercise caution while using them.

Do not release or destroy the session or the frame.
Do not pause the session.
Do not reconfigure the session.

NOTE: Doing any of the above will cause Vuforia's handling of the information from the provider to fail in undefined ways.

Camera ownership

Vuforia owns the camera throughout this process. In order to change any camera properties such as the video mode or autofocus mode, you need to use Vuforia Engine’s camera control APIs.

Pointer validity

A valid value for the session will be available after the camera has been started and will remain valid until the camera is stopped.
A valid value for the frame will be available after the camera has been started and will only be valid for the duration of one frame.
The caller needs to request the frame information for every new frame.

Platform-Specific aspects

No ARKit delegate callbacks are available on iOS.
- All information supplied through the callbacks is available via the ARSession.
No Java-binding support on Android.
- Developer must use the ARCore C-API and if needed should use JNI to access this functionality from Kotlin or Java applications.
Some parameters of the returned struct also have a platform-specific lifecycle.

Expected usage

It is expected that these calls will be most useful during the rendering phase of the application, and largely in a read-only mode from within an application's render loop.

Vuforia tracking Behavior

The behavior of Vuforia Engine has been modified to provide developers with more flexibility when making use of the vuPlatformControllerGetFusionProviderPlatformType() API call.

Engine behavior on ARCore and ARKit

In addition to horizontal planes, vertical planes will be enabled when VuPlatformARKitInfo or VuPlatformARCoreInfo are populated with the relevant API calls.
- Vuforia HitTest calls will continue to use only horizontal planes as specified by the VuHitTestHint.
On Device Pose Observer reset, only Anchors that have been created by Vuforia Engine will be destroyed.

ARKit Platform-specific Pointer

For complete details of the ARKit API, developers should refer to the Apple documentation.

As Vuforia owns the pointers, you must use the (__bridge ARSession*) cast in Objective-C to turn the void* into an arSession or arFrame. This informs Objective-C's reference counting that the pointer should not be handled by the application.

#import <ARKit/ARKit.h>
 
// Code omitted ...

ARSession* arSession = (__bridge ARSession*)info.mSession;
 
ARFrame* arSession = (__bridge ARFrame*)info.mFrame;
/*
 * Alternatively for ARKit the frame can also be obtained directly from the ARSession, using arSession.currentFrame;
 */
 
// Code omitted ...

The following code snippet in Swift is an extract from the native iOS sample; it demonstrates how to access the ARKit Session held by Vuforia.

/// Method to demonstrate how to get access to the ARKit session and frame objects held by Vuforia
func accessFusionProviderPointers()
{
    let info = getARKitInfo()
    if ((info.arSession == nil) ||
        (info.arFrame == nil))
    {
        NSLog("Fusion provider platform pointers are not set")
        return
    }
        
    let arSession = Unmanaged<ARSession>.fromOpaque(info.arSession).takeUnretainedValue()
        
    // The ARKit ARFrame pointer is provided as part of the ARKitInfo
    // It is preferred to extract it from the session, see below.
    let arFrame = Unmanaged<ARFrame>.fromOpaque(info.arFrame).takeUnretainedValue()
        
    // In ARKit the same ARFrame can also be extracted from the session
    // as shown below, and should be the preferred method.
    let arSessionFrame = arSession.currentFrame
    if (arFrame != arSessionFrame)
    {
        // This test would not normally be needed, it is here for
        // example purposes only
        NSLog("Error arFrame does not match the one on the ARSession")
        return
    }
    /// ...     
}

The getARKitInfo() method is declared in the VuforiaWrapper.h to allow the ARKit Session and Frame pointers to be retrieved by the Swift code.
getARKitInfo() is implemented in VuforiaWrapper.cpp
Employ a method such as the accessFusionProviderPointers() found in the native iOS sample in the VuforiaView.swift file to get access to the ARKit session.

You can test the sample by connecting to an iOS device with a USB cable and build the project to the device with the debugging option enabled and a Vuforia license key added to the AppController.cpp.

When running the application and clicking either the Model Target icon or the Image Target icon, a log is written in the debug console every 5 seconds displaying the tracking state.

ARCore Platform-specific Pointer

For complete details of the ARCore API, developers should refer to the Google documentation. As ARCore is a C-based API, it is sufficient to use a static_cast<> to turn the pointer from a void* into an ArSession or ArFrame. As this is an NDK pointer, this functionality cannot be used directly from within a Kotlin or Java Android application. If developers wish to use these APIs within a Kotlin or Java application, they will have to make the calls in a C/C++ source file invoked via JNI.

#include <arcore_c_api.h>
   
// Code omitted ...  

ArSession *session = static_cast<ArSession>(info.mSession);
   
/*
 * In ARCore developers must get the ArFrame using this call. Updating
 * the ArSession to get the ArFrame will cause the Vuforia library to
 * enter an undefined state.
 */
ArFrame *frame = static_cast<ArFrame *>(info.mFrame);
   
// Code omitted ...

To enable ARCore in an Android project, modify the build.gradle to include the ARCore native libraries. Please refer to the Google ARCore documentation or the Vuforia Android native sample’s build.gradle file on adding the library path. A summary of the steps to add ARCore native libraries is listed below.

Add a path to extract the ARCore native libraries.
Create a configuration to mark the aars to extract the .so files from.

def ARCORE_LIBPATH = "${buildDir}/arcore-native"

// Create a configuration to mark which aars to extract .so files from
// Part of enabling use of ARCore APIs in the App
configurations { natives }

Add dependencies for ARCore support.

implementation 'com.google.ar:core:1.23.0' // Add ARCore support
natives 'com.google.ar:core:1.23.0' // Enabling use of ARCore APIs in the App

If applicable, add an argument to externalNativeBuild to pass ARCore to your CMake library.

externalNativeBuild {
    cmake {
        cppFlags "-std=c++17"

        // Pass ARCore paths to cmake
        // Part of enabling use of ARCore APIs in the App
        arguments "-DARCORE_LIBPATH=${ARCORE_LIBPATH}/jni",
                "-DARCORE_INCLUDE=${project.rootDir}/arcore/include"
    }
}

Add a task to extractNativeLibraries from the native configuration to allow the NDK API to access the libraries.

// Extracts the shared libraries from aars in the natives configuration.
// This is done so that NDK builds can access these libraries.
// Method added as part of enabling use of ARCore APIs in the App
task extractNativeLibraries() {
    // Always extract, this insures the native libs are updated if the version changes.
    outputs.upToDateWhen { false }
    doFirst {
        configurations.natives.files.each { f ->
            copy {
                from zipTree(f)
                into ARCORE_LIBPATH
                include "jni/**/*"
            }
        }
    }
}

Add a task whenTaskAdded to enable the use of ARCore APIs in the application.

// Method added as part of enabling use of ARCore APIs in the App
tasks.whenTaskAdded {
    task-> if (task.name.contains("external") && !task.name.contains("Clean")) {
        task.dependsOn(extractNativeLibraries)
    }
}

In your CMakeLists.txt, add the directives to include the ARCore libraries.

# Import the ARCore (Google Play Services for AR) library.
# The following 2 directives are added as part of enabling use of ARCore APIs in the App
add_library(ARCORE_LIBRARY SHARED IMPORTED)
set_target_properties(ARCORE_LIBRARY PROPERTIES IMPORTED_LOCATION
        ${ARCORE_LIBPATH}/${ANDROID_ABI}/libarcore_sdk_c.so
        INTERFACE_INCLUDE_DIRECTORIES ${ARCORE_INCLUDE}
)

Include in your main application the arcore_c_api.h available from the ARCore-Android-SDK repository. In the native samples, this is done in the VuforiaWrapper.cpp. Employ a method to access the session pointers as demonstrated in the native Android sample’s accessFusionProviderPointer().

You can test the sample by connecting to an Android device with a USB cable and build the project to the device with the debugging option enabled and a license key added to the AppController.cpp.

When running the application and clicking either the Model Target icon or the Image Target icon, a log is written in the debug console every 5 seconds displaying the current tracking state.