Creating a Camera Driver

Developers can input camera data not otherwise supported by Vuforia Engine by implementing a Vuforia Driver that provides the camera data to Vuforia Engine for tracking.

A driver is a native C/C++ dynamic library (.so or .dll file) that implements the required API for Vuforia Engine to load the library and use it to provide image data, and optionally, device poses and anchors. See Creating a Custom Driver with a Device Tracker for information on providing device poses via a device tracker.

To get started, download the Native SDK package appropriate for your platform from the Developer Portal. We are mainly interested in the Driver.h file. This file defines several functions and classes which must be extended by the custom driver. For more information, refer to the Driver API Reference documentation and the README.md from the File Driver Sample.

Data Formats

Vuforia Engine requires data input from the Vuforia Driver in the following data formats.

  • Frames from the camera must be in formats YUYV, NV12, NV21, RGB888, RGBA8888, YUV420P, or YV12.

Image data

Follow the recommendations listed here to improve the tracking outcome and performance from your camera device.

  • Image resolution can vary, but in most cases, the best trade-off between image quality and performance is having a resolution of 720p for medium and small targets that you are able to view at a closer position and 1080p for targets that are further away, as is the case for head worn devices.
  • The image frame rate should be around 30 fps, it can be reduced to 15 fps for static targets.
  • We recommend delivering color images, although most targets are trackable with greyscale frames. Color images are required for Advanced Model Targets and may be required by other Vuforia Engine targets in future releases.
  • Any optical image stabilization may interfere with device pose data: The alignment between camera image motion and device tracker data determines the stability of tracking and user experience. Note that Vuforia Engine requests the device pose from the moment that the camera image is captured.

Pose data

Some Vuforia Engine features require 6DOF device poses in addition to camera images. See our guide on Creating a Custom Driver with a Device Tracker.

Vuforia Driver API Implementation

The Driver class is Vuforia Engine’s way to interact with your driver. Initialize your driver with vuforiaDriver_init() to construct and return an instance of the VuforiaDriver class. The memory and the lifetime of this object is owned by the library. The VuforiaDriver object is expected to remain valid until vuforiaDriver_deinit() is called.

Vuforia Engine constructs the VuforiaDriver instance with platformData and userData where platformData is platform-specific initialization data (for example the JavaVM and App Activity pointers on Android), and userData is a pointer to data that you can specify to allow information e.g., for settings to be passed through Engine to your Driver.

Vuforia Engine deinitializes the driver with vuforiaDriver_deinit() to destruct the specified instance of the VuforiaDriver object.

vuforiaDriver_getAPIVersion() returns a number containing the Driver Framework version which the library was built against and conforms to. This constant is defined in Driver.h as VUFORIA_DRIVER_API_VERSION. If it doesn’t match the version known to the version of Engine that you are using, then Engine initialization will fail.

Use vuforiaDriver_getLibraryVersion() to return a string containing the version describing the library version. The driver developer selects a version value that is non-empty and generally under 50 characters long (e.g., “AndroidUVC-1.0.0”). This should uniquely identify your driver and its version.

VuforiaDriver Class

VuforiaDriver::Driver is the interface for Vuforia Engine to interact and learn about the driver implementation. This includes creating instances for and interacting with the camera and external positional device pose tracker. See the lifecycle between Vuforia and the external camera implementation in the API reference library for the Driver class.

Vuforia Engine queries the driver with getCapabilities() for what supported data it can provide. There are two different capabilities supported by Vuforia Driver.

  • Vuforia::Driver::Capability::CAMERA_IMAGE for camera only sequence.
  • (Vuforia::Driver::Capability::CAMERA_IMAGE | Vuforia::Driver::Capability::CAMERA_POSE) for camera and pose sequence.

Vuforia Engine will request an ExternalCamera instance by calling createExternalCamera(). The memory and lifetime of this instance is owned by the library. The instance should remain valid until destroyExternalCamera() is called.

Vuforia Engine will request an ExternalPositionalDeviceTracker instance by calling createExternalPositionalDeviceTracker(), provided that CAMERA_POSE is supported. The memory and lifetime of this instance is owned by the library. The instance should remain valid until destroyExternalPositionDeviceTracker is called.

ExternalCamera Class

Vuforia Engine uses the ExternalCamera class to interact with the external camera. When an ExternalCamera instance is returned, Vuforia Engine calls open(). If it returns true, the driver will allocate any resources needed.

Vuforia Engine will query the total number of supported camera modes with getNumSupportedCameraModes(). The total number is then queried to return a supported camera mode with getSupportedCameraMode(index, cameraMode) where the index of the mode is in a range starting from 0 to the total number – 1. On return, the output parameter, cameraMode, will be populated with camera mode of the requested index number.

Vuforia Engine starts the camera by calling start(cameraMode, CameraCallback) with the specified camera mode and callback class where camera frames will be delivered to.

On shutdown, Vuforia Engine calls stop() to stop the camera from producing new camera frames to the callback, and close() to close the camera and release the allocated resources.

The ExternalCamera class also has access to the exposure and focus of the camera through various functions (e.g., gets/sets). For more information, refer to the ExternalCamera class reference.

CameraCallback Class

The driver provides camera frames to Vuforia Engine through a callback class. The driver pushes CameraFrame structure to Vuforia Engine through the CameraCallback::onNewCameraFrame function. Each CameraFrame structure points to a buffer containing the actual frame data. Frame data also includes an accurate timestamp, exposure time and frame width/height. Along with the frame data, camera details must also be provided through the intrinsics parameter of type CameraIntrinsics. For more details on the CameraIntrinsics structure, refer to the External Camera Calibration article. The information provided by the CameraFrame structure must be accurate to achieve optimal performance.

ExternalPositionalDeviceTracker Class

Extend your driver with pose data if the camera supports 6DoF (6 degrees of freedom). See Creating a Driver with a Device Tracker for details. Vuforia Engine queries whether camera poses are supported with getCapabilities().

Vuforia Engine calls open() to allow the driver to allocate resources. Only then will the supported positional device tracker be available.

Vuforia Engine calls start(PoseCallback) to start the pose data flow to the specified callback class .

When Vuforia Engine calls stop(), it ends the production of new pose data to the callback. Once the stop returns true, the PoseCallback should be considered invalid and additional poses being pushed to the class should be stopped as well.

Allocated resources will be freed when Vuforia Engine calls close().

PoseCallback Class

The driver provides pose data to Vuforia Engine through a callback class. The driver pushes Pose structure to Vuforia Engine through the PoseCallback::onNewPose function. Each Pose structure includes pose data (e.g., an accurate timestamp and translation (vector) along x, y and z axes followed by a 3*3 rotation matrix for the pose). For a pose and camera frame to be associated together, the timestamp in Pose and CameraFrame structures should match exactly. Driver implementation should first feed pose and then the matching camera frame. Pose is optional and, in such scenarios, driver implementation can just feed the camera frames.

Using a Driver in Vuforia Engine

After the driver is compiled, it may be shared with other developers who want to use the external system. To use a driver with Vuforia Engine, the application developer must make the following call to include the Driver to the Vuforia Engine configuration:

VuDriverConfig driverConfig = vuDriverConfigDefault();
driverConfig.driverName = "libFileDriver.so";
driverConfig.userData = nullptr;
vuEngineConfigSetAddDriverConfig (configSet, &driverConfig)

Android

driverConfig.driverName = "libFileDriver.so";

UWP

driverConfig.driverName = "FileDriver.dll";

iOS

driverConfig.driverName = "FileDriver.framework";

The call must be made before calling vuEngineCreate().

Driver configurations added to the Engine is set for the lifetime of the Engine instance. If you wish to run operations without the driver, the Engine should first be destroyed and then re-created.

Can this page be better?
Share your feedback via our issue tracker