Building and Using the File Driver Sample

The File Driver Sample demonstrates how to implement a driver which uses the External Camera and the External Device Tracking APIs to feed Vuforia Engine camera frames and device poses, respectively.

The File Driver sample can be downloaded from the Developer Portal. The File Driver should be configured to load an MP4 file that was recorded using the SessionRecorder API. The device tracking poses, which are embedded in the recording, are fed into Vuforia Engine along with camera frames from the video track.

NOTE: This article only describes usage of the MP4 recordings. File Driver still supports recordings in XML/XPGM format as well. However, this format is deprecated and will be removed in a future Vuforia Engine release.

The File Driver package includes two sets of example recordings: Stones Image Target and Astronaut Image Target. Native developers should use the Stones Image Target while Unity developers should use the Astronaut Image Target sequence.

Use the Session Recorder API to create your own sequences as MP4 files, See the Session Recorder API and Recording and Playback for details.

Building the File Driver

Prerequisites

Supported image file formats for the File Driver

The recording format from the Session Recorder API is MP4. Other image formats such as .xpgm, .png, and .jpg are deprecated but still supported in the File Driver. Other formats determined by the individual platform image loader are also deprecated. 

Setup

  1. Extract the File Driver package and the Vuforia SDK package.
  2. Run the appropriate build command:
    Note: The -vh  or -vf  command line options may be omitted if the driver is put in the SDK's samples directory.

Android:

python3 build.py android -vh [path to Vuforia SDK]/build/include

UWP:

python3 build.py uwp -vh [path to Vuforia SDK]/build/include

iOS:

python3 build.py ios -vf [path to Vuforia SDK]/build

Set the -vf option to the folder containing the VuforiaEngine.framework

  1. Output files should be placed in the build/bin directory by default: 
    • libFileDriver.so for Android, 
    • FileDriver.dll for Windows,
    • and FileDriver.framework for iOS

Using the File Driver

Android

  1. Add libFileDriver.so for each needed architecture from build/bin to your Android app project by modifying either Android.mk or CMakeLists.txt or build.gradle. Select the option that best fits your project:
    • Android.mk:

      Add libFileDriver prebuilt definition to your Android.mk file:

      include $(CLEAR_VARS)
LOCAL_MODULE := libFileDriver-prebuilt
LOCAL_SRC_FILES = [path-in-your-filesystem]/FileDriver/build/bin/Android/Release/$(TARGET_ARCH_ABI)/libFileDriver.so   
include $(PREBUILT_SHARED_LIBRARY)

      When defining your local module in the same Android.mk add libFileDriver-prebuilt as a dependency to your LOCAL_SHARED_LIBRARIES:

      LOCAL_SHARED_LIBRARIES := libFileDriver-prebuilt
    • CMakeLists.txt:

      Add libFileDriver prebuilt definition to your CMakeLists.txt file:

      add_library(VUFORIA_FILEDRIVER_LIBRARY SHARED IMPORTED)
set_property(TARGET VUFORIA_FILEDRIVER_LIBRARY PROPERTY IMPORTED_LOCATION
[path-in-your-filesystem]/FileDriver/build/bin/Android/Release/${ANDROID_ABI}/libFileDriver.so)

       

      When linking libraries in CMake, link libFileDriver library in the same CMakeLists.txt: 
      target_link_libraries(
VuforiaSample
${ANDROID_LIBRARY}
${LOG_LIBRARY}
${GLES3_LIBRARY}
VUFORIA_LIBRARY
VUFORIA_FILEDRIVER_LIBRARY
)
    • Gradle: 

      If using either Android.mk or CMakeLists.txt you will also need to update your build.gradle to include the library in the APK. Use the following code in your app/build.gradle:

      android {
    sourceSets.main {
        jniLibs.srcDirs += '[path-in-your-filesystem]/FileDriver/build/bin/Android/Release/'
    }
}

       

      NOTE: If you are using the Android Plugin version 4.x and a CMakeLists.txt then the above change to build.gradle is not required.
  2. Add FileDriver.jar from build/bin to your Android-app project. Use the following code in your app/build.gradle: 
    dependencies {
    implementation files("[path-in-your-filesystem]/FileDriver/build/bin/Android/Release/FileDriver.jar")
}

     

  3. Add the sample sequence from the data directory to your Android-app project. Use the following code in your app/build.gradle: 
    android {
    sourceSets.main {
        assets.srcDirs += '[path-in-your-filesystem]/FileDriver/data/stones'
    }
}

     

  4. Add then your configuration for the driver before creating a new Engine instance with vuEngineCreate() 
    VuDriverConfig driverConfig = vuDriverConfigDefault();
driverConfig.driverName = "libFileDriver.so";
driverConfig.userData = nullptr;
vuEngineConfigSetAddDriverConfig(configSet, &driverConfig);

     

UWP

  1. Add FileDriver.dll from build/bin/uwp into your Visual Studio UWP app project.
    1. Import the FileDriver.dll to the root of your project. Remember to use a .dll that matches your architecture (x86/x64) and build type (Debug/Release) configurations.
    2. Click the FileDriver.dll from the project file list and set the property Content to True.
  2. Add a sample sequence from the data directory into your Visual Studio UWP app project. For example, to use the “stones” sequence:
    1. Import the FileDriver/data/stones/FileDriverRecording.mp4 file into your project.
    2. Select the file and set the property Content to True.
  3. Add the following call before calling vuEngineCreate():
VuDriverConfig driverConfig = vuDriverConfigDefault();
driverConfig.driverName = "FileDriver.dll";
driverConfig.userData = nullptr;
vuEngineConfigSetAddDriverConfig(configSet, &driverConfig);

iOS

  1. In your Xcode project, open the build settings screen and select the General tab. 
  2. In the Embedded Binaries section, click the + icon to add a new binary. 
  3. From the file selection menu, locate the iOS File Driver Framework in bin/Release-iphoneos/FileDriver.framework
  1. In the Linked Frameworks and Libraries section, remove the FileDriver entry. The entry should not be listed in this section and the application should not try to link against it.
  1. Drag the FileDriverRecording.mp4 file into the left-hand project pane.
  2. Add the following call before calling vuEngineCreate():
VuDriverConfig driverConfig = vuDriverConfigDefault();
driverConfig.driverName = "FileDriver.framework";
driverConfig.userData = nullptr;
vuEngineConfigSetAddDriverConfig(configSet, &driverConfig);

 

Changing the Sequence

To select which sequence is used for playback, a pointer to a FileDriverUserData object can be passed to the driver using the driverConfig.userData argument.

struct FileDriverUserData
{
    const char* sequenceAbsolutePath {};
}
  • If driverConfig.userData or driverConfig.userData->sequenceAbsolutePath is NULL, the File Driver will look for a file named FileDriverRecording.mp4 in the main asset location (depending on platform).
  • If driverConfig.userData->sequenceAbsolutePath is set to a string starting with asset://, FileDriver will interpret the remaining part of the string as a path in the main application asset location (depending on platform).
  • Any other string will be interpreted by File Driver as a normal filesystem path.

Custom Sequences (Deprecated)

Please note that the custom file driver XML file format is deprecated, and this section will be removed in a future version of Vuforia Engine.

Developers looking to create their own sequence, should consider the following:

  • A top level XML file groups a device pose and a matching camera frame under one <Element> tag. This XML file also contains global attributes, such as total number of camera frames, width, and height, which apply to the entire sequence. For more information, refer to the Format of Camera and Pose XML Sequence File article.
  • The width and height of each frame should remain consistent and match the intrinsic information provided with each frame.
  • The framerate at which frames are provided to Vuforia Engine does not necessarily need to match the rate at which they were captured.
  • The samples currently use PNG format, but other image formats may be supported depending on the platform. Ensure that the files can be read at the rate which matches the desired framerate. For instance, if the desired framerate is 30 FPS, then the device needs to be able to read 30 files per second.
  • Avoid compression artifacts in the files.
  • Developers need to create/name their file sequences to easily process the sequence in the appropriate order. The sample uses 'filename#####.png' notation, where ##### represents the frame index, padded to 5-digits.

Playback Control API

The File Driver API exposes playback controls that can be used to configure the playback behavior of a recording.

Supported controls:

  • Playback start time
  • Playback end time
  • Looping playback
  • Changing playback mode

Set the FileDriverPlaybackMode() to FILEDRIVER_PLAYBACK_MODE_RESPECT for playing the sequence in real-time with possible frames being dropped or to FILEDRIVER_PLAYBACK_MODE_WAIT for playing the sequence with all frames but possibly slowed to let the Engine keep up. Please refer to the header API documentation for more details. It is available in the include/FileDriverPlaybackController.h file that is part of the File Driver sample package.

Using the playback control API

  1. After successfully creating the engine with File Driver, a native platform handle to the loaded library can be retrieved using vuPlatformControllerGetDriverLibraryHandle().
  2. This handle is used to load the File Driver playback controller functions using the native symbol loading mechanism of the current platform (GetProcAddress() on Windows/UWP, dlsym() on POSIX systems(known as Portable Operating System Interface).
  3. The retrieved function pointers are used to configure playback at any provided timestamp until the engine is destroyed and thus the library unloaded.

NOTE: Some playback control functions require that the engine is stopped and restarted before the change takes effect.

Example

VuController* platformController = nullptr;
vuEngineGetPlatformController(engine, &platformController);

void* handle = nullptr;
vuPlatformControllerGetDriverLibraryHandle(platformController, &handle);
 
#ifdef _WIN32
void* funcPtr = GetProcAddress(static_cast<HMODULE>(handle), "fileDriverSetPlaybackStartTime");
#else
void* funcPtr = dlsym(handle, "fileDriverSetPlaybackStartTime");
#endif
 
using FileDriverSetTimeFunc = bool(*)(uint64_t timestamp);
auto fileDriverSetPlaybackStartTime = reinterpret_cast<FileDriverSetTimeFunc>(funcPtr);

fileDriverSetPlaybackStartTime(100);

 

Use the File Driver in Unity

After you have built the File Driver, the output can be imported into Unity. Ensure the correct plugin platform options are set in the imported libraries. The option should match to the configured Unity build platform.

NOTE: This sample should be used together with a Session recording or images stored in the device storage. To use the File Driver and feed it image data from an external source, please see External Camera Support.

NOTE: File Driver cannot be used in Play Mode. To play back a Session recording in the Unity Editor, see Recording and Playback.

  1. Copy the compiled binaries from build/bin to:
    • Android: Assets/Plugins/Android/
    • UWP: Assets/Plugins/WindowsStoreApps/[x64, x86]
    • iOS: Assets/Plugins/iOS/
  1. Add the Session Recording to your Unity project. Copy your recording .MP4 or the sample sequence from YourDirectory/VuforiaDriver-x-y-z/Data/Astronaut into the Assets/StreamingAssets folder. This ensures it's included in the application build.

  1. In the VuforiaConfiguration window, enable Delayed initialization. This sets the driver library name before Vuforia Engine is initialized in order to use the file driver from Unity.
  2. Also, make sure to add a license key as the file driver will not work without it.

  1. The driver library can then be set from any app level script, e.g., in a Start() method. After that, Vuforia Engine can be initialized:
VuforiaApplication.Instance.Initialize(driverName, sUserDataPtr);

Now, the image sequence should play in a loop instead of the device camera being used.

On iOS, The resulting .xcodeproj should be modified to include the FileDriver.framework as an embedded framework. Remember to remove it from the list of linked libraries (refer to the step relating to linked libraries in the Using the File Driver iOS section).

Example script for initiating Vuforia from the file driver sequence.

using System;
using System.IO;
using System.Runtime.InteropServices;
using UnityEngine;
using UnityEngine.Networking;
using Vuforia;

public class LoadDriver : MonoBehaviour
{
    struct FileDriverUserData
    {
        public string sequenceDirectoryAbsolutePath;
    }

    static FileDriverUserData sUserData;
    static IntPtr sUserDataPtr = IntPtr.Zero;

    void Start()
    {
#if UNITY_WSA || UNITY_IOS
        var recordingPath = Path.Combine(Application.streamingAssetsPath, “myRecording.mp4”);
#else
        var recordingPath = "asset://myRecording.mp4"; //file name relative to StreamingAssets
#endif        
        sUserData = new FileDriverUserData { sequenceDirectoryAbsolutePath = recordingPath };
        sUserDataPtr = Marshal.AllocHGlobal(Marshal.SizeOf(typeof(FileDriverUserData)));
        Marshal.StructureToPtr(sUserData, sUserDataPtr, false);
        var driverName = "";
#if UNITY_ANDROID
        driverName = "libFileDriver.so";
#elif UNITY_WSA
        driverName = "FileDriver.dll";
#elif UNITY_IOS
        driverName = "FileDriver.framework";
#endif
        VuforiaApplication.Instance.Initialize(driverName, sUserDataPtr);
        VuforiaApplication.Instance.OnVuforiaDeinitialized += OnVuforiaDeinitialized;
    }

    static void OnVuforiaDeinitialized()
    {
        // Preserve the pointer to the user data for the whole Vuforia lifecycle 
       Marshal.FreeHGlobal(sUserDataPtr);
    }
}

 

Learn More

Vuforia Engine Driver Framework

External Camera

Format of Camera and Pose XML Sequence File

Building and Using the UVC Driver Sample

Creating a Custom Driver

External Camera Calibration

 

Can this page be better?
Share your feedback via our issue tracker