Recording and Playback

Vuforia Engine allows developers to record and play back video recordings of AR sessions. Camera frames and device tracking data is recorded in order to play back the AR session at a later time and enables developers to rapidly iterate on their app and test AR experiences efficiently. In particular, this is helpful for building AR experiences that use Model Targets or Area Targets, where the physical target is not always available for testing. Session Recordings can then be played back either on a device or using Unity Editor Play Mode.

Overview and Recommended Workflow

Vuforia Engine includes a SessionRecorder API that can be used directly in a native or Unity app to capture Session Recordings and transfer them from any application using Vuforia Engine. See the last section of this article for more information about this API. See Vuforia Play Mode for more details on how to replay recordings in Unity.

Capture Session Recordings with Vuforia targets in the scene; not having any visual feedback for target tracking removes any visual confirmation that Vuforia has detected the target.

See the video tutorial to quickly get started with Recording and Playback.

It is also worth mentioning that it is important to work out the scenario you wish to capture before recording. A clear strategy on what to capture; whether it is user centric, sequential steps, or targets under environmental constraints; planning ahead will reward you with an ideal recording that can help in the development process and result in a capable AR application.

Limitations

The following restrictions apply:

  • Currently devices with cameras that capture frames in the following pixel formats are supported:
    • NV21
    • NV12
    • YUV420P
    • YV12
  • Recording and playback of sessions is currently not supported for Ground Plane experiences.
  • The SessionRecorder Prefab requires you to add your own UI (User Interface) when building applications for Magic Leap
  • Enable Sharing is only supported on iOS and Android devices.
    • In additionmake sure you are only sharing to applications that do not modify the recording by compressing the video. Modifying the Session Recording would remove important metadata and make it unusable. OneDrive, Dropbox or Airdrop all support sharing session recordings without modifying them, while Slack or Google Photos is known to compress videos by default and should be avoided.
  • Record only in one view orientation (vertical or horizontal) during a session. Changing the orientation while recording will stop the recording and throw an error. See Record sessions in one view orientation for more details.

Session Recorder in Unity

In Unity, there is a readily available Session Recorder GameObject that can be added to any existing application. It will display a customizable UI for recording and will call the underlying API for you.

NOTE: For advanced use cases, recording can also be started and stopped using the scripting API. See the section further below for more details. 

  1. First, set up a Vuforia enabled scene with the Session Recorder GameObject, ARCamera GameObject, and a Vuforia Target. These are easily added by right-clicking in the Hierarchy panel and selecting Vuforia Engine.

Session Recorder configuration

From the Session Recorder Behaviour component, it is possible to tweak, add UI and calls to events. If not needed, you can hide the Clean button or disable sensor data recording in the Advanced Settings. However, recording sensor data is strongly advised.

The Overlay Canvas contains the UI for mobile devices, while the Holographic Canvas is used on HoloLens. On HoloLens, the voice commands “SessionRecorder StartSessionRecorder Stop” and “SessionRecorder Clean” control the Session Recorder’s functionality.

Enable Sharing to add a UI window that allows you to share the recording to a cloud storage through other applications on your device.

The Session Recorder Behaviour is fully customizable and can interact with any other MonoBehaviour in the scene. For inspiration, see the Model Target scene from the Vuforia Core Samples for Unity.

NOTE: For iOS builds, the UIFileSharingEnabled plist entry is automatically enabled when the Session Recorder GameObject is added to your scene. You can also configure it manually in Vuforia Configuration.  

  1. Set the build platform and Player Settings according to your connected device and press build and run.
  2. When the app is launched, it may be required to allow permission to use and access the camera. 
  3. Once the camera feed is present, press the record button and record your scene and Vuforia Target. 
    Please note the following limitations
    • Recording a session is only possible from a device, not from the Unity Editor with Vuforia’s Webcam Play Mode.(however, a Session Recording can be played back in Play Mode, which is described below in Using Recordings)
    • Recording a session is currently only possible in landscape left mode.
  4. Pressing the Stop button will conclude the recording.
  5. Between recordings, you can choose to remove existing recorded sessions by pressing the Clean button to free up space on the device.

NOTE: The Model Target scene in the Vuforia Engine Core Unity Samples include an example of a customized recording API.

Transferring Session Recordings

The Session Recording will be stored in the app’s private storage of your device and there are different procedures to retrieve it that depends on the device platform.

Retrieving a Session Recording is easiest on iOS and Android with the Enable Sharing property enabled.

  1. On Android devices, use the Enable Sharing in Unity to include a UI for sharing through OneDrive, Dropbox or any other application that does not modify the recording.
  2. The Session Recording can be accessed using the adb command or using a tool like Android File Transfer or Windows Explorer. The Session Recordings are found within the shared files directory of the app on your device (this is what the getExternalFilesDir Android API method returns). The exact path may vary from device to device. One known variant is /storage/emulated/0/Android/data, so the whole path to the Session Recordings might look like 

/storage/emulated/0/Android/data/[PACKAGE NAME]/files/Recordings

The section Querying the Location of a Recording  in the Session Recorder API Overview article explains how to obtain the path to each Session Recording. The [PACKAGE NAME] will have a form like com.mycompany.MyApps.MyCoolApp (specified within the AndroidManifest.xml of the app using the package property). 

To copy a Session Recording to the current working directory on your computer, you can use the following adb command : 

adb pull /storage/emulated/0/Android/data/[PACKAGE NAME]/files/Recordings/VuforiaSession-[DATE]-[TIME].mp4 ./

The Session Recording can alternatively be copied using Android Studio, Android File Transfer on Mac, or Windows File Explorer on Windows. 

In Android Studio, select View -> Tool Windows -> Device File Explorer and browse to the media files and right-click them and select Save As… It may be necessary to right-click and select Synchronize first, if no files appear in the Data folder.

For Windows, you can use the File Explorer and browse the MTP device from This PC. Connect your device with a USB and press the Allow on the pop-up message on the device as it connects. The Session Recordings can now be transferred.

For macOS, you can use the Android File Transfer software from Google to browse to the Data folder and retrieve the recordings.

  1. On iOS devices, use the Enable Sharing in Unity to include a UI for sharing through OneDrive, Dropbox, Airdrop or any other application that does not modify the recording.
  2. The Session Recordings can be accessed using iTunes, macOS’s Finder or a tool like iExplorer. iExplorer will list the app under the Apps menu on the bar on the left hand-side. Clicking on the app’s name reveals a number of directories on the main window belonging to the app. The Session Recordings are found within the Documents/Recordings sub-directory and can simply be copied using the export functionality of iExplorer. For this, right-click on the media files you’d like to copy and select the Export to Folder-> option. They can also be dragged-and-dropped into a directory that is open in Finder.
  1. On UWP devices, the Session Recordings can be accessed using the operating system’s file browser. The media files are found in the recording app’s private storage space. The path to the Session Recordings directory has the form: 

C:\Users\[USER NAME]\AppData\Local\Packages\[PACKAGE NAME]_[PACKAGE FAMILY]\LocalState\Recordings.

The [PACKAGE NAME] will have a form like My.Apps.MyCoolApp and [PACKAGE FAMILY] will be a cryptic string containing numbers and letters.  

  1. On the HoloLens, the Device portal API of Microsoft (see Microsoft's device portal api reference article) can be used for downloading the Session Recording on to a computer. This can be done in two steps using HTTP GET calls:
  1. Get a list of all files in the Recordings directory,
  2. Get all the files in the list the first call returns.

Please see below for the details of the HTTP calls.

For step 1:

  • URL: http://[HOLO LENS IP]:10080/api/filesystem/apps/files
  • Parameters:
    • knownfolderid = LocalAppData
    • packagefullname = [PACKAGE NAME]_[PACKAGE FAMILY]
    • path = \\LocalState\Recordings (note the double backslashes, which might need to be escaped in a string)

For step 2:

  • URL: http://[HOLO LENS IP]:10080/api/filesystem/apps/file (note the singular file, in contrast to files above)
  • Parameters:
    • knownfolderid = LocalAppData
    • packagefullname = [PACKAGE NAME]_[PACKAGE FAMILY]
    • path = [PATH TO FILE FROM RETRIEVED LIST]
    • filename = [FILENAME FROM RETRIEVED LIST]

Please note that you will need to include your HoloLens credentials (username and password) as part of this HTTP call. For the [PACKAGE NAME] and [PACKAGE FAMILY], please see the explanation in the UWP section above.

  1. On the Magic Leap, you can transfer the Session Recordings by using the Device Bridge tool from the Magic Leap Lab. For more information, see Magic Leap’s Working with the File System.
    • Connect your device and use the Device Bridge to transfer the media files to your desktop machine.
    • The Session Recordings are located in the default directory [APP_NAME FOLDER]/files/Recordings.
    • All Session Recordings are saved to the [user]/MagicLeap/downloads folder by default.

Using SessionRecordings

Using Session Recordings in Unity Editor Play Mode

Using a Session Recording in Unity Editor Play Mode allows you to rapidly prototype your AR experience without building to a mobile device, and, without having to be close to the physical target that you use for testing.

Follow the steps below to use a recording in Play Mode:

  1. Once the Session Recording is transferred to your workstation, the Vuforia Configuration can be set to Play Mode type – RECORDING.
  2. Browse to the directory of the saved media files and select the Session Recording you wish to use.
  3. The Session Recording will be played back once Play Mode is started in the editor.

See Vuforia Play Mode in Unity for more information.

Using Recordings in a mobile application

In cases where you want to test your application on the actual mobile target device, but still do not want to test on the physical object / location (e.g. because it is not available), you can play back a recording using the File Driver sample in a native or Unity application.

See the File Driver Sample for more information.

Creating Recordings using the SessionRecorder API

Using the Unity scripting API

Instead of using the Session Recorder GameObject, the recording functionality can also be used by directly calling into the SessionRecorder scripting API. This allows you to create custom interfaces and i.e. instructions of what to scan and how long.

Start by calling the SessionRecorder.Instance and initiate a recording using the SessionRecorder.StartRecording call. While recording, you can check the status with GetRecordingStatus and enlist conditions depending on the returned status. Conclude the recording with SessionRecorder.StopRecording and call the method GeRecordingPath to identify the location of the stored Session Recording.

For more information, please refer to the Session Recorder API Overview.

Creating Recordings in a native application

To use the Session Recorder in native, start by retrieving the VuSessionRecorderController using vuEngineGetSessionRecorderController. You can then create a VuRecording with vuSessionRecorderControllerCreateRecording. The recording can either be automatically started on creation or explicitly started with vuRecordingStart.

To stop the recording, use the vuRecordingStop call which will then output the recording data consisting of an MP4 file which contains the recorded video and sensor data. The file will be named VuforiaSession followed by time and date stamps.

The frame images are associated with device pose data and similar information that helps you in testing and troubleshooting on your device and scenario. The types of data that can be recorded are identified with vuSessionRecorderControllerGetSupportedRecordingDataFlags.

Use vuSessionRecorderControllerGetDefaultRecordingDataFlags to get the sources that are recommended for recording on the current device. Using the recommended set of sources will always result in a recording that can be used for AR sequence playback.

After a recording is started, the location of the MP4 file can be queried with vuRecordingGetPath. To retrieve the file, please see the above section and note the different approaches for different operating systems.

NOTE: If you plan to record with an iOS device, the UIFileSharingEnabled plist entry needs to be set in Xcode.

For additional Session Recorder methods, please see Session Recorder API Overview, which illustrates in detail how these can be used with examples.

Learn More

Session Recorder API Overview

How to Use the File Driver Sample

Vuforia Play Mode in Unity