Working with the Camera in Unity

Adjust the camera to deliver in-focus camera frames to the Vuforia Engine. Use the Image class returned by the CameraDevice to use the texture from the camera frames.

Set Focus and Camera Modes

To adjust the camera for performance mode and focus modes, please see Device Performance Optimization.

The FocusMode is set by the CameraDevice class. We recommend using the continuous auto focus mode for most scenarios. The CameraMode can be adjusted for performance or quality. In normal use cases, a default mode can be set with:

bool focusModeSet = VuforiaBehaviour.Instance.CameraDevice.SetFocusMode(FocusMode.FOCUS_MODE_CONTINUOUSAUTO);
if (!focusModeSet)
   {
        Debug.Log("Failed to set focus mode" + focusModeSet);
   }

You can configure the autofocus with a script that inherits from Monobehaviour. This registers a callback with the VuforiaBehaviour that will set a focus mode when the Vuforia Engine has started.

void Start()
{
    VuforiaApplication.Instance.OnVuforiaStarted += OnVuforiaStarted;
    VuforiaApplication.Instance.OnVuforiaPaused += OnPaused;
}

private void OnVuforiaStarted()
{
    VuforiaBehaviour.Instance.CameraDevice.SetFocusMode(
        FocusMode.FOCUS_MODE_CONTINUOUSAUTO);
    VuforiaBehaviour.Instance.CameraDevice.SetCameraMode(Vuforia.CameraMode.MODE_DEFAULT);
}

private void OnPaused(bool paused)
{
    if (!paused) // Resumed
    {
        // Set again autofocus mode when app is resumed
        VuforiaBehaviour.Instance.CameraDevice.SetFocusMode(
            FocusMode.FOCUS_MODE_CONTINUOUSAUTO);
    }
}

Apply a Shader to the Camera Images

The Image class provides the camera pixels as a byte array. This approach is useful for some image processing tasks, but sometimes it is preferable to obtain the image as a Unity Texture2D, as shown in the example above. You can also apply the image to a texture in a Material that is used in a shader.

Access the Camera Image

Use the Vuforia Image class to access and set the desired camera image format. Use the images as an OpenGL texture or apply the texture to a Unity material. Register for the desired image format using the Vuforia.PixelFormat declaration.

VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(PixelFormat.RGB888, true);

NOTE: The Vuforia Namespace Reference page lists the available pixel formats.

Call this method after Vuforia Engine was initialized and started; to this aim, it is recommended to register an OnVuforiaStarted callback in the Start() method of your MonoBehaviour script:

private void OnVuforiaStarted()
{
    PixelFormat pixelFormat = PixelFormat.RGB888;
    bool success = VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(pixelFormat, true);
    
    // Vuforia has started, now register camera image format
    if (success)
    {
        Debug.Log("Successfully registered pixel format " + pixelFormat.ToString());
    }
    else
    {
        Debug.LogError(
            "Failed to register pixel format " + pixelFormat.ToString() +
            "\n the format may be unsupported by your device;" +
            "\n consider using a different pixel format.");
    }
}

We can extend this script to retrieve the camera images after setting the format, and during a callback. In this way, you ensure that you retrieve the latest camera image that matches the current frame.

Also, make sure that the camera image is not null, since it can take a few frames for the image to become available after registering for an image format.

Unregister the camera image format whenever the Engine is stopped and register it again when the Engine is started again.

Apply camera images to a RawImage GameObject in your Unity scene.

  1. Create an Empty GameObject and attach the below script CameraImageAccess.
  2. Create a RawImage GameObject from UI -> RawImage.
  3. Drag the RawImage GameObject to the public field in the CameraImageAccess script.
using UnityEngine;
using UnityEngine.UI;
using Vuforia;
using Image = Vuforia.Image;

public class CameraImageAccess : MonoBehaviour
{
    const PixelFormat PIXEL_FORMAT = PixelFormat.RGB888;
    const TextureFormat TEXTURE_FORMAT = TextureFormat.RGB24;

    public RawImage RawImage;

    Texture2D mTexture;
    bool mFormatRegistered;

    void Start()
    {
        // Register Vuforia Engine life-cycle callbacks:
        VuforiaApplication.Instance.OnVuforiaStarted += OnVuforiaStarted;
        VuforiaApplication.Instance.OnVuforiaStopped += OnVuforiaStopped;
        if (VuforiaBehaviour.Instance != null)
            VuforiaBehaviour.Instance.World.OnStateUpdated += OnVuforiaUpdated;
    }

    void OnDestroy()
    {
        // Unregister Vuforia Engine life-cycle callbacks:
        if (VuforiaBehaviour.Instance != null)
            VuforiaBehaviour.Instance.World.OnStateUpdated -= OnVuforiaUpdated;

        VuforiaApplication.Instance.OnVuforiaStarted -= OnVuforiaStarted;
        VuforiaApplication.Instance.OnVuforiaStopped -= OnVuforiaStopped;

        if (VuforiaApplication.Instance.IsRunning)
        {
            // If Vuforia Engine is still running, unregister the camera pixel format to avoid unnecessary overhead
            // Formats can only be registered and unregistered while Vuforia Engine is running
            UnregisterFormat();
        }

        if (mTexture != null)
            Destroy(mTexture);
    }

    /// 
    /// Called each time the Vuforia Engine is started
    /// 
    void OnVuforiaStarted()
    {
        mTexture = new Texture2D(0, 0, TEXTURE_FORMAT, false);
        // A format cannot be registered if Vuforia Engine is not running
        RegisterFormat();
    }

    /// 
    /// Called each time the Vuforia Engine is stopped
    /// 
    void OnVuforiaStopped()
    {
        // A format cannot be unregistered after OnVuforiaStopped
        UnregisterFormat();
        if (mTexture != null)
            Destroy(mTexture);
    }

    /// 
    /// Called each time the Vuforia Engine state is updated
    /// 
    void OnVuforiaUpdated()
    {
        var image = VuforiaBehaviour.Instance.CameraDevice.GetCameraImage(PIXEL_FORMAT);

        // There can be a delay of several frames until the camera image becomes available
        if (Image.IsNullOrEmpty(image))
            return;

        Debug.Log("\nImage Format: " + image.PixelFormat +
                  "\nImage Size: " + image.Width + " x " + image.Height +
                  "\nBuffer Size: " + image.BufferWidth + " x " + image.BufferHeight +
                  "\nImage Stride: " + image.Stride + "\n");

        // Override the current texture by copying into it the camera image flipped on the Y axis
        // The texture is resized to match the camera image size
        image.CopyToTexture(mTexture, true);

        RawImage.texture = mTexture;
        RawImage.material.mainTexture = mTexture;
    }

    /// 
    /// Register the camera pixel format
    /// 
    void RegisterFormat()
    {
        // Vuforia Engine has started, now register camera image format
        var success = VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(PIXEL_FORMAT, true);
        if (success)
        {
            Debug.Log("Successfully registered pixel format " + PIXEL_FORMAT);
            mFormatRegistered = true;
        }
        else
        {
            Debug.LogError("Failed to register pixel format " + PIXEL_FORMAT +
                           "\n the format may be unsupported by your device;" +
                           "\n consider using a different pixel format.");
            mFormatRegistered = false;
        }
    }

    /// 
    /// Unregister the camera pixel format
    /// 
    void UnregisterFormat()
    {
        Debug.Log("Unregistering camera pixel format " + PIXEL_FORMAT);
        VuforiaBehaviour.Instance.CameraDevice.SetFrameFormat(PIXEL_FORMAT, false);
        mFormatRegistered = false;
    }
}

Use an OpenGL texture

Other image processing tasks might require you to obtain the image as an OpenGL texture. You can obtain the image as an OpenGL texture using the approach demonstrated in the OcclusionManagement sample.

  1. Register a Texture2D object to be filled with the camera pixels at each frame instead of letting Vuforia Engine render the camera image natively at each frame, using the VuforiaRenderer.VideoBackgroundTexture API
  2. See the OcclusionManagement sample scripts for an example of this technique.

Accessing the Raw Pixels

For custom processes involving CV (Computer Vision), you can access the raw pixels from the camera with the Image class. See the following example method on retrieving the pixel size:

private byte[] GetPixels(Image image)
{
    var pixelBufferPtr = image.PixelBufferPtr;
    int imageSize = image.Stride * image.Height;
    byte[] pixels = new byte[imageSize];
    System.Runtime.InteropServices.Marshal.Copy(pixelBufferPtr, pixels, 0, imageSize);
    return pixels;
}