Rendering in Native

Render Augmented Reality content correctly overlaying physical objects in the real world. This article introduces the concepts and APIS Vuforia Engine provides when rendering in native application on supported platforms

To render an AR scene on devices like phones and tablets, first the video background needs to be rendered. To render the augmentation itself, a projection matrix and a modelView matrix is used which must be in sync with the video feed from the camera. Vuforia Engine provides a set of utility methods to ensure that rendering in relation to a target’s pose in the real world is in sync at all times with the video background image. Vuforia Engine only provides support to rendering functionality on the supported platforms. See the Graphics API Support in Supported Versions.

  • VuRenderConfig – sets the choice of video background type for rendering on a supported platform.
  • VuCameraConfig – sets the video mode presets and focus modes. See Camera API Overview for more details.

Setup the VuRenderConfig

The render configuration can be set to default values as presented in the code example below. Set these before starting the vuEngineStart().

// Engine setup
VuEngineConfigSet configSet;
vuEngineConfigSetCreate(&configSet);

// Render setup configuration with default values
VuRenderConfig config = vuRenderConfigDefault();
config.vbRenderBackend = VU_RENDER_VB_BACKEND_DEFAULT;
config.vbViewportMode = VU_VIDEOBG_VIEWPORT_MODE_SCALE_TO_FILL;

vuEngineConfigSetAddRenderConfig(configSet, &config);

Alternatively, the viewport mode can also be set at runtime using the controller class and defined vbMode.

VuVideoBackgroundViewportMode vbMode = VU_VIDEOBG_VIEWPORT_MODE_SCALE_TO_FIT;
vuRenderControllerSetVideoBackgroundViewportMode(&controller, vbMode);

Video background type

The VuRenderVBBackendType vbRenderBackend defines which platform the video background is running on. If set to _DEFAULT, the supported platforms will be set to:

  • DirextX 11 for UWP
  • Metal for iOS
  • Open GL ES 3.x for Android and iOS

NOTE: You may specify different backend configuration settings for your application provided that the platform supports it.

Video background mode

The aspect ratio of the screen images is defined by the VuVideoBackgroundViewportMode vbViewportMode. You may configure it to:

  • _FILL – will stretch and crop to fill the screen.
  • _FIT – will adjust to the aspect ratio adding black banding around the image.
  •  _NATIVE_VIDEO – will use the native video mode and not apply an aspect ratio. This mode is recommended only for debugging purposes.

Use the RenderController

Rendering specific functionalities can be adjusted with the Render Controller class. You can adjust screen resolution, video background settings, and video background render data inputs.

Configure the render view resolution

// Set the render view configuration
VuRenderViewConfig rvConfig;
rvConfig.resolution.data[0] = width;
rvConfig.resolution.data[1] = height;

Retrieve the video background texture size

This example code is available in the native samples as well.

bool getVideoBackgroundTextureSize(VuVector2I& textureSize)
{
    VuVideoBackgroundViewInfo vbViewInfo;
    if (vuRenderControllerGetVideoBackgroundViewInfo(mRenderController, &vbViewInfo) != VU_SUCCESS)
    {
        LOG("Error getting video background view info");
        return false;
    }

    textureSize = vbViewInfo.vBTextureSize;
    return true;
}

Get video background view information

VuRenderVideoBackgroundData renderVBData;
vuRenderControllerUpdateVideoBackgroundTexture(controller, &state, renderVBData);

Update the orientation

There may also be cases where orientation changes occur, (i.e., rotating a device from landscape to portrait) which result in wrongly oriented matrices that are no longer aligned. When the device orientation changes, the rendering matrices need to be updated. To accommodate this, you can set at configuration steps and then update the VuViewOrientation and then communicate this to the VuRenderState.

Set the orientation before rendering, and update it every time view orientation changes:

VuViewOrientation orientation = VU_VIEW_ORIENTATION_LANDSCAPE_LEFT;
vuPlatformControllerSetViewOrientation(controller, &orientation);

The available orientation values correspond with the following:

  • VU_VIEW_ORIENTATION_LANDSCAPE_LEFT – the device is viewed in a landscape view coming from a 90 degrees counter-clockwise rotation from a portrait view. On iOS devices, this would be _LANDSCAPE_RIGHT.
  • VU_VIEW_ORIENTATION_LANDSCAPE_RIGHT – the device is viewed in a landscape view coming from a 90 degrees clockwise rotation from a portrait view. On iOS devices, this would be _LANDSCAPE_LEFT.
  • VU_VIEW_ORIENTATION_PORTRAIT – the device is viewed in portrait view. The screen height is taller than it is wide.
  • VU_VIEW_ORIENTATION_PORTRAIT_UPSIDEDOWN – the device is viewed in a portrait view but in the opposite direction from the _ORIENTATION_PORTRAIT.

NOTE: The orientation must be set even if no rendering is performed as Vuforia Engine is dependent on this value to report accurate device poses in the state. If no orientation is set, the value defaults to VU_VIEW_ORIENTATION_LANDSCAPE_LEFT.

For platform specific view orientation descriptors, a conversion method is available:

vuPlatformControllerConvertPlatformViewOrientation(const VuController* controller, const void* platformOrientation, VuViewOrientation* vuOrientation);

The conversion requires a platformOrientation of one of the following valid types:

  • Android – takes an integer array of two values. [0] is the origin value on an orientation change. [1] is the amount of rotation.
  • iOS – will deliver a UIInterfaceOrientation value.
  • UWP – will deliver a DisplayOrientation_ value depending on your build configuration (C++/CX, C++/WinRT, or other).

Please refer to the reference library for more information on retrieving platform specific orientation descriptors.

The call will return VU_SUCCESS if the passed platform orientation was successfully converted to a valid Vuforia view orientation.

Update the VuRenderState

In the VuRenderState, contains all the data you need for rendering, including video background and matrices. The video background viewport should be set before getting the render frame from the state using vuRenderControllerSetRenderViewConfig().

vuStateGetRenderState(mVuforiaState, &mCurrentRenderState);

ProjectionMatrix

The render state frame will provide the projection matrix that corresponds to the physical camera parameters. The projection matrix utilizes the near and far planes and holds default values fitting to the most common use cases.

To adjust the near and far plane values, and consequently affecting the projection matrix, use the following get/set calls.

vuRenderControllerSetProjectionMatrixNearFar(mRenderController, NEAR_PLANE, FAR_PLANE);

Metal-based rendering code on iOS uses the same style matrix convention as OpenGL ES. The projection matrix can, therefore, be used interchangeably.

DirectX rendering code needs to be converted to the DirectX matrix convention before it can be used in DirectX rendering calls.

Video background mesh

The vbMesh from the render state can be used to manipulate the video background mesh with shaders. On devices that does not require video background rendering, for example see-through devices like the HoloLens, the vbMesh will be null.

Freeze the state

The State update mechanisms can also be used to freeze the camera frame in the current AR session. Use acquireLatestState() and reuse it for rendering as long as you want to freeze the state. Then a renderVideoBackground() function can use that state to render as the video background. For more information on AR session management, please refer to Continued AR Experiences.