How to Create Model Targets from 3D Scans

Vuforia Image

This article explains how to create a Model Target from 3D-scanning a physical object. This approach replaces the need to have the CAD model of the object and you may find that 3D-scanned models can be a great start for exploring the possibilities with Model Targets.

Before heading out to scan and capture objects with your scanning equipment or camera, we would like to point at two methods that we have found to work well. However, note that the accuracy and robustness of tracking Model Targets made from scanned objects might differ and be less consistent compared to Model Targets made from a CAD model.

The two methods for achieving Model Targets from scanned models are:

  1. Hardware aided scanning with the Occipital Structure Sensor.
    This is the recommended approach both in terms of simplicity as well as scanning quality.
  2. If you don’t have access to the Structure Sensor, photogrammetric model generation from Meshroom or RealityCapture is also an option, but will require more work with less reliable results.

Both scanning methods have their advantages and limitations.
Another option is to use the Vuforia Object Scanner to create an Object Target.

The following should serve as a guide when to use which technology:

Model Targets

Model Target creation requires a CAD model or mesh of the object. Model Target detection and tracking is based on the shape of the object and therefore also works if the object is not well textured or its appearance varies – e.g. different colors in cars or different textures for toys.
Ideally, Model Targets are created from CAD files that closely resemble the physical object.

Object Target Creation with Vuforia Object Scanner

This method works best for smaller, toy-sized objects that have a feature-rich texture. Detection is based on the physical appearance of the object, so this technology is very sensitive to different textures or color variations between objects.
Object Targets do not provide a mesh that can be used for authoring or occlusion.
Compared to Model Targets, Object Targets have lower performance requirements. See Object Recognition for more information.

Model Targets created from 3D Scans

If no CAD data is available for a Model Target, a mesh can be created from a 3D scan. Ideally, this is done using hardware aided scanning, e.g. using the Occipital Structure Sensor. This hardware allows scanning of objects even if the target is not well textured, e.g. with a coffee machine in the example below.

Without dedicated hardware, using photogrammetric model generation is also an option. However, this will require your model to be well textured. Depending on your use case, creating an Object Target using the Vuforia Object Scanner may yield better results in this case. Creating a Model Target using photogrammetry has some advantages though, such as having a mesh representation of your object for authoring and occlusion. Also, in cases where your object is highly textured, but texture varies significantly between objects (e.g. for wooden toys), Model Targets will give better results.

The guide below explains how to create a Model Target using hardware aided 3D scanning. If you plan to use a photogrammetric approach, please refer to this guide.

When scanning an object, keep in mind that objects with black and/or reflective surfaces tend to not work well with either method. Make sure that you are capturing the object in a well and evenly lit environment. Avoid hard shadows (direct sunlight) and strongly colored light sources.

Hardware aided 3D Scanning

Various dedicated hardware to 3D-scan an object is available on the market.
The following section describes using the Structure Sensor but should also be applicable to similar equipment.

The Structure Sensor (Mark II) by Occipital is a hardware accessory for iOS devices. It uses a depth camera to scan objects or environments and produces textured models at correct scale.

Requirements

  • Original Structure Sensor or Structure Sensor (Mark II)
  • Bracket Adapter fitting your iOS tablet
  • Apple type charging cable
  • Download the Calibrator and the Scanner application from the Apple Store

Setup

To get started, attach the sensor to your iOS tablet as shown in Occipital’s guide.

Connect the hardware to your iOS device and open the calibrator application. Proceed to calibrate the sensor by following Occipital’s YouTube guide.

Capturing objects

To start capturing an object, open the scanner application and point it towards the object you wish to scan. The structure sensor will automatically detect the object and render a box around it.

  • Adjust this box to contain the object within by using two fingers to zoom, rotate, or minimize .
  • Press scan and move around the object to capture all sides and details.
  • Once finished, press the stop button and review the model. Click the color button to see the model with texture.

The app allows you to send the model to an e-mail address, but you can alternatively connect the tablet with an Apple Mac via USB and use the iExplorer to retrieve the model from the app’s model folder.

NOTE: The Scanner app only saves one model at a time. Scanning again will delete prior scans. Therefore, make sure that the scan was transferred before making a new scan.

Vuforia Image

Model Preparation

After you have obtained a model from the structure sensor, a clean-up of the model is necessary. This includes removing the surroundings that was captured during the scanning of the object, and it should be removed before importing the model into the Model Target Generator.

NOTE: Reflective or transparent surfaces on objects are more difficult to capture with the structure sensor. Ensure that the object is under sufficient lighting without glare.

Vuforia Image

Nespresso Machine captured with Occipital Original Structure Sensor

Use your preferred modelling software to edit away the surroundings. Additionally, inspect the model for missing parts or holes.

  • Patch up holes and missing vertices in the model. However, be attentive that patching the model that you plan to use with texture can introduce inaccuracies on the texture and, consequently, affect the tracking of the object.
  • The structure sensor delivers the model to scale but we recommend verifying if the scale is accurate with a measuring tool in your modeling software. If necessary, re-scale the model to match the size of the physical object before using it to create a Model Target.

Create a Model Target

Once a model of the scanned object is freed from its surrounding mesh and it realistically resembles the real-life object, it is time to create the Model Target. For a detailed guide on how to use the Model Target Generator (MTG), please refer to the How to Create a Model Target and the Model Target Generator User Guide.

Model Targets with Texture from a scan

When scanning a physical object, we recommend testing Model Target creation with and without including the texture from the scan to compare detection and tracking performance. See Model Targets Supported Objects and CAD Model Best Practices for detailed information on using textures.

Generally, we advise against using inaccurate textures that present shadows or incorrect surface colors. Advanced Model Targets that are trained for i.e. 360-degree detection are affected more by inaccurate textures. If you choose to create an Advanced Model Target with texture, do not forget to select Realistic model appearance when creating a training session in the MTG. Finish by proceeding to load your Model Target in the Model Target Test App to test its tracking robustness.

Vuforia Image