Model Targets with Apple's Object Capture

Use the Apple’s Object Capture API  to create 3D scans of objects that can be used to create Model Targets.

Apple Object Capture API is a part of the Apple AR RealityKit framework. Object Capture lets you procure and process images into a textured 3D model. Object Capture includes a sample iOS app for taking photos and a Command-line MacOS app for configuring the process and output.

While it is possible reconstruct objects from images of any source, we recommend using the sample app on iOS devices with a Lidar sensor for best results and accurate scale of the object.

Requirements

See the hardware and software requirements necessary to run the software and build tools at Apple’s Object Capture PhotogrammetrySession page.

In addition, it’s required to have MacOS 12.0+ and Xcode 13.0+ to run the RealityKit Object Capture framework.

Supported image formats are PNG, JPEG, or HEIC format.

Installation

Get the Command-line macOS app here.

Get the Object Capture app for iOS devices here. Other camera devices and image sources can also be used; but using this app on an iOS device with Lidar sensor should be preferred.

Image Capturing Guide

Images can be taken with the Object Capture sample app for iOS devices or from other sources such as digital, DLSR or mirrorless camera. If you have an iOS device with LiDAR, you can use the sample Capture app to also capture depth data alongside the images; it will help in generating the 3D model in more accurate scale.

Listed here are a few guidelines and practices for capturing the object. For an in depth guide, please see Capturing Photographs for RealityKit Object Capture:

Object Selection

  • Select an object that is static – it should not have movable or deformable parts.
  • Avoid objects that are highly reflective, transparent, or translucent. Choose instead objects with varying color and with texture.

Image sequence

  • Take images in a sequence that sets the adjacent shots with an overlap of 70% or more.
  • Each image should contain the entirety of the object.
  • Take a minimum of 100 images.
  • Set your camera to the highest resolution and a narrow aperture to keep the object in focus and the background blurred.

Image Processing

Consistent camera settings should be kept for all images. Maintain the same focal length, shutter speed, aperture, and ISO throughout the image sequencing.

Use then the PhotogrammetrySession with the sequence of images as input to reconstruct a 3D model from it.

Use the Photogrammetry Command-Line App and see this guide for configuring the generation process.

The Command-line app has the following parameters:

  • input_folder: The path to the directory where the object pictures are located.
  • output_folder: The path to the output directory where the reconstructed model will be stored.
  • -d: The level of detail (e.g., "reduced, medium, full").
  • -o: The order of the photos (e.g., "unordered" or "sequential").
  • -f: The feature sensitivity (e.g., "normal, high").

The choice of parameters naturally depends on the type of object, therefore, inspect the output 3D model according to the CAD model best practices. In general, the detail can be set to full and the sensitivity to normal.

Example output command

./HelloPhotogrammetry ./Wooden_bike_photos ./Wooden_bike_model -d full -o sequential -f normal

The reconstructed 3D model is reconstructed as OBJ and USDZ format with accompanying texture images and material file.

Model Preparation

Before the 3D model is imported into the Model Target Generator (MTG), it should be cleaned up from any undesired mesh artifacts and its scale should be validated and corrected if necessary.

NOTE: If you have used images from a camera that also captured depth data, the scale will likely be correct to that of the real object, and rescaling will not be necessary. Smaller objects might have a larger scale offset.

Open the OBJ in a mesh editor and inspect the model, and if necessary, adjust it uniformly so that it corresponds to the object. Save the file in one of the formats that the MTG supports.

See the Create a Model Target from a Reconstruction for the final steps.