---
title: "How To: Run AI Object Matching on Camera Images (ctrlX CORE with Hailo)"
description: "From a trained network to detections on camera images - upload the model, configure the module, use the Hailo accelerator of the ctrlX CORE, run the inference and read the result."
product: "LumiScan AI Object Matching"
component: "lumiscan-ai-object-matching"
version: "2.1.0"
display_version: "2.1.0.9"
latest_version: true
language: "en"
url: "https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/how-tos/run-ai-object-matching-on-a-camera-image.html"
---

# How To: Run AI Object Matching on Camera Images (ctrlX CORE with Hailo) – LumiScan AI Object Matching

This guide sets up LumiScan AI Object Matching on a ctrlX CORE so that it detects objects in the images of LumiScan Cam Driver with a network trained in LumiScan Training, accelerated by the Hailo TPU of the ctrlX CORE.

When you have finished:

- the trained network is loaded on the ctrlX CORE and runs on the Hailo TPU,
- the module reads the camera image directly from LumiScan Cam Driver,
- one write to `inference/execute` detects the objects in the current image,
- the detections - position, class, score and, depending on the network, orientation - are on the Data Layer.

## Before you start {#prerequisites}

**Hardware**

- ctrlX CORE X3 or X7 with a Hailo TPU
- A camera connected to LumiScan Cam Driver
- Engineering PC with a browser and WinSCP

**Software**

- LumiScan AI Object Matching with a licence, see [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html)
- LumiScan Cam Driver, configured, connected and able to capture images
- For the Hailo TPU: the ctrlX app **Hardware Support** in version 2.6.2
- Optional: LumiScan Cockpit, to look at the images and results

**Data**

- The trained network from LumiScan Training: the model file (`.onnx`) and its description file (`.json`)

## How it fits together {#overview}

The settings of LumiScan AI Object Matching are files in its folder `lumiscan-ai-object-matching`:

- `ai-object-matching.json` - the application configuration. Its `modules` list names the module file to load. Currently, one module is supported.
- A module file, for example `ai-object-matching-module.json` - the module’s name, its network and its image source.
- The folder `models` - the network: the model file and its description file, `model.json`, with the class names, the thresholds and the inference provider.

The module’s `name` is its address on the Data Layer: `hdvisionsystems/lumiscan-ai-object-matching/<name>/…`. With `externalImagePath`, the module reads the image directly from another application, here LumiScan Cam Driver. Each inference uses the image as it is at that moment.

The files are read when the application starts. Restart it after every change.

## Step 1 - Check the Hailo requirements {#step-1-hailo}

1. In the ctrlX web interface, open **Settings > Apps**.
2. Check that the app **Hardware Support** is installed in version 2.6.2.
3. Compare the versions with the table **TPU Acceleration Requirements** in [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html).

The TPU is optional. Without it, the network runs on the CPU (`provider` `CPU` in step 4), only slower.

## Step 2 - Upload the network {#step-2-upload}

The model files are uploaded with WebDAV.

1. Start WinSCP and create a connection:
   - **File protocol**: `WebDAV`
   - **Encryption**: `TLS/SSL`
   - **Host name**: the address of the ctrlX CORE, with your ctrlX user name and password
2. Connect and go to `/solutions/webdav/appdata/lumiscan-ai-object-matching/models`.
3. Copy the model file (`.onnx`) and its description file (`.json`) into this folder.

The steps in detail, with screenshots, are in [Uploading Data with WebDAV](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html#upload-using-webdav).

## Step 3 - Configure the application and the module {#step-3-module}

1. In the ctrlX web interface, go to **Home > Manage app data**, click **Open file view** and open `lumiscan-ai-object-matching`.

2. Open `ai-object-matching.json` and check that the `modules` list names your module file:
   ```json
   "modules": [
     "ai-object-matching-module.json"
   ]
   ```

3. Open the module file and set:
   ```json
   {
     "name": "example",
     "neuralNetworkConfig": "models/model.json",
     "externalImagePath": "hdvisionsystems/lumiscan-cam-driver/camera1/output/image"
   }
   ```
   - `name` - the module’s name on the Data Layer. Only lower-case letters, digits, `-` and `_`.

   - `neuralNetworkConfig` - the path of the description file you uploaded, relative to the application folder.

   - `externalImagePath` - the image node of the camera module, here `camera1` of LumiScan Cam Driver. The node must provide `data`, `width`, `height` and `pixel-type`.

     Keep the `$schema` and `meta` entries of the file. All parameters are described in [Module Configuration File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html).

## Step 4 - Select the Hailo TPU in the model description {#step-4-model}

Open `models/model.json`. Its parameters are generated by LumiScan Training; change only these:

```json
{
  "pathToNetwork": "model.onnx",
  "provider": "HAILO",
  "deviceIndex": 0
}
```

- `pathToNetwork` - the model file, relative to `model.json`.
- `provider` - `HAILO` for the TPU, `CPU` without it.
- `deviceIndex` - currently always `0`.

In the same file, `thresholdList` sets for each class the minimum score, from `0.0` to `1.0`, a detection needs. See [Model.json File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html#model-json-file).

## Step 5 - Restart and check the module {#step-5-restart}

1. Save all changed files.
2. Under **Settings > Apps**, select **LumiScan AI Object Matching** and click **Restart**.
3. Open **Settings > Data Layer** and go to `hdvisionsystems/lumiscan-ai-object-matching`.

Expected: a folder with the module’s `name`, here `example`, and `example/control/inference/ready` is `true` once the network is loaded.

If the module is missing or `ready` stays `false`, read `example/control/status/text` and the application log. Common causes are a wrong path in the module file or in `model.json`, a `name` with upper-case letters, and a missing licence. See [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html).

## Step 6 - Run the inference {#step-6-inference}

1. Capture an image with LumiScan Cam Driver, for example with a write to `hdvisionsystems/lumiscan-cam-driver/camera1/control/capture/execute`.
2. Write to `hdvisionsystems/lumiscan-ai-object-matching/example/control/inference/execute`. The write returns when the inference has finished.
3. Read `example/control/status/code`.

Expected: the status code is `0`, and `example/output/package-identifier` has grown by one.

- `100001` means that there was no image to read. Capture an image and run the inference again.
- `-1` means that the inference failed. The module then accepts only a reset: write to `example/control/reset/execute`, and check the log.

> [!NOTE]
> A client waits only a few seconds for an `execute` call to return. If the inference on a large image takes longer, start it with the ready/request handshake on `example/control/inference` instead. The handshake always ends at status code `3`; tell success by `output/package-identifier`, which grows only after a successful inference. See [Initializing the Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html).

## Step 7 - Read the detections {#step-7-result}

The detections are on

```
hdvisionsystems/lumiscan-ai-object-matching/example/output/result
```

For each detected object, the result gives its position, class and score and, if the network was trained for it, its orientation. How to read the result in a PLC program is described in [Accessing Result Data with FlatBuffers](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html).

To check the detections visually:

- In LumiScan Cockpit, a module that infers on the camera’s image stream shows the boxes on the live frames. See [Inference on an Image Stream (Experimental)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html).
- For a single inference, `example/control/save-result-image/execute` saves the result image for debugging.

## Result {#result}

- The trained network runs on the Hailo TPU of the ctrlX CORE.
- The module reads the camera image of LumiScan Cam Driver directly.
- One write to `inference/execute` detects the objects in the current image.
- The detections are on `output/result`, ready for the PLC or the robot.

In production, a cycle is: capture with LumiScan Cam Driver, write to `inference/execute`, check the status code, read `output/result`.