# LumiScan AI Object Matching 2.1.0.9 – complete documentation

> All pages of the LumiScan AI Object Matching 2.1.0.9 manual (language: en) by HD Vision Systems GmbH, in navigation order. Each page starts with a level-1 heading followed by its URL; cite that URL. Manual: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/

# Description – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/description.html

LumiScan AI Object Matching uses neural networks to detect trained objects in images and reports their position and orientation in pixel coordinates.

The application runs on ctrlX OS (ctrlX CORE or ctrlX IPC) or on a PC (Windows 11 or Ubuntu 24.04), and is controlled through the Data Layer: the ctrlX Data Layer on ctrlX OS, LumiScan Keryx on a PC. It is configured using application and model configuration files. On ctrlX OS it is installed through the App Store.

What is the same for every LumiScan app - how addresses are built, the handshake, execute and stream call patterns, status codes, settings and tools - is described in the [LumiScan Keryx](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/index.html) manual. This manual covers only what is specific to LumiScan AI Object Matching; the pages of this manual describe its own nodes.

Unlike rule-based image processing, AI Object Matching relies on a trained neural network. The model and its accompanying description file are created with the LumiScan Training Framework and uploaded to the application together.

## Where the Images Come From

The application does not acquire images directly from a camera. It reads image data published by another application through the Data Layer.

For example, images can be provided by the LumiScan Cam Driver:

`hdvisionsystems/lumiscan-cam-driver/camera1/output/image`

The configured image source must provide the image data and the required image information in the expected format.

## Models

A trained model defines which objects the application can detect. Models can be created using the LumiScan Annotation Framework and LumiScan Training Framework.

Each model consists of the neural network file and its corresponding description file. These files belong together and should always be uploaded and handled as one unit.

The model configuration defines parameters such as the neural network path, detection thresholds, and inference provider. See [Model.json File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html#model-json-file).

## Modules

A **module** represents one configured instance of AI Object Matching. It defines the image source, the model configuration to use, and where the corresponding results and controls are available in the Data Layer.

Each module therefore combines:

- An input image source
- A trained neural network model
- Detection settings
- Data Layer controls and results

Currently, only one module can be active at a time.

# Requirements – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html

To run LumiScan, the following components are required:

**On ctrlX OS:**

- ctrlX CORE or ctrlX IPC with the ctrlX Data Layer

**On a PC:**

- PC with Windows 11 or Ubuntu 24.04 and LumiScan Keryx
- NVIDIA GPU with the CUDA Toolkit 12.9 and cuDNN 9.11.1 for CUDA 12.9 (the Windows installer requires both)

**On both:**

- Neural Network Models provided by LumiScan Training Framework software

## Licence {#licence}

How LumiScan AI Object Matching is licensed depends on where you bought it and where it runs:

| Bought from | Runs on | Licensing |
| --- | --- | --- |
| HD Vision Systems | Windows 11 or Ubuntu 24.04 | The licence is held on your LumiScan Key. It is a one-time purchase and needs no renewal.<br><br>[Licensing path: bought from HD Vision Systems](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/path-hdvs.html) |
| ctrlX Store | Windows 11 | The licence is held in ctrlX WORKS. The LumiScan Key is required in addition; it protects the software against reverse engineering.<br><br>[Licensing path: ctrlX Store, Windows](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/path-ctrlx-windows.html) |
| ctrlX Store | ctrlX OS (ctrlX CORE or IPC) | ctrlX licence `SWL-W-XCx-OBJxMATCHxHDVSx-V2NN`, one licence per app instance.<br><br>[Licensing path: ctrlX Store, ctrlX OS](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/path-ctrlx-os.html) |

Each device on which LumiScan AI Object Matching runs at the same time needs its own LumiScan Key. Keys can be exchanged between devices.

How to install CodeMeter Runtime and how to request, renew and move licences is described in [LumiScan Licensing](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/index.html).

On ctrlX OS, the application shuts down shortly after starting if the licence is missing; the reason is written to the system log.

## Optional: TPU Acceleration (ctrlX OS)

TPU acceleration can be used to improve neural network inference performance, but it is not required for running LumiScan.

If TPU acceleration is enabled, make sure the ctrlX CORE includes a TPU and meets the following software and hardware support requirements:

**Table 1. TPU Acceleration Requirements**

| ctrlX CORE Hardware | Minimum App Version | App Hardware Support Version |
| --- | --- | --- |
| X3 | \>= v1.2 | == 2.6.2 |
| X7 | \>= v1.1 | == 2.6.2 |

## Optional: GPU Acceleration (PC)

On a PC, inference can be accelerated with an NVIDIA GPU: set `provider` to `CUDA` in the model description file, see [Model.json File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html#model-json-file).

# How To: Run AI Object Matching on Camera Images (ctrlX CORE with Hailo) – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/how-tos/run-ai-object-matching-on-a-camera-image.html

This guide sets up LumiScan AI Object Matching on a ctrlX CORE so that it detects objects in the images of LumiScan Cam Driver with a network trained in LumiScan Training, accelerated by the Hailo TPU of the ctrlX CORE.

When you have finished:

- the trained network is loaded on the ctrlX CORE and runs on the Hailo TPU,
- the module reads the camera image directly from LumiScan Cam Driver,
- one write to `inference/execute` detects the objects in the current image,
- the detections - position, class, score and, depending on the network, orientation - are on the Data Layer.

## Before you start {#prerequisites}

**Hardware**

- ctrlX CORE X3 or X7 with a Hailo TPU
- A camera connected to LumiScan Cam Driver
- Engineering PC with a browser and WinSCP

**Software**

- LumiScan AI Object Matching with a licence, see [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html)
- LumiScan Cam Driver, configured, connected and able to capture images
- For the Hailo TPU: the ctrlX app **Hardware Support** in version 2.6.2
- Optional: LumiScan Cockpit, to look at the images and results

**Data**

- The trained network from LumiScan Training: the model file (`.onnx`) and its description file (`.json`)

## How it fits together {#overview}

The settings of LumiScan AI Object Matching are files in its folder `lumiscan-ai-object-matching`:

- `ai-object-matching.json` - the application configuration. Its `modules` list names the module file to load. Currently, one module is supported.
- A module file, for example `ai-object-matching-module.json` - the module’s name, its network and its image source.
- The folder `models` - the network: the model file and its description file, `model.json`, with the class names, the thresholds and the inference provider.

The module’s `name` is its address on the Data Layer: `hdvisionsystems/lumiscan-ai-object-matching/<name>/…`. With `externalImagePath`, the module reads the image directly from another application, here LumiScan Cam Driver. Each inference uses the image as it is at that moment.

The files are read when the application starts. Restart it after every change.

## Step 1 - Check the Hailo requirements {#step-1-hailo}

1. In the ctrlX web interface, open **Settings > Apps**.
2. Check that the app **Hardware Support** is installed in version 2.6.2.
3. Compare the versions with the table **TPU Acceleration Requirements** in [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html).

The TPU is optional. Without it, the network runs on the CPU (`provider` `CPU` in step 4), only slower.

## Step 2 - Upload the network {#step-2-upload}

The model files are uploaded with WebDAV.

1. Start WinSCP and create a connection:
   - **File protocol**: `WebDAV`
   - **Encryption**: `TLS/SSL`
   - **Host name**: the address of the ctrlX CORE, with your ctrlX user name and password
2. Connect and go to `/solutions/webdav/appdata/lumiscan-ai-object-matching/models`.
3. Copy the model file (`.onnx`) and its description file (`.json`) into this folder.

The steps in detail, with screenshots, are in [Uploading Data with WebDAV](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html#upload-using-webdav).

## Step 3 - Configure the application and the module {#step-3-module}

1. In the ctrlX web interface, go to **Home > Manage app data**, click **Open file view** and open `lumiscan-ai-object-matching`.

2. Open `ai-object-matching.json` and check that the `modules` list names your module file:
   ```json
   "modules": [
     "ai-object-matching-module.json"
   ]
   ```

3. Open the module file and set:
   ```json
   {
     "name": "example",
     "neuralNetworkConfig": "models/model.json",
     "externalImagePath": "hdvisionsystems/lumiscan-cam-driver/camera1/output/image"
   }
   ```
   - `name` - the module’s name on the Data Layer. Only lower-case letters, digits, `-` and `_`.

   - `neuralNetworkConfig` - the path of the description file you uploaded, relative to the application folder.

   - `externalImagePath` - the image node of the camera module, here `camera1` of LumiScan Cam Driver. The node must provide `data`, `width`, `height` and `pixel-type`.

     Keep the `$schema` and `meta` entries of the file. All parameters are described in [Module Configuration File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html).

## Step 4 - Select the Hailo TPU in the model description {#step-4-model}

Open `models/model.json`. Its parameters are generated by LumiScan Training; change only these:

```json
{
  "pathToNetwork": "model.onnx",
  "provider": "HAILO",
  "deviceIndex": 0
}
```

- `pathToNetwork` - the model file, relative to `model.json`.
- `provider` - `HAILO` for the TPU, `CPU` without it.
- `deviceIndex` - currently always `0`.

In the same file, `thresholdList` sets for each class the minimum score, from `0.0` to `1.0`, a detection needs. See [Model.json File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html#model-json-file).

## Step 5 - Restart and check the module {#step-5-restart}

1. Save all changed files.
2. Under **Settings > Apps**, select **LumiScan AI Object Matching** and click **Restart**.
3. Open **Settings > Data Layer** and go to `hdvisionsystems/lumiscan-ai-object-matching`.

Expected: a folder with the module’s `name`, here `example`, and `example/control/inference/ready` is `true` once the network is loaded.

If the module is missing or `ready` stays `false`, read `example/control/status/text` and the application log. Common causes are a wrong path in the module file or in `model.json`, a `name` with upper-case letters, and a missing licence. See [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html).

## Step 6 - Run the inference {#step-6-inference}

1. Capture an image with LumiScan Cam Driver, for example with a write to `hdvisionsystems/lumiscan-cam-driver/camera1/control/capture/execute`.
2. Write to `hdvisionsystems/lumiscan-ai-object-matching/example/control/inference/execute`. The write returns when the inference has finished.
3. Read `example/control/status/code`.

Expected: the status code is `0`, and `example/output/package-identifier` has grown by one.

- `100001` means that there was no image to read. Capture an image and run the inference again.
- `-1` means that the inference failed. The module then accepts only a reset: write to `example/control/reset/execute`, and check the log.

> [!NOTE]
> A client waits only a few seconds for an `execute` call to return. If the inference on a large image takes longer, start it with the ready/request handshake on `example/control/inference` instead. The handshake always ends at status code `3`; tell success by `output/package-identifier`, which grows only after a successful inference. See [Initializing the Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html).

## Step 7 - Read the detections {#step-7-result}

The detections are on

```
hdvisionsystems/lumiscan-ai-object-matching/example/output/result
```

For each detected object, the result gives its position, class and score and, if the network was trained for it, its orientation. How to read the result in a PLC program is described in [Accessing Result Data with FlatBuffers](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html).

To check the detections visually:

- In LumiScan Cockpit, a module that infers on the camera’s image stream shows the boxes on the live frames. See [Inference on an Image Stream (Experimental)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html).
- For a single inference, `example/control/save-result-image/execute` saves the result image for debugging.

## Result {#result}

- The trained network runs on the Hailo TPU of the ctrlX CORE.
- The module reads the camera image of LumiScan Cam Driver directly.
- One write to `inference/execute` detects the objects in the current image.
- The detections are on `output/result`, ready for the PLC or the robot.

In production, a cycle is: capture with LumiScan Cam Driver, write to `inference/execute`, check the status code, read `output/result`.

# General Concept of Handshake – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html

The communication process in the LumiScan AI Object Matching application is based on the general concept of handshake. The naming convention for handshake parameters usually consists of the **functionality** and the names **-ready** or **-request**.

The handshake works the same on the ctrlX Data Layer and on LumiScan Keryx. Its general rules are described in [Handshake](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/handshake.html); this page describes how AI Object Matching follows it and where it differs.

**Table 1. General Concept of Handshake: Address**

| Address | Description |
| --- | --- |
| `<parent address>/<functionality name>/ready` | Read-Only, controlled by our app to indicate whether the relevant functionality is allowed to be executed. |
| `<parent address>/<functionality name>/request` | Writable. It is used by user to trigger our application to execute relevant functionality. |

## Functionality Execution

> [!NOTE]
> To avoid repeating the entire address, in this section the **invariant part** containing the parent address and the **variable functionality name** (inference, reset) have been replaced by **XXX**.
>
> Thus, the address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/ready` will be written as **XXX/ready**.
>
> Accordingly, the address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/request` will be written as **XXX/request**.

In order to execute a functionality, you need to run the following steps (see also image [Functionality Execution Steps](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#functionality-execution-steps)):

1. Wait until XXX/ready is set to **True**.
2. Set XXX/request to **True**.
   > [!NOTE]
   > Make sure there is only one process is accessing XXX/request at a time.
   >
   > The result of execution is **undefined** when:
   >
   > a. ProcessA set XXX/request to True.
   >
   > b. ProcessB set XXX/request to False without considering ProcessA.
   >
   > c. Our application might therefore miss XXX/request from ProcessA.
3. Wait until XXX/ready changes to **False**: the application has seen the request.
4. Set XXX/request to **False**.
5. Wait until XXX/ready is **True** again: the functionality has finished.
6. Read `control/status/code` and the results.

**Functionality Execution Steps**

```mermaid
%% The ready/request handshake. Rendered by Mermaid in the browser; edit and preview at https://mermaid.live
sequenceDiagram
  autonumber
  participant C as Client (PLC or program)
  participant M as Module
  M-->>C: ready = true: the function can be started
  C->>M: request = true
  M-->>C: ready = false: the request was seen
  C->>M: request = false
  Note over M: runs the function
  M-->>C: ready = true: the function has finished
  C->>M: read control/status/code and output/
```

## When a Functionality Runs {#when-a-functionality-runs}

Not every functionality waits for XXX/request to fall before it starts:

**Table 2. When each functionality runs**

| Functionality | Runs | `status/code` at the end |
| --- | --- | --- |
| `inference` | At once, as soon as the application sees the request. `status/code` is `2` (running) while it runs, then `3` (waitingForClient) while the application waits for XXX/request to fall. | **`3`, whether the inference succeeded or not.** See [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference). |
| `reset` | At once, without waiting for XXX/request to fall. It runs again on every cycle while XXX/request stays **True**. | `0` |
| `list-result-files`, `load`, `save`, `save-result-image`, `remove` | After XXX/request has fallen. `status/code` is `2` from the request until they finish. | `0`, or an error (below `0`) |

While the module is in error, it ignores every request but `reset`. XXX/ready still shows **True**, but the request is never acknowledged: XXX/ready does not fall. Request `reset`.

While the image stream runs, `inference/ready` is **False**. An inference requested anyway sets `status/code` to `100002` (StreamActive) and is not acknowledged.

## Outcome of an Inference {#outcome-of-an-inference}

An inference requested through the handshake ends at `status/code` `3`, whatever its outcome. Tell the outcome by `hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/package-identifier`: it grows by one with every successful inference, and only then.

If it did not grow, the inference failed. Either:

- the image could not be read: there was no image, or it was empty. Provide an image and request the inference again; or
- the inference itself failed. The module is then in error and ignores every request but `reset` (see above). Request `reset`, and check the application log for the cause.

To read the outcome directly, start the inference with `control/inference/execute` instead. `status/code` then keeps it: `0`, `100001` or `-1`. See [Execute a Method](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#execute-a-method).

# Initializing the Inference – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html

This page lists the control nodes of a module and describes how to run an inference. The nodes of the image stream are listed in [Inference on an Image Stream](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html), the `system` nodes in [Settings on the Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html).

## Data Layer {#data-layer}

LumiScan AI Object Matching offers its functions on the Data Layer under `hdvisionsystems/lumiscan-ai-object-matching`.

On ctrlX OS (ctrlX CORE or ctrlX IPC), LumiScan AI Object Matching uses the [ctrlX Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/ctrlx-data-layer.html).

On a PC with Windows 11 or Ubuntu 24.04, LumiScan AI Object Matching uses [LumiScan Keryx](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/keryx.html).

The node addresses, types and call patterns are the same on both. LumiScan AI Object Matching offers:

- [ready/request handshakes](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/handshake.html)
- [execute methods](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/execute.html)
- a [stream](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/streams.html) that publishes its results (experimental)
- a [stream](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/streams.html) subscription to one source (experimental)

How addresses are built, how to call functions, the status codes and the tools are described in [LumiScan Keryx](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/index.html).

## Addresses

**Table 1. Functionality Addresses**

| Functionality | Permission | Address |
| --- | --- | --- |
| Inference | Read-Only | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/ready` |
| Inference | Writable | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/request` |
| Inference | Write-Only (method) | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/execute` |
| Reset | Read-Only | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/reset/ready` |
| Reset | Writable | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/reset/request` |
| Reset | Write-Only (method) | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/reset/execute` |
| Status | Read-Only | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/status/code` |
| Status | Read-Only | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/status/text` |
| Result counter | Read-Only | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/package-identifier` |

`output/package-identifier` grows by one with every successful inference, and only then.

## Steps To Run Inference with External Image

> [!NOTE]
> To avoid repeating the entire address, in this section the **invariant part** containing the parent address has been replaced by **XXX**.
>
> Thus, the address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/ready` will be written as **XXX/inference/ready**.
>
> Accordingly, the address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/request` will be written as **XXX/inference/request**.

The application reads the image from the external address configured for the network, as it is when the inference starts. It does not check whether the image is new: an image that has not changed since the last inference is inferred again.

To start running inference:

1. Check XXX/inference/ready = True.[^1]

> [!NOTE]
> The application will automatically load configurations and setup neural networks. XXX/inference/ready is **False** while they are set up, while the image stream runs, without a licence, and after the settings were changed on the Data Layer (`status/code` `-5`, restart required). Please check XXX/status/code and the log for more information.

2. Note the value of `output/package-identifier`.
3. Set XXX/inference/request := True
4. Wait until XXX/inference/ready = False. The inference is already running.

> [!NOTE]
> If the application hasn’t seen that the variable has changed:
>
> a. Make sure that only one process is accessing XXX/inference/request at a time.
>
> b. If the last inference failed, the module ignores every request but reset. Request a reset.
>
> c. Contact hdvisionsystems.

5. Set XXX/inference/request := False
6. Wait until XXX/inference/ready = True.
7. Check that `output/package-identifier` has grown by one.\
   `status/code` is `3` now, whether the inference succeeded or not: see [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference).\
   You can now check result at `hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/result`

## Steps to Run Inference with Input Image

> [!NOTE]
> To avoid repeating the entire address, in this section the **invariant part** containing the parent address has been replaced by **XXX**.
>
> Thus, the address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/ready` will be written as **XXX/inference/ready**.
>
> Accordingly, the address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/request` will be written as **XXX/inference/request**.

To set input image:

1. Find the following address in the datalayer[^2]: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/image`
2. Copy raw image into the following address: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/image`
3. Set corresponding image width and height and PixelFormat under the path `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/image`

> [!NOTE]
> For the RGB image, data layout is height, width, channel. In other words, the RGB value of a pixel is displayed next to each other in the raw data.

To start running inference:

1. Check XXX/inference/ready = True.[^1]

> [!NOTE]
> The application will automatically load configurations and setup neural networks. As long as XXX/inference/ready is **True**, you can run inference.

2. Note the value of `output/package-identifier`.
3. Set XXX/inference/request := True
4. Wait until XXX/inference/ready = False. The inference is already running.

> [!NOTE]
> If the application hasn’t seen that the variable has changed:
>
> a. Make sure that only one process is accessing XXX/inference/request at a time.
>
> b. If the last inference failed, the module ignores every request but reset. Request a reset.
>
> c. Contact hdvisionsystems.

5. Set XXX/inference/request := False
6. Wait until XXX/inference/ready = True.
7. Check that `output/package-identifier` has grown by one.\
   `status/code` is `3` now, whether the inference succeeded or not: see [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference).\
   You can now check result at `hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/result`

## Execute a Method {#execute-a-method}

Each function can also be started with a single write to its `execute` node instead of the handshake:

`hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/<function>/execute`

for `inference`, `reset`, `list-result-files`, `load`, `save`, `save-result-image` and `remove`. The write returns when the function has finished. It runs the same code as the handshake, and its results appear on the same output nodes.

Two results have to be kept apart:

- **Whether the function could be started at all** is reported by the write itself. A refused write is never queued. The results have different names on the two Data Layers; their meaning is the same (see also [Results of a method call](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/status-codes.html#method-results)):

  | ctrlX Data Layer | LumiScan Keryx | Meaning | What to do |
  | --- | --- | --- | --- |
  | `DL_OK` | `Ok` | The function ran. | Read `control/status/code` for its outcome. |
  | `DL_TOO_MANY_OPERATIONS` | `TooManyOperations` | Another function of the module runs, through the handshake or another `execute`. | Retry. |
  | `DL_INVALID_OPERATION_MODE` | `InvalidOperationMode` | The function’s `ready` is **False**, for example `inference` while the image stream runs. | Bring the module into the state first; retrying alone keeps failing. |
  | `DL_RESOURCE_UNAVAILABLE` | `ResourceUnavailable` | The application is shutting down; or the settings were changed on the Data Layer and a restart is required (`status/code` `-5`); or the module is in error after a failed inference. `reset/execute` is never refused for the error. | After an error, run `reset`. After a settings change, restart the application, see [Settings on the Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#restart-required). |

- **Whether the function succeeded** is reported on `control/status/code` and `control/status/text`. Unlike the handshake, `execute` leaves the outcome there: after `inference/execute` it is `0`, `100001` or `-1`.

Keep each call short: a client gives up waiting after a few seconds (on LumiScan Keryx after 2 s by default, see [Execute Methods](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/execute.html#timeouts)). Inference on a large image or a slow device can take longer; use the handshake then.

## Status Code Values {#status-code-values}

The values `0` to `3` and `-1` to `-5` are the module status codes common to the LumiScan applications, see [Status Codes and Results](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/status-codes.html#module-status). `100001` and `100002` are specific to AI Object Matching.

**Table 2. Status Code Values Description**

| Status Code Number | String | Meaning |
| --- | --- | --- |
| 0 | "ready" | The last functionality is successfully executed, the application is ready to execute the next functionality. |
| 1 | "settingUp" | The module or function is still setting up. |
| 2 | "running" | The function is currently running. |
| 3 | "waitingForClient" | Waiting for the client (usually for XXX/request to fall). Also the end of every inference requested through the handshake, whether it succeeded or not: see [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference). |
| 100001 | "ImageQueueEmpty" | The image could not be read: the image node (`input/image`, or the external image address) has no data, or could not be read. Not an error: provide an image and run the inference again. |
| 100002 | "StreamActive" | An inference was requested while the image stream runs. It is refused, not an error. See [Inference on an Image Stream](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html). |
| -1 | "undefinedError" | Something went wrong and we did not create a special error code for it. |
| -2 | "invalidSetting" | The supplied settings file could not be loaded. |
| -3 | "invalidInput" | One of the input fields was not valid. |
| -4 | "MissingLicense" | Make sure you have the correct licence installed. |
| -5 | "restartRequired" | The settings files were changed on the Data Layer: restart the application. See [Settings on the Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#restart-required). |

[^1]: The meaning of all status code values is explained in the [Status Code Values](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#status-code-values) table at the end of this chapter.
[^2]: If the application is configured to use an input image, the specified address should be there.

# Accessing Result Data with FlatBuffers – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html

This page is for ctrlX OS (ctrlX CORE or ctrlX IPC): it describes how a ctrlX PLC program reads the result.

LumiScan AI Object Matching publishes the result of each inference on the ctrlX Data Layer as a binary FlatBuffers message. A PLC program can only interpret this message with the schema that describes its layout. The application therefore provides the schema file `object_detection.fbs`.

This chapter describes how to obtain the schema, how to import it into ctrlX PLC Engineering, and how to read the result, including the position and score of a detected object.

## Result Node

**Table 1. Result Node Properties**

| Property | Value |
| --- | --- |
| Node address | `hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/result` |
| Permission | Read-Only |
| Data type | FlatBuffers |
| Registered type address | `types/hdvisionsystems/lumiscan-ai-object-matching/object_detection` |
| Schema file | `object_detection.fbs` |
| Schema namespace | `hdv.object_detection2d` |
| Root type | `Result` |

The node holds the result of the latest inference. Read it only after the inference has finished and succeeded: with the handshake, `output/package-identifier` has grown by one (`status/code` is `3` at the end of every inference requested through the handshake); with `control/inference/execute`, `status/code` is `0`. See [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference) and [Status Code Values](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#status-code-values).

## Prerequisites

- ctrlX PLC Engineering 2.6 or later.
- The libraries `CXA_Datalayer` and `CXA_Flatbuffers` are referenced in the **Library Manager** of the PLC project. `CXA_Datalayer` reads nodes from the Data Layer. `CXA_Flatbuffers` provides the base function blocks of the FlatBuffers function blocks that the import generates. After the import, check the Library Manager. If a library is missing, add it there (see also [Install the ctrlX Data Layer Library](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-install-datalayer-library)).
- LumiScan AI Object Matching is installed and has been started at least once with a valid licence (see [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html)).

## Step 1: Obtain the Schema File

The application copies the schema to its application data folder on the ctrlX CORE each time it starts. To download it:

1. Log in to the ctrlX OS web interface and open **Manage app data** (see [LumiScan AI Object Matching Folder Structure](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/folder-structure.html)).
2. Open the folder `lumiscan-ai-object-matching/schemas/flatbuffers`.
3. Click the download icon in the **Actions** column of `object_detection.fbs` and save the file on the engineering PC.

Alternatively, download the file over WebDAV (see [Uploading Data](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html)).

> [!NOTE]
> The `schemas` folder is created while the application starts, after the licence check. If the folder does not exist, the application has not started successfully. Check that a valid licence is installed and read the system log. See [The schemas folder does not exist](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-schemas-folder-missing).
>
> The application deletes and recreates the `schemas` folder on every start. Do not edit files in this folder; changes are lost. The schema always matches the installed application version.

## Step 2: Import the Schema into the PLC Project

1. In ctrlX PLC Engineering, open the PLC project.
2. In the device tree, right-click **Application**.
3. Select **Import from FlatBuffers file…**.
4. Select the downloaded `object_detection.fbs`.

![ctrlX PLC Engineering device tree with the Application context menu open on Import from FlatBuffers file](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/plc-import-flatbuffers.png)

*Figure 1. Import from FlatBuffers file*

PLC Engineering adds the following function blocks to the application. Their names consist of the schema namespace and the type name.

**Table 2. Function Blocks Created by the Import**

| Function block | Purpose |
| --- | --- |
| `hdv_object_detection2d_Result` | Root of the message. Provides the best score and class, and access to the array of instances. |
| `hdv_object_detection2d_Instance` | One detected object: class index, score and bounding box. |
| `hdv_object_detection2d_OrientedBoundingBox` | Center, size and angle of the bounding box of an instance. |
| `hdv_object_detection2d_InstanceTVec` | Not needed for reading results (see [Known Message: Identifier 'VectorT' Not Defined](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-known-message-vectort)). |

### Known Message: Identifier 'VectorT' Not Defined {#lumiscan-known-message-vectort}

Depending on the version of ctrlX PLC Engineering, the editor of the function block `hdv_object_detection2d_InstanceTVec` shows the message `Identifier 'VectorT' not defined`. The message does not affect reading results. Causes and steps are described in [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html):

- [The message "Identifier 'VectorT' not defined" appears after the import](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-vectort-not-defined)

### Install the ctrlX Data Layer Library {#lumiscan-install-datalayer-library}

If **Import from FlatBuffers file…** is not available in the context menu, the ctrlX Data Layer library is missing. Install it as follows, then repeat Step 2:

1. In the PLC Engineering menu bar, select **Tools** > **Library Repository…**.

![ctrlX PLC Engineering Tools menu with the Library Repository command highlighted](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/plc-install-datalayer-1.png)

*Figure 2. Library Repository command*

2. Search for `CXA_Datalayer`, select the library and click **Open**.

![Find Library dialog with the search term CXA_Datalayer and the library CXA_Datalayer selected](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/plc-install-datalayer-2.png)

*Figure 3. Searching for the CXA_Datalayer library*

## Step 3: Read the Result in the PLC Program

1. Run an inference with the handshake described in [Initializing the Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html).
2. Wait until the inference has finished and check that it succeeded: `inference/ready` is **True** again and `output/package-identifier` has grown by one. With `control/inference/execute` instead, check that `status/code` is 0.
3. Read the node `hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/result` with the function block `DL_ReadNodeValue` of the `CXA_Datalayer` library. The function block returns the value in a `DL_NodeValue`.
4. Pass the address and size of the buffer (`GetData` and `GetSize` of the `DL_NodeValue`) to `getRootAsResult` of `hdv_object_detection2d_Result`.
5. Read the fields of the result with the `get` methods of the generated function blocks.

> [!NOTE]
> The generated function blocks read the received buffer in place; they do not copy it. Use the values or copy them before the node is read again.

## Example: Reading Position and Score of an Instance {#lumiscan-example-instance}

The `instances` array of the result is sorted by score in descending order. Instance 0 is therefore the instance with the highest score. The example reads the position (x, y), the score and the class index of this instance.

Declaration:

```text
PROGRAM PRG_ReadObjectMatchingResult
VAR
    // Data Layer access (CXA_Datalayer)
    fbReadResult : DL_ReadNodeValue;
    nvResult     : DL_NodeValue;    // receives the FlatBuffers buffer
    bRead        : BOOL;            // set to TRUE to read the node once
    sNode        : STRING(255) := 'hdvisionsystems/lumiscan-ai-object-matching/<network name>/output/result';

    // FlatBuffers readers, generated by "Import from FlatBuffers file..."
    fbResult     : hdv_object_detection2d_Result;
    fbInstance   : hdv_object_detection2d_Instance;
    pBox         : POINTER TO hdv_object_detection2d_OrientedBoundingBox;

    // Values of the instance with the highest score
    bFound       : BOOL;
    rX           : REAL;            // x of the bounding box center [pixel]
    rY           : REAL;            // y of the bounding box center [pixel]
    rScore       : REAL;            // score [0.0 - 1.0]
    uClassIndex  : ULINT;           // index into class-names
END_VAR
```

Implementation:

```text
// 1. Read the node. A rising edge on bRead starts one read.
fbReadResult(Execute := bRead, NodeName := sNode, NodeValue := nvResult);

IF fbReadResult.Done THEN
    bFound := FALSE;

    IF nvResult.GetSize() > 0 THEN
        // 2. Point the generated reader at the received buffer.
        fbResult.getRootAsResult(data := nvResult.GetData(), size := nvResult.GetSize());

        // 3. The instances are sorted by score, so instance 0 has the highest score.
        IF fbResult.getInstancesLength() > 0 THEN
            IF fbResult.getInstances(j := 0, result := ADR(fbInstance)) THEN
                rScore      := fbInstance.GetScore();
                uClassIndex := fbInstance.GetClassIndex();

                // 4. The position is part of the bounding box.
                pBox := fbInstance.getOrientedBoundingBox();
                IF pBox <> 0 THEN
                    rX     := pBox^.GetCenterX();
                    rY     := pBox^.GetCenterY();
                    bFound := TRUE;
                END_IF
            END_IF
        END_IF
    END_IF
END_IF

// 5. End the access.
IF fbReadResult.Done OR fbReadResult.Error THEN
    bRead := FALSE;
    fbReadResult(Execute := bRead, NodeName := sNode, NodeValue := nvResult);
END_IF
```

Replace `<network name>` in `sNode` with the name of the network. Set `bRead` to TRUE only after the inference has finished and succeeded (see Step 3). If `fbReadResult.Error` is TRUE, evaluate `fbReadResult.ErrorID` and `fbReadResult.ErrorIdent`.

**Table 3. Methods Used in the Example**

| Value | Method | Type | Unit or range |
| --- | --- | --- | --- |
| Number of instances | `fbResult.getInstancesLength()` | `UDINT` |   |
| Instance by index | `fbResult.getInstances(j, result)` | `BOOL` | `TRUE` if the instance exists |
| Score | `fbInstance.GetScore()` | `REAL` | 0.0 to 1.0 |
| Class index | `fbInstance.GetClassIndex()` | `ULINT` | index into `class-names` |
| Bounding box | `fbInstance.getOrientedBoundingBox()` | `POINTER TO hdv_object_detection2d_OrientedBoundingBox` |   |
| x of the center | `pBox^.GetCenterX()` | `REAL` | pixel |
| y of the center | `pBox^.GetCenterY()` | `REAL` | pixel |

> [!NOTE]
> The score order is the behaviour of the current version. The schema allows the sort order to become configurable in a future version. `getBestIndex()` of `hdv_object_detection2d_Result` always returns the index of the instance with the highest score; it is 0 while the instances are sorted by score. Use it instead of the fixed index 0 if the program has to remain independent of the sort order.

### Reading All Instances

To evaluate all instances, loop over the array. Because the instances are sorted by score, the loop can stop at the first instance below a minimum score. The following fragment replaces step 3 of the implementation. It needs the additional declarations `i : UDINT;` and `rMinScore : REAL := 0.5;`.

```text
i := 0;
WHILE i < fbResult.getInstancesLength() DO
    IF fbResult.getInstances(j := i, result := ADR(fbInstance)) THEN
        rScore := fbInstance.GetScore();

        IF rScore < rMinScore THEN
            EXIT;   // sorted by score: no instance with a higher score follows
        END_IF

        pBox := fbInstance.getOrientedBoundingBox();
        IF pBox <> 0 THEN
            rX := pBox^.GetCenterX();
            rY := pBox^.GetCenterY();
            // Use or store the values of instance i here.
        END_IF
    END_IF
    i := i + 1;
END_WHILE
```

Do not use a `FOR` loop with the unsigned count (`FOR i := 0 TO count - 1`): if the count is 0, the upper limit wraps around to the largest `UDINT` value.

## Result Structure

The following tables describe the content of `object_detection.fbs`. The schema file is the authoritative definition.

**Table 4. Result (table, root type)**

| Field | Type | Description |
| --- | --- | --- |
| `instances` | `[Instance]` | Detected objects, sorted by score in descending order (highest score first). The array can contain objects of different classes; the PLC program has to select the required class (see [Class Names and Class Filters](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-class-names-and-filters)). |
| `best_index` | `ulong` | Index of the instance with the highest score in `instances`. Always 0 while the instances are sorted by score. |
| `best_score` | `float` | Score of the best instance, in the range 0.0 to 1.0. |
| `best_class_index` | `ulong` | Class index of the best instance. |
| `image_id` | `ulong` | Package identifier of the input image the result belongs to. In versions before October 2026 it was a 16-bit `short`. |
| `image_name` | `string` | Not used. |
| `requested_timestamp` | `ulong` | Time at which the application accepted the inference request that the PLC had set to **True**. |
| `timestamp` | `ulong` | Timestamp of the input image. Type and resolution depend on the timestamp source of the `lumiscan-cam-driver` application that delivered the image. |
| `status` | `ulong` | Not used. Use `status/code` to check the outcome (see [Status Code Values](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#status-code-values)). |

**Table 5. Instance (struct)**

| Field | Type | Description |
| --- | --- | --- |
| `oriented_bounding_box` | `OrientedBoundingBox` | Position, size and orientation of the detected object in the image. |
| `class_index` | `ulong` | Index of the detected class (see [Class Names and Class Filters](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-class-names-and-filters)). |
| `score` | `float` | Detection score, in the range 0.0 to 1.0. |

**Table 6. OrientedBoundingBox (struct)**

| Field | Type | Unit | Description |
| --- | --- | --- | --- |
| `center_x` | `float` | pixel | Horizontal position of the box center. |
| `center_y` | `float` | pixel | Vertical position of the box center. |
| `width` | `float` | pixel | Width of the box. |
| `height` | `float` | pixel | Height of the box. |
| `angle` | `float` | degree | Orientation of the box. The current implementation can return negative angles; do not assume that the value is normalized to the range 0.0 to 360.0. |

## Class Names and Class Filters {#lumiscan-class-names-and-filters}

With the default class mapping, `class_index` identifies a class by its position in the array of class names configured for the network. Use the following nodes to resolve and restrict classes:

> [!NOTE]
> If custom class-index remapping is configured, the published class index follows that mapping and may differ from the position in `setting/class-names`. Interpret the result using the configured mapping.

**Table 7. Class-Related Nodes**

| Address | Permission | Description |
| --- | --- | --- |
| `hdvisionsystems/lumiscan-ai-object-matching/<network name>/setting/class-names` | Read-Only | Class names of the network. With the default mapping, the position of a name in the array is its class index. |
| `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/class-filters` | Writable | Class names to include in the result. Leave the array empty to disable filtering. |

## Updating the Application

After an update of the application, download the schema again and re-import it into the PLC project, so that the PLC program uses the schema of the installed version.

# Inference on an Image Stream (Experimental) – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html

> [!IMPORTANT]
> Inference on an image stream is **experimental** (from v2.2.0). Its nodes may still change. Use it for evaluation, and verify it with your network and camera before relying on it.

The general concept of streams - producers and consumers, the result envelope and its lineage - is described in [Streams](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/streams.html). This page describes the stream of AI Object Matching. It works the same on the ctrlX Data Layer and on LumiScan Keryx.

On request, a module infers on one image per request: your program writes the image, or the Cam Driver captures one, and then asks for the inference. With an image stream, the Cam Driver publishes every image it takes on `control/publish/output/stream/frame`, with its size, its pixel format and its lineage (see the Cam Driver manual, *Streaming Images*). A module subscribes to the stream of one camera - its source - and infers on the images as they arrive, and publishes one result per image it infers. Each result carries the lineage of its image - which camera took it, in which run of its stream, as which image of the run - with the module’s own step added, see [The lineage of a result](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#lineage).

By default the module never falls behind the camera. When images arrive faster than the network infers, it infers on the newest image each time the previous inference is finished. The images in between are counted as skipped. How many images may wait is set with the queue size, see [The queue size](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#queue-size).

A module has one source. To infer on the images of two cameras, use two modules.

On both Data Layers every image reaches the module, in order. Images dropped on the way are counted in `statistics/lost-frames`: each image carries its number in its camera’s stream.

## Setting it up {#setting-up}

Every module has the stream’s nodes below, whether it has a source yet or not: no restart is needed to start using them. Either

- in the module configuration file, set the `stream` section, for example
  ```json
  "stream": {
      "enable": true,
      "address": "hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream"
  }
  ```

  See [Module Configuration File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html). A changed file applies at the next start or reset; or

- at run time, write the inputs of `control/subscribe` and call it, see [Subscribing at run time](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#subscribing).

The image stream is inferred with the module’s network, class filters and remapping. For another network or other settings, use another module.

### Kept in the settings {#kept-in-the-settings}

Stream mode and the source are settings of the module, in the `stream` section of its configuration file. Without that section, or without `enable` in it, the module is not in stream mode. Stream mode needs a source: a module configuration file with `enable` set to `true` and no `address` is not valid, and the module is not loaded. Neither is a module with a step in `stream.onEntry`: the section has the steps of every LumiScan application’s stream, and AI Object Matching has none yet.

`control/subscribe/execute` and `control/unsubscribe/execute` save `enable`, `address`, `tag`, `queueSize` and `keepLatestOutput` before they apply them. After a restart the module subscribes to the same source and, when `stream.enable` is `true`, starts its stream at once - nobody has to call the method again. Saving changes only those values: the rest of the file stays as it was, comments included.

## Addresses {#addresses}

All addresses below start with `hdvisionsystems/lumiscan-ai-object-matching/<network name>/`. The stream follows the layout of an execute method: `control/subscribe` owns its inputs and outputs.

**Table 1. Methods**

| Address | Meaning |
| --- | --- |
| `control/subscribe/execute` | Checks all of `control/subscribe/input/…` together, saves them in the module configuration file, then applies them: the stream follows on its next cycle, with no restart. See [Subscribing at run time](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#subscribing). |
| `control/unsubscribe/execute` | Drops the source named by `control/unsubscribe/input/address`, which ends stream mode too, and saves that. The tag stays. |

**Table 2. Inputs**

| Address | Type | At start | Meaning |
| --- | --- | --- | --- |
| `control/subscribe/input/enable` | bool | `stream.enable`, or `false` | `true` infers on the images of the source; `false` stops it. |
| `control/subscribe/input/address` | string | `stream.address`, or empty | The stream folder of the source, holding the camera’s `frame`, for example `hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream`. |
| `control/subscribe/input/tag` | string | `stream.tag`, or empty | The tag the module puts on its results: the module’s step in the lineage of each result carries it, so whoever reads a result downstream sees which module made it. Empty: the module id, the last segment of the module’s address - `example` for `hdvisionsystems/lumiscan-ai-object-matching/example`. A new tag marks the next result; the source stays subscribed. |
| `control/subscribe/input/queue-size` | uint32 | `stream.queueSize`, or 1 | How many images may wait for the inference, 1 to 64. See [The queue size](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#queue-size). |
| `control/subscribe/input/keep-latest-output` | bool | `stream.keepLatestOutput`, or `false` | `true` also puts each stream result on the regular output. See [Keeping the latest output](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#keep-latest-output). |
| `control/unsubscribe/input/address` | string | empty | The source `control/unsubscribe/execute` drops: the address on `setting/stream/address`. |

Writing an input changes nothing by itself: only `control/subscribe/execute` applies the inputs. They start as what the module applies, and a reset sets them to the file’s values again, so a client can change one input and call the method.

What the source must hold: read `hdvisionsystems/lumiscan-ai-object-matching/system/meta/subscription-schema` (string, read only), the same for every module. It is a JSON Schema (draft-04) of the `frame` node of the source, a `stream_image` with its `lineage` and its `image`, in its JSON form. The node is experimental.

**Table 3. Outputs**

| Address | Type | Meaning |
| --- | --- | --- |
| `control/subscribe/output/stream/active` | bool | `true` while the module infers on its image stream. |
| `control/subscribe/output/stream/result` | flatbuffer `stream_detections` | The result of each image inferred, in the same format as `output/result`, and its lineage. Subscribe to it to receive each one. See [The result of a streamed image](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#result). |
| `control/subscribe/output/stream/statistics/…` | uint64 | Counters of the current or last stream, see [Reading the statistics](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#statistics). |

**Table 4. Setting**

| Address | Type | Meaning |
| --- | --- | --- |
| `setting/stream/enable`, `setting/stream/address`, `setting/stream/tag`, `setting/stream/queue-size`, `setting/stream/keep-latest-output` | as the inputs | What the module applies now, as saved in the `stream` section. A client finds the source’s images here: LumiScan Cockpit uses `setting/stream/address` to draw the results on the live images. |
| `setting/settings-hash` | string | The hash of the settings the results are produced with, as 16 hex digits. See [The settings hash](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#settings-hash). |

Unless `keep-latest-output` is set, the results of the stream are published only on `control/subscribe/output/stream/result`, and `output/result`, `output/package-identifier`, `output/timestamp` and `output/image` keep the result of the last inference on request.

## The result of a streamed image {#result}

`control/subscribe/output/stream/result` holds a FlatBuffers table of type `stream_detections`, registered on the Data Layer as `types/hdvisionsystems/lumiscan-ai-object-matching/stream_detections`. Its compiled schema is installed as `bfbs/stream_detections.bfbs`. It has two fields: `result`, the same table as `output/result` (`object_detection`), and `lineage`, see [The lineage of a result](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#lineage).

Three fields of `result` describe the image:

| Field | Meaning |
| --- | --- |
| `image_id` | The number of the image since the stream started: 1, 2, 3 and so on. A gap means the images in between were skipped or failed. A 64-bit number (`ulong`). In versions before October 2026 it was a 16-bit `short`, and wrapped to negative numbers after 32767 images. |
| `timestamp` | When the image arrived, in microseconds since the epoch. |
| `requested_timestamp` | When the inference on the image started, in microseconds since the epoch. The difference to `timestamp` is how long the image waited. |

The class filters in `input/class-filters` and the class index remapping of the module apply as they do on request. A class the network description does not name is published with its index.

## Keeping the latest output {#keep-latest-output}

With `keep-latest-output` set, each stream result is also written to the module’s regular output, as an inference on request writes it: `output/result`, `output/timestamp`, and `output/package-identifier`, counted on from the inferences on request. A client that reads the regular output - a PLC program written for inference on request, for example - then sees the newest stream result there. The output image (`output/image`) is not drawn.

It costs a copy of every result, so it is off by default. The stream result on `control/subscribe/output/stream/result` is published first.

## The lineage of a result {#lineage}

`lineage.steps` lists where the result came from, origin first:

1. The camera’s step, as the Cam Driver made it: `producer` is the camera’s address, `hdvisionsystems/lumiscan-cam-driver/<camera name>`, `run` when its stream started, `sequence` the image’s number in that run, `external_id` the camera’s own frame number, `settings_hash` the camera’s settings, and `tag` the Cam Driver’s tag for the camera, by default its module id, for example `camera1`.
2. The module’s step: `producer` is the module’s address, `hdvisionsystems/lumiscan-ai-object-matching/<network name>`; `run` when the module’s stream started (each start is a new run); `sequence` the result’s number in the run; `settings_hash` the module’s settings, see [The settings hash](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#settings-hash); `elapsed_us` the microseconds from the image’s arrival to the result’s publication; `time_us` when it was published; `tag` the module’s tag, `control/subscribe/input/tag`, or the module id when it is empty.

So a result names its image exactly: the camera by `producer`, the image by `run` and `sequence`. A source changed while the stream runs keeps the module’s run: the results of the new camera are numbered on, and the `producer` of their first step names it. Two results of the same camera image, before and after a change of settings, differ in the `settings_hash` of the module’s step.

An image that carries no lineage - published by something other than the Cam Driver - gives a result with the module’s step alone. A value on the stream’s `frame` node that is not a `stream_image` is ignored, and logged once.

## The settings hash {#settings-hash}

`setting/settings-hash`, and the `settings_hash` of the module’s step in each result, identify what the results are produced with. The hash changes when one of these changes:

- the module configuration file’s settings, as the module was started with - except the `stream` section and `externalImagePath`, which choose the images rather than what is made of them;
- the network description, and the network file it names, by its content - a new network under the old file name is a new hash; with `simulateResult`, that file’s content instead;
- `input/class-filters`;
- the application’s version.

A reset reads the network description and the network file again. The first time a hash appears, the module writes the settings it stands for to `settings-history/<network name>-<hash>.json` in the application’s configuration folder, so a hash found in a result can be looked up later.

> [!NOTE]
> A result carries the hash only. A different hash says that something the results are produced with changed, not what changed. To find out, compare the two files in `settings-history` that the hashes name, by hand. The same holds for the camera’s hash in the camera’s step; see the Cam Driver manual, *Streaming Images*, for what it follows.

## Subscribing at run time {#subscribing}

A PLC or another client changes the source and stream mode while the application runs, with or without a running stream:

1. Write the inputs that change to `control/subscribe/input/…` - for another camera, `control/subscribe/input/address`; for another mark on the results, `control/subscribe/input/tag`; to start or stop, `control/subscribe/input/enable`.
2. Write `true` to `control/subscribe/execute`.

The method checks all inputs together, saves them in the `stream` section, then applies them: `setting/stream/…` shows them at once, and a running stream follows on its next cycle. Another address replaces the source - no unsubscribe is needed first -, and the images the old camera had waiting are counted as skipped. The tag, the queue size and `keep-latest-output` apply without subscribing again: a new tag marks the next result.

To drop the source, write its address to `control/unsubscribe/input/address` and `true` to `control/unsubscribe/execute`. That ends stream mode too, and saves `enable` as `false` with no address. The tag stays: it marks the module’s results, not the source. `control/subscribe/input/…` are set to that as well, so a later `control/subscribe/execute` that changes another input does not bring the source back. A `/` at either end of an address does not matter.

The methods answer:

| ctrlX Data Layer | LumiScan Keryx | When |
| --- | --- | --- |
| `DL_OK` | `Ok` | Done. |
| `DL_INVALID_VALUE` | `Rejected` | `control/subscribe`: `enable` is `true` and the address is empty, or the queue size is not 1 to 64. `control/unsubscribe`: the address is not the source, or there is no source. Nothing changed. |
| `DL_CREATION_FAILED` | `CreationFailed` | The module configuration file could not be written. Nothing changed: the stream stays as it was. The answer and the log say why. |
| `DL_RESOURCE_UNAVAILABLE` | `ResourceUnavailable` | The settings were edited on `system/settings` and the application waits for a restart. |

A reset (`control/reset`) reads the `stream` section of the module configuration file again and applies it, and sets the inputs to it: a running stream follows. So a file edited by hand applies with a reset.

When the subscription cannot be made, for example because the camera is not running yet, the module tries it again every 5 seconds while the stream runs, and logs the first failure.

## The queue size {#queue-size}

`control/subscribe/input/queue-size`, saved as `stream.queueSize`, sets how many images may wait for the inference: 1 to 64, 1 by default. An image that arrives when that many wait pushes out the oldest of them, which is counted in `statistics/skipped-frames`.

| Queue size | Effect |
| --- | --- |
| 1 (default) | The module infers on the newest image. It never falls behind; when the network is slower than the camera, it skips images. |
| More than 1 | Short bursts of images are all inferred, and fewer are skipped. When the network is slower than the camera for longer, the module falls behind: it infers on images up to that many inferences old. Each waiting image holds a whole image in memory. |

A changed queue size applies to the next image that arrives, with no restart of the stream.

## Running a stream {#running}

1. Make sure the Cam Driver’s stream runs: its `control/publish/output/stream/active` is `true`.
2. Subscribe to `control/subscribe/output/stream/result`.
3. Write `true` to `control/subscribe/input/enable` - and the source’s address, if the module has none yet - and `true` to `control/subscribe/execute`. Wait until `control/subscribe/output/stream/active` is `true`. The results arrive from then on.
4. To stop, write `false` to `control/subscribe/input/enable` and `true` to `control/subscribe/execute`.

Both are saved: the module starts again as it was left. The stream can be started before the Cam Driver’s stream; results arrive once images do.

## Inference on request while the stream runs {#while-streaming}

Both use the same network, so they never run at the same time:

- `control/inference/ready` is `false` while the stream runs.
- A request on `control/inference/request` is answered with status `100002` (`StreamActive`). It does **not** latch an error, and no reset is needed.
- `control/inference/execute` answers `DL_INVALID_OPERATION_MODE` (LumiScan Keryx: `InvalidOperationMode`). Do not retry it in a loop: stop the stream first.
- Reset, list, load, save, save-result-image and remove keep working. A reset loads the network anew and the stream goes on with it.

## Measured rates {#measured-rates}

Measured with one simulated camera (1440 x 1080, Mono8), a detection network with a 640 x 480 input on an NVIDIA GeForce RTX 2060 SUPER with the CUDA provider, under Linux. The figures show the order of magnitude only.

- **ctrlX Data Layer:** the Data Layer made no difference to the figures.
- **LumiScan Keryx on a PC (Windows):** on the same GPU, an inference took about 53 ms, and 18.4 of 30 images per second were inferred; the table shows the run under Linux.

| Camera | Received | Inferred | Skipped | Inference time |
| --- | --- | --- | --- | --- |
| 10 images/s | 10 /s | 10 /s | 0 | about 49 ms |
| 30 images/s | 30 /s | 19.4 /s | 10.7 /s | about 51 ms |

No image was lost or failed.

## Reading the statistics {#statistics}

The counters are under `control/subscribe/output/stream/statistics/`. They start from zero each time the stream starts, and are updated about four times a second while it runs.

| Counter | Meaning |
| --- | --- |
| `elapsed-ms` | Milliseconds since the stream started. Divide a counter by it for a rate. |
| `received-frames` | Images the subscription delivered. |
| `inferred-frames` | Images inferred, each with a result published on `control/subscribe/output/stream/result`. |
| `skipped-frames` | Images never inferred: pushed out of the queue by newer ones while inferences ran (see [The queue size](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#queue-size)), or waiting when the source was replaced. |
| `failed-frames` | Images that could not be decoded or inferred, for example because their pixels do not fill the size and pixel format they carry. The log says why. |
| `lost-frames` | Images dropped before they arrived, counted from gaps in the camera’s `sequence` within one run of its stream. A new run of the camera’s stream, or a new source, starts the count again. |
| `inference-us` | Microseconds all inferences took. Divide by `inferred-frames` for the mean. |

# LumiScan AI Object Matching Folder Structure

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/folder-structure.html

The settings of the application are files in its folder `lumiscan-ai-object-matching`. There are three files by default (You can add new files later):

- **ai-object-matching.json** contains the **application configuration**. It is used to specify how many neural network modules to load.\
  Currently, only one module is supported.
- **\<file name x>.json** (here as an example: **ai-object-matching-module.json**) This file contains the **module configuration**. It is used to configure how the network module evaluates image data.
- **models** contains two subfiles with detection parameters and neural network model.

The settings can also be read and changed on the Data Layer, see [Settings on the Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html).

## On ctrlX OS: Edit with Built-In Website

1. Log in to the ctrlX OS website.
2. Go to **Home**.
3. Select **Manage app data**.

![ctrlX OS Home screen with Manage app data highlighted](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/ctrlx-os-home-manage-app-data.png)

4. Click the three dots and select **File View**.

![Manage app data screen with the file actions menu open on File View](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/manage-app-data-file-view-menu.png)

5. Go to the relevant application folder.

![Active folder listing showing the datalayer and lumiscan-ai-object-matching folders](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/manage-app-data-active-folder.png)

6. The folder holds the three files described above.

![lumiscan-ai-object-matching folder listing showing models](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/lumiscan-folder-contents.png)

7. Left-clicking on the desired file name will automatically open the contents of the file in the right window.

![ai-object-matching.json opened in the file content viewer](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/json-file-content-view.png)

## On a PC: Settings Folder

On a PC the settings folder is:

- (Windows) `%APP_RESOURCE_DIR%\etc\lumiscan-ai-object-matching`, when the environment variable `APP_RESOURCE_DIR` is set;
- (Windows) otherwise `C:\ProgramData\HD Vision Systems GmbH\LumiScan Vision Suite\lumiscan-ai-obj-mat\etc\lumiscan-ai-object-matching`.

Open the files in any text editor, preferably one that checks JSON against the schemas in the `schemas` folder.

> [!NOTE]
> More detailed information about each file can be found in the corresponding sections later in this manual.

# Application Configuration File – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/application-configuration-file.html

This application can run multiple neural network object detection modules. Each of the modules requires a corresponding network module configuration (.json).

The general settings of the LumiScan AI Object Matching application are located in the **ai-object-matching.json** file. This file is used to specify how many neural network modules to load.

An example of application configuration JSON settings is shown below.

```json
{
    "meta": {
        "inputFileType": "LumiScan-CtrlX-AI-Object-Matching",
        "inputFileVersion": "1.0.0",
        "targetMachine": "MachineName",
        "programName": "AI-Object-Matching",
        "programVersion": "1.0.0"
    },
    "modules": [
        "ai-object-matching-module.json"
    ]
}
```

The description of all the parameters mentioned above can be found in the following [Table 1](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/application-configuration-file.html#table-1).

**Table 1. Application Configuration File: JSON Settings Description**

| Parameter | Description |
| --- | --- |
| meta | Place to set meta information: inputFileType and inputFileVersion are required. |
| inputFileType | In this case it must be: `LumiScan-CtrlX-AI-Object-Matching` |
| inputFileVersion | Currently, only version 1.0.0 is supported. |
| targetMachine | Optional. Enter the name of the machine in use here to indicate that the current application configuration is being used for this particular machine. |
| programName | Optional. This parameter indicates by which application the current file is used. In this case: `AI-Object-Matching` |
| programVersion | Optional. Currently used program version. |
| modules | Contains created model configuration files. |

> [!NOTE]
> After changing the parameters, the LumiScan AI Object Matching application should be restarted.

## Complete reference {#complete-reference}

Every setting of this file as its JSON Schema `lumiscan-ai-object-matching.schema.json` defines it, generated for version 2.1.0:

| Setting | Type | Default | Allowed values | Description |
| --- | --- | --- | --- | --- |
| `meta` | object |   |   | **Required.** Meta information about the application and current setting. This is mainly for information purpose and not used by the application itself |
| `meta/inputFileType` | string |   | not empty | **Required.** |
| `meta/inputFileVersion` | string |   | not empty | **Required.** |
| `meta/targetMachine` | string |   | not empty | **Required.** |
| `meta/programName` | string |   | not empty | **Required.** |
| `meta/programVersion` | string |   | not empty | **Required.** |
| `modules` | string[] |   | at least 1 entry | **Required.** List of module setting files that will be loaded by the application |
| `system` | object |   |   |   |
| `system/debug` | boolean | `false` |   | Enable debug mode for the application. |
| `system/logLevel` | string | `INFO` | `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`, `FATAL` | Log level for the application. |
| `system/storage` | object |   |   | Configuration for storage. The storage is used to store the images when save function is called. |
| `system/storage/type` | string |   | `Samba` | **Required.** The storage is a Samba storage. |
| `system/storage/parameters` | object |   |   | Parameters for the storage. The parameters are different for each storage type. |
| `system/storage/parameters/ip` | string |   | not empty | **Required.** The IP address of the Samba server. |
| `system/storage/parameters/remotePath` | string |   | not empty | **Required.** |
| `system/storage/parameters/domain` | string |   | not empty | **Required.** |
| `system/storage/parameters/username` | string |   | not empty | **Required.** |
| `system/storage/parameters/password` | string |   | not empty | **Required.** |

# Module Configuration File – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html

The module configuration settings are located in the **\<file name x>.json** file (in this example, **ai-object-matching-module.json**), and are used to configure how the network module evaluates image data.

An example of module configuration JSON settings is shown below.

```json
{
    "$schema": "schemas/lumiscan-ai-object-matching-module.schema.json",
    "meta":
    {
        "inputFileType": "LumiScan-CtrlX-AI-Object-Matching-Module",
        "inputFileVersion": "1.0.0",
        "targetMachine": "Example",
        "programName": "AI-Object-Matching",
        "programVersion": "1.0.0"
    },
    "name": "example",
    "neuralNetworkConfig": "models/model_description.json",
    "externalImagePath": "hdvisionsystems/lumiscan-cam-driver/camera1/output/image"
}
```

The description of the parameters mentioned above can be found in the following [Table 6](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html#table-6).

**Table 1. Module Configuration: JSON Settings Description**

| Parameter | Description |
| --- | --- |
| meta | Place to set meta information: inputFileType and inputFileVersion are required. |
| inputFileType | In this case it must be: `LumiScan-CtrlX-AI-Object-Matching-Module` |
| inputFileVersion | Currently, only version 1.0.0 is supported. |
| targetMachine | Optional. Enter the name of the machine in use here to indicate that the current application configuration is being used for this particular machine. |
| programName | Optional. This parameter indicates by which application the current file is used. In this case: `AI-Object-Matching` |
| programVersion | Optional. Currently used program version. |
| name | A unique identifier for the neural network, which is also used as the network name in the datalayer address: `hdvisionsystems/lumiscan-ai-object-matching/<name>/...`. Only lower-case letters, digits, `-` and `_` are allowed. A module whose name has an upper-case letter or any other character is not loaded; the log says which name to write instead (`Network1` → `network1`). |
| neuralNetworkConfig | The file path to the neural network model parameters. These parameters are generated by the LumiScan Training Framework. |
| externalImagePath | Path to datalayer address that contains image. given image address must contains **data, height, width, pixel-type**.<br> <br> If not specified, the network module will create an additional path under `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input` to allow users to upload raw images to the datalayer without the need for an additional application. |
| stream | **Experimental** (from v2.2.0). Optional. Inference on an image stream, from one source: the settings below. Without this section, or without `enable` in it, the module is not in stream mode. `control/subscribe/execute` and `control/unsubscribe/execute` save `enable`, `address`, `tag`, `queueSize` and `keepLatestOutput` here before they apply them; the rest of the file stays as it was, comments included. The section has the same keys in every LumiScan application that subscribes to a stream. See [Inference on an Image Stream](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html). |
| stream.enable | `true`: the module starts in stream mode, and infers on the images of its source from its start. `false` by default. Needs `stream.address`: a module with `enable` set to `true` and no address is not loaded. |
| stream.address | The image stream to infer on: its folder, for example `hdvisionsystems/lumiscan-cam-driver/camera1/control/publish/output/stream`. The module subscribes to the `frame` node of the folder, whose images carry their size, pixel format and lineage; the lineage of every result names the camera. |
| stream.tag | The tag on the module’s results: the module’s step in the lineage of each stream result carries it, so whoever reads a result downstream sees which module made it. Optional. Left out or empty: the module id, the last segment of the module’s address - `example` for `hdvisionsystems/lumiscan-ai-object-matching/example`. |
| stream.queueSize | How many images may wait for the inference: 1 to 64, 1 by default. 1 infers on the newest image; more keeps more images when the network falls behind. See [The queue size](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#queue-size). |
| stream.keepLatestOutput | `true`: each stream result also goes to the regular output (`output/result`, the timestamp and `output/package-identifier`). `false` by default. See [Keeping the latest output](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#keep-latest-output). |
| stream.onEntry | The steps applied to each image as it arrives, before the inference, each an object with a `kind`. AI Object Matching has none yet: leave it out, or empty (`[]`). A step keeps the module from loading, because it would infer without the step its file asks for; the log names it. |
| stream.onExit | Reserved for steps applied to each result. Not read yet: the application logs a warning, and the steps are not applied. |

> [!NOTE]
> After changing the parameters, the LumiScan AI Object Matching application should be restarted. A reset is enough for the `stream` section.
>
> The stream layouts of earlier test builds are no longer read: the top-level `subscriptions` and `streamQueueSize`, and `stream.subscriptions` (a map of several sources). A module has one source now: move its address into `stream.address`, and `streamQueueSize` into `stream.queueSize`. The application logs a warning while a module file still has one of them.

## Automatic Copying of External Image

To automatically copy an external image from another application, the application must provide an address containing the following parameters:

**Table 2. Parameters of External Image**

| Address | Type |
| --- | --- |
| `<address>/data` | array of uint8_t |
| `<address>/height` | uint32_t |
| `<address>/width` | uint32_t |
| `<address>/pixel-type` | string, possible values: Mono8, RGB8 |

**\<address>** is the image field specified in `externalImagePath` in the module configuration file.

> [!NOTE]
> This interface is designed and tested for use with the LumiScan Cam Driver. Compatibility with image data provided by other applications cannot be guaranteed.

## Complete reference {#complete-reference}

Every setting of this file as its JSON Schema `lumiscan-ai-object-matching-module.schema.json` defines it, generated for version 2.1.0:

| Setting | Type | Default | Allowed values | Description |
| --- | --- | --- | --- | --- |
| `meta` | object |   |   | **Required.** |
| `meta/inputFileType` | string |   | not empty | **Required.** |
| `meta/inputFileVersion` | string |   | not empty | **Required.** |
| `meta/targetMachine` | string |   | not empty | **Required.** |
| `meta/programName` | string |   | not empty | **Required.** |
| `meta/programVersion` | string |   | not empty | **Required.** |
| `name` | string |   | pattern `^[a-zA-Z0-9_-]+$` | **Required.** only [a-Z], [0-9], '-' and '\_' are allowed. Unique ID of the camera configuration. This is used to identify different camera on datalayer. |
| `neuralNetworkConfig` | string |   | not empty | **Required.** Path to network description file. |
| `externalImagePath` | string |   | not empty | If specified, the image will be loaded from the specified path instead of the ai-object-matching app managed input image. |
| `cropImage` | object |   |   | If specified, input image will be cropped and then used for inference. Note that the detected pixel position is related to original image. Use it if you want to accelerate the inference by reducing the image size. |
| `cropImage/startX` | number |   | ≥ 0 | **Required.** |
| `cropImage/startY` | number |   | ≥ 0 | **Required.** |
| `cropImage/width` | number |   |   | **Required.** |
| `cropImage/height` | number |   |   | **Required.** |
| `simulateResult` | string |   | not empty | If specified, the result will be simulated. This is useful for testing the application without the actual neural network. |
| `classIndexRemapping` | object |   |   | If specified, the detected object's index will be remapped to other index. Ke |
| `classIndexRemapping/<name>` | integer |   |   |   |

## JSON Schemas {#json-schemas}

The application keeps the JSON Schemas of its settings files in its `schemas` folder; those copies always match the installed version. The schemas of LumiScan AI Object Matching 2.1.0 can also be downloaded: [JSON Schemas of LumiScan AI Object Matching 2.1.0](https://docs.hdvisionsystems.com/schemas/lumiscan-ai-object-matching/2.1.0/index.html).

To have an editor such as Visual Studio Code check a copy of a settings file that you edit on your PC, point its `$schema` entry at the published schema:

```json
{
  "$schema": "https://docs.hdvisionsystems.com/schemas/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching-module.schema.json",
  ...
}
```

Leave the `$schema` entry of the files on the device as the application wrote it.

# Models Folder – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html

![Models folder containing model.json and model.onnx](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/models-folder-contents.png)

The Models folder contains two files generated by the LumiScan Training Framework:

- **\<file name>.json** (for example, **model.json**) contains the detection parameters and references the neural network model.
- **\<file name>.onnx** (for example, **model.onnx**) contains the trained neural network model.

**On ctrlX OS:** The model files must be uploaded using WebDAV.[^1]

**On a PC:** Copy the model files into the `models` folder of the settings folder, see [Uploading Data](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html#upload-on-a-pc).

## Model.json File {#model-json-file}

The `model.json` file contains the parameters required to configure the neural network model used for detection.

The following example shows the JSON structure:

```json
{
    "classList": {
        "0": "nockeIO",
        "1": "nockeVerbogen"
    },
    "thresholdList": {
        "nockeIO": 0.02,
        "nockeVerbogen": 0.02
    },
    "resizedImageShape": {
        "height": 431,
        "width": 571
    },
    "normalizationMean": [
        123.67500305175781,
        116.27999877929688,
        103.52999877929688
    ],
    "normalizationStd": [
        58.39500045776367,
        57.119998931884766,
        57.375
    ],
    "spacialNMSRadius": 3,
    "pathToNetwork": "model.onnx",
    "provider": "CPU",
    "deviceIndex": 0
}
```

All parameters are generated by the LumiScan Training Framework.

Only `thresholdList`, `pathToNetwork`, `provider`, and `deviceIndex` should be modified manually. These parameters are described in [Table 8](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html#table-8).

**Table 1. Model.json File: JSON Settings Description**

| Parameter | Description |
| --- | --- |
| `thresholdList` | Defines the minimum confidence threshold for each detection class.<br><br>- Values must be between `0.0` and `1.0`.<br>- A higher value requires a higher detection confidence.<br>- A value of `1.0` represents 100% confidence. |
| `pathToNetwork` | Specifies the path to the trained neural network model used for object detection.<br><br>The path must be relative to the location of the current JSON file. |
| `provider` | Specifies the inference provider. Possible values:<br><br>- On ctrlX OS: `CPU` and `HAILO`. If `HAILO` is used, see [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html).<br>- On a PC: `CPU` and `CUDA` (an NVIDIA GPU). |
| `deviceIndex` | Specifies the device index. Currently, this value must be `0`. |

> [!NOTE]
> Restart the LumiScan AI Object Matching application after changing any of these parameters.

## Complete reference {#complete-reference}

Every setting of this file as its JSON Schema `network-description.schema.json` defines it, generated for version 2.1.0:

| Setting | Type | Default | Allowed values | Description |
| --- | --- | --- | --- | --- |
| `meta` | object |   |   | **Required.** |
| `meta/inputFileType` | string |   | not empty | **Required.** |
| `meta/inputFileVersion` | string |   | not empty | **Required.** |
| `meta/targetMachine` | string |   | not empty |   |
| `meta/programName` | string |   | not empty |   |
| `meta/programVersion` | string |   | not empty |   |
| `networkType` | string |   |   | **Required.** Current only support 'OrientedObjectDetection' |
| `classList` | object |   |   | [Deprecated] Please rename to 'classes' |
| `thresholdList` | object |   |   | [Deprecated] Please rename to 'thresholds' |
| `classes` | object |   |   | **Required.** Key: Index as string type, Value: Class name |
| `thresholds` | object |   |   | **Required.** Key: Class Name, Value: <0-1 float> threshold value between 0 and 1 |
| `resizedImageShape` | object |   |   | **Required.** |
| `resizedImageShape/height` | number |   |   | **Required.** |
| `resizedImageShape/width` | number |   |   | **Required.** |
| `normalizationMean` | number[] |   | exactly 3 entries | **Required.** |
| `normalizationStd` | number[] |   | exactly 3 entries | **Required.** |
| `provider` | string |   | `CPU`, `HAILO` | **Required.** Type of provider, currently only CPU is supported. |
| `deviceIndex` | number |   |   | **Required.** Index of the device to use. If the provider is CPU, this value is ignored. If the provider is GPU, this value is the index of the GPU to use. |
| `spacialNMSRadius` | number |   |   | **Required.** |
| `pathToNetwork` | string |   | not empty | **Required.** Path to the network file. The network file must be in ONNX format and the path is relative to the description file. |

[^1]: For information about uploading data with WebDAV, see [Uploading Data with WebDAV](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html#upload-using-webdav).

# Adding New Module – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/adding-new-module.html

A newly trained model is provided together with its corresponding description file. These files belong together and should always be uploaded and handled as one unit.

The steps below cover ctrlX OS and a PC; the screenshots show the ctrlX OS web interface.

1. Upload the new model and its description file as described in [Uploading Data](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html).

   For details about the model files, see [Models Folder](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html).

2. Open the **ai-object-matching.json** application settings:
   - **On ctrlX OS:** left-click the **ai-object-matching.json** file name. The JSON settings open on the right.
   - **On a PC:** open the file in the settings folder with a text editor.

   ![ai-object-matching.json settings with the new module file listed](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/application-json-with-new-module.png)

3. Enter the name of the uploaded module description file in the `modules` field.

   ![modules field in ai-object-matching.json highlighting the new module file name](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/new-module-json-field-detail.png)
   > [!NOTE]
   > Currently, only one module is supported at a time. Remove the previous module from the `modules` field when switching to a new one.

4. Open the uploaded module description file:
   - **On ctrlX OS:** left-click its file name. The module settings open on the right.
   - **On a PC:** open the file with a text editor.

   ![New module file listing with its JSON content open on the right](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/new-module-file-listing-json-panel.png)

5. Verify that the `neuralNetworkConfig` parameter points to the newly uploaded model description JSON file.

   ![neuralNetworkConfig field highlighted in the new module’s JSON content](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/new-module-neural-network-config-detail.png)

6. Verify the remaining settings as described in [Module Configuration File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html).

7. Save the changes.

> [!NOTE]
> Restart the LumiScan AI Object Matching application after changing the configuration.

A module can also be created and activated through the Data Layer, see [Adding and removing modules](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#adding-and-removing-modules).

# Settings on the Data Layer – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html

## Data Layer {#data-layer}

LumiScan AI Object Matching offers its functions on the Data Layer under `hdvisionsystems/lumiscan-ai-object-matching`.

On ctrlX OS (ctrlX CORE or ctrlX IPC), LumiScan AI Object Matching uses the [ctrlX Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/ctrlx-data-layer.html).

On a PC with Windows 11 or Ubuntu 24.04, LumiScan AI Object Matching uses [LumiScan Keryx](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/keryx.html).

The node addresses, types and call patterns are the same on both. LumiScan AI Object Matching offers:

- [ready/request handshakes](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/handshake.html)
- [execute methods](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/execute.html)
- a [stream](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/streams.html) subscription to one source (experimental)

How addresses are built, how to call functions, the status codes and the tools are described in [LumiScan Keryx](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/index.html).

LumiScan AI Object Matching shows the settings it was started with on the Data Layer, on `hdvisionsystems/lumiscan-ai-object-matching/system/settings`. You can read them there without access to the configuration folder, and change them there: the application checks a change, saves it to its settings files, and asks to be restarted. `hdvisionsystems/lumiscan-ai-object-matching/system/control/restart/execute` restarts it.

This works the same on the ctrlX Data Layer and on LumiScan Keryx. The `system` node of the LumiScan applications is described in general in [Settings, Modules and Restart](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/settings-and-modules.html). Where a method answers differently on the two Data Layers, both names are given below as ctrlX Data Layer / LumiScan Keryx.

## What the node shows {#what-the-node-shows}

`settings` holds one JSON document, as text:

- the application settings file, `lumiscan-ai-object-matching.json`, as it was read;
- in its `modules` list, each module configuration file’s content instead of the file’s name. `"$file"` names the file each module came from;
- a module file that could not be read appears as `{"$file": ..., "$error": ...}`, where `"$error"` says why. That is also why the module did not start;
- every value named `password` shows as `"***"`.

For example:

```json
{
  "$schema": "schemas/lumiscan-ai-object-matching.schema.json",
  "meta": { ... },
  "modules": [
    {
      "$file": "lumiscan-ai-object-matching-example-module.json",
      "$schema": "schemas/lumiscan-ai-object-matching-module.schema.json",
      "name": "ood",
      "neuralNetworkConfig": "models/ood-description.json",
      ...
    }
  ],
  "system": { ... }
}
```

Comments in the files are not shown.

The node shows the files as the application read them at start. A module’s reset reads its module file again (see [General Concept of Handshake](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html)), and the node then shows that file as it was read. A reset applies only part of the file: the network and the `stream` section. What `control/subscribe/execute` and `control/unsubscribe/execute` save - the `stream` section - shows here at once.

## Changing the settings {#changing-the-settings}

Read `settings`, change what you need, and write the whole document back. These rules apply:

- Each module keeps its `"$file"`. It must name one of the application’s module files: a module cannot be added here. To add one, see [Adding and removing modules](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#adding-and-removing-modules).
- A module left out of `modules` is no longer listed in `lumiscan-ai-object-matching.json`. Its file stays where it is.
- `"***"` for a password keeps the password the file has. Any other value replaces it.
- A file whose content does not change is not written. A changed file is changed only where its content changed: comments and layout stay. A new key or list element goes after the last one there; a removed key goes with its line, a comment on that line included. Only an object that both gained and lost keys, or a list that lost an element, is written anew, without the comments inside it. The previous file is kept beside it as `<file>.bak`.
- A key such as `"_comment"` is content like any other, so it is shown and kept.

The application checks the change before anything is written: the application configuration file and each changed module configuration file have to pass the checks the application makes at start. For example, the network description has to exist. An edit the check refuses, or that is not valid JSON, is answered with `DL_INVALID_VALUE` / `Rejected`. The reason is in the log, and nothing is written.

An edit that writes nothing answers `DL_OK` / `Ok` and changes nothing.

### Restart required {#restart-required}

A change that was saved takes effect only with the next start. Until then the application runs with settings that are no longer those of its files, so it stops working:

- `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required` is `true`;
- `hdvisionsystems/lumiscan-ai-object-matching/system/status-text` says which files changed;
- every module reports status `-5` (`restartRequired`). It answers no request, and its `ready` signals are `false`;
- every `execute` method of a module answers `DL_RESOURCE_UNAVAILABLE` / `ResourceUnavailable`. That includes `reset`, which would otherwise load part of the new settings;
- a running image stream stops.

Only a restart ends this. A reset does not. The log has the message at error level.

Further changes are possible in the meantime. After a change, `settings` shows the files as saved, which is what the next start loads, not what the application runs with.

If a file could not be saved, the write answers `DL_CREATION_FAILED` / `CreationFailed`. Files changed before it stay changed, so the application requires a restart all the same. Correct the files, then restart.

## Adding and removing modules {#adding-and-removing-modules}

Three methods of `hdvisionsystems/lumiscan-ai-object-matching/system` handle modules, each under `control/<method>/` as `control/restart` is. Each takes the module’s id, the name of its file without `.json`, in `control/<method>/input/module-id`:

| Method | What it does |
| --- | --- |
| `create-module` | Writes the module file `<module-id>.json` with the JSON in `control/create-module/input/content`. The application checks it as it checks a changed module file. The module is not active yet, so nothing has to be restarted. An existing file is not overwritten: change it through `settings`. |
| `activate-module` | Adds `<module-id>.json`, which has to exist, to the `modules` list, after the application has checked the module. It runs from the next start, so the application then requires a restart (see [Restart required](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#restart-required)). |
| `deactivate-module` | Takes `<module-id>.json` off the `modules` list. The file stays, so `activate-module` can list it again. It stops with the next start, so the application then requires a restart. |

> [!IMPORTANT]
> AI Object Matching runs one module. `activate-module` is refused while a module is active: deactivate it first.

Each method reports on `control/<method>/output/status-code` and `control/<method>/output/status-text`:

| Code | Meaning |
| --- | --- |
| 1 | Done. |
| -2 | The input or the module was refused: `status-text` says why. Nothing was written. |
| -6 | A file could not be written. |
| -9 | `activate-module`: there is no such file. `deactivate-module`: the module is not active. |
| -10 | `create-module`: the file exists. `activate-module`: the module is active already. |

The method answers `DL_OK` / `Ok` when it is done, `DL_INVALID_VALUE` / `Rejected` when it refused, and `DL_CREATION_FAILED` / `CreationFailed` when a file could not be written. For example, to add the module `ood2`:

1. Write the module’s JSON to `hdvisionsystems/lumiscan-ai-object-matching/system/control/create-module/input/content` and `ood2` to `control/create-module/input/module-id`, then call `control/create-module/execute`.
2. Write `ood2` to `control/activate-module/input/module-id`, then call `control/activate-module/execute`.
3. Call `hdvisionsystems/lumiscan-ai-object-matching/system/control/restart/execute`.

## Restarting the application {#restarting}

`hdvisionsystems/lumiscan-ai-object-matching/system/control/restart/execute` ends the application and starts it again. It reads its settings files anew, which also applies changes made to the files by hand, and ends `restart-required`.

- **On ctrlX OS:** the application ends, and ctrlX OS starts it again after about five seconds.
- **On a PC:** the application starts itself again once it has released everything it held. It uses the same program, arguments and working folder. It is a new process, in the same console, and its output goes where the first one’s went.

The method answers `DL_OK` / `Ok` as soon as the restart has begun. The application’s nodes then disappear until it is up again: wait until `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required` can be read again.

> [!NOTE]
> A restart is not tied to a change. It also restarts an application whose settings did not change.

## Starting from an Example (PC) {#generate-settings}

On a PC, the AI Object Matching writes an example of its settings into a folder of your choice, and ends without starting:

**Example (Windows)**

```shell
lumiscan-ai-obj-mat --generate-settings C:\work\ai-object-matching-settings
```

Without a folder (`-g` alone) it writes into the current folder. The example is the application settings file `lumiscan-ai-object-matching.json`, its example module file `lumiscan-ai-object-matching-example-module.json`, the network description that module names, `models\example-network-description.json`, and the five schemas in `schemas\` that an editor uses to check them. The model file the description names, `example.onnx`, is not part of it: that one is yours to supply. Existing files are never overwritten: when one of them is in the folder already, nothing is written and the AI Object Matching says which.

The AI Object Matching then names the folder it reads its settings from, by default (Windows) `C:\ProgramData\HD Vision Systems GmbH\LumiScan Vision Suite\lumiscan-ai-obj-mat\etc\lumiscan-ai-object-matching`. Edit the example, copy it there, and start the AI Object Matching.

# Uploading Data – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html

How the settings files and the neural network models get into the application folder depends on the platform.

## On ctrlX OS

> [!NOTE]
> Most data can be uploaded directly through the ctrlX Web UI. Use WebDAV only when direct upload is not available or when you need to access the application folder structure.

![ctrlx web ui upload](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/ctrlx-web-ui-upload.png)

Most files can be uploaded using the upload service in the ctrlX Web UI. WebDAV is available as an alternative when files need to be uploaded directly to the application data folders.

### Upload Using the ctrlX Web UI

Use the upload service in the ctrlX Web UI to upload the required neural network model and other supported files.

![ctrlx web ui upload](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/ctrlx-web-ui-upload.png)

### Upload Using WebDAV {#upload-using-webdav}

To upload files using WebDAV:

1. Go to the WinSCP website.

![WinSCP download website](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/winscp-website.png)

2. Download and install the latest version of WinSCP.
3. Start WinSCP.
4. On the login page, select **WebDAV** as the **File protocol**.
5. Select **TLS/SSL** for **Encryption**.
6. Enter your **user name** and **password**.

![WinSCP WebDAV login dialog](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/winscp-login.png)

7. Navigate to `/solutions/webdav/appdata`.

![WinSCP location profiles dialog navigated to the WebDAV appdata path](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/winscp-location-profiles.png)

8. The application folder structure is displayed in the right panel.

![WinSCP dual-pane view showing the WebDAV appdata folder structure](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/winscp-folder-view.png)

Drag and drop the required data or configuration files into the corresponding application folder.

## On a PC {#upload-on-a-pc}

On a PC the settings files and the neural network models are ordinary files. Copy them into the folder the application reads its settings from:

- (Windows) `%APP_RESOURCE_DIR%\etc\lumiscan-ai-object-matching`, when the environment variable `APP_RESOURCE_DIR` is set;
- (Windows) otherwise `C:\ProgramData\HD Vision Systems GmbH\LumiScan Vision Suite\lumiscan-ai-obj-mat\etc\lumiscan-ai-object-matching`.

Put the neural network model and its description file into the `models` folder there. Restart the application after changing a file.

# FAQ – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/faq.html

Short answers to common questions about LumiScan AI Object Matching. Each entry links to the page with the details. Problems and error messages are described in [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html).

> [!TIP]
> Each entry has a fixed anchor (for example `#faq-run-inference`). To find an error message, search the [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html) page for it.

## Getting started

### What does LumiScan AI Object Matching do, and what does it need? {#faq-what-it-does}

- **Answer:** It uses a trained neural network to detect objects in images and reports their position and orientation in pixel coordinates. It runs on ctrlX OS (ctrlX CORE or ctrlX IPC) with the ctrlX Data Layer, or on a PC (Windows 11 or Ubuntu 24.04) with LumiScan Keryx, and is controlled through the Data Layer; the node addresses are the same on both. It does not acquire images itself; it reads images published by another application, for example the LumiScan Cam Driver (`hdvisionsystems/lumiscan-cam-driver/camera1/output/image`). The model is created with the LumiScan Annotation Framework and LumiScan Training Framework.
- **More:** [Description](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/description.html), [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html)

### Can I use TPU acceleration, and what are the requirements? {#faq-tpu-acceleration}

- **Applies to:** ctrlX OS
- **Answer:** TPU acceleration is optional. If it is used, the ctrlX CORE must include a TPU, and the following versions apply: ctrlX CORE X3 with app version v1.2 or later, ctrlX CORE X7 with app version v1.1 or later, and App Hardware Support exactly 2.6.2. With a HAILO TPU, downgrading Hardware Support to v2.6.2 is mandatory. In the model JSON file, set `provider` to `HAILO`. On a PC, use an NVIDIA GPU with `provider` set to `CUDA` instead.
- **More:** [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html), [Release Notes: v1.2.0](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/release-notes/v1-2-0.html)

## Configuration

### Where are the configuration files? {#faq-config-files}

- **Answer:** In the folder `lumiscan-ai-object-matching`. On ctrlX OS, open **Home** > **Manage app data** in the ctrlX OS web interface, select **File View** and open that folder. On a PC it is the settings folder of the application: on Windows `%APP_RESOURCE_DIR%\etc\lumiscan-ai-object-matching`, or without `APP_RESOURCE_DIR` `C:\ProgramData\HD Vision Systems GmbH\LumiScan Vision Suite\lumiscan-ai-obj-mat\etc\lumiscan-ai-object-matching`. Restart the application after changing any file. The settings can also be read and changed on the Data Layer at `hdvisionsystems/lumiscan-ai-object-matching/system/settings`. The folder contains:
  - `ai-object-matching.json`: application configuration (`inputFileType` `LumiScan-CtrlX-AI-Object-Matching`); `modules` lists the module configuration files.
  - A module configuration file (for example `ai-object-matching-module.json`, `inputFileType` `LumiScan-CtrlX-AI-Object-Matching-Module`). Its `name` is the network name used in the Data Layer address, and `neuralNetworkConfig` points to the model’s JSON file.
  - `models`: the model files.
- **More:** [LumiScan AI Object Matching Folder Structure](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/folder-structure.html), [Settings on the Data Layer](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html)

### What is in the models folder, and which model parameters may I change? {#faq-model-files}

- **Answer:** The models folder holds two files generated by the LumiScan Training Framework, which belong together: a `.onnx` file with the trained network and a `.json` file with the detection parameters. Only these JSON parameters should be changed manually; restart the application afterwards:
  - `thresholdList`: minimum confidence per class, between `0.0` and `1.0` (`1.0` = 100 %).
  - `pathToNetwork`: path to the `.onnx` file, relative to the JSON file.
  - `provider`: `CPU` or `HAILO` on ctrlX OS, `CPU` or `CUDA` (an NVIDIA GPU) on a PC.
  - `deviceIndex`: currently must be `0`.
- **More:** [Models Folder](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/models-folder.html)

### How do I use a newly trained model? {#faq-new-model}

- **Answer:** Upload the new model together with its description file (they are always handled as one unit); on a PC, copy both into the `models` folder of the settings folder. Make sure the module configuration file is listed in `modules` of `ai-object-matching.json` and that its `neuralNetworkConfig` points to the new model’s JSON file. Check the remaining module settings, save, and restart the application.
- **More:** [Adding New Module](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/adding-new-module.html), [Uploading Data](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/uploading-data.html)

### Which image formats and image sources are supported? {#faq-image-source}

- **Answer:** An external image address configured in the module must provide `<address>/data` (array of `uint8_t`), `<address>/height` and `<address>/width` (`uint32_t`) and `<address>/pixel-type` (`Mono8` or `RGB8`). This interface is designed and tested for the LumiScan Cam Driver; other sources are not guaranteed to work. If no external image address is configured, the module creates `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/image`, where you write the raw image and set its width, height and pixel format. RGB data is laid out as height, width, channel. From v2.2.0 a module can also infer on the image stream of the Cam Driver (experimental).
- **More:** [Module Configuration File](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/module-configuration-file.html), [Inference on an Image Stream](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html)

## Running inference

### How do I start an inference and get the result? {#faq-run-inference}

- **Answer:** Use the ready/request handshake on `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/`: wait until `ready` is `True`, note the value of `<network name>/output/package-identifier`, set `request` to `True`, wait until `ready` is `False` (the inference is already running), set `request` to `False`, and wait until `ready` is `True` again. If `output/package-identifier` has grown by one, the inference succeeded: read the result at `<network name>/output/result`. `status/code` is `3` at the end of every inference requested through the handshake, whether it succeeded or not. From v2.1.0, you can instead write once to `<network name>/control/inference/execute`, which blocks until the operation is done; `status/code` then keeps the outcome: `0`, `100001` or `-1`. Only one process may write `request` at a time.
- **More:** [Initializing the Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html), [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference), [Release Notes: v2.1.0](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/release-notes/v2-1-0.html)

### How do I restrict the result to certain classes? {#faq-class-filters}

- **Answer:** Write the class names to include into `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/class-filters`; leave the array empty to disable filtering. With the default class mapping, `class_index` in the result is the position of the class in the network’s class-names array. The `instances` array can contain objects of different classes, so the PLC program has to select the required class.
- **More:** [Class Names and Class Filters](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-class-names-and-filters)

## Reading results in ctrlX PLC Engineering

### Where do I get the FlatBuffers schema for the result? {#faq-flatbuffers-schema}

- **Applies to:** ctrlX OS
- **Answer:** The result node `<network name>/output/result` is a FlatBuffers message described by `object_detection.fbs`. The application copies this file on each start to `lumiscan-ai-object-matching/schemas/flatbuffers`; download it in **Manage app data** with the download icon in the **Actions** column, or over WebDAV. Do not edit files in `schemas`, because the folder is recreated on every start. After an application update, download and re-import the schema.
- **More:** [Accessing Result Data with FlatBuffers](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html)

### How do I import the schema into my PLC project? {#faq-import-schema}

- **Answer:** You need ctrlX PLC Engineering 2.6 or later and the libraries `CXA_Datalayer` and `CXA_Flatbuffers` in the **Library Manager**. In the device tree, right-click **Application**, select **Import from FlatBuffers file…** and choose `object_detection.fbs`. If **Import from FlatBuffers file…** is not in the context menu, the ctrlX Data Layer library is missing: install `CXA_Datalayer` via **Tools** > **Library Repository…** and repeat the import.
- **More:** [Install the ctrlX Data Layer Library](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-install-datalayer-library)

### How do I read the position and score of the best detection? {#faq-read-position-score}

- **Answer:** After the inference has finished and succeeded (handshake: `output/package-identifier` has grown by one; `inference/execute`: `status/code` is `0`), read the result node with `DL_ReadNodeValue` and pass `GetData` and `GetSize` of the `DL_NodeValue` to `getRootAsResult` of `hdv_object_detection2d_Result`. Instances are sorted by score (highest first); `getBestIndex()` always returns the best one. From the instance, `GetScore()` gives the score (0.0 to 1.0) and `getOrientedBoundingBox()` gives the box with `GetCenterX()`/`GetCenterY()` in pixels; the angle is in degrees and can be negative. The function blocks read the buffer in place, so use or copy the values before the next read.
- **More:** [Example: Reading Position and Score of an Instance](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-example-instance)

## Troubleshooting {#troubleshooting}

Problems and error messages are described step by step in [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html):

- [inference/ready stays False, or the inference ends with status 100001](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-no-new-external-image)
- [status/code is 3 after every inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-status-3-after-inference)
- [A write to execute is refused](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-execute-refused)
- [An inference request is answered with status 100002 (StreamActive)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-stream-active)
- [Every module reports status -5 (restartRequired)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-restart-required)
- [The schemas folder does not exist](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-schemas-folder-missing)
- [Import from FlatBuffers file… is not in the context menu](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#ts-import-command-missing)
- [The message "Identifier 'VectorT' not defined" appears after the import](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-vectort-not-defined)

## Licensing and support

### Which licence do I need, and what happens without one? {#faq-licence}

- **Answer:** The ctrlX licence for LumiScan AI Object Matching is `SWL-W-XCx-OBJxMATCHxHDVSx-V2NN`. Bought in the ctrlX Store for ctrlX OS, it is a ctrlX licence and needs no LumiScan Key. On ctrlX OS, the application shuts down shortly after starting if the licence is missing; the reason is in the system log. A missing licence is reported as status `-4` `"MissingLicense"`.
- **More:** [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html#licence), [Licensing FAQ](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/faq.html)

### Where can I find more answers or get support? {#faq-support}

- **Answer:** See the site-wide [LumiScan FAQ](https://docs.hdvisionsystems.com/en/faq.html). Contact HD Vision Systems customer success at [customer.success@hdvisionsystems.com](mailto:customer.success@hdvisionsystems.com) or +49 6221 6721905; for licences, [licenses@hdvisionsystems.com](mailto:licenses@hdvisionsystems.com).
- **More:** [Licensing FAQ](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/faq.html)

# Troubleshooting – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html

This page starts with the general checks that apply to every problem. The problems follow, grouped into running inference and reading results in ctrlX PLC Engineering. Do the general checks first, then find your symptom.

## General checks {#general-checks}

- Read the status of the module: `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/status/code` and `hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/status/text`. The meaning of each code is listed in [Status Code Values](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#status-code-values).
- Read the status of the application: `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required` and `hdvisionsystems/lumiscan-ai-object-matching/system/status-text`. If `restart-required` is `true`, see [Problem: Every module reports status -5 (restartRequired)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-restart-required).
- If a module did not start, read `hdvisionsystems/lumiscan-ai-object-matching/system/settings`. A module file that could not be read appears there as `{"$file": ..., "$error": ...}`; `"$error"` says why. See [What the node shows](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#what-the-node-shows).
- Read the log. On ctrlX OS, the application writes its messages to the ctrlX OS system log. Read them in the Logbook of the ctrlX web interface, or with `journalctl` (see [ctrlX Data Layer on ctrlX OS](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/ctrlx-data-layer.html)).
- Check the licence. On ctrlX OS, the application shuts down shortly after starting if the licence is missing; the reason is in the system log. A missing licence is reported as status `-4` (`MissingLicense`). See [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html#licence).
- After you change a configuration file, restart the application. A change to the settings on the Data Layer also takes effect only with the next start.
- Problems that all LumiScan apps share - an app or module missing from the tree, a node that is "not found", a method call that is refused - are described in [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/troubleshooting.html) of the LumiScan Keryx manual.

## Running inference

### Problem: inference/ready stays False, or the inference ends with status 100001 {#faq-no-new-external-image}

**Symptoms**

- `<network name>/control/inference/ready` stays `False`, so you cannot request an inference.
- `inference/ready` is `True`, but the request is never acknowledged: `ready` does not fall after you set `request` to `True`.
- `control/status/code` is `100001` (`ImageQueueEmpty`). Earlier versions reported status `10001` (`noNewExternalImage`); that code is no longer used.

**Possible causes**

- Status `100001`: the image could not be read. The image node (`input/image`, or the external image address) has no data, or could not be read. This is not an error.
- The application is still loading the configuration and setting up the network.
- The image stream of the module runs.
- No licence is installed.
- The settings were changed on the Data Layer, and the application waits for a restart (status `-5`).
- `ready` is `True`, but the request is not acknowledged: the last inference failed, and the module is in error. It ignores every request but `reset`.
- More than one process writes `control/inference/request`.

**Diagnostic procedure**

1. Read `control/status/code` and `control/status/text` of the module and look the code up in [Status Code Values](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#status-code-values):
   - `1` (`settingUp`): the module is still setting up.
   - `100001` (`ImageQueueEmpty`): the image could not be read.
   - `100002` (`StreamActive`): the image stream runs, see [Problem: An inference request is answered with status 100002 (StreamActive)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-stream-active).
   - `-4` (`MissingLicense`): the licence is missing.
   - `-5` (`restartRequired`): a restart is required, see [Problem: Every module reports status -5 (restartRequired)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-restart-required).
2. For status `100001`, check the image node. With an external image address (`externalImagePath` in the module configuration file), check that `<address>/data`, `<address>/height`, `<address>/width` and `<address>/pixel-type` (`Mono8` or `RGB8`) have values. Without one, check `hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/image`.
3. Read `control/subscribe/output/stream/active`. Expected for inference on request: `false`.
4. If `ready` is `True` but does not fall after your request, read the log for a failed inference.
5. Check that only one process accesses `control/inference/request`.

**Resolution**

- **If the image could not be read (status `100001`):** provide an image from the image source, for example capture an image with the LumiScan Cam Driver, and run the inference again. For an image on `input/image`, write the raw image and set its width, height and pixel format, see [Initializing the Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html). The application does not check whether the image is new: an image that has not changed since the last inference is inferred again.
- **If the module is still setting up:** wait until `inference/ready` is `True`.
- **If the image stream runs:** stop the stream first, see [Problem: An inference request is answered with status 100002 (StreamActive)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-stream-active).
- **If no licence is installed:** install the licence, see [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html#licence).
- **If a restart is required (status `-5`):** restart the application, see [Problem: Every module reports status -5 (restartRequired)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-restart-required).
- **If the module is in error after a failed inference:**
  1. Request `reset` (with the handshake on `control/reset/`, or with `control/reset/execute`).
  2. Check the application log for the cause.
  3. Request the inference again.
- **If more than one process writes `request`:** make sure that only one process accesses `control/inference/request` at a time. Otherwise the result of the handshake is undefined.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the values of `control/status/code`, `control/status/text`, `control/inference/ready` and `output/package-identifier`, and whether you use the handshake or `control/inference/execute`.

### Problem: status/code is 3 after every inference {#faq-status-3-after-inference}

**Symptoms**

After every inference requested through the handshake, `control/status/code` is `3` (`waitingForClient`), whether the inference succeeded or not.

**Possible causes**

- This is how the handshake works: an inference requested through the handshake ends at status `3`, whatever its outcome. `status/code` does not show whether it succeeded.

**Diagnostic procedure**

1. Before you request the inference, note the value of `<network name>/output/package-identifier`.
2. When `inference/ready` is `True` again, read `output/package-identifier` again. Expected: it has grown by one. It grows by one with every successful inference, and only then (see [Outcome of an Inference](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/general-concept-of-handshake.html#outcome-of-an-inference)).
3. If it did not grow, the inference failed. Request the next inference: if `inference/ready` does not fall, the module is in error after a failed inference.

**Resolution**

- **If `output/package-identifier` has grown by one:** the inference succeeded. Read the result at `<network name>/output/result`.
- **If it did not grow because the image could not be read (there was no image, or it was empty):** provide an image and request the inference again.
- **If it did not grow because the inference itself failed:** the module is in error and ignores every request but `reset`. Request `reset`, and check the application log for the cause.
- **If you want to read the outcome on `status/code` directly:** start the inference with `control/inference/execute` instead (from v2.1.0). `status/code` then keeps the outcome: `0`, `100001` or `-1`. See [Execute a Method](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#execute-a-method).

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the value of `output/package-identifier` before and after the inference.

### Problem: A write to execute is refused {#faq-execute-refused}

**Symptoms**

From v2.1.0. A write to `<network name>/control/<function>/execute` is answered with one of the following results. The function does not run; a refused write is never queued.

| ctrlX Data Layer | LumiScan Keryx |
| --- | --- |
| `DL_TOO_MANY_OPERATIONS` | `TooManyOperations` |
| `DL_INVALID_OPERATION_MODE` | `InvalidOperationMode` |
| `DL_RESOURCE_UNAVAILABLE` | `ResourceUnavailable` |

**Possible causes**

- `DL_TOO_MANY_OPERATIONS` / `TooManyOperations`: another function of the module runs, through the handshake or another `execute`.
- `DL_INVALID_OPERATION_MODE` / `InvalidOperationMode`: the `ready` signal of the function is `False`, so the module cannot run it - for example `inference` while the image stream runs.
- `DL_RESOURCE_UNAVAILABLE` / `ResourceUnavailable`: the application is shutting down; or the settings were changed on the Data Layer and a restart is required (status `-5`); or the module is in error after a failed inference. `reset/execute` is never refused for the error.

**Diagnostic procedure**

1. Note the exact result of the write, and look it up in [Execute a Method](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/initializing-the-inference.html#execute-a-method) (see also [A method call is refused](https://docs.hdvisionsystems.com/en/lumiscan-keryx/1.0.0/troubleshooting.html#refused) in the LumiScan Keryx manual).
2. For `DL_INVALID_OPERATION_MODE` / `InvalidOperationMode`: read the `ready` node of the function (for an inference, `control/inference/ready`) and `control/status/code`. Read `control/subscribe/output/stream/active`: `true` means the image stream runs.
3. For `DL_RESOURCE_UNAVAILABLE` / `ResourceUnavailable`: read `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required`. `true` means a restart is required.
4. Read the log. The apps log the first refusal of a method, then at most one line every ten seconds for the same method and result.

**Resolution**

- **If another function runs (`DL_TOO_MANY_OPERATIONS` / `TooManyOperations`):** retry. The write succeeds once that operation finishes.
- **If the `ready` signal is `False` (`DL_INVALID_OPERATION_MODE` / `InvalidOperationMode`):** bring the module into the required state first; retrying alone keeps failing. Do not retry in a loop. If the image stream runs, stop it first, see [Problem: An inference request is answered with status 100002 (StreamActive)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-stream-active).
- **If a restart is required:** restart the application, see [Problem: Every module reports status -5 (restartRequired)](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#faq-restart-required).
- **If the module is in error after a failed inference:** run `reset` (`control/reset/execute`), then retry.
- **If the application is shutting down:** the function cannot run. When the application has started again, its nodes can be read again; then retry.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the exact result of the write, the function you called and the value of `control/status/code`.

### Problem: An inference request is answered with status 100002 (StreamActive) {#faq-stream-active}

**Symptoms**

From v2.2.0 (image stream, experimental).

- After a request on `control/inference/request`, `control/status/code` is `100002` (`StreamActive`), and the request is not acknowledged.
- `control/inference/ready` is `False`.
- `control/inference/execute` answers `DL_INVALID_OPERATION_MODE` (LumiScan Keryx: `InvalidOperationMode`).

**Possible causes**

- The module infers on its image stream. The stream and inference on request use the same network, so they never run at the same time (see [Inference on request while the stream runs](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/inference-on-an-image-stream.html#while-streaming)). This is not an error, and no reset is needed.
- The module started in stream mode: `stream.enable` is `true` in the module configuration file. `control/subscribe/execute` saves `enable`, so after a restart the module starts its stream again at once.

**Diagnostic procedure**

1. Read `control/subscribe/output/stream/active`. `true` means the module infers on its image stream.
2. Read `setting/stream/enable` and `setting/stream/address`: they show what the module applies now, as saved in the `stream` section of the module configuration file.

**Resolution**

- **If you need inference on request:**
  1. Stop the stream: write `false` to `control/subscribe/input/enable` and `true` to `control/subscribe/execute`. This is saved: the module starts again as it was left.
  2. Check that `control/subscribe/output/stream/active` is `false` and `control/inference/ready` is `True`.
  3. Request the inference.
- **If you need the results of the stream:** subscribe to `control/subscribe/output/stream/result` and read the results there.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the values of `control/subscribe/output/stream/active` and `setting/stream/enable`, and the `stream` section of the module configuration file.

### Problem: Every module reports status -5 (restartRequired) {#faq-restart-required}

**Symptoms**

- Every module reports status `-5` (`restartRequired`). It answers no request, and its `ready` signals are `false`.
- `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required` is `true`.
- Every `execute` method of a module answers `DL_RESOURCE_UNAVAILABLE` / `ResourceUnavailable`. That includes `reset`.
- A running image stream stops.
- The log has the message at error level.

**Possible causes**

- The settings were changed on the Data Layer (`hdvisionsystems/lumiscan-ai-object-matching/system/settings`). The change is saved, but takes effect only with the next start (see [Restart required](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/settings-on-the-data-layer.html#restart-required)).
- `activate-module` or `deactivate-module` was called. The change applies from the next start.
- A file could not be saved: the write answered `DL_CREATION_FAILED` / `CreationFailed`. Files changed before it stay changed, so the application requires a restart all the same.

**Diagnostic procedure**

1. Read `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required`. Expected in this state: `true`.
2. Read `hdvisionsystems/lumiscan-ai-object-matching/system/status-text`. It says which files changed.
3. Read `hdvisionsystems/lumiscan-ai-object-matching/system/settings`. It shows the files as saved, which is what the next start loads.
4. If a write to `settings` answered `DL_CREATION_FAILED` / `CreationFailed`, read the log for the file that could not be saved.

**Resolution**

A reset does not end this state; only a restart does.

- **If the settings were changed on the Data Layer, or a module was activated or deactivated:**
  1. Call `hdvisionsystems/lumiscan-ai-object-matching/system/control/restart/execute`. It answers `DL_OK` / `Ok` as soon as the restart has begun. On ctrlX OS, ctrlX OS starts the application again after about five seconds.
  2. Wait until `hdvisionsystems/lumiscan-ai-object-matching/system/restart-required` can be read again. Expected: `false`.
- **If a file could not be saved (`DL_CREATION_FAILED` / `CreationFailed`):** correct the files, then restart the application as described above.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the content of `system/status-text` and `system/settings`.

## Reading results in ctrlX PLC Engineering

### Problem: The schemas folder does not exist {#faq-schemas-folder-missing}

**Symptoms**

On ctrlX OS, in **Manage app data**, the folder `lumiscan-ai-object-matching/schemas` does not exist, so you cannot download `object_detection.fbs` from `lumiscan-ai-object-matching/schemas/flatbuffers`.

**Possible causes**

- The application has not started successfully. The `schemas` folder is created during start-up, after the licence check.
- The licence is missing. The application then shuts down shortly after starting.

**Diagnostic procedure**

1. Read the ctrlX OS system log in the Logbook of the ctrlX web interface, or with `journalctl`. If the licence is missing, the reason is written there.
2. Check that the ctrlX licence `SWL-W-XCx-OBJxMATCHxHDVSx-V2NN` is installed.

**Resolution**

- **If the licence is missing:** install a valid licence, see [Requirements](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html#licence) and [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-licensing/1.0.4/troubleshooting.html) of the LumiScan Licensing manual. Then check in **Manage app data** that the folder `lumiscan-ai-object-matching/schemas/flatbuffers` exists.
- **If the application did not start for another reason:** the reason is in the system log. Contact support.

Do not edit files in `schemas`: the application deletes and recreates the folder on every start.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the messages of the application from the ctrlX OS system log.

### Problem: Import from FlatBuffers file… is not in the context menu {#ts-import-command-missing}

**Symptoms**

In ctrlX PLC Engineering, you right-click **Application** in the device tree. The context menu does not contain **Import from FlatBuffers file…**.

**Possible causes**

- The ctrlX Data Layer library `CXA_Datalayer` is missing.

**Diagnostic procedure**

1. Check that you use ctrlX PLC Engineering 2.6 or later.
2. Check in the **Library Manager** of the PLC project that `CXA_Datalayer` and `CXA_Flatbuffers` are referenced.

**Resolution**

- **If the ctrlX Data Layer library is missing:**
  1. In the PLC Engineering menu bar, select **Tools** > **Library Repository…**.

  2. Search for `CXA_Datalayer`, select the library and click **Open**.

  3. Repeat the import: right-click **Application**, select **Import from FlatBuffers file…** and select `object_detection.fbs`.

     The screenshots are in [Install the ctrlX Data Layer Library](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-install-datalayer-library).

- **If another library is missing after the import:** add it in the **Library Manager**.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the version of ctrlX PLC Engineering and the libraries listed in the **Library Manager**.

### Problem: The message "Identifier 'VectorT' not defined" appears after the import {#faq-vectort-not-defined}

**Symptoms**

After the import of `object_detection.fbs`, the editor of the function block `hdv_object_detection2d_InstanceTVec` shows the message `Identifier 'VectorT' not defined` (observed with ctrlX PLC Engineering 3.6).

![Editor of hdv_object_detection2d_InstanceTVec with the message Identifier 'VectorT' not defined on the line FUNCTION_BLOCK hdv_object_detection2d_InstanceTVec EXTENDS flatbuffers.VectorT](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/plc-vectort-not-defined.png)

*Figure 1. Message in the editor of hdv_object_detection2d_InstanceTVec*

**Possible causes**

- The function block `hdv_object_detection2d_InstanceTVec` extends `flatbuffers.VectorT`. The version of the `CXA_Flatbuffers` library that is available for ctrlX PLC Engineering 3.6 (0.1.3.3) does not contain this function block.

**Diagnostic procedure**

1. Open the editor of `hdv_object_detection2d_InstanceTVec`. Expected: the message is shown on the line `FUNCTION_BLOCK hdv_object_detection2d_InstanceTVec EXTENDS flatbuffers.VectorT`.
2. Check which version of `CXA_Flatbuffers` the PLC project references in the **Library Manager**.

**Resolution**

The message does not affect reading results.

- **If the message is shown:**
  - Do not use `hdv_object_detection2d_InstanceTVec` in your program. It is not needed to read results. Access individual instances with `getInstances` of `hdv_object_detection2d_Result`, see [Example: Reading Position and Score of an Instance](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/accessing-flatbuffers-data.html#lumiscan-example-instance).
  - If the message is disturbing, delete the function block `hdv_object_detection2d_InstanceTVec` from the application. No other generated function block refers to it.

**If the problem persists**

Send the information listed under [Information for support](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html#support-information) to support, together with the version of ctrlX PLC Engineering, the version of `CXA_Flatbuffers` and the exact message.

## Information for support {#support-information}

If a problem persists, send the following information to support:

- The version of LumiScan AI Object Matching.
- The platform: ctrlX OS on a ctrlX CORE or ctrlX IPC, with the ctrlX OS version (with TPU acceleration, also the version of App Hardware Support); or a PC with Windows 11 or Ubuntu 24.04 and LumiScan Keryx.
- For problems in ctrlX PLC Engineering: the version of ctrlX PLC Engineering and of the libraries `CXA_Datalayer` and `CXA_Flatbuffers`.
- The log of the application. On ctrlX OS, the messages of the application from the ctrlX OS system log (Logbook of the ctrlX web interface, or `journalctl`). On a PC, the log of the application.
- The date and time of the error.
- The Data Layer status: which Data Layer you use (ctrlX Data Layer or LumiScan Keryx; on a PC, the Data Layer address), the values of `control/status/code` and `control/status/text` of the module, `system/restart-required` and `system/status-text`, and, for a refused method call, the exact result of the write.
- The configuration files from the folder `lumiscan-ai-object-matching`: the application configuration file `ai-object-matching.json`, the module configuration file and the model description file (`.json`) in `models`. Alternatively, send the content of `hdvisionsystems/lumiscan-ai-object-matching/system/settings`: every value named `password` shows there as `"***"`.

Contact HD Vision Systems customer success at [customer.success@hdvisionsystems.com](mailto:customer.success@hdvisionsystems.com) or +49 6221 6721905. For licences, contact [licenses@hdvisionsystems.com](mailto:licenses@hdvisionsystems.com).

# v1.2.0 - 2025.07.01 – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/release-notes/v1-2-0.html

**Added**

- Functionality: User can export/import detection results with following function `control/save/control/load/control/list-result-files` function to import/export detection results from configured storage.
- Functionality: Full control on GenICam feature. Additional parameter `initialFeatureFile` in camera module setting, which load space separated `GENICAM_FEATURE_NAME VALUE` pair in text file.
- Usability: support internal storage, SD Card, USB storage
- Usability: User can save image via calling function `control/save-result-image` for debug purpose.
- Functionality: add `input/class-filters` to be able to filter out results
- Usability: add `settings/class-names` to show what is the object names configured in settings.

**Bugfix**

- High CPU usage when shutting down program if no license detected, which paralyzes CtrlX Core.

> [!NOTE]
> Before using any version of the LumiScan AI Object Matching, ensure that the Hardware Support version is downgraded to v2.6.2.
>
> If a HAILO TPU is used to accelerate object detection, it is mandatory to downgrade to Hardware Support v2.6.2.

# AI Object Matching v2.1.0 – LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/release-notes/v2-1-0.html

## AI Object Matching v2.1.0

**Added** - Result now can be saved/load with json format - More descripted node description - Every operation now also exposes a method node `<network>/control/<operation>/execute` (inference, reset, load, save, remove, list-result-files, save-result-image). Writing to it runs the operation once and blocks until done — an alternative to the `ready`/`request` handshake, which still works. Results appear on the same output nodes and the outcome on `status/code`; the datalayer write returns `DL_TOO_MANY_OPERATIONS` when the module is busy with another operation (retry, it will succeed once that one finishes), `DL_INVALID_OPERATION_MODE` when the operation’s `ready` signal is false and the module is therefore in a state where it cannot run at all (retrying does not help), and `DL_RESOURCE_UNAVAILABLE` while shutting down or, for the guarded operations, while the module is in an error state awaiting a `reset`.

# LumiScan AI Object Matching

URL: https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/index.html

|   |   |   |
| --- | --- | --- |
| ![Neural network and its model file model.onnx](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/intro/step-1.svg)<br><br>**Bring a trained model.** Trained on your parts with LumiScan Training. | ![Mixed parts, each outlined with an oriented box, two with class name and score](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/_images/intro/step-2.svg)<br><br>**Detect objects in the image.** Each one with class, position, orientation and score. | Example result<br><br><br><br><br><br><br>Instances<br>4<br><br><br>Center<br>642, 388 px<br><br><br>Angle<br>−20.5°<br><br><br>Score<br>0.96<br><br>**Hand the result to your PLC.** Sorted by score, ready for the PLC or robot. |

LumiScan AI Object Matching uses neural networks to detect trained objects in camera images and reports each one with its class, position, orientation and score. It handles mixed, varied or hard-to-describe parts where rule-based matching struggles. You train the model on your own parts with LumiScan Training; the app runs it on ctrlX OS, optionally on a Hailo TPU, or on a Windows or Ubuntu PC with an NVIDIA GPU.

For licensing, see [Licence](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/lumiscan-ai-object-matching/requirements.html#licence). Questions? See the [FAQ](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/faq.html). A problem or an error message? See [Troubleshooting](https://docs.hdvisionsystems.com/en/lumiscan-ai-object-matching/2.1.0/troubleshooting.html).
