FAQ

It uses a trained neural network to detect objects in images and reports their position and orientation in pixel coordinates. It runs on ctrlX OS (ctrlX CORE or ctrlX IPC) with the ctrlX Data Layer, or on a PC (Windows 11 or Ubuntu 24.04) with LumiScan Keryx, and is controlled through the Data Layer; the node addresses are the same on both. It does not acquire images itself; it reads images published by another application, for example the LumiScan Cam Driver (hdvisionsystems/lumiscan-cam-driver/camera1/output/image). The model is created with the LumiScan Annotation Framework and LumiScan Training Framework.

Description, Requirements

ctrlX OS

TPU acceleration is optional. If it is used, the ctrlX CORE must include a TPU, and the following versions apply: ctrlX CORE X3 with app version v1.2 or later, ctrlX CORE X7 with app version v1.1 or later, and App Hardware Support exactly 2.6.2. With a HAILO TPU, downgrading Hardware Support to v2.6.2 is mandatory. In the model JSON file, set provider to HAILO. On a PC, use an NVIDIA GPU with provider set to CUDA instead.

Requirements, Release Notes: v1.2.0

In the folder lumiscan-ai-object-matching. On ctrlX OS, open Home > Manage app data in the ctrlX OS web interface, select File View and open that folder. On a PC it is the settings folder of the application: on Windows %APP_RESOURCE_DIR%\etc\lumiscan-ai-object-matching, or without APP_RESOURCE_DIR C:\ProgramData\HD Vision Systems GmbH\LumiScan Vision Suite\lumiscan-ai-obj-mat\etc\lumiscan-ai-object-matching. Restart the application after changing any file. The settings can also be read and changed on the Data Layer at hdvisionsystems/lumiscan-ai-object-matching/system/settings. The folder contains:

LumiScan AI Object Matching Folder Structure, Settings on the Data Layer

The models folder holds two files generated by the LumiScan Training Framework, which belong together: a .onnx file with the trained network and a .json file with the detection parameters. Only these JSON parameters should be changed manually; restart the application afterwards:

Models Folder

Upload the new model together with its description file (they are always handled as one unit); on a PC, copy both into the models folder of the settings folder. Make sure the module configuration file is listed in modules of ai-object-matching.json and that its neuralNetworkConfig points to the new model’s JSON file. Check the remaining module settings, save, and restart the application.

Adding New Module, Uploading Data

An external image address configured in the module must provide <address>/data (array of uint8_t), <address>/height and <address>/width (uint32_t) and <address>/pixel-type (Mono8 or RGB8). This interface is designed and tested for the LumiScan Cam Driver; other sources are not guaranteed to work. If no external image address is configured, the module creates hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/image, where you write the raw image and set its width, height and pixel format. RGB data is laid out as height, width, channel. From v2.2.0 a module can also infer on the image stream of the Cam Driver (experimental).

Module Configuration File, Inference on an Image Stream

Use the ready/request handshake on hdvisionsystems/lumiscan-ai-object-matching/<network name>/control/inference/: wait until ready is True, note the value of <network name>/output/package-identifier, set request to True, wait until ready is False (the inference is already running), set request to False, and wait until ready is True again. If output/package-identifier has grown by one, the inference succeeded: read the result at <network name>/output/result. status/code is 3 at the end of every inference requested through the handshake, whether it succeeded or not. From v2.1.0, you can instead write once to <network name>/control/inference/execute, which blocks until the operation is done; status/code then keeps the outcome: 0, 100001 or -1. Only one process may write request at a time.

Initializing the Inference, Outcome of an Inference, Release Notes: v2.1.0

Write the class names to include into hdvisionsystems/lumiscan-ai-object-matching/<network name>/input/class-filters; leave the array empty to disable filtering. With the default class mapping, class_index in the result is the position of the class in the network’s class-names array. The instances array can contain objects of different classes, so the PLC program has to select the required class.

Class Names and Class Filters

ctrlX OS

The result node <network name>/output/result is a FlatBuffers message described by object_detection.fbs. The application copies this file on each start to lumiscan-ai-object-matching/schemas/flatbuffers; download it in Manage app data with the download icon in the Actions column, or over WebDAV. Do not edit files in schemas, because the folder is recreated on every start. After an application update, download and re-import the schema.

Accessing Result Data with FlatBuffers

You need ctrlX PLC Engineering 2.6 or later and the libraries CXA_Datalayer and CXA_Flatbuffers in the Library Manager. In the device tree, right-click Application, select Import from FlatBuffers file…​ and choose object_detection.fbs. If Import from FlatBuffers file…​ is not in the context menu, the ctrlX Data Layer library is missing: install CXA_Datalayer via Tools > Library Repository…​ and repeat the import.

Install the ctrlX Data Layer Library

After the inference has finished and succeeded (handshake: output/package-identifier has grown by one; inference/execute: status/code is 0), read the result node with DL_ReadNodeValue and pass GetData and GetSize of the DL_NodeValue to getRootAsResult of hdv_object_detection2d_Result. Instances are sorted by score (highest first); getBestIndex() always returns the best one. From the instance, GetScore() gives the score (0.0 to 1.0) and getOrientedBoundingBox() gives the box with GetCenterX()/GetCenterY() in pixels; the angle is in degrees and can be negative. The function blocks read the buffer in place, so use or copy the values before the next read.

Example: Reading Position and Score of an Instance

The ctrlX licence for LumiScan AI Object Matching is SWL-W-XCx-OBJxMATCHxHDVSx-V2NN. Bought in the ctrlX Store for ctrlX OS, it is a ctrlX licence and needs no LumiScan Key. On ctrlX OS, the application shuts down shortly after starting if the licence is missing; the reason is in the system log. A missing licence is reported as status -4 "MissingLicense".

Requirements, Licensing FAQ

See the site-wide LumiScan FAQ. Contact HD Vision Systems customer success at customer.success@hdvisionsystems.com or +49 6221 6721905; for licences, licenses@hdvisionsystems.com.

Licensing FAQ