How To: Run AI Object Matching on Camera Images (ctrlX CORE with Hailo)

This guide sets up LumiScan AI Object Matching on a ctrlX CORE so that it detects objects in the images of LumiScan Cam Driver with a network trained in LumiScan Training, accelerated by the Hailo TPU of the ctrlX CORE.

When you have finished:

the trained network is loaded on the ctrlX CORE and runs on the Hailo TPU,
the module reads the camera image directly from LumiScan Cam Driver,
one write to inference/execute detects the objects in the current image,
the detections - position, class, score and, depending on the network, orientation - are on the Data Layer.

Before you start

Hardware

ctrlX CORE X3 or X7 with a Hailo TPU
A camera connected to LumiScan Cam Driver
Engineering PC with a browser and WinSCP

Software

LumiScan AI Object Matching with a licence, see Requirements
LumiScan Cam Driver, configured, connected and able to capture images
For the Hailo TPU: the ctrlX app Hardware Support in version 2.6.2
Optional: LumiScan Cockpit, to look at the images and results

Data

The trained network from LumiScan Training: the model file (.onnx) and its description file (.json)

How it fits together

The settings of LumiScan AI Object Matching are files in its folder lumiscan-ai-object-matching:

ai-object-matching.json - the application configuration. Its modules list names the module file to load. Currently, one module is supported.
A module file, for example ai-object-matching-module.json - the module’s name, its network and its image source.
The folder models - the network: the model file and its description file, model.json, with the class names, the thresholds and the inference provider.

The module’s name is its address on the Data Layer: hdvisionsystems/lumiscan-ai-object-matching/<name>/…. With externalImagePath, the module reads the image directly from another application, here LumiScan Cam Driver. Each inference uses the image as it is at that moment.

The files are read when the application starts. Restart it after every change.

Step 1 - Check the Hailo requirements

In the ctrlX web interface, open Settings > Apps.
Check that the app Hardware Support is installed in version 2.6.2.
Compare the versions with the table TPU Acceleration Requirements in Requirements.

The TPU is optional. Without it, the network runs on the CPU (provider CPU in step 4), only slower.

Step 2 - Upload the network

The model files are uploaded with WebDAV.

Start WinSCP and create a connection:
- File protocol: WebDAV
- Encryption: TLS/SSL
- Host name: the address of the ctrlX CORE, with your ctrlX user name and password
Connect and go to /solutions/webdav/appdata/lumiscan-ai-object-matching/models.
Copy the model file (.onnx) and its description file (.json) into this folder.

The steps in detail, with screenshots, are in Uploading Data with WebDAV.

Step 3 - Configure the application and the module

In the ctrlX web interface, go to Home > Manage app data, click Open file view and open lumiscan-ai-object-matching.
Open ai-object-matching.json and check that the modules list names your module file:
```
"modules": [
  "ai-object-matching-module.json"
]
```
Open the module file and set:
```
{
  "name": "example",
  "neuralNetworkConfig": "models/model.json",
  "externalImagePath": "hdvisionsystems/lumiscan-cam-driver/camera1/output/image"
}
```
- name - the module’s name on the Data Layer. Only lower-case letters, digits, - and _.
- neuralNetworkConfig - the path of the description file you uploaded, relative to the application folder.
- externalImagePath - the image node of the camera module, here camera1 of LumiScan Cam Driver. The node must provide data, width, height and pixel-type.
  
  Keep the $schema and meta entries of the file. All parameters are described in Module Configuration File.

Step 4 - Select the Hailo TPU in the model description

Open models/model.json. Its parameters are generated by LumiScan Training; change only these:

{
  "pathToNetwork": "model.onnx",
  "provider": "HAILO",
  "deviceIndex": 0
}

pathToNetwork - the model file, relative to model.json.
provider - HAILO for the TPU, CPU without it.
deviceIndex - currently always 0.

In the same file, thresholdList sets for each class the minimum score, from 0.0 to 1.0, a detection needs. See Model.json File.

Step 5 - Restart and check the module

Save all changed files.
Under Settings > Apps, select LumiScan AI Object Matching and click Restart.
Open Settings > Data Layer and go to hdvisionsystems/lumiscan-ai-object-matching.

Expected: a folder with the module’s name, here example, and example/control/inference/ready is true once the network is loaded.

If the module is missing or ready stays false, read example/control/status/text and the application log. Common causes are a wrong path in the module file or in model.json, a name with upper-case letters, and a missing licence. See Troubleshooting.

Step 6 - Run the inference

Capture an image with LumiScan Cam Driver, for example with a write to hdvisionsystems/lumiscan-cam-driver/camera1/control/capture/execute.
Write to hdvisionsystems/lumiscan-ai-object-matching/example/control/inference/execute. The write returns when the inference has finished.
Read example/control/status/code.

Expected: the status code is 0, and example/output/package-identifier has grown by one.

100001 means that there was no image to read. Capture an image and run the inference again.
-1 means that the inference failed. The module then accepts only a reset: write to example/control/reset/execute, and check the log.

A client waits only a few seconds for an execute call to return. If the inference on a large image takes longer, start it with the ready/request handshake on example/control/inference instead. The handshake always ends at status code 3; tell success by output/package-identifier, which grows only after a successful inference. See Initializing the Inference.

Step 7 - Read the detections

The detections are on

hdvisionsystems/lumiscan-ai-object-matching/example/output/result

For each detected object, the result gives its position, class and score and, if the network was trained for it, its orientation. How to read the result in a PLC program is described in Accessing Result Data with FlatBuffers.

To check the detections visually:

In LumiScan Cockpit, a module that infers on the camera’s image stream shows the boxes on the live frames. See Inference on an Image Stream (Experimental).
For a single inference, example/control/save-result-image/execute saves the result image for debugging.

Result

The trained network runs on the Hailo TPU of the ctrlX CORE.
The module reads the camera image of LumiScan Cam Driver directly.
One write to inference/execute detects the objects in the current image.
The detections are on output/result, ready for the PLC or the robot.

In production, a cycle is: capture with LumiScan Cam Driver, write to inference/execute, check the status code, read output/result.