Yes, we have completed these steps and successfully set up the image as suggested.
Since the SDK has not been published yet, could you please guide us on the next steps? In particular, is there any available reference, documentation, or recommended workflow for model quantization and conversion?
We would like to understand how we can convert our existing CV pipeline/models into formats and methods supported by Axon, so that we can start testing the deployment on the device while waiting for the SDK release.
Any available examples, references, or preliminary documentation would be very helpful.
Understood. We will go through the U-Boot and Kernel development documentation and explore the available options.
Meanwhile, could you please let us know when we can expect the RootFS build guide and AI SDK documentation to be published?
Also, is the existing U-Boot and Kernel configuration in the current Axon Ubuntu image sufficient for AI accelerator usage, or do we need to update or customize them before deploying our CV models?
This will help us plan our next steps for model quantization, conversion, and CV pipeline deployment on Axon.
Generally, AI accelerator usage, Ubuntu 24 image is sufficient to run and deploy projects.
We are working on it rootfs build guide part, we will mostly give to you at last resort by ending of this months, but if every check is passed than we will update you within this week only.
We are using RKNN Toolkit to convert and quantize our models and have deployed our complete CV pipeline on the board. Currently, we are achieving around 8–10 FPS per RTSP stream using 3 NPU cores , including detection, stream resizing, tracking, and business logic.
We would like to understand whether this is close to the board’s performance limit, or if we can achieve better FPS through AI SDK optimizations, faster inference, or pipeline/code optimization .
Could we have a discussion with your engineering team to understand the possible optimization options?
Please help me with the following so that we can diagnose the issue:
RKNN Toolkit version used: Please share the version of the RKNN Toolkit you used on the host (first to convert the model) and on Axon (Both the librknn.so and the Python package).
The exact RKNN conversion process you have used: Have you converted your models from, say, .pt/.pth or .tf/.tflite to RKNN directly, or have you gone the ONNX → RKNN route?
It is generally advisable to follow the ONNX → RKNN route, as it is better supported. Please also let me know if you have used any model optimizations (e.g., ONNX graph slimming using onnxslim, quantization, pruning, etc.). These greatly improve throughput by reducing operations and model size. Some of these are available here.
There are other techniques as well, but let’s start with these to diagnose.
Pipelining of tasks and utilization of NPU + CPU: How have you distributed the CV tasks across the NPU? Are you giving each model in your pipeline 1 core? Are you distributing the same model across 3 NPU cores? What batch size are you using?
You can determine and log the usage of the NPU during inference using rktop.
If you notice that, even when running your entire CV pipeline at full force, NPU usage isn’t saturated, there is a good chance your CV pipeline isn’t optimal, and you are leaving utilization and performance on the table.
RTSP stream delivery itself: A bit far-fetched, and assuming you have followed everything above thus far, you would also probably want to investigate the consistency of incoming frames and the camera FPS itself. Generally, this is more associated with network stability and how you buffer and pipeline to ensure the CPU + NPU never stall/wait on data.
Temperature of the SoC: Self-explanatory; make sure that Axon is sufficiently cooled, using a heat sink or active fan if the temperatures seem to get too hot; you may experience thermal throttling.
It is advisable first to determine the root causes; personally, I would do things in the following order:
Optimize the model itself as much as possible before converting to RKNN, as mentioned earlier.
During conversion, RKNN also offers additional optimization using optimization_level in the config method (by default, it is set to max, so you probably don’t have to mess with this)
Establish a single-model baseline first; see how running just 1 model in your pipeline at a time behaves so that down the line you can determine which model is utilizing the least and most resources, which is critical in time or waits on output from another model, etc. Knowing the critical path in your pipeline will help identify the bottlenecks, so that you can tackle them accordingly
If possible, do this on a local video first to rule out the RTSP stream itself.
Investigate and increase batch size for inference; this is generally determined empirically, whether or not you need it to begin with. Set this during model compilation using rknn_batch_size=N (N is the target batch size) in rknn.config() or rknn.build(), and apply it during inference.
Personally, I have noticed that you would want to set the batch size as large as the number of cores itself for a single model (i.e size as 3 for 3 NPU cores)
Pinning a CPU single thread (or a set of them) to a single NPU core has also been found to be helpful; try this after determining the root cause, of course
Plan a suitable pipeline based on the critical path; if it is a single model that is holding up inference down the line, increase its batch size if needed, or even change the architecture if lowering precision even further or pruning is not feasible. If it is the data stream that is the bottleneck, investigate that too; there is no point in optimizing the model inference if it is starved to begin with (the RTSP stream itself, as referenced earlier, and how data moves in your pipeline in general)
This is mostly trial and error until you can reasonably saturate the CPU + NPU. In case the NPU genuinely is the bottleneck, consider moving the smaller ones to the CPU so the NPU has more space (do this after you have tried everything above ONLY)
Please let me know if these steps help or if you’d like further clarity on any of the points above.
Try to make changes in phases to measure the impact of each step/optimization independently and save time.