We run an edge inference workload on an AXON V0.2 (RK3588, 8 GB RAM, eMMC).
The box hangs under multi-model RKNN load. The pstore shows an NPU IOMMU
page fault and a kernel oops. We would like to know whether this is a known
driver/runtime issue and which versions you recommend.
Environment
| Item | Value |
|---|---|
| Board | Vicharak AXON V0.2, RK3588, 8 GB RAM, eMMC |
| OS | Ubuntu 24.04.5 LTS, aarch64 |
| Kernel | 6.1.75-axon #axon SMP Mon Oct 5 19:20:01 IST 2026 |
| NPU driver | RKNPU driver: v0.9.8 |
| Runtime | librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33) |
| Workload | Python edge service, 5 RTSP streams from an 8-channel analog DVR |
| Models | YOLO11s-640 (3 contexts), SCRFD-640, ArcFace-112, OSNet-256x128, kitchen-PPE, kitchen-gear, pose-640 |
| Thermal | 85 °C passive trip; 85–90 °C under the 5-stream load; fan at max |
Observed failure
- After 12 minutes to 4 hours, the box stops. There is no heartbeat, no SSH
and no further log. - With the SoC watchdog armed, it resets every 12–60 minutes under the
5-stream load. The resets become a boot loop. fdab9000is the NPU IOMMU forfdab0000.npu(rockchip,rk3588-rknpu).
It is not the VPU decoder.
pstore console:
rk_iommu fdab9000.iommu: Page fault at 0x... of type write
rk_iommu fdab9000.iommu: iova = 0x...; dte_index ... pte ... valid: 1
pstore dmesg:
Internal error: Oops - Undefined instruction: 0000000002000000 [#2] SMP
CPU: 3 PID: 2021 Comm: python Tainted: G D 6.1.75-axon #axon
pc : percpu_ref_put_many.constprop.0+0x0/0x8c
lr : obj_cgroup_uncharge_pages+0x98/0x120
Call trace:
percpu_ref_put_many
refill_obj_stock
obj_cgroup_uncharge
kmem_cache_free
file_free_rcu
rcu_do_batch
The NPU DMA appears to write to a page the kernel has already freed. The
kernel oopses during memory reclaim or process exit.
The VPU also logs:
mpp_platform: client 4 driver is not ready!
That is a separate VPU probe warning. It is not the NPU fault.
What we ruled out
- eMMC:
life_time=0x01,pre_eol_info=0x01; a storage canary writes
and reads back every 60 s with no error; nommcorEXT4-fserrors. - Power: a second device on the same supply stayed up.
- Clean shutdown: no shutdown request, power-key event, OOM kill or
thermal-critical message. The ext4 recovery is the result of the resets. - VPU decoder: the fault persisted after we disabled hardware decode
and ran 3 cameras on software decode. The NPU inference was the common
path.
Workaround
We reduced the load to:
- 1 camera,
- only YOLO11s-640,
- one RKNN context,
- optional pose/PPE/face/reID models off.
The box has been stable for 20+ minutes with no fdab9000 fault, at
73–74 °C. This suggests a multi-model or multi-context issue, but we have
not isolated the exact model.
Questions
- Is RKNPU driver v0.9.8 the correct driver for
librknnrt2.3.0 on kernel
20261005-axon? Do you recommend a newer driver/runtime pair? - Is there a known IOMMU page-fault or use-after-free issue with multiple
RKNN contexts or multiple models on RK3588? - Is there newer AXON V0.2 NPU firmware or
librknnrtthat we should use? - Are there debug knobs (NPU IOMMU logs,
rknnmemory flags,
RKNN_LOG_LEVEL) that would help isolate the failing model? - Is sustained 85–90 °C at the 85 °C passive trip expected to be safe, or
could it contribute to the fault?