AXON V0.2 (RK3588): NPU IOMMU page fault (fdab9000) and kernel oops under multi-model RKNN load

General
1

We run an edge inference workload on an AXON V0.2 (RK3588, 8 GB RAM, eMMC).
The box hangs under multi-model RKNN load. The pstore shows an NPU IOMMU
page fault and a kernel oops. We would like to know whether this is a known
driver/runtime issue and which versions you recommend.

Environment

Item Value
Board Vicharak AXON V0.2, RK3588, 8 GB RAM, eMMC
OS Ubuntu 24.04.5 LTS, aarch64
Kernel 6.1.75-axon #axon SMP Mon Oct 5 19:20:01 IST 2026
NPU driver RKNPU driver: v0.9.8
Runtime librknnrt version: 2.3.0 (c949ad889d@2024-11-07T11:35:33)
Workload Python edge service, 5 RTSP streams from an 8-channel analog DVR
Models YOLO11s-640 (3 contexts), SCRFD-640, ArcFace-112, OSNet-256x128, kitchen-PPE, kitchen-gear, pose-640
Thermal 85 °C passive trip; 85–90 °C under the 5-stream load; fan at max

Observed failure

  • After 12 minutes to 4 hours, the box stops. There is no heartbeat, no SSH
    and no further log.
  • With the SoC watchdog armed, it resets every 12–60 minutes under the
    5-stream load. The resets become a boot loop.
  • fdab9000 is the NPU IOMMU for fdab0000.npu (rockchip,rk3588-rknpu).
    It is not the VPU decoder.

pstore console:

rk_iommu fdab9000.iommu: Page fault at 0x... of type write
rk_iommu fdab9000.iommu: iova = 0x...; dte_index ... pte ... valid: 1

pstore dmesg:

Internal error: Oops - Undefined instruction: 0000000002000000 [#2] SMP
CPU: 3 PID: 2021 Comm: python Tainted: G      D   6.1.75-axon #axon
pc : percpu_ref_put_many.constprop.0+0x0/0x8c
lr : obj_cgroup_uncharge_pages+0x98/0x120
Call trace:
  percpu_ref_put_many
  refill_obj_stock
  obj_cgroup_uncharge
  kmem_cache_free
  file_free_rcu
  rcu_do_batch

The NPU DMA appears to write to a page the kernel has already freed. The
kernel oopses during memory reclaim or process exit.

The VPU also logs:

mpp_platform: client 4 driver is not ready!

That is a separate VPU probe warning. It is not the NPU fault.

What we ruled out

  • eMMC: life_time=0x01, pre_eol_info=0x01; a storage canary writes
    and reads back every 60 s with no error; no mmc or EXT4-fs errors.
  • Power: a second device on the same supply stayed up.
  • Clean shutdown: no shutdown request, power-key event, OOM kill or
    thermal-critical message. The ext4 recovery is the result of the resets.
  • VPU decoder: the fault persisted after we disabled hardware decode
    and ran 3 cameras on software decode. The NPU inference was the common
    path.

Workaround

We reduced the load to:

  • 1 camera,
  • only YOLO11s-640,
  • one RKNN context,
  • optional pose/PPE/face/reID models off.

The box has been stable for 20+ minutes with no fdab9000 fault, at
73–74 °C. This suggests a multi-model or multi-context issue, but we have
not isolated the exact model.

Questions

  1. Is RKNPU driver v0.9.8 the correct driver for librknnrt 2.3.0 on kernel
    20261005-axon? Do you recommend a newer driver/runtime pair?
  2. Is there a known IOMMU page-fault or use-after-free issue with multiple
    RKNN contexts or multiple models on RK3588?
  3. Is there newer AXON V0.2 NPU firmware or librknnrt that we should use?
  4. Are there debug knobs (NPU IOMMU logs, rknn memory flags,
    RKNN_LOG_LEVEL) that would help isolate the failing model?
  5. Is sustained 85–90 °C at the 85 °C passive trip expected to be safe, or
    could it contribute to the fault?