can someone help me figure out what i am missing?I tried running gemma4 e2b on the axon using rk-llama.cpp and it loaded perfectly fine. but the npu has never utilized to its fullest. I tried changing the governor to performance and change the clock speed to max, but it still remain the same as before. Is it normal with axon or am i missing anything important?
Which compiler have you used? And which runtime versions have you used?
From this I can see clang21 produces more optimized code.
Also as far as I have gone through the code, they are delegating the matmul and vector operations to NPU via rknnrt, rknnmrt & rkllmrt and for rest using llama.cpp cpu backend, so there might be too many memcpy and cross platform procedure call overheads. And due to this utilizing NPU at full is not possible.
can you check with this one https://www.reddit.com/r/RockchipNPU/comments/1sc8fir/deploy_the_newest_qwen35_and_gemma4_models_of_any/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
I am trying it now, using rknn runtime version 2.4.3 and rkllm runtime version 1.2.3. and gcc-13. And set ddr frequency to max, and running gemma4 e2B, it is utilizing 22% NPU across all cores.
Are you using latest runtime libraries and have set the ulimit -n 635535 ?
You can also try clang for better CPU’s simd units (sve / sme) utilizations. Clang also optimizes cache and memory overheads, so it would increase NPU utilization if zero copy is done most of the time.
My current RKNN version is 2.3.0 and RKLLM is 1.2.3. with these i was able to attain max 20% utilization. But is there a way to go beyond 50 or 60% min? So far i was getting ~4 Tps with qwen3 4B and 8 Tps with gemma4 e2b. If i can use 60% then it will drastically improve the performance. Am i right? Correct me if i’m wrong.
using rkllm-toolkit would certainly increase the TPS. Also the benchmark they are sharing using llama.cpp, is using lpddr5 16/32 GB ram which can run on 3.2GHz, while the RAM we have runs on 2.1GHz, so they are benchmarking on 50% faster RAM and in LLMs RAM bandwidth is the bottleneck.
Compiling using Clang21 might decrease some overheads and so increase NPU utilizations.
But running using rkllm-toolkit will be the fastest, though it would take more development time.
Thanks mate. Now i get it why i was not able to be on par with those guys. Also i gonna try with clang 21 and post if any significant improvements were found.
Glad to be helpful. Meanwhile, when I get time, I will try to generate .rkllm file for gemma4 E2B, and see what TPS that gives.
Qwen3 model conversion is directly supported via rkllm-toolkit, so you can try that.
What TPS you are aiming for, right now, with gemma4 e2b?
1 Like
Anything above 20 Tps would good. Also when using the .rkllm the context length is limited to 16k. Is there any way to overcome that.
Searched for it, but as of now there is no way to increase that context length, also inference would become too slow upon hitting even that limit.
Also, rkllm-toolkit supports only int8 quantization for rk3588 and not less bit width like 4bit quantization, so only 1B or lesser parameters model can run with 20+ TPS generation speed (prompt processing speed is easily higher). But for running more parameters model at that speed, you would require to run 4bit or less bit wide parameters, and rk-llama.cpp can run that but rkllm-toolkit can’t as of now.
1 Like
As of now, gemma4 e2b was the only model that can perform better around that 1-2B mark, So i would like to stick with that, Later if rknn extends support for gemma4 models would move to it. Until then i would prefer rk-llama.cpp.
Once again thanks for your support.
2 Likes