How to set up LLM Model ( Vicharak-chat ) in Axon?

General
, , ,
1

Description

Vicharak-Chat: LLM Chat Interface for Axon SBCs
Vicharak-Chat is a streamlined solution designed exclusively for Axon SBCs, powered by the RK3588 chip. It enables users to run large language models (LLMs) efficiently on the device’s NPU. With Vicharak-Chat, users can transform their Axon SBC into a powerful AI conversational system.

Key Features

  • Optimized Performance: Harnesses the Axon SBC’s NPU to deliver high-efficiency LLM inference.
  • Effortless Setup: Packaged as a .deb file for easy installation and deployment.
  • Interactive Front-End: A web-based interface accessible via localhost:<port> for seamless interaction with deployed models.

Steps need to follow

  1. update apt package list using sudo apt update
  2. check if NPU driver version >= 0.9.8 using sudo cat /sys/kernel/debug/rknpu/version
    • if driver version < 0.9.8, update the kernel as follows
    • check kernel version using uname -a
    • if kernel version is 6.*.*, then run either sudo upgrade to upgrade all packages or run sudo apt reinstall linux-image-6.1.75-axon linux-headers-6.1.75-axon to upgrade kernel only
    • if kernel version is 5.*.* , then run sudo apt install linux-image-5.10.238-axon
    • if kernel version is 5.*.*, then make sure to remove older kernel after installing newer kernel, like sudo apt remove linux-image-5.10.160-axon
    • after updating kernel reboot Axon with command sudo reboot
  3. install vicharak chat by running: sudo apt install vicharak-chat
  4. to remove and free up space run sudo apt remove vicharak-chat
  5. in future to update vicharak-chat to get latest feature, run:
    sudo apt update
    sudo apt reinstall vicharak-chat
  • To Run Vicharak-Chat in Linux:

  1. First, you need to go into the root user by running the command su in terminal. you need to enter root password.

  2. Need to run the script , you can download by running below command.
    wget https://docs.vicharak.in/downloads/_static/files/fix_freq_rk3588.sh

  3. Make sure you have made this script executable.
    chmod +x fix_freq_rk3588.sh

  4. Run this script : ./fix_freq_rk3588.sh in root user.

  5. Open Terminal ( Ctrl + alt + t )

  6. Run command vicharak-chat

  7. It takes a few minutes to download LLM Model (/opt/vicharak-chat/models). After downloading, you can explore it on a local network.

    localhost:3060

    If you are using axon on ssh server then,

    <axon's IP>:3060

  • To Remove Vicharak-Chat
  1. sudo apt remove vicharak-chat

  • In future, new updated version will be released, you can apply it by running below command:

    1. sudo apt update
    2. sudo apt install --only-upgrade vicharak-chat

  • For further configuration

If socket port is modified through a command line argument for example if run as vicharak-chat --socket_port <port_num> then frontend would appear on localhost:3060/?socket_port=<port_num>