How to set up LLM Model ( Vicharak-chat ) in Axon?

General
, , ,
1

  • Description

    Vicharak-Chat: LLM Chat Interface for Axon SBCs
    Vicharak-Chat is a streamlined solution designed exclusively for Axon SBCs, powered by the RK3588 chip. It enables users to run large language models (LLMs) efficiently on the device’s NPU. With Vicharak-Chat, users can transform their Axon SBC into a powerful AI conversational system.

    Key Features:

    • Effortless Setup: Packaged as a .deb file for easy installation and deployment.
    • Optimized Performance: Harnesses the Axon SBC’s NPU to deliver high-efficiency LLM inference.
    • Interactive Front-End: A web-based interface accessible via localhost:<port> for seamless interaction with deployed models.

  • Steps need to follow :

  1. sudo apt update

uname -r : Check which version you are using. If it is not 5.10.228 or >5.10.228, you need to install it using the below command.

  1. sudo apt install linux-image-5.10.228-axon

And Remove older kernel version

  1. sudo apt remove linux-image-5.10.<version-num>-axon

  2. Before running this command sudo reboot, make sure you have removed the previous kernel version and installed 5.10.228 version.

  3. sudo apt install vicharak-chat

  • To Run Vicharak-Chat in Linux:

  1. First, you need to go into the root user by running the command su in terminal. you need to enter root password.

  2. Need to run the script , you can download by running below command.
    wget https://docs.vicharak.in/downloads/_static/files/fix_freq_rk3588.sh

  3. Make sure you have made this script executable.
    chmod +x fix_freq_rk3588.sh

  4. Run this script : ./fix_freq_rk3588.sh in root user.

  5. Open Terminal ( Ctrl + alt + t )

  6. Run command vicharak-chat

  7. It takes a few minutes to download LLM Model (/opt/vicharak-chat/models). After downloading, you can explore it on a local network.

    localhost:3060

    If you are using axon on ssh server then,

    <axon's IP>:3060

  • To Remove Vicharak-Chat
  1. sudo apt remove vicharak-chat

  • In future, new updated version will be released, you can apply it by running below command:

    1. sudo apt update
    2. sudo apt install --only-upgrade vicharak-chat

  • For further configuration

If socket port is modified through a command line argument for example if run as vicharak-chat --socket_port <port_num> then frontend would appear on localhost:3060/?socket_port=<port_num>