daslearning/Embedding-Onnx

OnLLM 🧠

OnLLM is the platform to run Large Language Model (LLM) or SLM models using OnnxRuntime directly on low-end devices like low power computers, mobile phones etc. It is cross-platform using Kivy & open-source.

The name is derived from: On from Onnx + LLM. Our approach is always local & offline first (you will need internet to download the model files for the first time). Arificial Intelligence (AI) to be accesible to everyone.

Features

1. ✅️ An inbuild chatbot to chat with the AI model(s).
2. ✅️ A local offline RAG to query on our selected document (pdf/docx) directly in the chatbot.
3. ✅️ No trackers, no Ads, No data collection, completely Private.

📽️ Demo

You can click on the below Image or this Youtube Link to see the demo. Please let me know in the comments, how do you feel about this App.

🖧 Our Scematic Architecture

To be added...

🧑‍💻 Quickstart Guide

📱 Download & Run the Android App

You can check the Releases and downlaod the latest version of the android app (APK file) on your phone.

💻 Download & Run the Windows or Linux App

You can check the Releases and downlaod the latest version of the application on your computer. If you are on Windows, download the OnLLM_X.X.X.exe file & double click to run it. If you are on Linux, download OnLLM_Linux_XXX file and run it.

Notes:

• Some Antivirus softwares might give you detection alert for the exe (happens for PyInstaller apps), you need to add an exception in that case.
• On Linux you may need to change file permission to execute it.

chmod +x OnLLM_Linux_XXX
./OnLLM_Linux_XXX

🐍 Run with Python

1. Clone the repo

git clone https://github.com/daslearning-org/OnLLM.git

2. Run the application

cd OnLLM/app/
pip install -r requirements.txt # virtual environment is recommended
python main.py

🦾 Build your own App

The Kivy project has a great tool named Buildozer which can make mobile apps for Android & iOS

📱 Build Android App

A Linux environment is recommended for the app development. If you are on Windows, you may use WSL or any Virtual Machine. As of now the buildozer tool works on Python version 3.11 at maximum. I am going to use Python 3.11

Install Rust & Add that in path

curl https://sh.rustup.rs -sSf | sh # then follow the prompt
export PATH="$HOME/.cargo/bin:$PATH"
# optional
rustup default stable

Install other dependencies & build APK

# add the python repository
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update

# install all dependencies.
sudo apt install -y ant autoconf automake ccache cmake g++ gcc libbz2-dev libffi-dev libltdl-dev libtool libssl-dev lbzip2 make ninja-build openjdk-17-jdk patch patchelf pkg-config protobuf-compiler python3.11 python3.11-venv python3.11-dev

# optionally we can default to python 3.11
sudo ln -sf /usr/bin/python3.11 /usr/bin/python3
sudo ln -sf /usr/bin/python3.11 /usr/bin/python
sudo ln -sf /usr/bin/python3.11-config /usr/bin/python3-config

# optionally you may check the java installation with below commands
java -version
javac -version

# install python modules
git clone https://github.com/daslearning-org/OnLLM.git
cd OnLLM/app/
python3.11 -m venv .env # create python virtual environment
source .env/bin/activate
pip install -r req_android.txt

# build the android apk
buildozer android debug # this may take a good amount of time for the first time & will generate the apk in the bin directory

🖳 Build Computer Application (Windows / Linux / MacOS)

A Python virtual environment is recommended and will use PyInstaller to package it.

# install pyinstaller
git clone https://github.com/daslearning-org/OnLLM.git
cd OnLLM/app/
python3.11 -m venv .env # create python virtual environment
source .env/bin/activate
pip install -r requirements.txt
pip install pyinstaller

# then update the spec file as needed
# then build your app which will be native to the OS i.e. Linux or Windows or MAC
pyinstaller desktopApp.spec

🔗 LLM Models used in this project

LLM (QA) models:

The models are being taken from different sources & below are the models available as on today. Size is the compressed size (approax). Few models which are marked with warning (⚠️) symbol may be very slow or unresponsive on Android or low power devices.

Model	Size	Android	Desktop
SmolLM2-135M	95MB	✅️	✅️
SmolLM2-360M	241MB	✅️	✅️
Gemma3-1B	645MB	⚠️	✅️

Embed models (Document RAG):

It follows the same priciples as above.

Model	Size	Android	Desktop
all-MiniLM-L6-V2	85MB	✅️	✅️