OnLLM 🧠
OnLLM is the platform to run Large Language Model (LLM) or SLM models using OnnxRuntime directly on low-end devices like low power computers, mobile phones etc. It is cross-platform using Kivy & open-source.
The name is derived from:
Onfrom Onnx +LLM. Our approach is always local & offline first (you will need internet to download the model files for the first time). Arificial Intelligence (AI) to be accesible to everyone.
Features
- An inbuild chatbot to chat with the AI model(s).
- No trackers, no Ads, No data collection, completely Private.
📽️ Demo
You can click on the below Image or this Youtube Link to see the demo. Please let me know in the comments, how do you feel about this App.
🖧 Our Scematic Architecture
To be added...
🧑💻 Quickstart Guide
📱 Download & Run the Android App
You can check the Releases and downlaod the latest version of the android app (APK file) on your phone.
💻 Download & Run the Windows or Linux App
You can check the Releases and downlaod the latest version of the application on your computer. If you are on Windows, download the OnLLM_X.X.X.exe file & double click to run it. If you are on Linux, download OnLLM_Linux_XXX file and run it.
Notes:
- Some Antivirus softwares might give you detection alert for the exe (happens for PyInstaller apps), you need to add an exception in that case.
- On Linux you may need to change file permission to execute it.
chmod +x OnLLM_Linux_XXX
./OnLLM_Linux_XXX
🐍 Run with Python
- Clone the repo
git clone https://github.com/daslearning-org/OnLLM.git
- Run the application
cd OnLLM/app/
pip install -r requirements.txt # virtual environment is recommended
python main.py
🦾 Build your own App
The Kivy project has a great tool named Buildozer which can make mobile apps for Android & iOS
📱 Build Android App
A Linux environment is recommended for the app development. If you are on Windows, you may use WSL or any Virtual Machine. As of now the buildozer tool works on Python version 3.11 at maximum. I am going to use Python 3.11
Install Rust & Add that in path
curl https://sh.rustup.rs -sSf | sh # then follow the prompt
export PATH="$HOME/.cargo/bin:$PATH"
# optional
rustup default stable
Install other dependencies & build APK
# add the python repository
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
# install all dependencies.
sudo apt install -y ant autoconf automake ccache cmake g++ gcc libbz2-dev libffi-dev libltdl-dev libtool libssl-dev lbzip2 make ninja-build openjdk-17-jdk patch patchelf pkg-config protobuf-compiler python3.11 python3.11-venv python3.11-dev
# optionally we can default to python 3.11
sudo ln -sf /usr/bin/python3.11 /usr/bin/python3
sudo ln -sf /usr/bin/python3.11 /usr/bin/python
sudo ln -sf /usr/bin/python3.11-config /usr/bin/python3-config
# optionally you may check the java installation with below commands
java -version
javac -version
# install python modules
git clone https://github.com/daslearning-org/OnLLM.git
cd OnLLM/app/
python3.11 -m venv .env # create python virtual environment
source .env/bin/activate
pip install -r req_android.txt
# build the android apk
buildozer android debug # this may take a good amount of time for the first time & will generate the apk in the bin directory
🖳 Build Computer Application (Windows / Linux / MacOS)
A Python virtual environment is recommended and will use PyInstaller to package it.
# install pyinstaller
git clone https://github.com/daslearning-org/OnLLM.git
cd OnLLM/app/
python3.11 -m venv .env # create python virtual environment
source .env/bin/activate
pip install -r requirements.txt
pip install pyinstaller
# then update the spec file as needed
# then build your app which will be native to the OS i.e. Linux or Windows or MAC
pyinstaller desktopApp.spec
🔗 LLM Models used in this project
The models are being taken from different sources & below are the models available as on today. Size is the compressed size (approax).
| Model | Size | Android | Desktop |
|---|---|---|---|
| SmolLM2-135M | 95MB | ✅️ | ✅️ |
| SmolLM2-360M | 241MB | ✅️ | ✅️ |
| Gemma3-1B | 645MB | ❌ | ✅️ |