README.md

---
language:
- en
- zh
- ja
- ko
- fr
- es
- pt
- de
- it
- ru
- ar
- vi
- th
tags:
- text-generation
- ternary
- bitnet
- 1.58bit
- cpu
- gguf
- qwen2.5
- deepseek
- efficient
- low-memory
- jirack
- web-ui
- routing
- tool-call
- robotics
license: mit
---

# JiRack Ultra 7B (CPU)

A fast and efficient 7B model optimized for CPU inference. The model was refactored with BitNet features and an updated tokenizer that includes new **Routing**, **Media**, **Vision** , **Sound** ,**Tool call**, and **Robotics** tags. Built on a DeepSeek R1 -7B architecture with native ternary (BitNet-style) support and ready-to-run GGUF quantizations.

- JiRack is a cloud-ready model that helps save money on cloud infrastructure. It can be used as an expert model in RAG deployments, with the ONNX JiRack Java server as an alternative.
- Subscription: **$1 per month per user** (updated license for non-company use).
- Corp Subscription: **$3 per month per user** (updated license for company use).
- It works without subscription but send message about subscription


# JiRack sevice options
- Current quantizations were done from the FP16 model, but the model allows for more compression thanks to its ternary architecture. 
- If you need to do ternary compression, please write to me and I'll perform QAT from your dataset, tailored specifically to your task.
- Plus double QAT via ONNX QAT.


# JiRack RoboTech
- Advanced Tokenizer with  Robotics & Routing & Tool calls Tokenizer and other
- [CMSManhattan/JiRackPrecisionTokenizer](https://huggingface.co/CMSManhattan/JiRackPrecisionTokenizer)
    

## Available Variants

| Tag | Quant | Size | Approx. RAM | Description |
|-----|-------|------|-------------|-------------|
| `cmsmanhattan/jirack-ultra-7b-cpu:latest` | Full | 28.1 GB | ~12.2 GB | Full precision reference |
| `cmsmanhattan/jirack-ultra-7b-cpu-q4:latest` | Q4_K_M | 10.1 GB | ~4.8 GB | Recommended balance |
| `cmsmanhattan/jirack-ultra-7b-cpu-q3:latest` | Q3_K_M | 8.42 GB | ~4.0 GB | Good quality / size trade-off |
| `cmsmanhattan/jirack-ultra-7b-cpu-q2:latest` | Q2_K | 6.81 GB | ~3.2 GB | Maximum compression |

## Quick Start

### Run with Docker

**Default CPU (Q4 recommended)**
```bash
docker run -d \
  --name jirack_ultra_7b \
  -p 7869:7869 \
  --restart unless-stopped \
  cmsmanhattan/jirack-ultra-7b-cpu-q4:latest
```

**Q3**
```bash
docker run -d \
  --name jirack_ultra_7b \
  -p 7869:7869 \
  --cpus=16 \
  -e THREADS=16 \
  -e THREADS_BATCH=16 \ 
  --restart unless-stopped \
  cmsmanhattan/jirack-ultra-7b-cpu-q3:latest
```

**Q2 (lowest memory)**
```bash
docker run -d \
  --name jirack_ultra_7b \
  -p 7869:7869 \
  --cpus=16 \
  -e THREADS=16 \
  -e THREADS_BATCH=16 \ 
  --restart unless-stopped \
  cmsmanhattan/jirack-ultra-7b-cpu-q2:latest
```

**Full precision**
```bash
docker run -d \
  --name jirack_ultra_7b \
  -p 7869:7869 \
  --cpus=16 \
  -e THREADS=16 \
  -e THREADS_BATCH=16 \ 
  --restart unless-stopped \
  cmsmanhattan/jirack-ultra-7b-cpu:latest
```

**Multi CPU**
```bash
docker run -d \
  --name jirack_ultra_7b \
  -p 7869:7869 \
  --cpus=16 \
  -e THREADS=16 \
  -e THREADS_BATCH=16 \ 
  --restart unless-stopped \
  --memory=16g \
  --cpus=8 \
  cmsmanhattan/jirack-ultra-7b-cpu-q4:latest
```

### Docker Compose Example
```yaml
services:
  jirack:
    image: cmsmanhattan/jirack-ultra-7b-cpu-q4:latest
    container_name: jirack_ultra_7b
    ports:
      - "7869:7869"
    volumes:
      - .:/app
      - ./web:/app/web
    environment:
      - MAX_TOKENS=2048
      - TEMPERATURE=0.7
      - TOP_P=0.9
      - DEFAULT_STREAM=False
      - INTRA_THREADS=4
      - USE_ENV_ALLOCATOR=1
      - THREADS=16 
      - THREADS_BATCH=16 
    deploy:
      resources:
        limits:
          memory: 16g
```

## Access the UI

Once the container is running, open your browser and navigate to:

`http://localhost:7869`

This opens the JiRack UI — a clean web interface.

## Changing the Port

The listening port can be easily modified directly from the **Settings** panel within the JiRack UI.

## Licensing

- The JiRack Ultra 7B model is provided under a commercial license ($12 per user per year).
- All JiRack UI clients are provided under a commercial license.
- However, the UI clients can be used for free when running together with the official JiRack Docker containers, as long as they are not redistributed separately.

For commercial licensing, cluster deployment, or enterprise use of JiRack models, please contact us.

- **JiRack MS Windows 11 Desktop Client (with Ollama API):**
  https://huggingface.co/kgrabko/JiRackTernary_1b/resolve/main/jirack-chat.zip
- **Live email chat with the model:** support@cmsmanhattan.com

## Hardware Recommendations

### Recommended Hardware for JiRack Ultra 7B (single Docker container)

| Use Case          | CPU                          | RAM      | Recommended Quant | Expected Speed      | Recommendation |
|-------------------|------------------------------|----------|-------------------|---------------------|----------------|
| Recommended       | Ryzen 7 / Intel i7           | 16 GB    | Q4_K_M            | Good interactive    | Best choice    |
| High Performance  | Ryzen 9 / Intel i9           | 24–32 GB | Full / Q4         | Excellent           | Excellent      |
| Low Memory        | Modern 6+ core CPU           | 8–12 GB  | Q3_K_M or Q2_K    | Usable              | Acceptable     |
| Edge / Minimal    | Laptop CPU                   | 8 GB     | Q2_K              | Acceptable          | Budget option  |

## Important Memory Notes

Even though the quantized 7B models are small, we recommend the following for best experience:

- Q4_K_M: 8–12 GB system RAM minimum
- Q3_K_M / Q2_K: 6–10 GB system RAM
- Full precision: 16 GB+ system RAM recommended

Reasons for extra headroom:
- KV-cache consumption during generation
- Runtime overhead and temporary buffers
- System stability and avoiding out-of-memory errors
- Room for larger context windows

**Minimum recommended (Q4):** 12 GB system RAM  
**Ideal:** 16–24 GB system RAM

I added the default model in full precision. This serves as the base for quantization, allowing us to find the optimal balance between model size and performance.

## Architecture Notes

- **Refactored with BitNet features**: Native BitLinear ternary path (b1.58-style) with λ-warmup STE
- **Updated tokenizer**: Extended with new special tags for **Routing**, **Tool call**, and **Robotics**
- Base: Qwen2.5-7B style (Hidden 3584, 28 layers, GQA 28/4, vocab 152064)
- RoPE θ = 10000, RMSNorm ε = 1e-6
- Ready-to-run GGUF quantizations (Q2_K, Q3_K_M, Q4_K_M)

## 📧 Contact & Licensing

For joint venture opportunities, hardware integration, or licensing inquiries:

- **Email:** grabko@cmsmanhattan.com
- **Phone:** +1 (516) 777-0945
- **Location:** New York, USA



## License
MIT License