| # Lite LLM |
|
|
| Lite LLM is a deterministic, tiered-parameter, hierarchical sparse expert (HSER) language model runtime designed to scale from **1B → 1T parameters** and beyond (up to quadrillion-scale parameter universes) while keeping **active compute bounded** per token. |
|
|
| The Github organization hosts the **specification corpus**, **reference implementations**, and **operational tooling** for building and deploying Lite LLM as an enterprise / reference-grade system. |
|
|
| Model optimization for the LiteCore Coherent Silicon Photonic Complex Multiply-Accumulate (CSP-cMAC) Unit Cell Hardware focuses on maximizing inference efficiency under tight memory and power constraints by combining compression, quantization, and memory aware execution. LiteCore is a fundamental photonic compute primitive purpose-built for large language model (LLM) inference at quadrillion-parameter scales. LiteCore leverages silicon-on-insulator (SOI) photonics to perform complex-valued multiply-accumulate operations at <1 fJ energy and 1–10 ps latency—representing 500–2,000× energy and 1,000–10,000× latency improvements over state-of-the-art electronic GPUs. |
|
|
| --- |
|
|
| ## What makes Lite LLM different |
|
|
| ### Deterministic by design |
| Lite LLM treats determinism as a first-class requirement: |
| - Stable top‑k routing with seeded tie‑breaking |
| - Deterministic collectives and reproducible distributed execution |
| - Deterministic audit logs and replayable training runs |
|
|
| ### Tiered Parameter Architecture (TPA) |
| Parameters are partitioned across storage tiers: |
| - **Hot** (HBM / GPU) |
| - **Warm** (DRAM) |
| - **Cold** (NVMe) |
| - **Archive** (Object Store) |
|
|
| Only the TierSet for a request is eligible for routing; everything else has **zero activation probability**. |
|
|
| ### Hierarchical Sparse Expert Routing (HSER) |
| Routing is hierarchical: |
| **Tier → Group → Expert** |
| with bounded activation: |
| `k_tier × k_group × k_expert` experts per token per layer. |
|
|
| This enables extreme parameter scaling while keeping per-token compute predictable. |
|
|
| ### Enterprise runtime focus |
| Lite LLM is not only a model architecture—it is a runtime system: |
| - Distributed execution protocols |
| - Storage hierarchy and prefetching |
| - Secure loading and integrity verification |
| - Multi-tenant isolation, quotas, and compliance readiness |
|
|
| --- |
|
|
| ## Repositories |
|
|
| ### Specifications (authoritative) |
| - `lite-llm-specs` — Enterprise Runtime Engineering Specification Corpus (SPEC‑001…SPEC‑060) |
| - `lite-llm-schemas` — JSON/YAML schemas for manifests, telemetry, policies |
| - `lite-llm-rfcs` — Design proposals and evolution process (RFCs) |
|
|
| ### Reference implementations |
| - `lite-llm-runtime` — Rust runtime (routing, caches, dispatch, TierSet engine) |
| - `lite-llm-train` — Training orchestration, checkpointing, determinism harness |
| - `lite-llm-kernels` — Device kernels + safe wrappers (CUDA/HIP/Metal/CPU) |
| - `lite-llm-comm` — Transport abstraction (RDMA / NCCL / QUIC), collectives |
| - `lite-llm-storage` — Shards, manifests, tier placement, streaming + prefetch |
|
|
| ### Tooling |
| - `lite-llm-cli` — Operator CLI (inspect checkpoints, tier policies, telemetry) |
| - `lite-llm-observability` — Metrics exporters, dashboards, tracing |
| - `lite-llm-deploy` — Helm charts, Terraform modules, bare‑metal playbooks |
|
|
| > The organization may not yet contain all repositories listed above; this is the intended long-term structure. |
|
|
| --- |
|
|
| ## Getting started |
|
|
| ### 1) Read the specs |
| Start with: |
| - **SPEC‑001** Runtime Architecture Overview |
| - **SPEC‑003** Deterministic Routing Engine |
| - **SPEC‑004** Tiered Parameter Architecture (TPA) |
| - **SPEC‑005** Hierarchical Sparse Expert Routing (HSER) |
| - **SPEC‑006** Active Compute Bounding Model |
| - **SPEC‑021…030** Storage hierarchy (hot/warm/cold/archive) |
| - **SPEC‑041…050** Inference runtime (TierSet selection, dispatch, KV cache) |
|
|
| ### 2) Implement the contracts |
| The specs are written to be directly implementable: |
| - Deterministic routing + stable sorting |
| - Tier placement policies and shard formats |
| - All‑to‑all dispatch and imbalance handling |
| - Audit logging and integrity verification |
|
|
| ### 3) Validate determinism |
| Before performance optimization: |
| - Ensure cross-node routing reproducibility |
| - Validate deterministic collectives |
| - Use the replay engine during training |
|
|
| --- |
|
|
| ## Contribution |
|
|
| We welcome contributions in: |
| - Spec clarifications and testable invariants |
| - Rust runtime modules (memory model, routing, dispatch, caching) |
| - Deterministic training harness and replay tooling |
| - Storage tier orchestration and prefetch algorithms |
| - Security hardening and audit improvements |
|
|
| Please read: |
| - `CONTRIBUTING.md` for workflow and standards |
| - `CODE_OF_CONDUCT.md` for community expectations |
| - `SECURITY.md` for vulnerability reporting |
|
|
| --- |
|
|
| ## Security |
|
|
| Lite LLM emphasizes: |
| - Memory-safe runtime design in Rust |
| - Secure checkpoint loading and integrity verification |
| - Encryption at rest for tier storage |
| - Key management and auditability |
| - Sandboxing and capability isolation for extensions |
|
|
| See `SECURITY.md` to report vulnerabilities responsibly. |
|
|
| --- |
|
|
| ## Governance |
|
|
| The specification corpus is the **normative authority**. |
| Changes to the corpus should go through the RFC process: |
| 1. Open an RFC in `lite-llm-rfcs` |
| 2. Discuss and iterate |
| 3. Land a spec patch with tests, invariants, and migration notes |
|
|
| --- |
|
|
| ## License |
|
|
| Lite-LLM is distributed under the Dust Open Source License |
|
|
| license: other |
| license_name: dosl-iie-1.0 |
| license_link: https://github.com/lite-llm/lite-llm/raw/refs/heads/main/LICENSE |
|
|
| --- |
|
|
| ## Contact |
|
|
| - Security: see `SECURITY.md` |
| - General: open an issue in the relevant repository |
|
|
| --- |
|
|
|
|
