Instructions to use Premchan369/Q-TensorFormer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Premchan369/Q-TensorFormer with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Premchan369/Q-TensorFormer", trust_remote_code=True)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Premchan369/Q-TensorFormer", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Premchan369/Q-TensorFormer with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Premchan369/Q-TensorFormer"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Premchan369/Q-TensorFormer",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Premchan369/Q-TensorFormer

SGLang

How to use Premchan369/Q-TensorFormer with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Premchan369/Q-TensorFormer" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Premchan369/Q-TensorFormer",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Premchan369/Q-TensorFormer" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Premchan369/Q-TensorFormer",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Premchan369/Q-TensorFormer with Docker Model Runner:
```
docker model run hf.co/Premchan369/Q-TensorFormer
```

Q-TensorFormer / reports /COLD_START_EVALUATION.md

Premchandyadav369

Complete 16-phase scientific hardening, strict checkpoint audit, empirical benchmarks, and documentation

7aefd9e 13 days ago

preview code

Raw History Blame Contribute Delete

4.74 kB

# Phase Eight — Cold-Start & Model-Loading Evaluation Report

## 1. Motivation & Practical Edge Context

This evaluation is motivated by the repository author's practical developer experience when deploying medical and edge AI models on memory-constrained hardware: **loading a 1.5B-parameter model (such as MedGemma 1.5B) into constrained VRAM took approximately 200 seconds before the first response could be generated.**
*(Attribution Note: The ~200s MedGemma 1.5B figure is cited strictly as the developer's motivating user experience and is not claimed as a benchmark run inside this repository.)*

On edge devices (phones, point-of-care medical diagnostic units, drones, embedded controllers), total user-perceived response time is governed by two distinct regimes:

$$T_{\text{total}}(N_{\text{tok}}) = \underbrace{T_{\text{disk\_read}} + T_{\text{deserialize}} + T_{\text{init}} + T_{\text{load\_state\_dict}} + \text{TTFT}}_{\text{Cold-Start Overhead } T_{\text{cold}}} + \underbrace{N_{\text{tok}} \times \text{TPOT}}_{\text{Steady-State Decode}}$$
---
## 2. Measured Cold-Start Breakdown (`RUN-20261013-QTF-HARDENING`)
Data source: `results/raw/cold_start_and_edge.json` (`MEASURED` on Intel 20-thread CPU, Windows 10, PyTorch `2.10.0+cpu`, 5 repeats per model variant, $d_{\text{model}}=256, L_{\text{layers}}=4, H=8, V=10000$).

| Model Variant | Dtype | Total Params | Checkpoint Size (MB) | Model Init (ms) | Disk Deserialization (ms) | `load_state_dict` (`strict=True`, ms) | First Inference `TTFT` ($L=16$, ms) | Total Cold-Start to 1st Token (ms) | Classification |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| `DenseBaseline_FP32` | `float32` | `5,719,552` | `22.086 MB` | `115.25 ± 1.49` | `25.87 ± 5.59` | `5.19` | `17.76` | `164.08 ± 24.90 ms` | `MEASURED` |
| `QTensorFormer_Balanced_r8_FP32` | `float32` | `3,927,288` | `15.320 MB` (**-30.6%**) | `137.66 ± 48.97` | `58.97 ± 15.22` | `10.59` | `37.63` | `244.87 ± 83.37 ms` | `MEASURED` |
| `QTensorFormer_Edge_r2_FP32` | `float32` | `3,726,648` | `14.553 MB` (**-34.1%**) | `108.96 ± 0.56` | `54.79 ± 5.71` | `10.08` | `50.48` | `224.32 ± 37.31 ms` | `MEASURED` |
| `HF_QTensorFormer_GQA_r8_FP32` | `float32` | `5,852,928` | `22.353 MB` | `125.15 ± 2.92` | `32.85 ± 12.52` | `4.88` | `25.82` | `188.70 ± 17.42 ms` | `MEASURED` |
| `HF_QTensorFormer_GQA_r8_FP16` | `float16` | `5,852,928` | `11.189 MB` (**-49.3%**) | `132.72 ± 1.05` | `21.64 ± 5.81` (**fastest**) | `4.27` (**fastest**) | `335.66` (CPU FP16 emulation) | `494.30 ± 11.10 ms` | `MEASURED` |

---

## 3. Honest Scientific Analysis of Cold-Start Findings

1. **Checkpoint Storage & Disk Footprint Reduction (`MEASURED`):**
   - Replacing dense FFN matrices with nested TT cores reduces on-disk checkpoint size from `22.086 MB` (`DenseBaseline_FP32`) to `15.320 MB` (`QTensorFormer_Balanced_r8_FP32`, **-30.63%**) and `14.553 MB` (`QTensorFormer_Edge_r2_FP32`, **-34.11%**).
   - Storing the Hugging Face GQA variant in `float16` (`HF_QTensorFormer_GQA_r8_FP16`) cuts checkpoint size to `11.189 MB` (**-49.34%**) and achieves the fastest disk deserialization (`21.64 ms` vs `25.87 ms`) and `load_state_dict` (`4.27 ms` vs `5.19 ms`).
   - At the individual FFN layer level ($512 \times 2048$, where embedding tables do not dominate parameter count), `TTLinear` reduces weight storage from `4.20 MB` to `0.53 MB` at $r=8$ (**-87.28%**) and `0.14 MB` at $r=2$ (**-96.68%**).

2. **Object-Count Deserialization Overhead in PyTorch Zip Archives (`MEASURED`):**
   - Although `QTensorFormer_Balanced_r8_FP32` has `30.6%` fewer bytes on disk than `DenseBaseline_FP32`, its state-dict has more individual tensor objects (`185` keys vs `38` keys in `DenseBaseline` due to 4 TT cores per projection, information probes, and allocator MLPs). On fast local SSD cache, `torch.load` zip-entry parsing overhead (`54.8–59.0 ms`) exceeds the byte-read savings at small model scales (`15–22 MB`).
   - **Implication for Large Edge Models ($\ge 1\text{B}$ parameters):** When checkpoints are gigabytes in size and I/O-bound by flash storage or PCIe/VRAM paging (as in the MedGemma 1.5B scenario), byte volume dominates zip-entry metadata overhead, so a $3\times$–$10\times$ weight byte reduction directly reduces flash-to-VRAM staging time.

3. **CPU FP16 Emulation Penalty (`MEASURED`):**
   - While `HF_QTensorFormer_GQA_r8_FP16` deserializes and loads into memory fastest (`25.9 ms` combined load), executing `float16` `einsum` contractions on an x86 CPU without native FP16 tensor cores triggers software upcasting/downcasting (`TTFT = 335.66 ms`). On CPU, `float32` (`TTFT = 25.82 ms`) should be used for execution while `float16` can be used for on-disk storage.

	# Phase Eight — Cold-Start & Model-Loading Evaluation Report

	## 1. Motivation & Practical Edge Context

	This evaluation is motivated by the repository author's practical developer experience when deploying medical and edge AI models on memory-constrained hardware: loading a 1.5B-parameter model (such as MedGemma 1.5B) into constrained VRAM took approximately 200 seconds before the first response could be generated.
	(Attribution Note: The ~200s MedGemma 1.5B figure is cited strictly as the developer's motivating user experience and is not claimed as a benchmark run inside this repository.)

	On edge devices (phones, point-of-care medical diagnostic units, drones, embedded controllers), total user-perceived response time is governed by two distinct regimes:

	$$T_{\text{total}}(N_{\text{tok}}) = \underbrace{T_{\text{disk\_read}} + T_{\text{deserialize}} + T_{\text{init}} + T_{\text{load\_state\_dict}} + \text{TTFT}}_{\text{Cold-Start Overhead } T_{\text{cold}}} + \underbrace{N_{\text{tok}} \times \text{TPOT}}_{\text{Steady-State Decode}}$$

	---

	## 2. Measured Cold-Start Breakdown (`RUN-20261013-QTF-HARDENING`)

	Data source: `results/raw/cold_start_and_edge.json` (`MEASURED` on Intel 20-thread CPU, Windows 10, PyTorch `2.10.0+cpu`, 5 repeats per model variant, $d_{\text{model}}=256, L_{\text{layers}}=4, H=8, V=10000$).

	\| Model Variant \| Dtype \| Total Params \| Checkpoint Size (MB) \| Model Init (ms) \| Disk Deserialization (ms) \| `load_state_dict` (`strict=True`, ms) \| First Inference `TTFT` ($L=16$, ms) \| Total Cold-Start to 1st Token (ms) \| Classification \|
	\| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \|
	\| `DenseBaseline_FP32` \| `float32` \| `5,719,552` \| `22.086 MB` \| `115.25 ± 1.49` \| `25.87 ± 5.59` \| `5.19` \| `17.76` \| `164.08 ± 24.90 ms` \| `MEASURED` \|
	\| `QTensorFormer_Balanced_r8_FP32` \| `float32` \| `3,927,288` \| `15.320 MB` (-30.6%) \| `137.66 ± 48.97` \| `58.97 ± 15.22` \| `10.59` \| `37.63` \| `244.87 ± 83.37 ms` \| `MEASURED` \|
	\| `QTensorFormer_Edge_r2_FP32` \| `float32` \| `3,726,648` \| `14.553 MB` (-34.1%) \| `108.96 ± 0.56` \| `54.79 ± 5.71` \| `10.08` \| `50.48` \| `224.32 ± 37.31 ms` \| `MEASURED` \|
	\| `HF_QTensorFormer_GQA_r8_FP32` \| `float32` \| `5,852,928` \| `22.353 MB` \| `125.15 ± 2.92` \| `32.85 ± 12.52` \| `4.88` \| `25.82` \| `188.70 ± 17.42 ms` \| `MEASURED` \|
	\| `HF_QTensorFormer_GQA_r8_FP16` \| `float16` \| `5,852,928` \| `11.189 MB` (-49.3%) \| `132.72 ± 1.05` \| `21.64 ± 5.81` (fastest) \| `4.27` (fastest) \| `335.66` (CPU FP16 emulation) \| `494.30 ± 11.10 ms` \| `MEASURED` \|

	---

	## 3. Honest Scientific Analysis of Cold-Start Findings

	1. Checkpoint Storage & Disk Footprint Reduction (`MEASURED`):
	- Replacing dense FFN matrices with nested TT cores reduces on-disk checkpoint size from `22.086 MB` (`DenseBaseline_FP32`) to `15.320 MB` (`QTensorFormer_Balanced_r8_FP32`, -30.63%) and `14.553 MB` (`QTensorFormer_Edge_r2_FP32`, -34.11%).
	- Storing the Hugging Face GQA variant in `float16` (`HF_QTensorFormer_GQA_r8_FP16`) cuts checkpoint size to `11.189 MB` (-49.34%) and achieves the fastest disk deserialization (`21.64 ms` vs `25.87 ms`) and `load_state_dict` (`4.27 ms` vs `5.19 ms`).
	- At the individual FFN layer level ($512 \times 2048$, where embedding tables do not dominate parameter count), `TTLinear` reduces weight storage from `4.20 MB` to `0.53 MB` at $r=8$ (-87.28%) and `0.14 MB` at $r=2$ (-96.68%).

	2. Object-Count Deserialization Overhead in PyTorch Zip Archives (`MEASURED`):
	- Although `QTensorFormer_Balanced_r8_FP32` has `30.6%` fewer bytes on disk than `DenseBaseline_FP32`, its state-dict has more individual tensor objects (`185` keys vs `38` keys in `DenseBaseline` due to 4 TT cores per projection, information probes, and allocator MLPs). On fast local SSD cache, `torch.load` zip-entry parsing overhead (`54.8–59.0 ms`) exceeds the byte-read savings at small model scales (`15–22 MB`).
	- Implication for Large Edge Models ($\ge 1\text{B}$ parameters): When checkpoints are gigabytes in size and I/O-bound by flash storage or PCIe/VRAM paging (as in the MedGemma 1.5B scenario), byte volume dominates zip-entry metadata overhead, so a $3\times$–$10\times$ weight byte reduction directly reduces flash-to-VRAM staging time.

	3. CPU FP16 Emulation Penalty (`MEASURED`):
	- While `HF_QTensorFormer_GQA_r8_FP16` deserializes and loads into memory fastest (`25.9 ms` combined load), executing `float16` `einsum` contractions on an x86 CPU without native FP16 tensor cores triggers software upcasting/downcasting (`TTFT = 335.66 ms`). On CPU, `float32` (`TTFT = 25.82 ms`) should be used for execution while `float16` can be used for on-disk storage.