File size: 3,622 Bytes
c69238e 946cd52 c69238e 946cd52 c69238e 946cd52 c69238e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
license: cc-by-nc-sa-4.0
base_model: stanford-star/rt-j
tags:
- relational-deep-learning
- relational-databases
- tabular
- tabular-classification
- tabular-regression
- foundation-model
- in-context-learning
- quantized
- int8
- relational-transformer
datasets:
- stanford-star/the-join
- stanford-star/relbench
---
# RT-J — int8 (Q8_0-style) quantized checkpoints
Int8 quantizations of [stanford-star/rt-j](https://huggingface.co/stanford-star/rt-j),
the Relational Transformer foundation model for in-context learning over
relational databases. Produced and consumed by the
[RelativeDB](https://github.com/RelativeDB) native inference engine
(`cpp/rt_quantize --type q8`). Weights stay **quantized-resident** at
inference: the engine's CPU (Accelerate / portable SIMD) and Metal/MPS
kernels dequantize inside the GEMM, so DRAM weight traffic is the int8
payload.
Sibling repos: [rt-j-int4](https://huggingface.co/RelativeDB/rt-j-int4) ·
[rt-j-fp16](https://huggingface.co/RelativeDB/rt-j-fp16)
| File | Task head | Size |
|---|---|---|
| `classification/model.q8.safetensors` | classification / ranking (logits — apply sigmoid) | 86 MB |
| `regression/model.q8.safetensors` | regression / forecasting (normalized values) | 86 MB |
(vs. 171 MB bf16 upstream, 342 MB fp32 in memory.)
## Quantization format
Q8_0-style, plain safetensors — no custom container:
- Every **transformer-block projection** (`wq/wk/wv/wg`, `wo`, `ffn.w1/w2/w3`
across all 12 blocks — ~99% of parameters) is stored as an `I8` tensor with
a per-output-row symmetric fp32 scale in a `<name>.q_scale` companion
tensor: `W[o,i] ≈ q[o,i] * scale[o]`, `scale[o] = max|W[o,:]| / 127`.
- The value/col-name **encoders, decoder head, norms, biases and mask
embeddings stay fp32** (like llama.cpp's embedding/output layers):
input-side quantization error would propagate through all 12 blocks for a
negligible size win.
Any safetensors reader can load these files; dequantization is one line per
row. The RelativeDB engine keeps the int8 payload resident and dequantizes
64-row tiles (CPU) or in-register while staging GEMM tiles (Metal); the CUDA
backend is fp32-only for now.
## Accuracy
Measured on the RelativeDB golden batch (B=5, S=16 churn example) against the
PyTorch reference, identical on CPU and Metal/MPS:
| Metric | fp32 | int8 |
|---|---|---|
| `yhat` max abs. error vs. torch | 3.9e-3 | 1.1e-2 |
| `yhat` mean abs. error | 5.1e-4 | 3.5e-3 |
| target-score sign / ranking | preserved | preserved |
Worst per-row mean relative weight error: ≈5%.
## Usage (RelativeDB native engine)
```bash
# direct path
./build/rt_test testdata classification/model.q8.safetensors --quantized --device mps
# via the Java / Python / Rust bindings: place the .q8 file next to the fp32
# checkpoint (or point at a directory containing it) and opt in with
export RELATIVEDB_RT_QUANTIZED=q8 # (or 1/true; q4 and f16 select siblings)
```
The C ABI (`rt_model_load`) accepts these files directly — the format is
auto-detected from the tensor dtypes.
## Reproduce
```bash
cmake -B build -S cpp && cmake --build build -j
./build/rt_quantize <rt-j>/classification/model.safetensors classification/model.q8.safetensors
./build/rt_quantize <rt-j>/regression/model.safetensors regression/model.q8.safetensors
```
## License & attribution
Derivative of [stanford-star/rt-j](https://huggingface.co/stanford-star/rt-j)
(Stanford STAR lab), redistributed under the same
**CC-BY-NC-SA-4.0** license. Architecture and training details are described
in the upstream model card; only the weight storage format differs here.
|