EdgeRock-4B-INT8

EdgeRock's constant-memory reasoning architecture in a smaller weight package. This release uses bitsandbytes LLM.int8() weights and mixed-precision outlier handling. It retains the full recurrent-state, local-attention, and Rock Memory Anchors (RMA) architecture.

The source is Rock Matrix's EdgeRock, frozen at a6a53a5573f1. Quantization changes the large projection and MLP matrices. RMA weights, small recurrent gates, the tied embedding/output head, cache capacity, and recurrent-state settings retain their source values. This is mixed-precision quantization; the format name describes the quantized matrices.

Load and generate

Use a fresh Python environment and install this repository's requirements.txt. The pinned Transformers wheel contains the Qwen3.5 implementation used by EdgeRock. This package contains the text model and supports the PyTorch inference path.

pip install -r requirements.txt

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "hkust-aaron/EdgeRock-4B-INT8"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map={"": 0},
).eval()
messages = [{"role": "user", "content": "Calculate 17 * 23."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt", add_special_tokens=False).to("cuda")
output = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

The validated execution environment uses an NVIDIA Hopper GPU. Other devices have not been validated for this package.

The quantization configuration is embedded in config.json; loading does not require a second quantization pass.

Conversion validation

Check	BF16 source	This release
Weight files	8.527 GiB	5.211 GiB
Teacher-forced NLL on 32 GSM8K test examples	0.9892	0.9979
Maximum logit change after saving and reloading	—	0

The NLL check scores the final up-to-64 tokens of each question/answer sequence, truncated to 512 tokens. Lower is better; these examples are held out from conversion and are not calibration data. This is a conversion check, not a GSM8K accuracy score.

Fresh-process checks verified 168 protected tensors against the source bit for bit. All three arithmetic checks and both identifier-retrieval checks passed; retrieval prompts span approximately 2K and 8K tokens. A Chinese-to-English translation check is included. See validation.json for the exact protocol and outputs. Full benchmark accuracy and optimized serving throughput have not been remeasured for these quantized weights.

A separate generation screen uses GSM8K test rows 32–47, greedy decoding and a 4,096-token output budget. Correct complete answers: 14/16, compared with 15/16 for the BF16 source. Two additional memory update/key-disambiguation checks use 9,664 and 10,462 input tokens: 2/2, compared with 2/2 for BF16. A correct answer must match the expected value, close the thinking section, and reach EOS within the budget. These small screens do not establish unchanged full-benchmark accuracy.

The source model's multimodal frontend and native serving adapters are distributed in the original repository. This quantized package validates text inference; it does not bundle or validate the visual frontend. It is not a GGUF or Ollama model.

License and provenance

The weights retain the EdgeRock Model License 1.0; code retains Apache-2.0. Original notices are preserved in NOTICE. Changes in this derivative release consist of weight quantization, the packaged chat template and generation settings, and the HQQ loader where applicable. Source benchmark claims are described in the technical report.