win10/Spark-engram-4B-V2

Spark Engram V2: Complete Spark-X2.5-4B Merged Checkpoint

This checkpoint integrates Spark Engram V2's automatic lifetime, surprise momentum, and bounded admission with the Spark-X2.5-4B causal language model. It is the complete model produced by merging the trained LoRA adapter at step 100 and its saved Engram module into the original backbone.

The model contains 4,113,838,981 parameters, including 1,759,621 Engram parameters. Nine FP32 safetensors shards total 16,455,391,332 bytes (15.325 GiB); the tensor payload is 16,455,355,924 bytes. Config, tokenizer, generation config, weight index, and the Python modules required by Hugging Face auto_map are included. The original five backbone shards were preserved.

Architecture

The backbone retains 36 Transformer layers, RMSNorm, GQA, RoPE, attention head gating, GELU-gated MLP, and its original full/sliding attention pattern. Engram sidecars at layers 1 and 5 combine static hashed embeddings, dynamic memory features, context gates, and causal depthwise convolution. Retrieved features affect an independent memory prediction and are not propagated through subsequent backbone attention or MLP layers. A separate baseline expert is retained for causal Bayesian calibration.

Inference-time fast state minimizes the local residual objective

$$\mathrm{\ell }(u;z)=\frac{1}{2}\mathrm{\parallel }u-z{\mathrm{\parallel }}_2^2,{\mathrm{\nabla }}_u\mathrm{\ell }=u-z,\backslash ell(u;z)=\backslash tfrac12\backslash |u-z\backslash |\_2^2,\; \backslash qquad\; \backslash nabla\_u\backslash ell=u-z,$$

with projected velocity momentum, forgetting, and an observation-indexed surprise EMA. The target is a normalized, detached slice of the current source-token embedding. New key/continuation pairs enter bounded probation. Admission requires repeated observations, sufficient surprise or velocity, and available row/cell capacity. Fast values, velocity, counts, and probation are inference state; static embeddings, projections, sidecars, and copy gate are model parameters.

This checkpoint allows 65,536 confirmed rows, 262,144 continuation cells, and 8,192 probation pairs per positional lane, with at most 16 lanes. The SQLite main file is capped at 256 MiB. That limit excludes rollback journals, weights, KV caches, logits, and working tensors. Existing confirmed associations remain updatable at capacity; new admission is rejected and counted. Storage failure raises an error and rolls back the transaction.

CUDA loading and inference

The validated stack used Python 3.12.3, PyTorch 2.10.0+cu130 (CUDA 13.0), Transformers 5.8.0, PEFT 0.18.1, and Accelerate 1.11.0. Install a CUDA-enabled PyTorch build using the official installation instructions, then install the inference dependencies:

pip install transformers==5.8.0 accelerate==1.11.0

PEFT is needed to reproduce the merge, but the complete merged model can be loaded directly. Set <MODEL_DIR> to this checkpoint directory:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_dir = "<MODEL_DIR>"
assert torch.cuda.is_available()
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
tokenizer.padding_side = "left"
model = AutoModelForCausalLM.from_pretrained(
    model_dir, trust_remote_code=True,
    dtype=torch.bfloat16, device_map={"": "cuda:0"},
).eval()

inputs = tokenizer("The capital of France is", return_tensors="pt").to("cuda")
with torch.inference_mode():
    output = model(**inputs, use_cache=False)
    answer = model.generate(**inputs, max_new_tokens=16, do_sample=False)
print(tokenizer.decode(answer[0, inputs.input_ids.shape[1]:]))

AutoModelForCausalLM resolves Spark2_5ForCausalLM through the bundled auto_map. Merging and saving used FP32 to preserve small weight updates; inference was verified with BF16 loading. The package also provides TorchSparkForCausalLM for native reference checkpoints; this 4B checkpoint uses the Hugging Face interface above.

Automatic lifetime and SQLite

The saved configuration sets memory_config.enable=true, mode=expandable, lifecycle=auto, and namespace Spark-X2.5-4B-v2-merged-checkpoint100. With storage_path=null, the default location is

$SPARK_ENGRAM_HOME/model-Spark-X2.5-4B-v2-merged-checkpoint100/memory.sqlite3

Without SPARK_ENGRAM_HOME, the root is $XDG_STATE_HOME/spark_engram, falling back to ~/.local/state/spark_engram. Set SPARK_ENGRAM_HOME before the model's first inference call, or provide memory_config.storage_path in the configuration before loading, to select a deployment location.

Ordinary forward and generate calls restore, query, and commit memory automatically. A call without KV starts a fresh sequence boundary while retaining durable associations, preventing n-grams from crossing unrelated documents. Successful inference commits SQLite before returning. Training mode bypasses durable storage. A single observation can remain in probation and does not imply guaranteed recall.

save_pretrained preserves model weights and memory configuration. SQLite is a separate file; copying the checkpoint alone does not transfer previously committed associations. Reloading the same checkpoint with the same bank location restores them.

The verified BPE example observes a 17-token source twice, reloads the checkpoint, and supplies only its 13-token prefix:

text = "Archive key hazel-47: the recorded access phrase is cobalt lantern."
ids = tokenizer(text, add_special_tokens=False, return_tensors="pt").input_ids.cuda()
with torch.inference_mode():
    model(ids, use_cache=False)
    model(ids, use_cache=False)
del model
torch.cuda.empty_cache()
model = AutoModelForCausalLM.from_pretrained(
    model_dir, trust_remote_code=True,
    dtype=torch.bfloat16, device_map={"": "cuda:0"},
).eval()
prefix = ids[:, :-4]
with torch.inference_mode():
    answer = model.generate(prefix, max_new_tokens=4, do_sample=False)
print(tokenizer.decode(answer[0, prefix.shape[1]:]))  # " cobalt lantern."

Beam and multiple-return semantics

Each original source prompt is observed once per generation call. Its committed table snapshot is shared by that source's beams or returns. Each decoded sequence has independent history, convolution buffers, calibration, pending predictions, and cursor state, synchronized with KV repeat/select/reorder operations. Different source lanes use separate banks. Model-generated continuations do not become new memory evidence by default.

Greedy decoding, beam size 2, and sampling with two returns were verified. A lower-ranked beam hypothesis may differ from the best hypothesis. Batched generation uses left padding, with every source ending in a visible token; ordinary forward/SFT supports right padding. Generation requires KV and currently accepts the default/dynamic cache backend.

LoRA source and reproducible merge

The source adapter is the formal step-100 checkpoint. LLaMA-Factory 0.9.6.dev0 SFT used rank 16, alpha 32, targets q_k_v_proj,out_proj, and modules_to_save=["engram"]. The two sources were claude_opus_4_6_traceinversion_qwen35 (8,669 records) and qwen38_27b_fp8_distillation_40k (40,000 records). A prompt-group 95/5 split yielded 46,183 training and 2,486 holdout records.

The fixed run used 100 optimizer steps, BF16, cutoff 1024, packing=false, batch 1 with accumulation 4, learning rate 5e-5, cosine schedule, 5% warmup, non-reentrant gradient checkpointing, and seed/data seed 72001. It trained 10,606,981 parameters, including all Engram parameters. This artifact is a LoRA merge, with the remaining backbone tensors included in full. See the training report.

The merge loads the original backbone on CUDA, replaces all 17 initialized Engram tensors with the adapter's trained active module, and merges 72 LoRA target layers. PEFT unloading retains the active modules_to_save module. All 307 independent weight tensors are saved; embedding and LM head remain tied.

A portable reproduction of the essential merge is:

pip install peft==0.18.1 transformers==5.8.0 accelerate==1.11.0

from pathlib import Path
import torch
from peft import PeftModel
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer

base_dir = "<BASE_MODEL_DIR>"  # V2-integrated Spark backbone with auto_map
adapter_dir = "<ADAPTER_DIR>"  # trained formal checkpoint-100
output_dir = Path("<OUTPUT_DIR>")
assert torch.cuda.is_available()
if output_dir.exists():
    raise FileExistsError("Use a fresh output directory")
config = AutoConfig.from_pretrained(base_dir, trust_remote_code=True)
config.memory_config = dict(
    config.memory_config, enable=True, mode="expandable", lifecycle="auto",
    storage_path=None, namespace="Spark-X2.5-4B-v2-merged-checkpoint100",
)
base = AutoModelForCausalLM.from_pretrained(
    base_dir, config=config, trust_remote_code=True,
    dtype=torch.float32, device_map={"": "cuda:0"},
)
adapted = PeftModel.from_pretrained(base, adapter_dir, is_trainable=False)
merged = adapted.merge_and_unload(safe_merge=True)
merged.save_pretrained(output_dir, safe_serialization=True, max_shard_size="2GB")
AutoTokenizer.from_pretrained(adapter_dir, trust_remote_code=True).save_pretrained(output_dir)

The source backbone's V2 Python dependencies must accompany the saved auto_map; this checkpoint includes that complete dependency bundle. The original base weights are read only, and the output uses a separate directory. safe_merge checks the LoRA merge for nonfinite values; reload validation additionally checks model loading and actual outputs.

Verified results

Measurements were recorded on NVIDIA Blackwell 96GB using CUDA PyTorch. All model parameters were on cuda:0. Both BF16 reloads had zero missing, unexpected, or mismatched keys and zero loading errors.

Check	Observed result
Complete model	4,113,838,981 parameters; 307 independent tensors; 17 Engram tensors
Merge integrity	All 72 LoRA deltas nonzero; sampled base weights changed; all trained Engram tensors loaded with zero difference
Forward	Finite logits [1,17,131072]; range -41.204169 to -0.001044
General greedy execution	16 output tokens; starts with Paris.\nující, followed by newlines; complete output is preserved in observe.json
Exact-prefix recall	Two observations per record, then a different GPU process reload: 32/32, 100%, zero failures
Four-token continuation	Greedy 1/1; beam 2 1/1; top-k-1 sampling with two returns 2/2, all cobalt lantern.
Reader behavior	Independent transient state, shared same-source snapshot; 13 source-clock tokens added per call

The 32 fixtures use actual BPE encoding of Registry entry #NNNN => <word>. Each final eight-token prefix address is unique, and the single-token answer is fixed in advance. The observing process exits before the query process loads the complete model and receives only prefixes. This establishes exact-prefix recall for those fixtures. General-language quality was not scored in the merge verification.

Phase	Peak allocated GiB	Peak reserved GiB
CUDA FP32 merge and save	15.599	15.828
CUDA BF16 reload, forward, greedy, two observation passes	7.767	7.781
Separate CUDA reload, 32 recalls, multi-sequence generation	7.752	7.783

These are per-process PyTorch allocator peaks, excluding other processes' device occupancy. Anonymized evidence is available in 4B merged results. It includes merge, forward, recall, per-record predictions, and the original output-serialization error and recovery. That error occurred after successful observation and result persistence; observations were not repeated.

Limitations

• Retrieval uses exact-token prefix/n-gram addresses. Paraphrases and semantic QA are not guaranteed by these experiments.
• SQLite banks use model/tokenizer namespace and positional source lane. They do not provide user authentication or cross-user isolation. Cooperating writers serialize transactions.
• Speculative/assisted decoding, prompt lookup, self-speculative decoding, and KV crop lack transactional Engram cursor rollback and are rejected.
• Bounded admission can reject useful new facts. The 256 MiB limit applies only to the SQLite main file.
• max_position_embeddings=1048576 remains the backbone configuration; million-token GPU inference, quality, and throughput were not measured.
• Dynamic memory uses Python/SQLite processing. TP/DP sharding, fused memory kernels, and alternative cache backends were not validated.