ffurfaro/Titans-OLMo-1B-hf

Titanesque-OLMo-1B-hf

Titanesque version of allenai/OLMo-1B-hf with parallel linearized attention (TPTT 😊) and PEFT.

The architecture was presented in the paper TPTT: Transforming Pretrained Transformers into Titans.

Abstract

Transformer-based large language models (LLMs) have achieved strong performance across many natural language processing tasks. Nonetheless, their quadratic computational and memory requirements, particularly in self-attention layers, pose challenges for efficient inference on long contexts and for deployment in resource-limited environments. We present TPTT (Transforming Pretrained Transformers into Titans), a framework designed to augment pretrained Transformers with linearized attention (LiZA) and internal memory gating via Memory as Gate (MaG), applied without full retraining. TPTT supports parameter-efficient fine-tuning (LoRA) and integrates with standard toolkits such as Hugging Face Transformers. We evaluated TPTT on several pretrained models, including Llama-1B, OlMoE-1B-7B, Qwen2.5-1.5B, Gemma3-270m, OpenELM-1.3B, and Mistral-7B, in order to assess applicability across architectures of different scales. Experiments on models with approximately 1 billion parameters, evaluated primarily on the MMLU benchmark, suggest potential improvements in both efficiency and accuracy compared to baseline models. For example, Titans-Llama-1B exhibited up to a 20% relative increase in Exact Match scores in one-shot evaluation. An additional finding is that it is possible to convert a quadratic-attention model into a purely linear-attention model using the DeltaProduct mechanism. All training runs were carried out with modest computational resources. These preliminary findings indicate that TPTT may help adapt pretrained LLMs for long-context tasks with limited overhead. Further studies on larger models and a broader set of benchmarks will be necessary to evaluate the generality and robustness of the framework. Code is available at this https URL . Python package at this https URL .

Model list

Classic model parameter with LiZA injection :

Subfolder	Max Self Attn Length	Mag Weight	Cross Gate	Max Chunk Size	Bidirectional	LoRA	Description
delta_rule	8192 (default)	0.5	False	64	False	Yes	Parallel linearized attention with delta_rule operator
delta_rule_gelu	8192 (default)	0.5	False	64	False	Yes	Non-linear operator with gelu activation
delta_product	8192 (default)	0.5	False	64	False	Yes	Second order operator with derivative trick
delta_product_r	8192 (default)	0.5	False	64	False	Yes	Second order operator with rotative trick
delta_product_c	8192 (default)	0.5	False	64	False	Yes	Second order operator with combined trick

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
"ffurfaro/Titanesque-OLMo-1B-hf",
subfolder="tptt_subfolder", # see in repo tree
trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("ffurfaro/allenai/OLMo-1B-hf")

prompt = "Your prompt here"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs, skip_special_tokens=True))

Citation & Contact

If you use TPTT in your academic work, please cite Furfaro. For questions or support, please open an issue on the GitHub repository or contact the maintainer.

---