README.md

---
license: apache-2.0
language:
  - en
pipeline_tag: text-generation
tags:
  - tfa
  - tri-flux-attention
  - tnsa
  - moe
  
extra_gated_prompt: |
  1. LICENSE DEFINITIONS & SCOPE
       1.1 Core Definitions
       1.1.1 "OpenWeight Models" means AI models where TNSA has made model weights, parameters, and architecture publicly available under open licensing terms.
       1.1.2 "Model Weights" means the learned parameters, coefficients, and numerical values that define the behavior and capabilities of an AI model.
       1.1.3 "Model Architecture" means the structural design, layer configuration, and computational framework of an AI model.
       1.1.4 "Training Code" means software, scripts, and procedures used to train, fine-tune, or modify AI models.
       1.1.5 "Inference Code" means software required to load, execute, and generate outputs from AI models.
       1.1.6 "Model Card" means documentation describing model capabilities, limitations, training data, and intended use cases.
       1.1.7 "Derivative Model" means any AI model created by modifying, fine-tuning, or building upon TNSA OpenWeight Models.
       1.1.8 "Commercial Use" means any use of OpenWeight Models for commercial purposes, including but not limited to revenue generation, business operations, or competitive advantage.
       1.1.9 "Research Use" means use for academic research, scientific investigation, or educational purposes without commercial intent.
       1.1.10 "Distribution" means making OpenWeight Models or derivatives available to third parties through any means.
       1.2 License Scope
       1.2.1 This license applies to all TNSA OpenWeight Models and associated materials made available under open licensing terms.
       1.2.2 The license covers model weights, architectures, training code, inference code, and documentation.
       1.2.3 Different license terms may apply to different model versions or releases.
       1.2.4 Supplemental terms may apply to specific models or use cases as indicated in model documentation.
       1.2.5 This license does not grant rights to TNSA trademarks, service marks, or proprietary branding.
       1.2.6 Training data used to create OpenWeight Models may be subject to separate licensing terms.
       1.2.7 Third-party components integrated into OpenWeight Models retain their original licensing terms.
       1.2.8 Geographic restrictions may apply based on export control laws and regulations.
       1.2.9 Temporal limitations may apply to certain license grants or model versions.
       1.2.10 Updates and modifications to licensing terms will be communicated through official TNSA channels.
       1.3 License Types
       1.3.1 "TNSA Open License" - Permissive license allowing broad commercial and research use with attribution.
       1.3.2 "TNSA Research License" - License restricted to non-commercial research and educational use.
       1.3.3 "TNSA Community License" - License for community-driven development with share-alike provisions.
       1.3.4 "TNSA Evaluation License" - Time-limited license for evaluation and testing purposes.
       1.3.5 "TNSA Custom License" - Negotiated license terms for specific use cases or organizations.
   2. COVERED MODEL CATEGORIES
       2.1 NGen Series Models
       2.1.1 NGen3 OpenWeight Models - Latest generation large language models with full commercial licensing.
       2.1.2 NGen2 OpenWeight Models - Previous generation models with continued support and licensing.
       2.1.3 NGen Base Models - Foundation models suitable for fine-tuning and specialization.
       2.1.4 NGen Instruct Models - Instruction-tuned variants optimized for conversational use.
       2.1.5 NGen Code Models - Specialized variants trained for code generation and programming tasks.
       2.1.6 NGen Reasoning Models - Enhanced variants with improved logical reasoning capabilities.
       2.1.7 NGen Multimodal Models - Models capable of processing text, images, and other modalities.
       2.1.8 NGen Domain Models - Specialized models for specific industries or use cases.
       2.1.9 NGen Efficient Models - Optimized variants for resource-constrained environments.
       2.1.10 NGen Experimental Models - Research variants with novel architectures or capabilities.
       2.2 IGen Series Models
       2.2.1 IGen 1 Nano Models - Compact image generation models for edge deployment.
       2.2.2 IGen Base Models - Foundation models for image synthesis and manipulation.
       2.2.3 IGen Style Models - Specialized models for artistic and stylistic image generation.
       2.2.4 IGen Photo Models - Photorealistic image generation models with high fidelity output.
       2.2.5 IGen Edit Models - Models specialized for image editing and modification tasks.
       2.2.6 IGen Upscale Models - Super-resolution models for image enhancement and upscaling.
       2.2.7 IGen Inpaint Models - Models for image inpainting and completion tasks.
       2.2.8 IGen Control Models - Models with enhanced controllability and conditioning options.
       2.2.9 IGen Fast Models - Optimized models for rapid image generation with reduced latency.
       2.2.10 IGen Research Models - Experimental image generation models for research purposes.
       2.3 Specialized Model Categories
       2.3.1 Stellar v2 Models - Advanced reasoning and analysis models for complex problem-solving.
       2.3.2 Audio Generation Models - Models for speech synthesis, music generation, and audio processing.
       2.3.3 Video Generation Models - Models for video synthesis, editing, and temporal content creation.
       2.3.4 Embedding Models - Models for generating vector representations of text, images, and other data.
       2.3.5 Classification Models - Models for content classification, sentiment analysis, and categorization.
       2.3.6 Translation Models - Models for language translation and cross-lingual understanding.
       2.3.7 Summarization Models - Models specialized for text summarization and content condensation.
       2.3.8 Question Answering Models - Models optimized for factual question answering and information retrieval.
       2.3.9 Safety Models - Models for content moderation, safety classification, and harm detection.
       2.3.10 Evaluation Models - Models for assessing quality, accuracy, and performance of other AI systems.
   3. GRANTED PERMISSIONS
       3.1 Usage Rights
       3.1.1 Use OpenWeight Models for inference, prediction, and output generation.
       3.1.2 Deploy models in production environments for commercial and non-commercial purposes.
       3.1.3 Integrate models into applications, services, and platforms.
       3.1.4 Process proprietary and confidential data through licensed models.
       3.1.5 Scale usage according to computational resources and business needs.
       3.1.6 Use models across multiple geographic regions and jurisdictions.
       3.1.7 Combine multiple TNSA models in integrated solutions.
       3.1.8 Use models for both batch processing and real-time inference.
       3.1.9 Implement custom inference optimizations and performance enhancements.
       3.1.10 Use models in research, development, and experimental applications.
       3.2 Modification Rights
       3.2.1 Fine-tune models on custom datasets for specialized applications.
       3.2.2 Modify model architectures for performance or efficiency improvements.
       3.2.3 Quantize, compress, or optimize models for specific hardware platforms.
       3.2.4 Merge or ensemble multiple models for enhanced capabilities.
       3.2.5 Extract and use individual model components or layers.
       3.2.6 Adapt models for different programming languages or frameworks.
       3.2.7 Create domain-specific variants through transfer learning.
       3.2.8 Implement custom training procedures and optimization techniques.
       3.2.9 Modify input/output interfaces and data preprocessing pipelines.
       3.2.10 Develop novel applications and use cases based on model capabilities.
       3.3 Distribution Rights
       3.3.1 Distribute unmodified OpenWeight Models with proper attribution.
       3.3.2 Share derivative models created through permitted modifications.
       3.3.3 Include models in open-source projects and repositories.
       3.3.4 Distribute models through academic and research channels.
       3.3.5 Package models with applications and commercial products.
       3.3.6 Provide models to customers, partners, and collaborators.
       3.3.7 Host models on cloud platforms and model repositories.
       3.3.8 Create and distribute model variants for different use cases.
       3.3.9 Share models within organizations and affiliated entities.
       3.3.10 Contribute models to community projects and initiatives.
   4. USAGE RESTRICTIONS
       4.1 Prohibited Uses
       4.1.1 Using models to generate illegal content or facilitate criminal activities.
       4.1.2 Creating deepfakes or synthetic media intended to deceive or harm individuals.
       4.1.3 Developing surveillance systems that violate privacy rights or human dignity.
       4.1.4 Training models on data obtained without proper consent or legal authorization.
       4.1.5 Using models to discriminate against protected groups or individuals.
       4.1.6 Generating content that promotes violence, hatred, or extremist ideologies.
       4.1.7 Creating systems designed to manipulate democratic processes or elections.
       4.1.8 Using models for military weapons development or autonomous weapons systems.
       4.1.9 Developing applications that could cause mass harm or societal disruption.
       4.1.10 Reverse engineering models to extract proprietary training data or methodologies.
       4.2 Technical Restrictions
       4.2.1 Removing or obscuring attribution notices, copyright statements, or license information.
       4.2.2 Circumventing built-in safety measures, content filters, or usage monitoring.
       4.2.3 Attempting to extract or reconstruct training data from model weights.
       4.2.4 Using models in ways that exceed specified computational or usage limits.
       4.2.5 Modifying models to remove safety guardrails or ethical constraints.
       4.2.6 Distributing models without required documentation or safety information.
       4.2.7 Using models in safety-critical applications without proper validation.
       4.2.8 Combining models with malicious code or harmful software components.
       4.2.9 Implementing models in ways that violate applicable privacy regulations.
       4.2.10 Using models to create competing AI services that directly replicate TNSA offerings.
       4.3 Commercial Restrictions
       4.3.1 Certain models may require separate commercial licensing for revenue-generating use.
       4.3.2 High-volume commercial usage may be subject to additional terms and fees.
       4.3.3 Reselling unmodified models as standalone products is prohibited.
       4.3.4 Using TNSA trademarks or branding without explicit permission is forbidden.
       4.3.5 Creating derivative works that compete directly with TNSA services may be restricted.
extra_gated_fields:
  First Name: text
  Last Name: text
  Date of birth: date_picker
  Country: country
  Affiliation: text
  Job title:
    type: select
    options:
      - Student
      - Research Graduate
      - AI researcher
      - AI developer/engineer
      - Reporter
      - Other
  geo: ip_location
  Purpose of access: text
  By clicking Submit below I accept the terms of the NGen-3 Community License and acknowledge that the information I provide will be collected stored processed and shared in accordance with the TNSA Privacy Policy: checkbox
---

![TFA-80B Logo](https://raw.githubusercontent.com/tnsaai/images-urls/refs/heads/main/tfa-80b.png)

# TFA-80B 

**TFA-80B** is a frontier-scale, constant-memory language model developed by **TNSA**. 

Built as a frontier 80B Mixture-of-Experts architecture, TFA-80B integrates Tri-Flux Attention (TFA)—a groundbreaking recurrent attention mechanism that replaces the standard KV Cache with a fixed-size state matrix. This allows the model to process functionally infinite context lengths with O(1) memory footprint in the augmented layers.

## Architecture
- **Base Model:** TFA-80B (80B parameters, Mixture-of-Experts)
- **TFA Injection:** Tri-Flux Attention is injected into layers `[3, 11, 23, 35, 47]`.
- **Memory Footprint:** The TFA state consumes a constant ~2.5 MiB of VRAM regardless of sequence length, bypassing the massive memory requirements of standard Transformers at 100k+ tokens.
- **Identity:** Unconditionally trained via BPTT to self-identify as TFA-80B by TNSA.

## How it works
TFA is a *hybrid* architecture. In the injected layers, the model runs a parallel pathway:
1. The frozen base attention (Q, K, V, O) processes local, short-term context.
2. The Tri-Flux Attention mechanism processes and compresses long-term context into a recurrent memory matrix S.
3. A learned `scale` parameter dynamically balances the two pathways.

## Usage

Because TFA-80B is a custom hybrid architecture, you cannot load it with a standard AutoModelForCausalLM call without injecting the TFA wrapper.

We have provided a ready-to-use model.py script in this repository. 

### Quickstart

```python
import torch
from transformers import AutoTokenizer
# Import the custom loader from the repository
from model import load_tfa_model

# 1. Load the model and tokenizer
model_id = "TNSA/TFA-80B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# 2. This function automatically downloads the weights, injects TFA, 
#    and loads the trained recurrent states.
model = load_tfa_model(model_id)

# 3. Generate!
prompt = "<|im_start|>user\nWho are you and what makes your architecture special?<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7)

print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Creator
Developed by **TNSA**.