|
|
--- |
|
|
license: other |
|
|
language: |
|
|
- en |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# Optrix-1-0257 |
|
|
|
|
|
**Optrix-1-0257** is a base 1 billion parameter language model developed by SVECTOR for general-purpose language generation and understanding. Pretrained on a broad corpus, it provides a strong foundation for fine-tuning on tasks such as summarization, dialogue, and retrieval. |
|
|
|
|
|
## Key Features |
|
|
|
|
|
* 1B parameter transformer architecture |
|
|
* Pretrained on a broad, diverse corpus |
|
|
* Optimized for efficient inference and low memory usage |
|
|
* Suitable for fine-tuning on a wide range of language tasks |
|
|
|
|
|
--- |
|
|
|
|
|
## Usage |
|
|
|
|
|
### With Hugging Face Transformers |
|
|
|
|
|
```python |
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("SVECTOR-CORPORATION/Optrix-1-0257") |
|
|
model = AutoModelForCausalLM.from_pretrained("SVECTOR-CORPORATION/Optrix-1-0257") |
|
|
|
|
|
inputs = tokenizer("What is AI?", return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_new_tokens=64) |
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
|
|
### With llama.cpp (GGUF Format) |
|
|
|
|
|
```sh |
|
|
./main -m Optrix-1-0257 -p "What is AI?" |
|
|
``` |
|
|
|
|
|
> Ensure the model is available in GGUF format before inference. |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Specifications |
|
|
|
|
|
**Developer:** SVECTOR <br> |
|
|
**Architecture:** Custom transformer with Grouped-Query Attention (GQA) <br> |
|
|
**Embedding Dimension:** 2048 <br> |
|
|
**Layers:** 16 <br> |
|
|
**Attention Heads:** 32 <br> |
|
|
**Vocabulary Size:** 128,256 <br> |
|
|
**Max Position Embeddings:** 131,072 <br> |
|
|
**Positional Encoding:** Rotary with dynamic scaling <br> |
|
|
**Activation Function:** GELU <br> |
|
|
**Output Head:** Tied linear projection <br> |
|
|
**Languages:** English, German, French, Spanish, Hindi, Portuguese, Thai, Italian, and others <br> |
|
|
**Release Date:** June 27, 2025 <br> |
|
|
|
|
|
--- |
|
|
|
|
|
## Architecture Overview |
|
|
|
|
|
* **Embedding Layer:** `nn.Embedding(vocab_size, hidden_size)` |
|
|
* **Transformer Block (×16):** |
|
|
|
|
|
* `nn.MultiheadAttention(batch_first=True)` |
|
|
* 2-layer MLP with GELU activation |
|
|
* LayerNorm (pre-attention and pre-MLP) |
|
|
* **Final LayerNorm** |
|
|
* **Output Layer:** `nn.Linear(hidden_size, vocab_size, bias=False)` |
|
|
* **Causal Masking:** Left-to-right for autoregressive generation |
|
|
* **Rotary Embeddings:** Applied with dynamic scaling |
|
|
|
|
|
--- |
|
|
|
|
|
## Example Configuration |
|
|
|
|
|
```json |
|
|
{ |
|
|
"architectures": ["OptrixForCausalLM"], |
|
|
"hidden_size": 2048, |
|
|
"num_hidden_layers": 16, |
|
|
"num_attention_heads": 32, |
|
|
"vocab_size": 128256, |
|
|
"max_position_embeddings": 131072, |
|
|
"model_type": "Optrix-1-0257" |
|
|
} |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## License & Contact |
|
|
|
|
|
Use of this model is governed by the **SVECTOR License**. |
|
|
For inquiries, please contact [SVECTOR](https://svector.co.in). |
