raihan-js/orch-nextjs-350m-v2

ORCH 350M

ORCH (Orchestrated Recursive Code Hierarchy) - A code generation model trained from scratch.

Model Details

Property	Value
Parameters	286,966,784
Architecture	Decoder-only Transformer (LLaMA-style)
Hidden Size	1024
Layers	24
Attention Heads	16
KV Heads (GQA)	4
Vocab Size	16000
Max Sequence	4096

Architecture Features

• RoPE (Rotary Position Embeddings)
• GQA (Grouped Query Attention) for efficient inference
• SwiGLU activation function
• RMSNorm for layer normalization
• Tied embeddings (input/output share weights)

Usage

import torch
from tokenizers import Tokenizer
from orch import OrchForCausalLM

# Load model and tokenizer
model = OrchForCausalLM.from_pretrained("raihan-js/orch-nextjs-350m-v2")
tokenizer = Tokenizer.from_file("orch-tokenizer.json")

# Generate code
prompt = 'def fibonacci(n):'
encoding = tokenizer.encode(prompt)
input_ids = torch.tensor([encoding.ids])

output = model.generate(input_ids, max_new_tokens=512, temperature=0.7)
print(tokenizer.decode(output[0].tolist()))

Training

• Trained from scratch on synthetic code data
• Framework: Custom PyTorch implementation
• Hardware: RTX 3060 12GB

Links

• GitHub: raihan-js/orch
• Paper: Coming soon

Citation

@software{orch2025,
  author = {Raihan},
  title = {ORCH: Orchestrated Recursive Code Hierarchy},
  year = {2025},
  url = {https://github.com/raihan-js/orch}
}

License

MIT License