Maxenith/MAX-Neural-500M-Arabic

MAX Neural 500M Instruct (Arabic)

MAX Neural 500M Instruct is a dedicated Arabic language model built from scratch using raw PyTorch. It is specifically fine-tuned to follow instructions and answer questions in Arabic.

📊 Model Details

• Parameters: ~500 Million
• Architecture: Custom Transformer (PyTorch)
• Layers: 24
• Hidden Size: 1024
• Attention Heads: 16 (GQA with 4 KV heads)
• Context Length: 512 tokens
• Vocabulary Size: 32,000 (Custom BPE)
• Format: FP16 (Half Precision) for efficient inference.
• File Size: ~821 MB

🏗️ Architecture Highlights

• RoPE (Rotary Position Embeddings)
• SwiGLU activation function
• RMSNorm for normalization
• Grouped Query Attention (GQA) for memory efficiency

📚 Training Data

• Base Training: Trained on millions of tokens from the Arabic Wikipedia corpus.
• Fine-Tuning (Instruct): Fine-tuned on 30,000 high-quality Arabic instruction-following examples (Alpaca-GPT4 format).

⚠️ Limitations

As a 500M parameter model, it may occasionally hallucinate facts or generate repetitive text. It is highly recommended to use this model in conjunction with a RAG (Retrieval-Augmented Generation) pipeline for factual accuracy.

💻 Usage

The model requires the custom architecture defined in the training script. Load the weights using torch.load() and ensure the tokenizer.json is loaded using the tokenizers library.