🦅 Falcon-7B Model Card (MarkAI Hosted Version)

Model Overview

Falcon-7B is a 7 billion parameter causal decoder-only model developed by Technology Innovation Institute (TII). This repository hosts the original model weights as part of MarkAI's model collection.

Technical Specifications

Architecture

Component	Specification
Model Type	Causal Decoder-only
Attention Mechanism	Multi-Query + FlashAttention
Positional Embeddings	Rotary Positional Embeddings
Normalization	Single LayerNorm per block

Training Details

Parameter	Value
Training Tokens	1,500B (1.5 trillion)
Training Compute	384 × A100 40GB GPUs (P4d instances)
Training Time	~2 weeks
Precision	bfloat16

Usage Examples

Text Generation

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "ibrahimlasfar/MarkAI",
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("ibrahimlasfar/MarkAI")

inputs = tokenizer(
    "The future of artificial intelligence",
    return_tensors="pt"
).to("cuda")
outputs = model.generate(
    **inputs,
    max_length=100,
    do_sample=True,
    top_k=10
)
print(tokenizer.decode(outputs[0]))