Gabriel2502/deepseek-coder-33b-gclc-lava-v5

GCLC LAVA v5 — deepseek-coder-33b-instruct

LAVA (Latent Action with Variational Alignment) fine-tuned model untuk GCLC geometry code generation.

Model Info

Komponen	Model
Decoder (base)	deepseek-ai/deepseek-coder-33b-instruct
Encoder (backbone)	BAAI/bge-m3
LoRA-1 (merged)	Gabriel2502/deepseek-coder-33b-gclc-lora-v4
LoRA-2 (trainable)	dalam repo ini (decoder/)

Struktur Repo

deepseek-coder-33b-gclc-lava-v5/
├── encoder/                        ← LightweightEncoder (Stage-1)
│   ├── backbone/                   ← BAAI/bge-m3 fine-tuned
│   │   ├── config.json
│   │   ├── model.safetensors       ← ~1083 MB
│   │   └── ...
│   ├── projection.pt               ← Linear(1024→7168) weights
│   └── encoder_config.json         ← enc_hidden, dec_hidden, enc_model_id
│
├── decoder/                        ← LoRA-2 adapter (Stage-2)
│   ├── adapter_config.json
│   └── adapter_model.safetensors   ← ~235 MB
│
└── training_log_lava_v5.json

Cara Load & Inferensi

import torch, json, os
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoModel
from peft import PeftModel
from huggingface_hub import snapshot_download
import torch.nn as nn

HF_REPO = "Gabriel2502/deepseek-coder-33b-gclc-lava-v5"
DEVICE  = torch.device("cuda:0")

# 1. Download semua files
local_dir = snapshot_download(HF_REPO)

# 2. Load encoder
class LightweightEncoder(nn.Module):
    def __init__(self, save_dir, device):
        super().__init__()
        with open(os.path.join(save_dir, "encoder_config.json")) as f:
            cfg = json.load(f)
        self.backbone = AutoModel.from_pretrained(
            os.path.join(save_dir, "backbone"),
            trust_remote_code=True,
            torch_dtype=torch.bfloat16,
            device_map={"": str(device)},
        )
        self.projection = nn.Linear(cfg["enc_hidden"], cfg["dec_hidden"], bias=False)
        self.projection.load_state_dict(
            torch.load(os.path.join(save_dir, "projection.pt"), map_location=device))
        self.projection = self.projection.to(device=device, dtype=torch.bfloat16)

encoder = LightweightEncoder(os.path.join(local_dir, "encoder"), DEVICE)
encoder.eval()

# 3. Load decoder
base = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/deepseek-coder-33b-instruct",
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    device_map={"": str(DEVICE)},
)
decoder = PeftModel.from_pretrained(base, os.path.join(local_dir, "decoder"))
decoder.eval()

# 4. Load tokenizers
from transformers import AutoTokenizer
enc_tok = AutoTokenizer.from_pretrained("BAAI/bge-m3", trust_remote_code=True)
dec_tok = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-33b-instruct", trust_remote_code=True)

Hyperparameters

Stage	LR	Epochs	Batch (eff)
Stage-1 (encoder)	5e-05	10	16
Stage-2 (decoder)	2e-05	20	16

• Max sequence length: 512
• Latent tokens: max 64, compression 1:4
• Sparse KL labels: top-64
• LoRA r=8, alpha=16, targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Version

v5-complete-fixed — Memory fragmentation fix + complete encoder upload