cczzzyyy/DrugSpace-openfda-lora

DrugSpace OpenFDA LoRA

DrugSpace-openfda-lora is the Stage II contrastive-alignment adapter for DrugSpace. It was trained on OpenFDA-derived single-ingredient drug-label text and must be loaded on top of cczzzyyy/DrugSpace-mntp-8B.

This repository contains a PEFT LoRA adapter, not a standalone base model.

Usage

import torch
from llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained(
    base_model_name_or_path="cczzzyyy/DrugSpace-mntp-8B",
    peft_model_name_or_path="cczzzyyy/DrugSpace-openfda-lora",
    enable_bidirectional=True,
    pooling_mode="mean",
    max_length=512,
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

texts = [
    "A complete English description of a drug.",
    "Another drug description for comparison.",
]
embeddings = model.encode(texts, show_progress_bar=True)
print(embeddings.shape)

The saved llm2vec_config.json uses mean pooling, a maximum sequence length of 512, a document maximum length of 400, and skip_instruction=true.

Model details

• Base model: cczzzyyy/DrugSpace-mntp-8B
• Model type: PEFT LoRA adapter for bidirectional LLM2Vec embeddings
• Task: drug-description embedding, similarity, and retrieval
• Language: English
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
• Trainable parameters: 41,943,040

Training data

The adapter was trained from OpenFDA-derived single-ingredient labeling records. The text views were drawn from the following label sections when available:

• indications and usage
• clinical pharmacology
• mechanism of action
• pharmacodynamics
• pharmacokinetics
• contraindications
• warnings and cautions
• drug interactions
• adverse reactions

The processed source contained 1,867 records. Of these, 1,857 records with at least two distinct non-empty text views were eligible for contrastive training.

Training procedure

• Fixed sampling protocol with subset seed 42
• 3,714 constructed triplets; 3,712 used per epoch after dropping the final incomplete batch
• Fixed training set constructed once with seed 42 and reused for all 20 epochs
• Global batch size 64
• Learning rate 1e-4
• 300 warm-up steps
• Maximum sequence length 512
• bfloat16 training
• Final training step used as the released checkpoint; no alignment validation split or early stopping

Intended use

This model is intended for research involving English drug descriptions, including embedding generation, semantic similarity, and retrieval. Inputs should be complete, clinically meaningful descriptions and should use a consistent writing style when results are compared.

Limitations

• This adapter is not a standalone model and requires the exact MNTP base model listed above.
• It is intended for representation learning, not text generation.
• Training data and label-section availability may introduce coverage and documentation biases.
• Similarity in the embedding space does not establish therapeutic equivalence, safety, efficacy, or causal relationships.
• The model is for research use and must not be used as a substitute for professional medical judgment.

License

Use of this adapter and its base model is subject to the applicable Llama 3.1 license and acceptable-use terms.