cczzzyyy/DrugSpace-openfda-lora-eval

DrugSpace OpenFDA LoRA Eval

DrugSpace-openfda-lora-eval is the evaluation-oriented Stage II contrastive-alignment adapter for DrugSpace. It was trained on OpenFDA-derived single-ingredient drug-label text and must be loaded on top of cczzzyyy/DrugSpace-mntp-eval-8B.

This repository contains a PEFT LoRA adapter, not a standalone base model. The eval designation identifies the pre-2020 MNTP base model used for initialization; the OpenFDA Stage II data itself was not temporally split.

Usage

import torch
from llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained(
    base_model_name_or_path="cczzzyyy/DrugSpace-mntp-eval-8B",
    peft_model_name_or_path="cczzzyyy/DrugSpace-openfda-lora-eval",
    enable_bidirectional=True,
    pooling_mode="mean",
    max_length=512,
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

texts = [
    "A complete English description of a drug.",
    "Another drug description for comparison.",
]
embeddings = model.encode(texts, show_progress_bar=True)
print(embeddings.shape)

The saved llm2vec_config.json uses mean pooling, a maximum sequence length of 512, a document maximum length of 400, and skip_instruction=true.

Model details

• Base model: cczzzyyy/DrugSpace-mntp-eval-8B
• Model type: PEFT LoRA adapter for bidirectional LLM2Vec embeddings
• Task: drug-description embedding, similarity, and retrieval
• Language: English
• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
• Trainable parameters: 41,943,040

Training data

The adapter was trained from OpenFDA-derived single-ingredient labeling records. The text views were drawn from the following label sections when available:

• indications and usage
• clinical pharmacology
• mechanism of action
• pharmacodynamics
• pharmacokinetics
• contraindications
• warnings and cautions
• drug interactions
• adverse reactions

The processed source contained 1,867 records. Of these, 1,857 records with at least two distinct non-empty text views were eligible for contrastive training.

Training procedure

• Fixed training set constructed once with seed 42 and reused for all 20 epochs
• 3,714 constructed triplets; 3,712 used per epoch after dropping the final incomplete batch
• Global batch size 64
• Learning rate 1e-4
• 300 warm-up steps
• Maximum sequence length 512
• bfloat16 training
• Final training step used as the released checkpoint; no alignment validation split or early stopping

Intended use

This adapter is intended for evaluation-oriented research involving English drug descriptions, including embedding generation, semantic similarity, and retrieval. Inputs should be complete, clinically meaningful descriptions and should use a consistent writing style when results are compared.

Limitations

• This adapter is not a standalone model and requires the exact MNTP evaluation base model listed above.
• The OpenFDA Stage II training data was not restricted to pre-2020 records.
• It is intended for representation learning, not text generation.
• Training data and label-section availability may introduce coverage and documentation biases.
• Similarity in the embedding space does not establish therapeutic equivalence, safety, efficacy, or causal relationships.
• The model is for research use and must not be used as a substitute for professional medical judgment.

License

Use of this adapter and its base model is subject to the applicable Llama 3.1 license and acceptable-use terms.