| --- |
| base_model: Codemaster67/Olmo-7b-spe |
| datasets: |
| - Codemaster67/Causal_lm_chemistry_1M_rows |
| language: en |
| library_name: transformers |
| license: apache-2.0 |
| tags: |
| - chemistry |
| - smiles |
| - olmo |
| - causal-lm |
| - full-finetune |
| - fsdp |
| --- |
| |
| # OLMo-7B Full Fine-Tune — Chemistry SMILES CPT |
|
|
| ## Model Description |
|
|
| This model is a **full-parameter fine-tuned** version of |
| [Codemaster67/Olmo-7b-spe](https://huggingface.co/Codemaster67/Olmo-7b-spe) trained on chemistry |
| SMILES strings from the |
| [Codemaster67/Causal_lm_chemistry_1M_rows](https://huggingface.co/datasets/Codemaster67/Causal_lm_chemistry_1M_rows) dataset. |
|
|
| The base model's tokenizer was pre-extended with ~300 SPE (SMILES Pair |
| Encoding) chemistry tokens plus `<|start_of_smiles|>` / `<|end_of_smiles|>` |
| special tokens, and its embedding & LM-head layers were resized with |
| mean-initialised vectors for the new tokens. |
|
|
| ## Training Details |
|
|
| | Parameter | Value | |
| |---|---| |
| | **Method** | Full Fine-Tune (all weights updated) | |
| | **Parallelism** | FSDP (Fully Sharded Data Parallel) | |
| | **Epochs** | 1 | |
| | **Learning Rate** | 5e-06 | |
| | **Batch Size (per device)** | 16 | |
| | **Gradient Accumulation** | 1 | |
| | **Max Sequence Length** | 512 | |
| | **Warmup Ratio** | 0.1 | |
| | **Weight Decay** | 0.01 | |
| | **Scheduler** | Cosine | |
| | **Precision** | bf16 | |
| | **Augmentation** | OFF | |
| | **Training Samples** | 250000 | |
| | **Eval Samples** | 25000 | |
|
|
| ## Evaluation Results |
|
|
| | Metric | Value | |
| |---|---| |
| | **Final Eval Loss** | 0.9727568626403809 | |
| | **Final Eval Perplexity** | 2.645226943673604 | |
| | **Training Loss** | 1.1177 | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("harindhar10/olmo_chem_lora_cpt_LoRA_500k", trust_remote_code=True) |
| tokenizer = AutoTokenizer.from_pretrained("harindhar10/olmo_chem_lora_cpt_LoRA_500k", trust_remote_code=True) |
| |
| smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>" |
| inputs = tokenizer(smiles_input, return_tensors="pt") |
| outputs = model.generate(**inputs, max_new_tokens=128) |
| print(tokenizer.decode(outputs[0], skip_special_tokens=False)) |
| ``` |
|
|
| ## Intended Use |
|
|
| Chemistry-domain language modelling, SMILES generation and completion, |
| and downstream molecular property prediction via fine-tuning. |
|
|
| ## Limitations |
|
|
| - Trained primarily on SMILES strings; natural-language instruction-following |
| ability may degrade compared to the base OLMo checkpoint. |
| - Augmentation was disabled for this run. |
|
|
