ysuz/Mistral-Nemo-Base-2407-bandgap

About this model

This is a fine-tuned version of mistralai/Mistral-Nemo-Base-2407 for materials property prediction.
The model was trained to predict bandgaps using the Materials Project dataset.
Please refer to the example snippet for input templates and usage.

This model is part of the hands-on materials for DxMT AIMHack 2024.
For scripts related to fine-tuning and data retrieval, please visit the GitHub repository: llm4material tutorial.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


model_id = "ysuz/Mistral-Nemo-Base-2407-bandgap"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id,
                                             device_map="auto",
                                             torch_dtype=torch.float16,
                                            )

# example of input context
structure_text = """
Reduced Formula: BaSrI4
abc   :   5.807091   5.807091   8.251028
angles:  90.000000  90.000000  90.000000
pbc   :       True       True       True
space group: ('P4/mmm', 123)
Sites (6)
  #  SP      a    b         c    magmom
  0  Ba    0.5  0.5  0               -0
  1  Sr    0    0    0.5             -0
  2  I     0    0.5  0.257945         0
  3  I     0.5  0    0.257945         0
  4  I     0    0.5  0.742055         0
  5  I     0.5  0    0.742055         0

Output:
"""

prompt = f"Instruction: What is the bandgap value of following material?:\n{structure_text}\n\nOutput:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    tokens = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.5,
        top_p=0.9,
        repetition_penalty=1.05,
    )
generated_text = tokenizer.decode(tokens[0], skip_special_tokens=True)
print(f"Generated raw text:\n{generated_text}\n\n")

Model performance

This model delivers an MAE of 0.33 on the test set, which was randomly split from the Materials Project dataset, with 113,568 instances for training and 12,618 for testing. The results show promise as a baseline for materials property prediction models based on Large Language Models. For comparison, one of the current state-of-the-art models, CrystalFormer, achieves an MAE of around 0.20 eV using 60k training data from the Materials Project dataset.