Create README.md

README.md

@@ -0,0 +1,100 @@
+---
+language:
+- en
+license: mit
+library_name: transformers
+tags:
+- materials-science
+- crystallography
+- generative-ai
+- inverse-design
+- chemistry
+- unconditional
+datasets:
+- c-bone/lematerial_clean
+pipeline_tag: text-generation
+---
+
+# Model Card for CrystaLLM-pi_base
+
+## Model Details
+
+### Model Description
+
+**CrystaLLM-pi_base** is an unconditional generative model designed for the generation of valid inorganic crystal structures. It serves as the foundational pre-trained model for the `CrystaLLM-pi` framework. Based on a GPT-2 decoder-only architecture, it is trained on a large corpus of Crystallographic Information Files (CIFs) to learn the syntax, symmetry, and chemical rules governing crystalline matter.
+
+This model does not accept property conditioning vectors. It generates structures based on text prompts (e.g., chemical composition or space group) or unconditionally (ab-initio generation).
+
+- **Developed by:** Bone et al. (University College London)
+- **Model type:** Autoregressive Transformer (GPT-2)
+- **Language(s):** CIF (Crystallographic Information File) syntax
+- **License:** MIT
+
+### Model Sources
+
+- **Repository:** [GitHub: CrystaLLM-pi](https://github.com/C-Bone-UCL/CrystaLLM-pi)
+- **Paper:** [Discovery and recovery of crystalline materials with property-conditioned transformers (arXiv:2511.21299)](https://arxiv.org/abs/2511.21299)
+- **Dataset:** [HuggingFace: c-bone/lematerial_clean](https://huggingface.co/datasets/c-bone/lematerial_clean)
+
+## Uses
+
+### Direct Use
+
+The model is intended for:
+1. **Unconditional Generation:** Exploring the general chemical space of stable crystals.
+2. **Composition/Space Group Completion:** Generating valid structures given a partial prompt (e.g., a chemical formula).
+3. **Fine-tuning base:** Serving as the pre-trained initialization for property-conditional models (like `CrystaLLM-pi_bandgap` or `CrystaLLM-pi_density`).
+
+### Out-of-Scope Use
+
+- **Property Conditioning:** This model cannot be steered by properties like band gap or density. Use the specific fine-tuned variants for those tasks.
+- **Large Unit Cells:** Context window limit of 1024 tokens (~20 atoms/cell).
+
+## Bias, Risks, and Limitations
+
+- **Training Distribution:** The model reflects the biases present in the LeMaterial dataset. It is most effective at generating structures similar to known stable inorganic compounds.
+- **Validity:** While it learns CIF syntax robustly, it may still generate physically invalid structures (e.g., overlapping atoms) or chemically unstable compositions.
+
+## How to Get Started with the Model
+
+For instructions on how to load and run generation with this model, please refer to the `_load_and_generate.py` script in the [CrystaLLM-pi GitHub Repository](https://github.com/C-Bone-UCL/CrystaLLM-pi).
+
+## Training Details
+
+### Training Data
+
+The model was pre-trained on the **LeMaterial** dataset (specifically `c-bone/lematerial_clean`), a large-scale collection of ~4.35 million augmented CIFs derived from major materials databases.
+
+- **Source:** LeMaterial (via `c-bone/lematerial_clean`)
+- **Preprocessing:** CIFs are deduplicated, augmented (with symmetry operations), and tokenized.
+
+### Training Procedure
+
+- **Architecture:** GPT-2 Small (~25.9M parameters).
+- **Objective:** Causal Language Modeling (Next-token prediction).
+- **Loss Function:** Cross-entropy with specific weighting for fixed syntax tokens to accelerate learning of the CIF format.
+
+## Evaluation
+
+### Metrics
+
+The model is evaluated based on:
+1. **Validity:** The rate at which generated sequences can be parsed as valid CIF files.
+2. **Structural Consistency:** Adherence to space group symmetry and reasonable bond lengths.
+
+### Results
+
+The base model achieves high validity rates and effectively learns to generate chemically plausible structures, serving as a robust foundation for downstream property-conditioning tasks.
+
+## Citation
+
+```bibtex
+@misc{bone2025discoveryrecoverycrystallinematerials,
+      title={Discovery and recovery of crystalline materials with property-conditioned transformers}, 
+      author={Cyprien Bone and Matthew Walker and Kuangdai Leng and Luis M. Antunes and Ricardo Grau-Crespo and Amil Aligayev and Javier Dominguez and Keith T. Butler},
+      year={2025},
+      eprint={2511.21299},
+      archivePrefix={arXiv},
+      primaryClass={cond-mat.mtrl-sci},
+      url={[https://arxiv.org/abs/2511.21299](https://arxiv.org/abs/2511.21299)}, 
+}