Upload folder using huggingface_hub

README.md

@@ -1,95 +1,18 @@
 ---
-language:
-- en
-tags:
-- causal-lm
-- arxiv
-- research-assistant
-- small-language-model
-datasets:
-- FlameF0X/arXiv-AI-ML
-library_name: transformers
-license: other
-license_name: lfm1.0
-license_link: LICENSE
+language: [en]
+license: apache-2.0
+tags: [pytorch, causal-lm, arxiv, lfm2]
+datasets: [FlameF0X/arXiv-AI-ML]
 ---
+# LFM2 – Pretrained on arXiv AI/ML
 
-# LFM2-Research
-
-**A small causal language model pre-trained on arXiv AI/ML research papers.**
-
-## Overview
-
-LFM2-Research is a compact language model built on the [LFM2 (Liquid Foundation Model 2)](https://github.com/huggingface/transformers/tree/main/src/transformers/models/lfm2) -- i got the code of the model from `src/transformers/models/lfm2`, architecture and pre-trained exclusively on AI and machine learning research papers from arXiv. It is designed as a lightweight research assistant capable of engaging with technical literature in the AI/ML domain.
-
-> ⚠️ This model has not undergone RLHF, instruction tuning, or any alignment procedure. It is a raw pre-trained model and is best suited for experimentation and research purposes.
-
----
-
-## Model Architecture
-
-| Parameter | Value |
+| Param | Value |
 |---|---|
-| Model Type | Causal Language Model |
-| Architecture | LFM2 |
-| Hidden Size | 512 |
+| Hidden size | 512 |
 | Layers | 8 |
-| Attention Heads | 8 |
-| KV Heads | 4 (Grouped Query Attention) |
-| Max Sequence Length | 2048 |
-| Vocabulary Size | 50,257 |
-
----
-
-## Training Details
-
-| Parameter | Value |
-|---|---|
-| Dataset | [`FlameF0X/arXiv-AI-ML`](https://huggingface.co/datasets/FlameF0X/arXiv-AI-ML) |
-| Training Samples | 2,500 |
-| Batch Size | 4 |
-| Learning Rate | 3e-4 |
-| Epochs | 23 |
-| Final Loss | 0.3090 |
-
----
-
-## Intended Use
-
-- Exploring AI/ML concepts in a research context
-- Prototyping lightweight domain-specific language model pipelines
-- Studying the effect of narrow-domain pre-training on small models
-
-### Out-of-Scope Use
-
-This model is **not** intended for production use, general-purpose chat, or any application requiring safe, aligned, or factually reliable outputs. It has a very small training set (2,500 samples) and may produce repetitive, incoherent, or factually incorrect text.
-
----
-
-## Limitations
-
-- **Small training set:** Only 2,500 samples were used for pre-training, which significantly limits generalization, but because they are research papers are are quite rich.
-- **No alignment:** The model has not been fine-tuned with human feedback or instruction tuning of any kind.
-- **Potential overfitting:** Given the high number of epochs (23) relative to the dataset size, the model may have overfit to training examples.
-- **Narrow domain:** The model has only been exposed to AI/ML research text and will likely perform poorly on out-of-domain inputs.
-
----
-
-## Safety
-
-Because training data consists solely of academic research papers, the risk of harmful content generation is low. However, the lack of any alignment procedure means outputs are unpredictable and should not be treated as authoritative or safe for end-user-facing applications.
-
----
-
-## Citation
-
-If you use this model in your work, please cite it as:
-```bibtex
-@misc{lfm2-research,
-  author = {FlameF0X},
-  title  = {LFM2-Research: A Small Language Model Pre-trained on arXiv AI/ML Papers},
-  year   = {2025},
-  url    = {https://huggingface.co/FlameF0X/LFM2-Research}
-}
-```
-Or dont do it. It's up to you.
+| Attention heads | 8 |
+| KV heads | 4 |
+| Max seq len | 2048 |
+| Vocab size | 50257 |
+| Epochs | 12 |
+| Final loss | 0.3772 |

config.json

@@ -20,5 +20,5 @@
   "block_ffn_dim_multiplier": 1.0,
   "block_auto_adjust_ff_dim": true,
   "torch_dtype": "float32",
-  "transformers_version": "4.36.0"
+  "transformers_version": "4.50.0"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:460a4af2c49a4151fc5b2858dfce727117b7c8fa23cbc9886b2f947fa6935574
-size 203632480
+oid sha256:659ee87cd8c532c4344a41c2426bd37047a392fe54cdf46690cb739b904c217d
+size 203638464