Upload FalconH1ForCausalLM

README.md

@@ -1,129 +1,199 @@
 ---
 library_name: transformers
-tags:
-- falcon-h1
-- edge
-license: other
-license_name: falcon-llm-license
-license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
+tags: []
 ---
 
-<img src="https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/l1du02RjuAZJcksI5tQ-F.png" alt="drawing" width="800"/>
+# Model Card for Model ID
 
-#  Table of Contents
+<!-- Provide a quick summary of what the model is/does. -->
 
-0. [TL;DR](#TL;DR)
-1. [Model Details](#model-details)
-2. [Training Details](#training-details)
-3. [Usage](#usage)
-4. [Evaluation](#evaluation)
-5. [Citation](#citation)
 
-# TL;DR
 
-# Model Details
+## Model Details
 
-## Model Description
+### Model Description
 
-- **Developed by:** [https://www.tii.ae](https://www.tii.ae)
-- **Model type:** Causal decoder-only
-- **Architecture:** Hybrid Transformers + Mamba architecture
-- **Language(s) (NLP):** English
-- **Number of Parameters:** 90M
-- **License:** Falcon-LLM License
+<!-- Provide a longer summary of what this model is. -->
 
-# Training details
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
 
-For more details about the training protocol of this model, please refer to the [Tiny-H1 technical blogpost](https://huggingface.co/spaces/tiiuae/tiny-h1-blogpost).
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
 
-# Usage
+### Model Sources [optional]
 
-Currently to use this model you can either rely on Hugging Face `transformers`, `vLLM`, `sglang`, `llama.cpp`, `ollama` or `mlx` library.
+<!-- Provide the basic links for the model. -->
 
-## Inference
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
 
-### 🤗 transformers
+## Uses
 
-Refer to the snippet below to run H1 models using 🤗 transformers:
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
+### Direct Use
 
-model_id = "tiiuae/Falcon-H1-Tiny-R-0.6B-pre-GRPO"
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 
-model = AutoModelForCausalLM.from_pretrained(
-  model_id,
-  torch_dtype=torch.bfloat16,
-  device_map="auto"
-)
+[More Information Needed]
 
-# Perform text generation
-```
+### Downstream Use [optional]
 
-or
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-```bash
-transformers serve tiiuae/Falcon-H1-Tiny-R-0.6B-pre-GRPO
-```
+[More Information Needed]
 
-### `llama.cpp`
+### Out-of-Scope Use
 
-You can find all GGUF files compatible with `llama.cpp` under [our official collection]() - an example setup could be:
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 
-```bash
-brew install llama.cpp 
-pip install huggingface_hub 
-hf download tiiuae/Falcon-H1-Tiny-R-0.6B-pre-GRPO Falcon-H1-Tiny-R-0.6B-pre-GRPO-Q8_0.gguf --local-dir ./ 
-llama-cli ./ Falcon-H1-Tiny-R-0.6B-pre-GRPO-Q8_0.gguf -cnv 
-```
+[More Information Needed]
 
-### `ollama`
+## Bias, Risks, and Limitations
 
-```bash
-ollama run hf.co/tiiuae/Falcon-H1-Tiny-R-0.6B-pre-GRPO:Q8_0 
-```
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
-### Apple `mlx` 
+[More Information Needed]
 
-```bash
-mlx_lm.chat --model tiiuae/Tiny-H1-SF 
-```
+### Recommendations
 
-### vLLM
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-For vLLM, simply start a server by executing the command below:
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-```bash
-# pip install vllm>=0.9.0
-vllm serve tiiuae/Falcon-H1-Tiny-R-0.6B-pre-GRPO --tensor-parallel-size 2 --data-parallel-size 1
-```
+## How to Get Started with the Model
 
-### sglang
+Use the code below to get started with the model.
 
-```bash
-python -m sglang.launch_server \
-  --model ttiiuae/Falcon-H1-Tiny-R-0.6B-pre-GRPO \
-  --tensor-parallel-size 1 
-```
+[More Information Needed]
 
-# Evaluation
+## Training Details
 
-For detailed evaluation of Tiny-H1 series, please refer to our [technical blogpost](https://huggingface.co/spaces/tiiuae/tiny-h1-blogpost)
+### Training Data
 
-# Useful links
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
 
-- View [our release blogpost]([https://falcon-lm.github.io/blog/falcon-h1/](https://huggingface.co/spaces/tiiuae/tiny-h1-blogpost)).
-- Feel free to join [our discord server](https://discord.gg/trwMYP9PYm) if you have any questions or to interact with our researchers and developers.
+[More Information Needed]
 
-# Citation
+### Training Procedure
 
-If the Tiny-H1 family of models were helpful to your work, feel free to give us a cite.
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-```
-@misc{team2025_tiny_h1_a_series_of_extremely_small_yet_powerful_language_models_redefining_capabilities_at_small_scale,
-  title={Tiny-H1: A series of extremely small, yet powerful language models redefining capabilities at small scale},
-  author={Falcon-LLM Team},
-  year={2025}, 
-}
-```
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

config.json

@@ -0,0 +1,71 @@
+{
+  "architectures": [
+    "FalconH1ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attention_in_multiplier": 1.0,
+  "attention_out_multiplier": 1.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "embedding_multiplier": 1.0,
+  "eos_token_id": [
+    11,
+    228
+  ],
+  "expansion_factor": 2.0,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 2048,
+  "key_multiplier": 0.03125,
+  "lm_head_multiplier": 1.0,
+  "mamba_chunk_size": 128,
+  "mamba_conv_bias": true,
+  "mamba_d_conv": 4,
+  "mamba_d_head": 64,
+  "mamba_d_ssm": 1536,
+  "mamba_d_state": 128,
+  "mamba_expand": 2,
+  "mamba_n_groups": 1,
+  "mamba_n_heads": 24,
+  "mamba_norm_before_gate": false,
+  "mamba_proj_bias": false,
+  "mamba_rms_norm": false,
+  "mamba_use_mlp": true,
+  "max_position_embeddings": 262144,
+  "mlp_bias": false,
+  "mlp_multipliers": [
+    1.0,
+    1.0
+  ],
+  "model_type": "falcon_h1",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 44,
+  "num_key_value_heads": 2,
+  "num_logits_to_keep": 1,
+  "pad_token_id": 0,
+  "projectors_bias": false,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 100000000000.0,
+  "sliding_window": null,
+  "ssm_in_multiplier": 1.0,
+  "ssm_multipliers": [
+    1.0,
+    1.0,
+    1.0,
+    1.0,
+    0.03125
+  ],
+  "ssm_out_multiplier": 1.0,
+  "tie_word_embeddings": false,
+  "time_step_floor": 0.0001,
+  "time_step_max": 0.1,
+  "time_step_min": 0.001,
+  "time_step_rank": "auto",
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "vocab_size": 32768
+}

generation_config.json

@@ -0,0 +1,10 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": [
+    11,
+    228
+  ],
+  "pad_token_id": 0,
+  "transformers_version": "4.57.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1190f210b274c21ffed0710c19c76986d3e279a8f87179ae171b986f2b829b4
+size 1244732432