Update README.md

README.md

@@ -1,65 +1,60 @@
 ---
-base_model: liswei/OpenELM-1_1B-zh-cp
-tags:
-- llama-factory
-- full
-- generated_from_trainer
-model-index:
-- name: OpenELM-1_1B-zh-sft
-  results: []
+library_name: transformers
+license: apache-2.0
+datasets:
+- liswei/zhtw-news-and-articles-2B
+- liswei/PromptPair-TW
+- yentinglin/TaiwanChat
+base_model:
+- liswei/Taiwan-ELM-1_1B
+language:
+- zh
+pipeline_tag: text-generation
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
+<center>
+    <img src="https://huggingface.co/liswei/Taiwan-ELM/resolve/main/Taiwan%20ELM%20Logo.jpeg" alt="Efficient LLM for Taiwan">
+</center>
 
-# OpenELM-1_1B-zh-sft
+> Efficient LLM for Taiwan
 
-This model is a fine-tuned version of [liswei/OpenELM-1_1B-zh-cp](https://huggingface.co/liswei/OpenELM-1_1B-zh-cp) on the en2tw-alignment-sft and the TaiwanChat datasets.
-It achieves the following results on the evaluation set:
-- Loss: 1.3355
+# Taiwan ELM
 
-## Model description
+Taiwan ELM is a family of Efficient LLMs for Taiwan base on [apple/OpenELM](https://huggingface.co/apple/OpenELM).
+The project aims to provide an efficient model for researchers without access to large-scale computing resources.
 
-More information needed
+The model is trained using a custom fork of [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory) on 2B Traditional Chinese tokens and 500K instruction samples.
+We will extend the model to train on larger data sets and different base models if there is sufficient demand.
 
-## Intended uses & limitations
+## What is being released?
 
-More information needed
+We release both pre-trained base models and instruction tuned variants with 270M and 1.1B parameters.
+Along with the model, datasets used to train the base and instruction-tuned models are also released.
 
-## Training and evaluation data
+List of released models:
+* [Taiwan-ELM-270M](https://huggingface.co/liswei/Taiwan-ELM-270M)
+* [Taiwan-ELM-1_1B](https://huggingface.co/liswei/Taiwan-ELM-1_1B)
+* [Taiwan-ELM-270M-Instruct](https://huggingface.co/liswei/Taiwan-ELM-270M-Instruct)
+* [Taiwan-ELM-1_1B-Instruct](https://huggingface.co/liswei/Taiwan-ELM-1_1B-Instruct)
 
-More information needed
+List of released datasets:
+* [liswei/Taiwan-Text-Excellence-2B](https://huggingface.co/datasets/liswei/Taiwan-Text-Excellence-2B)
+* [liswei/PromptPair-TW](https://huggingface.co/datasets/liswei/PromptPair-TW)
 
-## Training procedure
+## Usage Examples
 
-### Training hyperparameters
+We adapt the LLaMA2 template:
+```jinja2
+<s>[INST] <<SYS>>
+{{ system_prompt }}
+<</SYS>>
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0001
-- train_batch_size: 12
-- eval_batch_size: 12
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 8
-- gradient_accumulation_steps: 4
-- total_train_batch_size: 384
-- total_eval_batch_size: 96
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 1.0
+{{ user_message }} [/INST]
+```
 
-### Training results
+The model could be load via `AutoModelForCausalLM` with `trust_remote_code=True`:
+```python
+taiwanelm_270m = AutoModelForCausalLM.from_pretrained("liswei/Taiwan-ELM-270M", trust_remote_code=True)
+```
 
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 1.4543        | 0.3527 | 500  | 1.4672          |
-| 1.3683        | 0.7053 | 1000 | 1.3570          |
-
-
-### Framework versions
-
-- Transformers 4.41.1
-- Pytorch 2.3.0+cu121
-- Datasets 2.19.1
-- Tokenizers 0.19.1
+We also support additional generation methods and speculative generation, please find reference at [OpenELM#usage](https://huggingface.co/apple/OpenELM#usage).