| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - stingning/ultrachat |
| | - TIGER-Lab/MathInstruct |
| | - ise-uiuc/Magicoder-Evol-Instruct-110K |
| | - OpenAssistant/oasst2 |
| | - teknium/openhermes |
| | - bigcode/commitpackft |
| | - Open-Orca/SlimOrca |
| | - ise-uiuc/Magicoder-OSS-Instruct-75K |
| | language: |
| | - en |
| | library_name: transformers |
| | base_model: |
| | - mllmTeam/PhoneLM-0.5B |
| | --- |
| | PhoneLM-0.5B-Instruct is a 0.5 billion parameter decoder-only language model. |
| |
|
| | ## Usage |
| | ```python |
| | from transformers import AutoTokenizer, AutoModelForCausalLM |
| | |
| | model_name = 'mllmTeam/PhoneLM-0.5B-Instruct' |
| | question = "Hello, who are you?" |
| | prompt = [{"role": "user", "content": question}] |
| | |
| | model = AutoModelForCausalLM.from_pretrained(model_name, device_map='cuda', trust_remote_code=True) |
| | |
| | tokenizer = AutoTokenizer.from_pretrained(model_name) |
| | input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True) |
| | |
| | inp = tokenizer(input_text, return_tensors="pt") |
| | inp = {k: v.to('cuda') for k, v in inp.items()} |
| | out = model.generate(**inp, |
| | max_length=256, |
| | do_sample=True, |
| | temperature=0.7, |
| | top_p=0.7 |
| | ) |
| | text = tokenizer.decode(out[0], skip_special_tokens=True) |
| | print(text) |
| | ``` |
| |
|
| |
|
| | ## Model Details |
| |
|
| | * **Developed by**: mllmTeam |
| | * **Model type**: `PhoneLM 0.5B` models are auto-regressive language models based on the transformer decoder architecture. |
| | * **Language(s)**: English |
| | * **Paper**: [PhoneLM Technical Report]() |
| | * **Library**: [PhoneLM](https://github.com/UbiquitousLearning/PhoneLM) |
| |
|
| | ### Model Architecture |
| |
|
| | The model is a decoder-only transformer architecture with the following modifications: |
| |
|
| | | Hidden Size | Layers | Heads | Sequence Length | |
| | |-------------|--------|-------|-----------------| |
| | | 1024 | 24 | 16 | 2048 | |
| |
|
| | * **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf). PhoneLM quantized the sin and cos values in Rotary Position Embeddings to 8-bit integers. |
| | * **Normalization**: LayerNorm ([Ba et al., 2016](https://arxiv.org/abs/1607.06450)) with learned bias terms as opposed to RMSNorm ([Zhang & Sennrich, 2019](https://arxiv.org/abs/1910.07467)). |
| | * **Biases**: We remove all bias terms from the feed-forward networks and multi-head self-attention layers, except for the biases of the query, key, and value projections ([Bai et al., 2023](https://arxiv.org/abs/2309.16609)). |
| | * **ReLU Activation Function**: ReLU([Glorot et al., 2011](https://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf)) activation functions are adopted in feed-forward networks. |
| | * **Tokenizer**: We use the SmolLM([Allal et al., 2024](https://huggingface.co/blog/smollm))'s tokenizer with a vocabulary size of 49,152. |
| |
|
| | ## License |
| | * This repository is released under the [Apache-2.0](https://huggingface.co/mllmTeam/PhoneLM-0.5B-Instruct/blob/main/LICENSE) License.、 |
| |
|
| | ## Citation |
| | ``` |
| | @misc{yi2024phonelmanefficientcapablesmall, |
| | title={PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training}, |
| | author={Rongjie Yi and Xiang Li and Weikai Xie and Zhenyan Lu and Chenghua Wang and Ao Zhou and Shangguang Wang and Xiwen Zhang and Mengwei Xu}, |
| | year={2024}, |
| | eprint={2411.05046}, |
| | archivePrefix={arXiv}, |
| | primaryClass={cs.CL}, |
| | url={https://arxiv.org/abs/2411.05046}, |
| | } |
| | ``` |
