File size: 2,216 Bytes
4548c35 bb8b382 68f7a27 4548c35 68f7a27 2ff18ec 68f7a27 2ff18ec 68f7a27 2ff18ec 68f7a27 2ff18ec 68f7a27 2ff18ec 68f7a27 2ff18ec 68f7a27 2ff18ec 68f7a27 2ff18ec |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 |
---
title: README
emoji: 🌖
colorFrom: blue
colorTo: red
sdk: static
pinned: false
---
# MidReal
We follow an extremely simple format to organize and manage our models aand data.
# Model
Repo should be named as `midreal/{model_function}_{train_method}_{train_technique}_{base_model}_{date}`
Model card should include ([example](https://huggingface.co/midreal/passage_sft_qlora_llama3_70b_0704)):
```
# Data
Dataset trained with
# Base
Base model trained on
# Template
Prompt/Response template of the model
# System
MidReal system version that's compatible with the model
# W&B
Tracking of the training procedure
# PIC
Person in charge
```
# Dataset
Repo should be named as `midreal/{model_function}_{train_method}_{status}_{date}`
Dataset card should include ([example](https://huggingface.co/datasets/midreal/passage_sft_lmflow_0704)):
```
# Data Schema
The schema that the data elements should follow
# PIC
Person in charge
```
Other information about the dataset could also be commented on dataset cards.
The production of dataset should somehow follow a pipeline from `raw_data` to `story_data` to `openai` or `lmflow` format.
`raw_data` is any data in its original appearance.
`story_data` currently follow [data_schema_0718](https://huggingface.co/datasets/midreal/data_schema_0718).
`openai` refers to [OpenAI Fine-tuning data format](https://platform.openai.com/docs/guides/fine-tuning/example-format).
`lmflow` refers to [LMFlow data format](https://optimalscale.github.io/LMFlow/examples/DATASETS.html#data-format).
# Usage
We suggest using Huggingface CLI ([docs](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli)).
Once you have installed huggingface-cli and login, models/datasets could be uploaded with:
```
huggingface-cli upload [midreal/repo_id] [local_path] ([path_in_repo]) (--repo-type=dataset)
```
e.g.
```
huggingface-cli upload midreal/model ./path/to/curated/data /data/train
huggingface-cli upload midreal/dataset . . --repo-type=dataset
```
and downloaded with:
```
huggingface-cli download midreal/model
huggingface-cli download midreal/dataset --repo-type dataset
```
If the repo doesn’t exist yet, it will be created automatically. |