---
title: README
emoji: 🌖
colorFrom: blue
colorTo: red
sdk: static
pinned: false
---

# MidReal

We follow an extremely simple format to organize and manage our models aand data.

# Model

Repo should be named as `midreal/{model_function}_{train_method}_{train_technique}_{base_model}_{date}`

Model card should include ([example](https://huggingface.co/midreal/passage_sft_qlora_llama3_70b_0704)):

```
# Data

Dataset trained with

# Base

Base model trained on

# Template

Prompt/Response template of the model

# System

MidReal system version that's compatible with the model

# W&B

Tracking of the training procedure

# PIC

Person in charge
```

# Dataset

Repo should be named as `midreal/{model_function}_{train_method}_{status}_{date}`

Dataset card should include ([example](https://huggingface.co/datasets/midreal/passage_sft_lmflow_0704)):

```
# Data Schema

The schema that the data elements should follow

# PIC

Person in charge
```

Other information about the dataset could also be commented on dataset cards. 

The production of dataset should somehow follow a pipeline from `raw_data` to `story_data` to `openai` or `lmflow` format. 

`raw_data` is any data in its original appearance.

`story_data` currently follow [data_schema_0718](https://huggingface.co/datasets/midreal/data_schema_0718).

`openai` refers to [OpenAI Fine-tuning data format](https://platform.openai.com/docs/guides/fine-tuning/example-format).

`lmflow` refers to [LMFlow data format](https://optimalscale.github.io/LMFlow/examples/DATASETS.html#data-format).

# Usage

We suggest using Huggingface CLI ([docs](https://huggingface.co/docs/huggingface_hub/main/en/guides/cli)).

Once you have installed huggingface-cli and login, models/datasets could be uploaded with:
```
huggingface-cli upload [midreal/repo_id] [local_path] ([path_in_repo]) (--repo-type=dataset)
```

e.g.
```
huggingface-cli upload midreal/model ./path/to/curated/data /data/train
huggingface-cli upload midreal/dataset . . --repo-type=dataset
```

and downloaded with:
```
huggingface-cli download midreal/model
huggingface-cli download midreal/dataset --repo-type dataset
```

If the repo doesn’t exist yet, it will be created automatically.