README.md

metadata

language:
  - ko
license: apache-2.0
model-index:
  - name: mptk-1b
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 22.7
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=team-lucid/mptk-1b
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 25.48
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=team-lucid/mptk-1b
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 27.11
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=team-lucid/mptk-1b
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 0
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=team-lucid/mptk-1b
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 49.72
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=team-lucid/mptk-1b
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 0
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=team-lucid/mptk-1b
          name: Open LLM Leaderboard

MPTK-1B

MPTK-1B는 한국어/영어코드 데이터셋에서 학습된 1.3B 파라미터의 decoder-only transformer 언어모델입니다.

이 모델은 구글의 TPU Research Cloud(TRC)를 통해 지원받은 Cloud TPU로 학습되었습니다.

Model Details

Model Description

다른 decoder-only transformer에서 일부 수정된 아키텍처인 MPT를 기반으로 합니다.

• ALiBi (Attention with Linear Biases)를 사용합니다
• bias를 사용하지 않습니다.

Hyperparameter	Value
n_parameters	1.3B
n_layers	24
n_heads	16
d_model	2048
vocab size	50432
sequence length	2048

Uses

How to Get Started with the Model

fp16으로 실행 시 NaN이 발생할 수 있습니다. 따라서 fp32 혹은 bf16로 실행하기를 권장합니다.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

tokenizer = AutoTokenizer.from_pretrained("team-lucid/mptk-1b")
model = AutoModelForCausalLM.from_pretrained("team-lucid/mptk-1b")

pipe = pipeline('text-generation', model=model, tokenizer=tokenizer, device='cuda:0')

with torch.autocast('cuda', dtype=torch.bfloat16):
    print(
        pipe(
            '대한민국의 수도는',
            max_new_tokens=100,
            do_sample=True,
        )
    )

Training Details

Training Data

OSCAR, mC4, wikipedia, namuwiki 등 한국어 데이터에 RefinedWeb, The Stack 에서 일부를 추가해 학습하였습니다.

Training Hyperparameters

Hyperparameter	Value
Precision	bfloat16
Optimizer	Lion
Learning rate	2e-4
Batch size	1024

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	20.84
AI2 Reasoning Challenge (25-Shot)	22.70
HellaSwag (10-Shot)	25.48
MMLU (5-Shot)	27.11
TruthfulQA (0-shot)	0.00
Winogrande (5-shot)	49.72
GSM8k (5-shot)	0.00