KorByte-128K / src /korbyte /challenges.py
dawncr0w's picture
Publish KorByte-128K tokenizer and reproducibility evidence
b3c2a26 verified
Raw
History Blame Contribute Delete
1.15 kB
"""Fixed multilingual and Korean round-trip challenge strings."""
from __future__ import annotations
CHALLENGE_TEXTS = (
"์˜ค๋Š˜ ์•„์นจ ์„œ์šธ์—๋Š” ๋น„๊ฐ€ ๋‚ด๋ ธ์ง€๋งŒ ์˜คํ›„์—๋Š” ๋ง‘๊ฒŒ ๊ฐฐ๋‹ค.",
"์ƒˆ ๊ธฐ๋Šฅ์„ ๋ฐฐํฌํ•˜๊ธฐ ์ „์— ํ…Œ์ŠคํŠธ ๊ฒฐ๊ณผ์™€ ์˜ค๋ฅ˜ ๋กœ๊ทธ๋ฅผ ๋‹ค์‹œ ํ™•์ธํ•ด ์ฃผ์„ธ์š”.",
"์‚ฌ์šฉ์ž๋Š” ํœด๋Œ€์ „ํ™”์—์„œ ์ „์ž์ฑ… ๋งํฌ๋ฅผ ๋ˆŒ๋Ÿฌ ์›ํ•˜๋Š” ํŽ˜์ด์ง€๋กœ ์ด๋™ํ•  ์ˆ˜ ์žˆ๋‹ค.",
"๊ฐ€๊ฒฉ์€ 12,500์›์ด๋ฉฐ ๋ฐฐ์†ก์€ 2026๋…„ 8์›” 5์ผ์— ์‹œ์ž‘ํ•œ๋‹ค.",
"์ง„์งœ ๋น ๋ฅด๋„คใ…‹ใ…‹ ์ด ์ •๋„๋ฉด ๋ฐ”๋กœ ์จ๋„ ๋˜๊ฒ ๋‹ค!",
"ํ•œ๊ธ€๊ณผ English, ์ˆซ์ž 123, ์ด๋ชจ์ง€ ๐Ÿ˜Š๋ฅผ ํ•œ ๋ฌธ์žฅ์— ์„ž์–ด ์“ด๋‹ค.",
"๋„์–ด์“ฐ๊ธฐ ๋‘ ์นธ\tํƒญ\n์ค„๋ฐ”๊ฟˆ๊ณผ Windows ์ค„๋ฐ”๊ฟˆ\r\n๊นŒ์ง€ ๋ณด์กดํ•œ๋‹ค.",
"แ„’แ…กแ†ซแ„€แ…ณแ†ฏ(NFD)๊ณผ ํ•œ๊ธ€(NFC)์€ ์ฝ”๋“œ ํฌ์ธํŠธ๊ฐ€ ๋‹ค๋ฅด๋‹ค.",
"URL https://example.com/๊ฒฝ๋กœ?q=ํ† ํฌ๋‚˜์ด์ €์™€ email@example.com",
"Python: def ์ธ์‚ฌ(์ด๋ฆ„: str) -> str: return f'์•ˆ๋…•, {์ด๋ฆ„}! ๐Ÿ‘‹'",
"ไธญๆ–‡ใ€ๆ—ฅๆœฌ่ชžใ€เธ เธฒเธฉเธฒเน„เธ—เธข, ั€ัƒััะบะธะน ัะทั‹ะบ, ุงู„ุนุฑุจูŠุฉ, เคนเคฟเคจเฅเคฆเฅ€",
"๊ฐ€๋‚˜๋‹ค๋ผ๋งˆ๋ฐ”์‚ฌ์•„์ž์ฐจ์นดํƒ€ํŒŒํ•˜" * 16,
)