llama-order-model / postprocessing.py
gahyunlee's picture
edit postprocessing logic
fba0152 verified
raw
history blame contribute delete
525 Bytes
def extract_relevant_text(text):
"""
์ฃผ์–ด์ง„ ํ…์ŠคํŠธ์—์„œ `[/INST]`๋กœ ์‹œ์ž‘ํ•˜๊ณ  ์ข…๋ฃŒ ํƒœ๊ทธ `</>` ์ด์ „๊นŒ์ง€์˜ ํ…์ŠคํŠธ๋ฅผ ์ถ”์ถœํ•˜๋Š” ํ•จ์ˆ˜.
"""
pattern = r"\[/INST\](.*?</)" # [/INST]๋กœ ์‹œ์ž‘ํ•˜๊ณ  </> ์ด์ „๊นŒ์ง€์˜ ํ…์ŠคํŠธ๋ฅผ ์ถ”์ถœ
match = re.search(pattern, text, re.DOTALL)
if match:
# `[/INST]` ์ดํ›„ `</s>` ์ „๊นŒ์ง€์˜ ํ…์ŠคํŠธ ๋ฐ˜ํ™˜
return match.group(1).strip().replace("</", "")
else:
return "๋งค์นญ๋˜๋Š” ํ…์ŠคํŠธ๊ฐ€ ์—†์Šต๋‹ˆ๋‹ค."