NickyNicky/Llama-1B-base-GRPO-miniThinky_v1

max tokens 512

full full fine tune

script base.

https://gist.github.com/willccbb/4676755236bb08cab5f4e54a0475d6fb

Note.

I have taken the script from the user 'willccbb' and I have modified it many things to be able to train it.
thanks to this script I already understand many things.

Semantic Comparison of Sentences and Mapping the Cosine Similarity to the [0 ... 0.123] Scale (rewards).

https://colab.research.google.com/drive/1hdWSImb9lJ7Q4Csh3ubLAblhD0EscDPh?usp=sharing

import torch,gc

import transformers
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, BitsAndBytesConfig

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging,
    GenerationConfig,
    TextIteratorStreamer,
)

from transformers import StoppingCriteria, StoppingCriteriaList

model_name= "NickyNicky/Llama-1B-base-GRPO-miniThinky_v1"

base_model = AutoModelForCausalLM.from_pretrained(model_name,
                                            #  quantization_config=bnb_config,
                                             device_map={"":0},
                                            #  token=hf_token,
                                             # attn_implementation="flash_attention_2",
                                             torch_dtype=torch.bfloat16
                                             )

max_seq_length=4048

base_tokenizer_0 = AutoTokenizer.from_pretrained(model_name,
                                          # token=access_token,
                                          max_length=max_seq_length)

prompt = "What do you mean by 'core' when referring to a subject or topic?"
sys="""Respond in the following format:

<reasoning>
...
</reasoning>
<answer>
...
</answer>
"""
messages = [
    {"role": "system", "content": sys},
    {"role": "user", "content": prompt}
]

device = "cuda" # the device to load the model onto
text = base_tokenizer_0.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = base_tokenizer_0([text], return_tensors="pt").to(device)

generated_ids = base_model.generate(
    model_inputs.input_ids,
    max_new_tokens=1200,
    pad_token_id= base_tokenizer_0.eos_token_id,
)
generated_ids = [
    output_ids[:] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = base_tokenizer_0.batch_decode(generated_ids, skip_special_tokens=True)[0]
messages.append({"role": "assistant", "content": response})
print(response)

'''
<|begin_of_text|><|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 31 Jan 2025

Respond in the following format:

<reasoning>
...
</reasoning>
<answer>
...
</answer><|eot_id|><|start_header_id|>user<|end_header_id|>

What do you mean by 'core' when referring to a subject or topic?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

<reasoning>

When referring to a "core" in a scientific context, it typically refers to the central, innermost part of the brain or central nervous system, which houses the brain's main processing centers, including the cerebrum, the hippocampus, and the amygdala. The core is involved in controlling and coordinating the body's functions, such as the regulation of the heart rate, blood pressure, and digestion, as well as the processing of sensory information from the environment.

In a more general sense, a core can also refer to the central part of the brain, which is the main processing center of the brain, often containing the brain's main neurons and is involved in the processing of complex information, including the integration of sensory data from the environment, the interpretation of emotions, and the generation of the body's response to the environment.

The core is also often used to describe the central nervous system's main processing centers, which are located in the brain and are responsible for controlling the body's functions, including the regulation of the body's internal and external environment, the processing of the senses, and the generation of the body's response to the environment.

</reasoning>
<answer>

The core is a central part of the brain that plays a vital role in controlling and coordinating the body's functions, and it is involved in the processing of complex information from the environment, the interpretation of emotions, and the generation of the body's response to the environment. It is a central processing center of the brain, and its functions are often compared to the functions of the heart, which is often referred to as the "heart of the brain." The core is also a key component of the brain's main processing centers, which are located in the cerebrum, the hippocampus, and the amygdala.

</answer>
<|eot_id|>
'''