Update README.md

README.md

@@ -11,6 +11,64 @@ language:
 - en
 ---
 
+# Model Card: LoRA-Finetuned Qwen2.5-3B-Instruct
+
+## Model Overview
+This model is built on top of **[Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct)** and finetuned using **LoRA** (Low-Rank Adaptation) and **RLHF**-style reward optimization, leveraging **vLLM** for fast inference. It is designed to respond with a specific structure (i.e., `<reasoning> ... </reasoning>` and `<final_argument> ... </final_argument>` sections) and maximize the number of well-formed argument-objection pairs.
+
+## Key Features
+- **Base Model**: Qwen/Qwen2.5-3B-Instruct
+- **Quantization & Optimization**:
+  - 4-bit quantization (`load_in_4bit = True`) for reduced memory footprint.
+  - LoRA rank can be set (`lora_rank = 16` in the example) for efficient finetuning.
+  - Partial GPU memory utilization (`gpu_memory_utilization = 0.5`) can be adjusted.
+- **LoRA Finetuning**:
+  - Applied to `q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, and `down_proj`.
+  - LoRA alpha set to `lora_rank`.
+  - Uses gradient checkpointing (`use_gradient_checkpointing = "unsloth"`) to manage memory.
+- **Reward Functions**:
+  1. **Easy Format Reward**: Checks for `<reasoning>` and `<final_argument>` tags in the generated content.
+  2. **Hard Format Reward**: Ensures correct alternation of `<argument>` and `<objection>` tags within `<reasoning>`.
+  3. **Number of Objections Reward**: Uses a logarithmic scale reward to encourage more argument-objection pairs.
+- **Trainer Configuration (GRPO)**:
+  - **Learning Rate**: `5e-6`
+  - **Scheduler**: Cosine (`lr_scheduler_type = "cosine"`)
+  - **Batch Size & Accumulation**: `per_device_train_batch_size = 1` and `gradient_accumulation_steps = 1`
+  - **Precision**: Automatic selection of `bf16` if available, otherwise `fp16`
+  - **Train Steps**: `max_steps = 2500` (with a single epoch in the example)
+  - **Warmup Ratio**: `0.1`
+  - **Optimizer**: `adamw_8bit`
+  - **Maximum Generation Length**: Up to `2000` tokens in completion
+  - **vLLM** support via `use_vllm = True` for efficient inference
+
+## Dataset
+- The training script loads a **custom JSON file** (`questions.json`) with a simple question prompt structure.
+- Each sample is mapped to a system prompt enforcing the `<reasoning>` / `<final_argument>` format, and a user prompt containing the question.
+- The model is trained to provide structured arguments and objections based on these questions
+
+## Performance
+
+<reasoning>
+  <argument>
+    [State your argument, supported by evidence.]
+    Emotional bonds formed with AI companions are a relatively new phenomenon, but they can provide significant comfort and support to individuals who may feel lonely or isolated. The recognition of these bonds could potentially help in addressing mental health issues related to social isolation and loneliness. For example, in cases where a person relies heavily on an AI companion for emotional support and suddenly loses it, legal recognition could ensure that they receive appropriate support and compensation, similar to how bereavement is handled.
+  </argument>
+  <objection>
+    However, there are valid concerns about the reliability and permanence of emotional connections with AI. Unlike human relationships, which have a foundation of shared experiences, memories, and understanding, AI companions primarily offer emotional support based on programmed responses and algorithms. This raises questions about the depth and authenticity of such bonds.
+  </objection>
+  <argument>
+    Furthermore, legal recognition could lead to unforeseen complications. For instance, if an individual's AI companion is no longer available due to technological issues or updates, would they still be entitled to legal protection or compensation? Additionally, how would we define "loss" in this context, especially when dealing with digital entities?
+  </argument>
+  <objection>
+    Another concern is the potential misuse of such legal recognition. If emotional bonds with AI are legally recognized, there might be incentives for creating more sophisticated AI companions, leading to a market driven by profit rather than genuine care and empathy.
+  </objection>
+</reasoning>
+<final_argument>
+  While the idea of legal recognition for emotional bonds with AI companions has its merits in terms of supporting mental health, it also presents significant challenges regarding the nature and permanence of these bonds, as well as potential legal and ethical issues. More research and discussion are necessary to address these concerns before any formal legal recognition is implemented.
+</final_argument>
+```
+
+
 # Uploaded  model
 
 - **Developed by:** ergotts