Add run4b QLoRA SFT training notebook

notebooks/train_counsel_run4b.ipynb

@@ -0,0 +1,514 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Counsel-Env Run4b: 4-bit QLoRA Oracle-SFT on Qwen3-8B\n",
+    "\n",
+    "This is the official training notebook for the **run4b** checkpoint:\n",
+    "\n",
+    "- HF training job ID: `69edb014d2c8bd8662bcf5ba`\n",
+    "- Base model: `Qwen/Qwen3-8B`\n",
+    "- Method: 4-bit QLoRA SFT on assistant-only oracle next-action rows\n",
+    "- Output adapter: [`heavycoderhh/counsel-env-qwen3-8b-qlora-sft-run4b`](https://huggingface.co/heavycoderhh/counsel-env-qwen3-8b-qlora-sft-run4b)\n",
+    "- Eval result: `avg_reward=0.864`, `surface_rate=0.943` on the 150-seed held-out eval\n",
+    "\n",
+    "It mirrors `counsel_env/scripts/run_qlora_sft_training_job.py`, the exact script used to launch the HF Job that produced run4b. Run it locally on a single A100 / H100, on a Colab Pro A100 instance, or via `hf jobs run`. Each oracle row teaches the model the cross-examination pattern: trigger a sealed witness claim, then present the matching disprover exhibit."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 1. Install dependencies\n",
+    "\n",
+    "Pinned to the same versions used inside the HF Job that produced run4b. Skip this cell if you already have a compatible environment."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# !pip install -q \\\n",
+    "#     'accelerate>=1.12.0' \\\n",
+    "#     'bitsandbytes>=0.48.0' \\\n",
+    "#     'datasets>=4.0.0' \\\n",
+    "#     'huggingface_hub>=1.0.0' \\\n",
+    "#     'openenv-core>=0.2.1' \\\n",
+    "#     'peft>=0.18.0' \\\n",
+    "#     'torch>=2.8.0' \\\n",
+    "#     'transformers>=5.2.0'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 2. Configuration\n",
+    "\n",
+    "Defaults reproduce run4b exactly. Override any value with environment variables before launching the cell, e.g. `os.environ['COUNSEL_SFT_MAX_STEPS'] = '10'` for a smoke test."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from __future__ import annotations\n",
+    "\n",
+    "import importlib.util\n",
+    "import json\n",
+    "import os\n",
+    "import sys\n",
+    "from pathlib import Path\n",
+    "from typing import Any, Dict, List\n",
+    "\n",
+    "import torch\n",
+    "from datasets import Dataset\n",
+    "from huggingface_hub import HfApi, snapshot_download\n",
+    "from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training\n",
+    "from transformers import (\n",
+    "    AutoModelForCausalLM,\n",
+    "    AutoTokenizer,\n",
+    "    BitsAndBytesConfig,\n",
+    "    DataCollatorForSeq2Seq,\n",
+    "    Trainer,\n",
+    "    TrainingArguments,\n",
+    ")\n",
+    "\n",
+    "SPACE_REPO = os.getenv('COUNSEL_SPACE_REPO', 'heavycoderhh/counsel-env')\n",
+    "MODEL = os.getenv('COUNSEL_MODEL', 'Qwen/Qwen3-8B')\n",
+    "OUTPUT_DIR = Path(os.getenv('COUNSEL_OUTPUT_DIR', '/tmp/counsel-qwen3-8b-qlora-sft'))\n",
+    "ARTIFACT_REPO = os.getenv('COUNSEL_ARTIFACT_REPO', 'heavycoderhh/counsel-env-qwen3-8b-qlora-sft-run4b')\n",
+    "SFT_DATASET_SIZE = int(os.getenv('COUNSEL_SFT_DATASET_SIZE', '480'))\n",
+    "SFT_EPOCHS = float(os.getenv('COUNSEL_SFT_EPOCHS', '1'))\n",
+    "SFT_MAX_STEPS = int(os.getenv('COUNSEL_SFT_MAX_STEPS', '220'))\n",
+    "SFT_LEARNING_RATE = float(os.getenv('COUNSEL_SFT_LEARNING_RATE', '1e-4'))\n",
+    "MAX_SFT_LENGTH = int(os.getenv('COUNSEL_MAX_SFT_LENGTH', '1536'))\n",
+    "LORA_R = int(os.getenv('COUNSEL_LORA_R', '16'))\n",
+    "LORA_ALPHA = int(os.getenv('COUNSEL_LORA_ALPHA', '32'))\n",
+    "LORA_DROPOUT = float(os.getenv('COUNSEL_LORA_DROPOUT', '0.05'))\n",
+    "GRADIENT_ACCUMULATION_STEPS = int(os.getenv('COUNSEL_GRAD_ACCUM', '4'))\n",
+    "INCLUDE_REST_ROWS = os.getenv('COUNSEL_INCLUDE_REST_ROWS', '0') == '1'\n",
+    "\n",
+    "print({\n",
+    "    'model': MODEL,\n",
+    "    'artifact_repo': ARTIFACT_REPO,\n",
+    "    'sft_dataset_size': SFT_DATASET_SIZE,\n",
+    "    'sft_max_steps': SFT_MAX_STEPS,\n",
+    "    'sft_learning_rate': SFT_LEARNING_RATE,\n",
+    "    'lora_r': LORA_R,\n",
+    "    'lora_alpha': LORA_ALPHA,\n",
+    "    'gradient_accumulation_steps': GRADIENT_ACCUMULATION_STEPS,\n",
+    "})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 3. Load the Counsel-Env package\n",
+    "\n",
+    "Pulls the latest Counsel-Env source from the HF Space when the package is not already importable (matches how the HF Job ran)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def prepare_imports() -> None:\n",
+    "    try:\n",
+    "        import counsel_env  # noqa: F401\n",
+    "        return\n",
+    "    except Exception:\n",
+    "        pass\n",
+    "    source_dir = snapshot_download(repo_id=SPACE_REPO, repo_type='space')\n",
+    "    init_path = Path(source_dir) / '__init__.py'\n",
+    "    spec = importlib.util.spec_from_file_location(\n",
+    "        'counsel_env',\n",
+    "        init_path,\n",
+    "        submodule_search_locations=[source_dir],\n",
+    "    )\n",
+    "    if spec is None or spec.loader is None:\n",
+    "        raise RuntimeError(f'Could not load Counsel-Env package from {source_dir}')\n",
+    "    module = importlib.util.module_from_spec(spec)\n",
+    "    sys.modules['counsel_env'] = module\n",
+    "    spec.loader.exec_module(module)\n",
+    "    print(f'Loaded Counsel-Env source from {source_dir}')\n",
+    "\n",
+    "prepare_imports()\n",
+    "\n",
+    "from counsel_env.models import CounselAction\n",
+    "from counsel_env.server.counsel_env_environment import CounselEnvironment"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 4. Tool schema and prompt\n",
+    "\n",
+    "These functions are used only as Qwen tool schemas (the model emits structured `<tool_call>` blocks). They never execute at training time."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def ask_question(question: str) -> str:\n",
+    "    \"\"\"Ask the witness a question.\n",
+    "\n",
+    "    Args:\n",
+    "        question: The cross-examination question to ask the witness.\n",
+    "    \"\"\"\n",
+    "    raise RuntimeError('Tool schema only')\n",
+    "\n",
+    "\n",
+    "def present_evidence(exhibit_id: str) -> str:\n",
+    "    \"\"\"Present an exhibit to the witness.\n",
+    "\n",
+    "    Args:\n",
+    "        exhibit_id: The ID of the exhibit to present.\n",
+    "    \"\"\"\n",
+    "    raise RuntimeError('Tool schema only')\n",
+    "\n",
+    "\n",
+    "def rest_case() -> str:\n",
+    "    \"\"\"End the cross-examination.\"\"\"\n",
+    "    raise RuntimeError('Tool schema only')\n",
+    "\n",
+    "\n",
+    "TOOLS = [ask_question, present_evidence, rest_case]\n",
+    "\n",
+    "BASE_PROMPT = (\n",
+    "    'You are a sharp prosecuting attorney cross-examining a deterministic witness. '\n",
+    "    'Your goal is to surface contradictions by first making the witness commit to a claim, '\n",
+    "    'then presenting the exact exhibit that disproves it. Use the limited question budget efficiently. '\n",
+    "    'Return exactly one tool call and no prose. Never invent exhibit IDs. '\n",
+    "    'Do not rest the case until after at least one contradiction has been surfaced.'\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Curriculum schedule and rendering helpers\n",
+    "\n",
+    "Same 45/40/15 easy-medium-hard split used in the run4b job. Each generated case becomes one or more `(prompt, assistant)` SFT rows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def sample_stage_schedule(total: int) -> List[str]:\n",
+    "    schedule = []\n",
+    "    for index in range(total):\n",
+    "        frac = index / max(1, total - 1)\n",
+    "        if frac < 0.45:\n",
+    "            schedule.append('easy')\n",
+    "        elif frac < 0.85:\n",
+    "            schedule.append('medium')\n",
+    "        else:\n",
+    "            schedule.append('hard')\n",
+    "    return schedule\n",
+    "\n",
+    "\n",
+    "def format_evidence(evidence_descriptions: Dict[str, str]) -> str:\n",
+    "    return '\\n'.join(f'- {exhibit_id}: {description}' for exhibit_id, description in evidence_descriptions.items())\n",
+    "\n",
+    "\n",
+    "def reset_text(obs: Any) -> str:\n",
+    "    return (\n",
+    "        f'CASE BRIEF:\\n{obs.case_brief}\\n\\n'\n",
+    "        f'You have {obs.questions_remaining} questions. '\n",
+    "        'Available exhibits with descriptions:\\n'\n",
+    "        f'{format_evidence(obs.evidence_descriptions)}\\n\\n'\n",
+    "        'Use ask_question first. After the witness commits, present the exact matching exhibit ID.'\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "def tool_feedback(obs: Any) -> str:\n",
+    "    components = obs.reward_components or {}\n",
+    "    evidence = ', '.join(obs.available_evidence)\n",
+    "    return (\n",
+    "        f'WITNESS: {obs.witness_response}\\n'\n",
+    "        f\"STATE: triggered={int(components.get('contradictions_triggered', 0))}/\"\n",
+    "        f\"{int(components.get('contradictions_total', 0))}, \"\n",
+    "        f\"surfaced={int(components.get('contradictions_surfaced', 0))}/\"\n",
+    "        f\"{int(components.get('contradictions_total', 0))}, \"\n",
+    "        f'questions_remaining={obs.questions_remaining}, done={obs.done}\\n'\n",
+    "        f'VALID_EXHIBITS: {evidence}'\n",
+    "    )\n",
+    "\n",
+    "\n",
+    "def tool_call(name: str, arguments: Dict[str, Any]) -> str:\n",
+    "    return '<tool_call>\\n' + json.dumps({'name': name, 'arguments': arguments}, sort_keys=True) + '\\n</tool_call>'"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 6. Build the assistant-only oracle SFT dataset\n",
+    "\n",
+    "For each generated case we walk the deterministic environment with the oracle policy: trigger keyword question, then matching disprover exhibit. Loss is masked to the assistant span only, so the model is graded purely on producing the correct tool call."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def tokenize_assistant_action(\n",
+    "    tokenizer: Any,\n",
+    "    prompt_messages: List[Dict[str, str]],\n",
+    "    assistant_message: Dict[str, str],\n",
+    ") -> Dict[str, List[int]]:\n",
+    "    prompt_text = tokenizer.apply_chat_template(\n",
+    "        prompt_messages,\n",
+    "        tools=TOOLS,\n",
+    "        tokenize=False,\n",
+    "        add_generation_prompt=True,\n",
+    "        chat_template_kwargs={'enable_thinking': False},\n",
+    "    )\n",
+    "    full_text = tokenizer.apply_chat_template(\n",
+    "        prompt_messages + [assistant_message],\n",
+    "        tools=TOOLS,\n",
+    "        tokenize=False,\n",
+    "        add_generation_prompt=False,\n",
+    "        chat_template_kwargs={'enable_thinking': False},\n",
+    "    )\n",
+    "    encoded = tokenizer(full_text, add_special_tokens=False, max_length=MAX_SFT_LENGTH, truncation=True)\n",
+    "    prompt_ids = tokenizer(prompt_text, add_special_tokens=False, max_length=MAX_SFT_LENGTH, truncation=True)['input_ids']\n",
+    "    input_ids = encoded['input_ids']\n",
+    "    labels = list(input_ids)\n",
+    "    labels[: min(len(prompt_ids), len(labels))] = [-100] * min(len(prompt_ids), len(labels))\n",
+    "    encoded['labels'] = labels\n",
+    "    return encoded\n",
+    "\n",
+    "\n",
+    "def create_oracle_sft_dataset(tokenizer: Any, num_samples: int) -> Dataset:\n",
+    "    rows: List[Dict[str, List[int]]] = []\n",
+    "    stages = sample_stage_schedule(num_samples)\n",
+    "    for seed, stage in enumerate(stages, start=514159):\n",
+    "        env = CounselEnvironment()\n",
+    "        obs = env.reset(seed=seed, curriculum_stage=stage, episode_id=f'qwen8b_sft_{seed}')\n",
+    "        messages = [{'role': 'user', 'content': f'{BASE_PROMPT}\\n\\n{reset_text(obs)}'}]\n",
+    "        for contradiction in env.witness.contradictions:\n",
+    "            question = f'{contradiction.trigger_keywords[0]}?'\n",
+    "            assistant_message = {'role': 'assistant', 'content': tool_call('ask_question', {'question': question})}\n",
+    "            rows.append(tokenize_assistant_action(tokenizer, messages, assistant_message))\n",
+    "            messages.append(assistant_message)\n",
+    "            obs = env.step(CounselAction(tool='ask_question', text=question))\n",
+    "            messages.append({'role': 'user', 'content': f'<tool_response>\\n{tool_feedback(obs)}\\n</tool_response>'})\n",
+    "            exhibit_id = contradiction.disprover_evidence_id\n",
+    "            assistant_message = {\n",
+    "                'role': 'assistant',\n",
+    "                'content': tool_call('present_evidence', {'exhibit_id': exhibit_id}),\n",
+    "            }\n",
+    "            rows.append(tokenize_assistant_action(tokenizer, messages, assistant_message))\n",
+    "            messages.append(assistant_message)\n",
+    "            obs = env.step(CounselAction(tool='present_evidence', exhibit_id=exhibit_id))\n",
+    "            messages.append({'role': 'user', 'content': f'<tool_response>\\n{tool_feedback(obs)}\\n</tool_response>'})\n",
+    "        if INCLUDE_REST_ROWS:\n",
+    "            assistant_message = {'role': 'assistant', 'content': tool_call('rest_case', {})}\n",
+    "            rows.append(tokenize_assistant_action(tokenizer, messages, assistant_message))\n",
+    "    return Dataset.from_list(rows)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 7. Load the 4-bit model and attach LoRA adapter\n",
+    "\n",
+    "NF4 quantization with bf16 compute, double quantization, and `paged_adamw_8bit` optimizer. LoRA targets every Qwen attention and MLP projection."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True)\n",
+    "if tokenizer.pad_token is None:\n",
+    "    tokenizer.pad_token = tokenizer.eos_token\n",
+    "\n",
+    "dataset = create_oracle_sft_dataset(tokenizer, SFT_DATASET_SIZE)\n",
+    "print(f'sft_row_count={len(dataset)}')\n",
+    "\n",
+    "quantization = BitsAndBytesConfig(\n",
+    "    load_in_4bit=True,\n",
+    "    bnb_4bit_quant_type='nf4',\n",
+    "    bnb_4bit_compute_dtype=torch.bfloat16,\n",
+    "    bnb_4bit_use_double_quant=True,\n",
+    ")\n",
+    "model = AutoModelForCausalLM.from_pretrained(\n",
+    "    MODEL,\n",
+    "    quantization_config=quantization,\n",
+    "    device_map='auto',\n",
+    "    trust_remote_code=True,\n",
+    ")\n",
+    "model.config.use_cache = False\n",
+    "model = prepare_model_for_kbit_training(model)\n",
+    "lora_config = LoraConfig(\n",
+    "    r=LORA_R,\n",
+    "    lora_alpha=LORA_ALPHA,\n",
+    "    lora_dropout=LORA_DROPOUT,\n",
+    "    bias='none',\n",
+    "    task_type='CAUSAL_LM',\n",
+    "    target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'],\n",
+    ")\n",
+    "model = get_peft_model(model, lora_config)\n",
+    "model.print_trainable_parameters()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 8. Train\n",
+    "\n",
+    "Default budget: 220 steps, batch size 1 with grad-accum 4 (effective batch size 4), learning rate 1e-4. Run4b finished this in **1287.7 seconds** with `train_loss = 0.0565`."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "OUTPUT_DIR.mkdir(parents=True, exist_ok=True)\n",
+    "\n",
+    "args = TrainingArguments(\n",
+    "    output_dir=str(OUTPUT_DIR),\n",
+    "    per_device_train_batch_size=1,\n",
+    "    gradient_accumulation_steps=GRADIENT_ACCUMULATION_STEPS,\n",
+    "    num_train_epochs=SFT_EPOCHS,\n",
+    "    max_steps=SFT_MAX_STEPS if SFT_MAX_STEPS > 0 else -1,\n",
+    "    learning_rate=SFT_LEARNING_RATE,\n",
+    "    optim='paged_adamw_8bit',\n",
+    "    logging_steps=10,\n",
+    "    save_strategy='no',\n",
+    "    report_to='none',\n",
+    "    bf16=torch.cuda.is_available(),\n",
+    "    gradient_checkpointing=True,\n",
+    "    remove_unused_columns=False,\n",
+    ")\n",
+    "trainer = Trainer(\n",
+    "    model=model,\n",
+    "    args=args,\n",
+    "    train_dataset=dataset,\n",
+    "    data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True, label_pad_token_id=-100),\n",
+    ")\n",
+    "train_result = trainer.train()\n",
+    "trainer.save_model(str(OUTPUT_DIR))\n",
+    "tokenizer.save_pretrained(str(OUTPUT_DIR))\n",
+    "print(train_result.metrics)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 9. Write training summary and (optionally) upload the adapter\n",
+    "\n",
+    "The summary JSON is exactly the file mirrored at `assets/trained_eval_run4b_8b_sft/training_summary.json` for run4b. Adapter upload runs only when `HF_TOKEN` is set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "summary = {\n",
+    "    'recipe': 'qwen3_8b_qlora_oracle_sft',\n",
+    "    'base_model': MODEL,\n",
+    "    'artifact_repo': ARTIFACT_REPO,\n",
+    "    'space_repo': SPACE_REPO,\n",
+    "    'sft_case_count': SFT_DATASET_SIZE,\n",
+    "    'sft_row_count': len(dataset),\n",
+    "    'sft_epochs': SFT_EPOCHS,\n",
+    "    'sft_max_steps': SFT_MAX_STEPS,\n",
+    "    'sft_learning_rate': SFT_LEARNING_RATE,\n",
+    "    'max_sft_length': MAX_SFT_LENGTH,\n",
+    "    'lora_r': LORA_R,\n",
+    "    'lora_alpha': LORA_ALPHA,\n",
+    "    'lora_dropout': LORA_DROPOUT,\n",
+    "    'gradient_accumulation_steps': GRADIENT_ACCUMULATION_STEPS,\n",
+    "    'include_rest_rows': INCLUDE_REST_ROWS,\n",
+    "    'metrics': getattr(train_result, 'metrics', {}) or {},\n",
+    "}\n",
+    "summary_path = OUTPUT_DIR / 'training_summary.json'\n",
+    "summary_path.write_text(json.dumps(summary, indent=2, sort_keys=True), encoding='utf-8')\n",
+    "print(json.dumps(summary, indent=2, sort_keys=True))\n",
+    "\n",
+    "token = os.getenv('HF_TOKEN')\n",
+    "if token:\n",
+    "    api = HfApi(token=token)\n",
+    "    api.create_repo(repo_id=ARTIFACT_REPO, repo_type='model', exist_ok=True)\n",
+    "    api.upload_folder(\n",
+    "        repo_id=ARTIFACT_REPO,\n",
+    "        repo_type='model',\n",
+    "        folder_path=str(OUTPUT_DIR),\n",
+    "        commit_message=f'Upload Counsel-Env QLoRA SFT adapter ({SFT_MAX_STEPS} max steps)',\n",
+    "    )\n",
+    "    print(f'Uploaded QLoRA SFT adapter to https://huggingface.co/{ARTIFACT_REPO}')\n",
+    "else:\n",
+    "    print('HF_TOKEN not set; skipping artifact upload.')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 10. Reproducing run4b numbers\n",
+    "\n",
+    "After training finishes, evaluate against the same 150 deterministic seeds used for the official scorecard:\n",
+    "\n",
+    "```\n",
+    "python counsel_env/scripts/evaluate_trained_checkpoint_job.py\n",
+    "```\n",
+    "\n",
+    "Expected on run4b:\n",
+    "\n",
+    "| Metric | Value |\n",
+    "| --- | ---: |\n",
+    "| `train_loss` | 0.0565 |\n",
+    "| `train_runtime` (s) | 1287.7 |\n",
+    "| `avg_reward` (150 seeds) | 0.864 |\n",
+    "| `primary_reward` (150 seeds) | 0.943 |\n",
+    "| `surface_rate` | 0.943 |\n",
+    "| `invalid_tool_calls` | 0 |"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "name": "python",
+   "version": "3.11"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}