Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/episode_flow.jpg filter=lfs diff=lfs merge=lfs -text
+assets/why-rag.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -9,7 +9,9 @@ pinned: false
 app_port: 8000
 base_path: /web
 tags:
-  - openenv
+  - OpenEnv
+  - RL
+---
 
 # Research -> Interactive Explainer Environment
 
@@ -25,108 +27,93 @@ happen.
 So this environment trains a model to create interactive explanations instead of only
 writing paragraphs.
 
-Given a STEM topic, the agent:
+Given a topic, the agent:
 
 1. researches the topic,
 2. builds a [Marimo](https://marimo.io/) notebook or [Manim](https://www.manim.community/) animation,
 3. receives validation feedback,
 4. gets one chance to repair the artifact.
 
-```text
-research -> create -> verify -> repair
-```
+Expected Episode Flow:
+![Expected Episode Flow](./assets/episode_flow.jpg)
 
-```mermaid
-flowchart TD
-    A[Topic assigned] --> B{Explore<br/>up to 3 times}
-    T[Research tools<br/>Wikipedia<br/>arXiv<br/>Semantic Scholar<br/>HF Papers<br/>HF Hub<br/>Docs] -.-> B
-    B --> C[Collected context]
-    C --> D[Generate code<br/>Marimo notebook or Manim animation]
-    D --> E{Lint / build<br/>passes?}
-    E -- yes --> F[Done]
-    E -- no --> G[Feedback returned<br/>errors + hints]
-    G --> H[Repair once]
-    H --> E
-```
+## Rewarding Better
 
+The rewards are designed to be verifiable. We do not need an LLM judge.
 
+For every step, we first reward the basics:
 
-## Why This Environment Exists
+- Is action a valid JSON? (fields as well)
+- Correct action in correct phase of episode?
 
-The goal is not just "make an LLM explain a concept."
+Then each action gets its own reward logic.
 
-The goal is to make the model produce something a learner can see and interact with.
-Marimo gives us sliders, plots, tables, and reactive notebooks. Manim gives us visual
-math and step-by-step animations.
+### `explore`
 
-This makes the environment a good fit for training models that teach through artifacts,
-not just fluent text.
+- **Relevance**
+  - Is relevant tool for the topic?
+  - Is search query related?
+  - Useful content in retrieved sources?
+  - Coverage of task keywords?
+- Avoidance of similar **repetitive** searches?
+- Too many **explore steps**?
 
-## What Was Challenging
+### `generate`
 
-Reward design was the hardest part.
+- Correct **format** selected: Marimo or Manim?
+- Artifact includes the important topic **keywords**?
+- Code **parses**?
+  - Marimo: `marimo check` pass?
+  - Manim: code defines a valid scene structure?
+  - Can the artifact actually run/export/render?
 
-If the reward is too vague, the model does not know what improved. If the reward is too
-easy to game, the model can collect points while still producing broken or useless
-artifacts.
+### `repair`
 
-So the rewards are mostly verifiable:
+- Error (lint/build) addressed?
+- Passes validation?
+- Avoided repeation?
 
-- Did the model return valid JSON?
-- Did it choose a useful research tool?
-- Did the search add new information?
-- Did the generated code parse?
-- Did `marimo check` pass?
-- Did Manim or Marimo actually run?
-- Did the repair fix the previous error?
+> More details in [rewards/README.md](rewards/README.md)
 
-This keeps the training grounded. A broken notebook should not score well just because
-it mentions the right keywords.
+## Quirks
 
-Training was the other challenge. A base model may know Python and may know a topic, but
-it does not automatically know this environment's workflow. It has to learn how to act:
-when to research, when to stop, how to emit actions, how to write Marimo/Manim code, and
-how to repair from feedback.
+### Why RAG is done here?
 
-## Why SFT First
+We are training SLMs.<br>
+This will be a long-horizon task.<br>
+Where we need to use a lot of context - which comes from the exploration steps.<br>
 
-We use SFT as a warm start before RL.
+To only keep relevant context in the observation:<br>
+Model: `bge-small-en-v1.5`
 
-The SFT data includes the task bank, synthetic explore/generate/repair examples, and
-real Marimo and Manim examples. This teaches the model the shape of valid artifacts
-before GRPO starts optimizing rewards.
+![RAG](./assets/why-rag.jpg)
 
-In simple terms:
+### Selection of the SLM
 
-- Pre-trained model: can talk about topics, but may not follow the environment.
-- SFT model: learns the expected action format and Marimo/Manim style.
-- RL model: should improve using environment rewards like successful execution and
-repair.
+- We need SLMs with long context
+- We need it to be < 3B parameters
+- We need it intelligent enough
 
-## Current Status
+We have selected - `Mistral-3-3B`
 
-Completed:
+### Why SFT is done in our case?
 
-- OpenEnv server and client.
-- Explore -> generate -> repair episode flow.
-- Research tools for web/docs/papers/HF Hub.
-- Marimo and Manim validation.
-- Verifiable reward components.
-- SFT data preparation.
-- SFT and GRPO training scripts.
+Even 8B models are not writing good Marimo/Manim code properly/correctly.<br>
+We are extractin tutorials/examples/guides code from the Marimo and Manim cloned repos.<br>
+Created samples.<br>
+And did and SFT to teach/align our SLM to the expected Marimo/Manim code style.<br>
 
-Still remaining:
+## Links
 
-- Run full SFT and GRPO training.
-- Add final pre-trained vs SFT vs RL reward comparisons.
-- Add reward curves and final model results.
+SFT Code: [train/sft_unsloth.py](https://gitlab.com/kgdrathan/openenv-explainer/-/blob/main/train/sft_unsloth.py)
+RL GRPO Code: [train/grpo_unsloth.py](https://gitlab.com/kgdrathan/openenv-explainer/-/blob/main/train/grpo_unsloth.py)
+Dashboard for interacting with the environment: [explainer-env-dashboard](https://kgdrathan-explainer-env-dashboard.hf.space/)
 
-## Links
+> Dashboard is for looking at logs and interacting with the environment.
+
+## Status
+
+Completed: Environment and SFT
+Remaining: RL GRPO training
 
-- Environment: [kgdrathan-explainer-env.hf.space](https://kgdrathan-explainer-env.hf.space)
-- Design notes: [design.md](../design.md)
-- Reward details: [rewards/README.md](rewards/README.md)
-- SFT data script: [train/prepare_data.py](https://gitlab.com/kgdrathan/openenv-explainer/-/blob/main/train/prepare_data.py)
-- SFT training script: [train/sft_unsloth.py](https://gitlab.com/kgdrathan/openenv-explainer/-/blob/main/train/sft_unsloth.py)
-- RL training script: [train/grpo_unsloth.py](https://gitlab.com/kgdrathan/openenv-explainer/-/blob/main/train/grpo_unsloth.py)