README.md

---
library_name: hd-ppo
tags:
- Pendulum-v1
- deep-reinforcement-learning
- reinforcement-learning
- hyperdimensional-computing
- fractional-power-encoding
- LTU-AI
model-index:
- name: HD-PPO
  results:
  - task:
      type: reinforcement-learning
      name: reinforcement-learning
    dataset:
      name: Pendulum-v1
      type: Pendulum-v1
    metrics:
    - type: mean_reward
      value: -125.19 +/- 88.07
      name: mean_reward
      verified: false
---

# **HD-PPO** Agent playing **Pendulum-v1**

This is a trained **HD-PPO** (Hyperdimensional Proximal Policy Optimization) agent
playing **Pendulum-v1** using **gradient-adaptive Fractional Power Encoding (FPE)**
with a prune-and-fine-tune pipeline.

Published by [LTU-AI](https://huggingface.co/LTU-AI).

## Pipeline

1. Train a teacher at **D=512** with gradient-adaptive single-beta FPE.
2. Prune by actor-weight importance through **D=512 → 128 → 32**.
3. Fine-tune each pruned checkpoint with PPO.

Published checkpoint: seed **2024**, compact **D=32** model
(held-out eval mean reward **-125.19 ± 88.07**).

## Usage

Install dependencies:

```bash
pip install -r requirements.txt
```

Evaluate the local checkpoint:

```bash
python enjoy.py --weights hdppo-Pendulum-v1/weights.npz --episodes 10
```

Render episodes:

```bash
python enjoy.py --weights hdppo-Pendulum-v1/weights.npz --render --episodes 3
```

Record a replay video:

```bash
python record_video.py --weights hdppo-Pendulum-v1/weights.npz --output replay.mp4
```

Load from Hugging Face Hub:

```bash
python enjoy.py --weights LTU-AI/hdppo-Pendulum-v1 --episodes 10
```

## Training pipeline

Reproduce the teacher → prune → fine-tune workflow:

```bash
python run_prune_finetune_5seed.py
```

## Hyperparameters

```python
{
    "env": "Pendulum-v1",
    "algo": "HD-PPO (gradient-adaptive FPE, continuous)",
    "teacher_D": 512,
    "pruned_D": 32,
    "beta_base": 2.5,
    "timesteps_per_stage": 1000000,
    "seed": 2024
}
```

## Environment Arguments

```python
{
    "render_mode": "rgb_array"
}
```

## Model files

| File | Description |
|------|-------------|
| `hdppo-Pendulum-v1/weights.npz` | Published actor (+ critic if HD) and FPE encoder (D=32) |
| `hdppo-Pendulum-v1/weights_D512_teacher.npz` | Teacher checkpoint (D=512) |
| `replay.mp4` | Sample rollout video from the published min-D checkpoint |
| `results.json` | Evaluation summary for the published checkpoint |
| `results_D512_teacher.json` | Evaluation summary for the teacher |
| `config.yml` | Training hyperparameters |
| `train_hdppo.py` / training modules | Self-contained training code |

## Citation

If you use this model, please cite the HD-PPO / Hybrid-HD-PPO work.