README.md

---
license: mit
language:
- en
tags:
- autonomous-driving
- self-driving-car
- robotics
- imitation-learning
- behavioral-cloning
- pilotnet
- esp8266
- esp32-cam
- pytorch
- classification
library_name: pytorch
pipeline_tag: image-classification
model-index:
- name: ActionNet
  results: []
---

# ActionNet — Autonomous RC Car Driving Model

A lightweight classification CNN that drives a small RC car by predicting discrete motor actions from raw camera frames. Trained through imitation learning — a human drives the car while the system records frames and commands, then the model learns to replicate that behavior.

Part of the [OpenBot PC Server Project](https://github.com/loki-smip/openbot-pc-server-projuct-).

---

## Model Description

ActionNet classifies a single 66×200 RGB camera image into one of 9 discrete driving actions. It replaces the traditional regression approach (predicting continuous steering angles) because keyboard-driven training data only contains a handful of unique command pairs. Classification with cross-entropy loss handles this much better than mean-squared-error regression, which tends to average everything toward zero.

**Input:** 66×200×3 RGB image (cropped from 800×600, top 40% removed)
**Output:** probability distribution over 9 driving actions

### The 9 Actions

| Index | Action | Left Motor | Right Motor | Description |
|---|---|---|---|---|
| 0 | STOP | 0 | 0 | Both motors off |
| 1 | FORWARD | +70 | +70 | Straight ahead |
| 2 | BACKWARD | -70 | -70 | Straight reverse |
| 3 | TURN LEFT | -49 | +49 | Pivot left (in place) |
| 4 | TURN RIGHT | +49 | -49 | Pivot right (in place) |
| 5 | FORWARD+LEFT | +21 | +70 | Arc forward-left |
| 6 | FORWARD+RIGHT | +70 | +21 | Arc forward-right |
| 7 | BACKWARD+LEFT | -21 | -70 | Arc backward-left |
| 8 | BACKWARD+RIGHT | -70 | -21 | Arc backward-right |

Motor values are shown at speed=70 and scale proportionally with the speed setting.

---

## Architecture

The convolutional backbone is based on NVIDIA's PilotNet (from the "End to End Learning for Self-Driving Cars" paper), modified with batch normalization, ELU activations, and a classification head.

```
Layer                          Output Shape      Parameters
─────────────────────────────────────────────────────────────
Input                          (B, 3, 66, 200)   —

Conv2d(3→24, 5×5, stride=2)   (B, 24, 31, 98)   1,824
BatchNorm2d(24)                                   48
ELU                                               —

Conv2d(24→36, 5×5, stride=2)  (B, 36, 14, 47)   21,636
BatchNorm2d(36)                                   72
ELU                                               —

Conv2d(36→48, 5×5, stride=2)  (B, 48, 5, 22)    43,248
BatchNorm2d(48)                                   96
ELU                                               —

Conv2d(48→64, 3×3, stride=1)  (B, 64, 3, 20)    27,712
BatchNorm2d(64)                                   128
ELU                                               —

Conv2d(64→64, 3×3, stride=1)  (B, 64, 1, 18)    36,928
BatchNorm2d(64)                                   128
ELU                                               —

Dropout2d(0.15)                                   —

Flatten                        (B, 1152)          —
Dropout(0.35)                                     —
Linear(1152→64)                (B, 64)            73,792
ELU                                               —
Dropout(0.35)                                     —
Linear(64→9)                   (B, 9)             585

─────────────────────────────────────────────────────────────
Total trainable parameters:    ~145,000
Model file size:               ~1–2 MB (.pth)
```

### Design Decisions

- **BatchNorm after every conv layer** — stabilizes training and allows higher learning rates without divergence
- **ELU instead of ReLU** — avoids dead neurons and produces smoother gradients, which matters when the model is small
- **Spatial Dropout2d (15%)** — drops entire feature maps instead of individual pixels, forcing the network to spread information across channels
- **Two-layer classification head with 35% dropout** — the bottleneck at 64 units forces compression and fights overfitting on small datasets
- **Kaiming initialization** — all conv and linear layers use He initialization (fan-out mode), which pairs well with ELU activations
- **Label smoothing (0.2)** — prevents the model from becoming overconfident on exact training labels. A STOP frame labeled as [1.0, 0.0, 0.0, ...] becomes [0.82, 0.02, 0.02, ...], which improves generalization

---

## Preprocessing

The full pipeline from raw camera frame to model input:

```
Raw 800×600 BGR frame from ESP32-CAM
             │
             ▼
    Crop top 40% of the image
    (removes ceiling, sky, and upper walls)
             │
             ▼
    Convert BGR → RGB
             │
             ▼
    Resize to 200×66 pixels
    (using INTER_AREA interpolation)
             │
             ▼
    ToTensor → normalize to [0, 1] float32
             │
             ▼
    Final shape: [batch, 3, 66, 200]
```

The `crop_and_resize()` function in `trainer.py` performs this transformation. The exact same function is called during both training and inference (in `autopilot.py`) to guarantee consistency.

Why crop the top 40%? Because the camera is mounted on a low car pointing forward. The top portion of every frame shows ceiling, walls, or sky — none of which help the model decide where to steer. Removing it reduces noise and lets the model focus on the ground, obstacles, and path ahead.

---

## Training Configuration

| Parameter | Value | Notes |
|---|---|---|
| Optimizer | AdamW | weight_decay=5e-3 for L2 regularization |
| Learning Rate | 0.001 | Peak rate, with OneCycleLR schedule |
| LR Schedule | OneCycleLR | 10% warmup, cosine anneal, div_factor=10 |
| Loss Function | CrossEntropyLoss | label_smoothing=0.2 |
| Batch Size | 32 | Fits comfortably in CPU memory |
| Gradient Clipping | max_norm=1.0 | Prevents gradient explosions |
| Early Stopping | 30 epochs patience | Monitored by validation accuracy |
| Class Balancing | WeightedRandomSampler | Inverse-frequency weights per class |
| Train/Val Split | 80% / 20% | Random split |

### Data Augmentation

Applied on-the-fly during training:

| Augmentation | Probability | Details |
|---|---|---|
| Horizontal flip | 50% | Action labels are mirrored (LEFT↔RIGHT) |
| Random shadow | 50% | Vertical band at random brightness (30–70%) |
| Random brightness | 50% | HSV V-channel scaled 0.6–1.4× |
| Gaussian blur | 30% | Kernel 3×3 or 5×5 |
| Random translation | 40% | Shift ±10% in X and Y |
| Random erasing | 50% | Rectangular cutout on tensor |

The horizontal flip augmentation automatically swaps left/right action labels using a predefined mirror table, so the model never sees contradictory labels.

---

## Inference

At runtime, the autopilot module:

1. Reads the latest camera frame from the MJPEG stream
2. Runs `crop_and_resize()` → converts to tensor
3. Forward pass through ActionNet → gets 9 logits
4. Applies softmax → picks the action with highest probability
5. Uses a 3-frame majority vote to smooth out flickering predictions
6. Maps the smoothed action to (left, right) motor commands at the configured speed
7. Sends the command to the ESP8266 over WebSocket

The inference loop runs at 10 FPS on a typical laptop CPU. No GPU required.

---

## Hardware Requirements

This model is designed for a specific hardware setup:

| Component | Role |
|---|---|
| ESP32-CAM (OV2640) | Streams 800×600 MJPEG video over HTTP |
| ESP8266 (NodeMCU) | Receives motor commands over WebSocket, drives L298N |
| L298N Motor Driver | Controls 2 DC gear motors (differential drive) |
| SG90 Servo (optional) | Camera pan |
| PC (any laptop/desktop) | Runs the server, training, and inference |

The PC does all the heavy lifting. The microcontrollers are just I/O — one for video, one for motors. Total hardware cost is around $25–30 USD.

---

## How to Use This Model

### Quickstart

```python
import torch
import torch.nn.functional as F
from torchvision import transforms
from model import ActionNet, action_to_command

# Load
device = torch.device("cpu")
model = ActionNet().to(device)
checkpoint = torch.load("trained_models/autopilot.pth", map_location=device)
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# Prepare a 66x200 RGB image as tensor
transform = transforms.ToTensor()
img_tensor = transform(your_66x200_rgb_image).unsqueeze(0).to(device)

# Predict
with torch.no_grad():
    logits = model(img_tensor)
    probs = F.softmax(logits, dim=1)
    action = torch.argmax(probs, dim=1).item()
    confidence = probs[0, action].item()

# Convert to motor command
left, right = action_to_command(action, speed=70)
print(f"Action: {action}, Motors: L={left} R={right}, Confidence: {confidence:.1%}")
```

### Within the Full System

The model is used automatically by the autopilot module. Start the server, record some training data through the dashboard, train from the dashboard, then click "Start Autopilot."

See the full [README](https://github.com/YOUR_USERNAME/openbot-pc-server-project) for step-by-step instructions including hardware assembly, firmware upload, and data collection.

---

## Training Your Own Model

1. Assemble the hardware (ESP8266 + ESP32-CAM + motors)
2. Flash firmware to both microcontrollers
3. Start the PC server: `python app.py`
4. Drive the car manually while recording data
5. Click "Train" in the dashboard — or the model trains through the API
6. The best checkpoint saves automatically to `trained_models/autopilot.pth`

Training runs on CPU. A dataset of 3,000 frames trains in under 5 minutes on a modern laptop. GPU is supported if available but not required.

---

## Limitations

- The model only knows what it has seen. If you train it in one room, it won't generalize to a different room without additional data.
- Keyboard inputs produce jerky, discrete commands. A joystick or gamepad would produce smoother training data.
- The 40% top-crop assumes the camera is mounted pointing roughly forward and slightly down. If your camera angle is very different, adjust the crop ratio in `trainer.py`.
- Performance depends heavily on lighting conditions matching between training and inference.
- The model has no notion of obstacles, goals, or maps. It purely replicates the visual patterns it was trained on.

---

## Citation

If you use this project in your work, a mention is appreciated but not required:

```
OpenBot PC Server Project — Autonomous RC Car with Imitation Learning
https://github.com/YOUR_USERNAME/openbot-pc-server-project
```

---

## License

MIT License — use it, modify it, ship it.