day3_4_exp2 - Visual Clustering ACT (VC-ACT)

Visual Clustering-based Action Chunking Transformer for pick_hanger task.

Model Info

• Architecture: VC-ACT (Visual Clustering + 10 specialized ACT models)
• Dataset: 26-03-04-pick_hanger_V30 (100 episodes, clustered into 10 groups)
• Training Steps: 100,010 per cluster
• Robot: Piper dual-arm (7D + 7D = 14D state/action)
• Cameras: fisheye_left, fisheye_right, wide_mid, wide_top (256x256)
• FPS: 30

Repository Structure

day3_4_exp2/
├── models/
│   ├── vc_k10_cluster0/pretrained_model/  # Cluster 0 ACT model
│   ├── vc_k10_cluster1/pretrained_model/  # Cluster 1 ACT model
│   ├── ...
│   └── vc_k10_cluster9/pretrained_model/  # Cluster 9 ACT model
├── cluster_classifier/                     # Cluster classifier (ResNet-18)
│   └── cluster_classifier_best.pth
├── scripts/                                # Training pipeline scripts
│   ├── 1_extract_features.py
│   ├── 2_cluster_and_visualize.py
│   ├── 3_split_dataset_by_cluster.py
│   └── 4_train_cluster_classifier.py
└── normalization/
    └── 26-03-04-pick_hanger_V30_norm.py

Method Overview

VC-ACT uses a two-stage approach:

1. Visual Clustering: K-means clustering (k=10) on visual features extracted from the dataset
2. Specialized Training: Train separate ACT models for each cluster
3. Runtime Classification: Use a trained classifier to select the appropriate model

Usage

Download model

pip install huggingface_hub
huggingface-cli download QRP123/day3_4_exp2 --repo-type model --local-dir ./day3_4_exp2

Load cluster classifier

import torch
from torchvision import models

# Load classifier
checkpoint = torch.load("./day3_4_exp2/cluster_classifier/cluster_classifier_best.pth")
classifier = models.resnet18(pretrained=False)
classifier.fc = torch.nn.Linear(classifier.fc.in_features, 10)
classifier.load_state_dict(checkpoint["model_state_dict"])
classifier.eval()

Load cluster-specific ACT model

from lerobot.policies.act.modeling_act import ACTPolicy

# Load model for cluster 0
cluster_id = 0
policy = ACTPolicy.from_pretrained(f"./day3_4_exp2/models/vc_k10_cluster{cluster_id}/pretrained_model")
policy.eval()

Load normalization (cluster-specific)

import sys
sys.path.append("./day3_4_exp2/normalization")

# Load normalization for cluster 0
from cluster0_v30_norm import STATE_MEAN, STATE_STD, ACTION_MEAN, ACTION_STD

# Or dynamically:
cluster_id = 0
norm_module = __import__(f"cluster{cluster_id}_v30_norm")
STATE_MEAN = norm_module.STATE_MEAN

Training Pipeline

The scripts/ directory contains the complete training pipeline:

1. 1_extract_features.py: Extract visual features from dataset using pre-trained ResNet
2. 2_cluster_and_visualize.py: Perform K-means clustering and visualize results
3. 3_split_dataset_by_cluster.py: Split dataset into cluster-specific subsets
4. 4_train_cluster_classifier.py: Train classifier to predict cluster from images

Training Config

• chunk_size: 100
• n_action_steps: 100
• batch_size: 8
• num_clusters: 10
• use_vae: true

Comparison with Other Methods

Model	Architecture	Parameters	Inference	Training
day3_4_base	Single ACT	~50M	Fast	Simple
day3_4_exp1	MoE-ACT (10 experts, top-2)	~200M	2x slower	Complex
day3_4_exp2	VC-ACT (10 models)	10×50M	Fast (1 model)	Modular

Advantages of VC-ACT:

• Modular: Each cluster model can be trained/updated independently
• Interpretable: Visual clustering provides insight into task structure
• Efficient: Only one model active at inference time
• Scalable: Easy to add/remove clusters

Citation

If you use this model, please cite:

@misc{day3_4_exp2,
  title={VC-ACT: Visual Clustering-based Action Chunking Transformer},
  author={QRP123},
  year={2026},
  howpublished={\url{https://huggingface.co/QRP123/day3_4_exp2}}
}