README: queues/ path for AAMAS27 launchers

5a0a92d verified 7 days ago

14.7 kB

license: mit
library_name: other
tags:
  - robotics
  - oat
  - imitation-learning
  - robomimic
  - metaworld
  - robocasa
  - benchmark
  - latency
pretty_name: AAAI OAT Paper Models + Eval Artifacts

AAAI OAT — canonical artifact pack (models + eval)

This repo is the primary copy for paper artifacts.
Training data lives separately in hackhackhack66666/aaai-datasets.

Paper track: Consensus Selection (CS-D) on OAT — matched baseline / BoN / AWR across RoboMimic, MetaWorld, RoboCasa.

Start here (reviewer navigation)

If you need…	Open this first
Final paper numbers + protocol	`docs/RESULTS.md`
Selector baselines on top of PDF (AAMAS27)	`docs/aamas27/README.md` (this pack) + `eval/aamas27_selector/`
RoboCasa-specific protocol / literal-5 seeds	`docs/RESULTS_ROBOCASA.md`
Machine-readable file list	`INVENTORY.txt`
Per-suite SR summaries	`eval/matched_s10000/<suite>/summary.json` or `.../robocasa/<task>/summary_literal5.json`
Latency tables	`eval/matched_s10000/table_c.json`, `table_c_fair_kv.json`
Checkpoints to rerun eval	`checkpoints/my_models/` + `checkpoints/selected_from_output/` + `checkpoints/all_training/`
Train configs	`hydra/<run>/.hydra/config.yaml`
Raw training data	datasets repo (not here)

Two-repo layout

Repo	Role	Size (approx)
`hackhackhack66666/aaai-datasets`	Zarr + retained HDF5 for training	~22 GB
`hackhackhack66666/aaai27-models` (this repo)	Checkpoints, matched eval, latency, logs, Hydra, docs	~50+ GB

Directory map

README.md                         # this file
MANIFEST.json                     # pack metadata
INVENTORY.txt                     # flat file listing (generated at upload)

docs/
  RESULTS.md                      # paper ledger (Table P, latency, manifests)
  RESULTS_ROBOCASA.md
  ROBOCASA.md / ROBOMIMIC.md / METAWORLD*.md
  AGENT_GUIDE_TABLE_C_PRIME_LATENCY.md

checkpoints/
  my_models/                      # Wave2 AWR + RoboCasa TopK locks (+ lock txt)
  selected_from_output/           # paper-primary tokenizer/base ckpts (subset)
  all_training/                   # ALL retained TopK/latest ckpts from output runs
  awr16_from_hf/                  # BoN16-distill ckpts mirrored from Mirageinv/AWR

hydra/
  output/<run>/.hydra/            # train configs for paper runs

eval/
  matched_s10000/                 # Table P source-of-truth eval outputs
  eval_out/                       # replan probes + RoboCasa obs audit artifacts
  aamas27_selector/               # CS vs KDPE / selector baselines (on top of PDF)
    aic4/                         # dump from lab-c-0 (only cluster with these runs)
      paired/                     # [E] paired vote/kdpe (+ LIBERO 6 methods)
      paired_seed/                # [C] --policy_seed 0
      aamas27_selector_baselines/ # vote/random/medoid/max_ll (+ RC)
      kdpe_n8/                    # earlier KDPE cells + episodes.jsonl
      latency/                    # Table 5 selector + full-forward JSON/MD/logs
      queues/                     # cell_*.sh / lane_*.sh launchers (reproduce evals)
    ccm/                          # dump from ccmplanner (V100 cells; RC discarded from paper)
      aamas27_selector_baselines/ # close-drawer vote/random/medoid json live here
      ccm_v100_paired/            # can/lift base+vote V100 paired
    INDEX.txt                     # every eval_log.json + latency json
    MANIFEST_AAMAS27.json

docs/aamas27/
  LIBERO_LONG_REZULTATY.md        # LIBERO SR + latency (Russian ledger)
  TABLE5_LATENCY.md
  CS_VS_KDPE_TABLE.md
  SELECTOR_BASELINES_STATUS.md
  AAMAS27_DUMP.md
  SELECTOR_BASELINES_PROTOCOL.md


logs/
  matched_s10000_*.log
  awr_s10000_*wave2*.log
  mw_*bon16_32*.log
  replan_probe_*.log

Table P — where the numbers live

Protocol (RM/MW): test_start_seed=10000, n_test=50, -n 5, OAT8, BoN --bon_free 8 --bon_signal vote.

Protocol (RoboCasa): literal 5 seeds 10000..10004, each -n 1 --n_test 50.

RoboMimic + MetaWorld (matched_s10000 layout)

Suite	Summary	Baseline	BoN8	AWR	Extra
can	`can/summary.json`	`can/baseline_n5/eval_log.json`	`can/bon_n8_n5/`	`can/awr_n5/`	`can/latency*.json`
lift (run A, ep900)	`lift/summary.json`	`lift/baseline_n5/`	`lift/bon_n8_n5/`	`lift/awr_n5/`	—
lift (run B, ep1400)	`lift_ep1400/summary.json`	`lift_ep1400/baseline_n5/`	`lift_ep1400/bon_n8_n5/`	`lift_ep1400/awr_n5/`	paper primary Lift
square	`square/summary.json`	`square/baseline_n5/`	`square/bon_n8_n5/`	`square/awr_n5/`	replan probe in `eval/eval_out/`
coffee-pull	`coffee-pull/summary.json`	…	…	…	BoN16/32: `bon_n16_n5/`, `bon_n32_n5/`
stick-pull	`stick-pull/summary.json`	…	…	…	BoN16/32
disassemble	`disassemble/summary.json`	…	…	…	BoN16/32
box-close	`box-close/summary.json`	…	…	…	BoN16/32

All paths under eval/matched_s10000/.

RoboCasa (literal-5 layout)

Task	Summary	Per-seed evals
close_drawer	`robocasa/close_drawer/summary_literal5.json`	`baseline_seed1000X/`, `bon_n8_seed1000X/`
coffee_press_button	`robocasa/coffee_press_button/summary_literal5.json`	same pattern
turn_off_sink_faucet	`robocasa/turn_off_sink_faucet/summary_literal5.json`	same
turn_off_microwave	`robocasa/turn_off_microwave/summary_literal5.json`	same

Latency (AWR16/KV extension): robocasa/<task>/latency_fair_kv_n16.json.

Checkpoints — paper-primary paths

Wave2 AWR (deployable distill)

Suite	Checkpoint
can	`checkpoints/my_models/awr_s10000_can.ckpt`
lift (run A)	`checkpoints/my_models/awr_s10000_lift.ckpt`
square	`checkpoints/my_models/awr_s10000_square.ckpt`
coffee-pull	`checkpoints/my_models/awr_s10000_coffee-pull.ckpt`
stick-pull	`checkpoints/my_models/awr_s10000_stick-pull.ckpt`
disassemble	`checkpoints/my_models/awr_s10000_disassemble.ckpt`
box-close	`checkpoints/my_models/awr_s10000_box-close.ckpt`

RoboCasa TopK locks (Wave1 base)

Task	Checkpoint	Lock file
close_drawer	`robocasa_close_drawer_topk_ep0500_sr0.700.ckpt`	`robocasa_close_drawer_topk_lock.txt`
coffee_press_button	`robocasa_coffee_press_button_topk_ep0500_sr0.600.ckpt`	`…_topk_lock.txt`
turn_off_sink_faucet	`robocasa_turn_off_sink_faucet_topk_ep0500_sr0.580.ckpt`	`…_topk_lock.txt`
turn_off_microwave	`robocasa_turn_off_microwave_topk_ep0500_sr0.620.ckpt`	`…_topk_lock.txt`

All under checkpoints/my_models/.

Base + tokenizer (selected paper ckpts)

Under checkpoints/selected_from_output/output/<run>/checkpoints/.
See docs/RESULTS.md § Per-suite train → paper base for the canonical mapping.

Full training TopK archive

All retained training checkpoints (TopK + latest.ckpt) are under:

checkpoints/all_training/home/askhabaliev_gs/mipt_paper/oat/output/<date>_<run>/checkpoints/*.ckpt

Note: paths preserve the original cluster absolute prefix from upload staging.
Paper-primary subset is also mirrored under the cleaner prefix:
checkpoints/selected_from_output/output/<date>_<run>/checkpoints/*.ckpt

Use all_training/ if you need non-primary epochs (e.g. Lift ep-0900 vs ep-1400, Square TopK trajectory).

BoN16-distill (latency / extended)

Mirrored under checkpoints/awr16_from_hf/:

robomimic_{can,lift,square}_awr_bon16_e100.ckpt
robocasa_coffee_press_button_awr_bon16_e100.ckpt

Latency (Table C / C')

Table	Aggregate	Per-suite
C (deployed path)	`eval/matched_s10000/table_c.json`	`<suite>/latency.json`
C′ (fair-KV)	`eval/matched_s10000/table_c_fair_kv.json`	`<suite>/latency_fair_kv.json`
C′ AWR16/KV ext	—	`robocasa/<task>/latency_fair_kv_n16.json` (+ RM/MW `*_n16.json`)

Protocol details: docs/AGENT_GUIDE_TABLE_C_PRIME_LATENCY.md.

AAMAS27 — selector baselines on top of the PDF (CS / KDPE / random / medoid / max_ll)

These are not Table P BoN-vs-base. They are the extra selector comparison (Consensus Selection vs KDPE and ablations) plus Table 5 H100 selector latency.

Clusters scanned: aicenter4/lab-c-0 (paper host, 102 eval_log.json + latency), ccmplanner (audit dump, 28 eval_log.json including close-drawer vote/random/medoid), aicenter1 (empty), cds2 (empty). aicenter3 not used.

Git branch: aamas27_selector_baselines
Code eval: cluster ~/oat_code_kdpe
Latency: cluster ~/oat_code_seed (LAST_CALL in kdpe.py)
Do not mix [E] (eval/aamas27_selector/aic4/paired/, no --policy_seed) with [C] (…/paired_seed/, --policy_seed 0).

Two clocks (latency)

Clock	What	Typical N=8 D=7	Paper table
Selector-only	`_bon_select` / KDPE on tensor `[N,R,D]`	CS 0.264–0.272 ms, KDPE 0.53–0.55 ms (~2×)	Table 5
Full forward	vision CNN + AR sample N + detokenize + select	CS/KDPE both ~28 ms (Δ ~0.3 ms, ~1%)	context only

Do not write “CS speeds up inference 2×”. That 2× is selector-only. LIBERO D=7 matches RoboMimic can. MetaWorld D=4: CS still ~0.27 ms, KDPE ~0.20 ms — do not pool with D=7.

Where the numbers live

Root on Hub: eval/aamas27_selector/aic4/
Cluster original: ~/oat_eval_out/ (same relative paths).

Need	Path under `eval/aamas27_selector/`
LIBERO-LONG [E] suite + per-task	`aic4/paired/libero10/{vote,medoid,kdpe,max_likelihood,base,random}/eval_log.json`
RM/MW [E] CS vs KDPE (paired)	`aic4/paired/<task>/{vote,kdpe}/eval_log.json` (+ `base/` on can/lift/square)
RM/MW [E] random/medoid/max_ll	`aic4/aamas27_selector_baselines/<task>/{vote,random,medoid,max_likelihood}_n8/eval_log.json`
RoboCasa [E] (aic4 trusted)	`aic4/aamas27_selector_baselines/{close-drawer,coffee-button,faucet-off,microwave-off}/`
Confirmatory [C]	`aic4/paired_seed/<task>/{vote,kdpe}/eval_log.json`
Episode-level jsonl (paired)	`aic4/paired/<task>/<method>/episodes.jsonl`
Table 5 canon (can, selector+gen)	`aic4/latency/selector_latency_gpu1_n32diag.json`
can full-forward replicate	`aic4/latency/selector_latency_gpu1_clean_20261006.json`
LIBERO latency (selector+gen+full)	`aic4/latency/selector_latency_gpu1_libero10.json`
lift/square + MW D=4 latency	`aic4/latency/selector_latency_gpu2_{lift,square,box-close,coffee-pull,stick-pull,disassemble}.json`
Human ledgers	`docs/aamas27/*.md`
Launch scripts (aic4)	`aic4/queues/` (`cell_.sh`, `lane_.sh`)
ccm close-drawer vote/random/medoid json	`ccm/aamas27_selector_baselines/close-drawer/{vote,random,medoid}_n8/eval_log.json`
ccm V100 paired can/lift	`ccm/ccm_v100_paired/{can,lift}/{base,vote}/eval_log.json`

Paper scalars:

Suite	File	Key
LIBERO	`paired/libero10/<method>/eval_log.json`	`mean_success_rate_mean` (± `…_std`, n=5)
RM/MW/RC	same `eval_log.json`	`mean_success_rate_mean` (reported as % in git docs)

LIBERO [E] (n_test=500, seed 10000, N=8): vote 0.665±0.025 · medoid 0.663±0.028 · kdpe 0.605±0.009 · max_ll 0.584±0.010 · base 0.558±0.018 · random 0.554±0.015.

LIBERO latency N=8 empty GPU1: selector CS 0.272 / KDPE 0.553; full 27.76 / 28.09; gen 22.73.

Discarded / do not use as paper

ccm RoboCasa coffee/microwave/faucet — dumped under ccm/ for audit, not paper (systematically low vs aic4).
aic4/latency/selector_latency_gpu0.json and gpu1.json (warmup-10 / occupied-GPU drafts). Canon = n32diag. Full can = *_clean_20261006.
aic4/paired/square/{vote2,kdpe2} extra repeats — keep for audit; primary square cells are vote/kdpe.
close-drawer vote/medoid: json on ccm, aic4 dirs may lack eval_log.json. Paper close-drawer vote 56.8±5.6 is the aic4-trusted number in STATUS (not the discarded ccm RC SR).

Download this pack only

huggingface-cli download hackhackhack66666/aaai27-models \
  --include "eval/aamas27_selector/**" \
  --include "docs/aamas27/**" \
  --local-dir ./aaai27-models

Known missing artifacts (documented honestly)

These were never saved locally during the latency workflow (download → measure → delete):

Missing file	Impact
`robocasa_close_drawer_awr_bon16_e100.ckpt`	cannot reload AWR16 weights; latency JSON exists
`robocasa_turn_off_sink_faucet_awr_bon16_e100.ckpt`	same
`robocasa_turn_off_microwave_awr_bon16_e100.ckpt`	same
`awr_s10000_lift_ep1400.ckpt`	Lift run B AWR weights not in `my_models/`; eval_log + summary exist

Not blocking for Table P SR reproduction (baseline/BoN/AWR Wave2 primary ckpts are present).
Blocks only exact re-run of those specific AWR16 latency measurements from weights.

How to download

pip install -U huggingface-hub

# full repo (large)
huggingface-cli download hackhackhack66666/aaai27-models --local-dir ./aaai27-models

# one suite eval only
huggingface-cli download hackhackhack66666/aaai27-models \
  eval/matched_s10000/can/summary.json \
  --local-dir ./aaai27-models

# one checkpoint
huggingface-cli download hackhackhack66666/aaai27-models \
  checkpoints/my_models/awr_s10000_can.ckpt \
  --local-dir ./aaai27-models

Datasets (separate):

huggingface-cli download hackhackhack66666/aaai-datasets --repo-type dataset --local-dir ./aaai-datasets

Datasets: https://huggingface.co/datasets/hackhackhack66666/aaai-datasets
Upstream OAT: https://github.com/Chaoqi-LIU/oat
External AWR16 mirror (partial): https://huggingface.co/Mirageinv/AWR

Changelog

Date	Note
2026-08-19	Initial upload: matched eval, my_models, selected ckpts, latency, logs, hydra, docs
2026-08-19	Added `checkpoints/all_training/` (full TopK archive), expanded README navigation
2026-10-06	Added `eval/aamas27_selector/` + `docs/aamas27/`: all aic4 selector-baseline evals, LIBERO-LONG, Table 5 + full-forward latency

hackhackhack66666
/

aaai27-models