paavohtl Claude Opus 4.6 (1M context) commited on
Commit
1b50411
·
1 Parent(s): c7778ea

feat: add headful replay testing to Rust test runner

Browse files

Merge replay-test.ps1's headful mode into openwa-test as a new
`headful` subcommand. Runs with graphics+sound, checks for crashes,
panics, and [GAMEPLAY PASS/FAIL] markers. Supports filter by name
or direct .WAgame path, with configurable --timeout (default 150s).

replay-test.ps1 is now a thin wrapper like run-tests.ps1.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

.claude/skills/replay-test/SKILL.md CHANGED
@@ -1,13 +1,13 @@
1
  ---
2
  name: replay-test
3
- description: Run automated replay-based testing - builds, launches WA.exe with a replay file, captures validation logs, and presents results. Supports headful (graphics + validation) and headless (pure simulation + log diff) modes.
4
  ---
5
 
6
  # Replay Test
7
 
8
  Run the replay-based automated testing pipeline.
9
 
10
- ## Quick Start — Run All Tests
11
 
12
  The fastest way to run all replay tests:
13
 
@@ -15,7 +15,7 @@ The fastest way to run all replay tests:
15
  .\run-tests.ps1
16
  ```
17
 
18
- This builds and runs the `openwa-test` binary via `cargo run`, which then builds the DLL + launcher, discovers all `testdata/replays/*.WAgame` files with matching `*_expected.log`, and runs them concurrently (default 4 workers) in headless mode. Output:
19
 
20
  ```
21
  Building... done (2.1s)
@@ -35,48 +35,45 @@ Options:
35
 
36
  ## Modes
37
 
38
- ### Headless test runner (primary)
39
 
40
  Uses WA's built-in `/getlog` mode: no rendering, pure CPU simulation. Compares the output log byte-for-byte against an expected baseline.
41
 
42
  ```bash
43
- # All tests:
44
  .\run-tests.ps1
45
-
46
- # Single replay via old script:
47
- powershell -ExecutionPolicy Bypass -File replay-test.ps1 -Headless testdata/replays/longbow.WAgame
48
  ```
49
 
50
- ### Headful (interactive) — graphics, sound, validation
51
 
52
- Tests hooks, struct layouts, and game state with the full rendering pipeline active. Use for debugging specific replays.
53
 
54
  ```bash
55
- powershell -ExecutionPolicy Bypass -File replay-test.ps1
 
 
 
56
  ```
57
 
 
 
58
  ## Steps (for Claude)
59
 
60
  1. Run the appropriate test command.
61
 
62
  2. Read the output and present results:
63
 
64
- **Headless (test runner):**
65
  - PASS/FAIL summary is printed directly
66
  - Per-run logs stored in `testdata/runs/<timestamp>/`
67
  - FAIL shows first differing lines between expected and actual
68
 
69
- **Headless (single replay via script):**
70
- - PASS means byte-identical output to `<replay>_expected.log`
71
- - FAIL shows the diff lines
72
-
73
- **Headful mode:**
74
- - If `testdata/logs/errorlog_latest.txt` exists, WA.exe crashed. Read it and report the crash
75
- - Read `testdata/logs/validation_latest.log`:
76
- - **Static checks** (`[STATIC PASS]`/`[STATIC FAIL]`): vtable addresses, struct offsets
77
- - **Gameplay checks** (`[GAMEPLAY PASS]`/`[GAMEPLAY FAIL]`): game init, match start, match completion
78
- - Quote any FAIL lines exactly
79
- - Check `testdata/logs/openwa_latest.log` for `[PANIC]` errors
80
 
81
  ## Environment Variables
82
 
@@ -84,15 +81,15 @@ powershell -ExecutionPolicy Bypass -File replay-test.ps1
84
  - `OPENWA_REPLAY_TEST=1` — (headful) Enable fast-forward mode (50x speed)
85
  - `OPENWA_LOG_PATH=<path>` — Override OpenWA.log location (per-instance isolation)
86
 
87
- All are set automatically by the test runner / replay-test.ps1.
88
 
89
  ## Key Paths
90
 
91
  - Replay files + expected logs: `testdata/replays/*.WAgame` + `*_expected.log`
92
  - Per-run output: `testdata/runs/<timestamp>/` (gitignored)
93
  - Test runner: `crates/openwa-test-runner/` (`openwa-test` binary)
94
- - Convenience script: `run-tests.ps1`
95
- - Single-replay script: `replay-test.ps1`
96
 
97
  ## Adding New Replay Tests
98
 
@@ -103,6 +100,7 @@ All are set automatically by the test runner / replay-test.ps1.
103
 
104
  ## Notes
105
 
106
- - Headful replay: ~15-30s; headless: ~5s per test
 
107
  - Concurrent headless tests use per-PID event names, log paths, and temp files (CreateFileA hook)
108
  - Named event: `OpenWA_HooksReady_{pid}` — ensures hooks installed before WA main thread runs
 
1
  ---
2
  name: replay-test
3
+ description: Run automated replay-based testing - builds, launches WA.exe with a replay file, and presents results. Supports headful (graphics + gameplay checks) and headless (pure simulation + log diff) modes via the openwa-test binary.
4
  ---
5
 
6
  # Replay Test
7
 
8
  Run the replay-based automated testing pipeline.
9
 
10
+ ## Quick Start — Run All Tests (Headless)
11
 
12
  The fastest way to run all replay tests:
13
 
 
15
  .\run-tests.ps1
16
  ```
17
 
18
+ This builds and runs the `openwa-test` binary, which builds the DLL + launcher, discovers all `testdata/replays/*.WAgame` files with matching `*_expected.log`, and runs them concurrently (default 4 workers) in headless mode. Output:
19
 
20
  ```
21
  Building... done (2.1s)
 
35
 
36
  ## Modes
37
 
38
+ ### Headless (primary)
39
 
40
  Uses WA's built-in `/getlog` mode: no rendering, pure CPU simulation. Compares the output log byte-for-byte against an expected baseline.
41
 
42
  ```bash
 
43
  .\run-tests.ps1
44
+ .\run-tests.ps1 longbow
 
 
45
  ```
46
 
47
+ ### Headful — graphics, sound, gameplay validation
48
 
49
+ Tests hooks, struct layouts, and game state with the full rendering pipeline active. Runs sequentially. The game window must be focused once for the replay to start advancing, after which it runs to completion on its own.
50
 
51
  ```bash
52
+ .\replay-test.ps1
53
+ .\replay-test.ps1 bots
54
+ .\replay-test.ps1 testdata/replays/bots.WAgame
55
+ .\replay-test.ps1 bots --timeout 600 # 10 minute timeout
56
  ```
57
 
58
+ Both scripts are thin wrappers around `openwa-test` / `openwa-test headful`.
59
+
60
  ## Steps (for Claude)
61
 
62
  1. Run the appropriate test command.
63
 
64
  2. Read the output and present results:
65
 
66
+ **Headless:**
67
  - PASS/FAIL summary is printed directly
68
  - Per-run logs stored in `testdata/runs/<timestamp>/`
69
  - FAIL shows first differing lines between expected and actual
70
 
71
+ **Headful:**
72
+ - PASS/FAIL summary is printed directly
73
+ - Per-run logs stored in `testdata/runs/headful-<timestamp>/`
74
+ - Checks OpenWA.log for `[PANIC]` errors and `[GAMEPLAY PASS]`/`[GAMEPLAY FAIL]` markers
75
+ - PASS requires: no crash, no panics, no gameplay fails, at least one gameplay pass
76
+ - CRASH shows NTSTATUS name and ERRORLOG.TXT content
 
 
 
 
 
77
 
78
  ## Environment Variables
79
 
 
81
  - `OPENWA_REPLAY_TEST=1` — (headful) Enable fast-forward mode (50x speed)
82
  - `OPENWA_LOG_PATH=<path>` — Override OpenWA.log location (per-instance isolation)
83
 
84
+ All are set automatically by the test runner.
85
 
86
  ## Key Paths
87
 
88
  - Replay files + expected logs: `testdata/replays/*.WAgame` + `*_expected.log`
89
  - Per-run output: `testdata/runs/<timestamp>/` (gitignored)
90
  - Test runner: `crates/openwa-test-runner/` (`openwa-test` binary)
91
+ - Headless script: `run-tests.ps1`
92
+ - Headful script: `replay-test.ps1`
93
 
94
  ## Adding New Replay Tests
95
 
 
100
 
101
  ## Notes
102
 
103
+ - Headful replay: ~15-30s per test; headless: ~5s per test
104
+ - Headful timeout defaults to 150s, configurable with `--timeout SECS`
105
  - Concurrent headless tests use per-PID event names, log paths, and temp files (CreateFileA hook)
106
  - Named event: `OpenWA_HooksReady_{pid}` — ensures hooks installed before WA main thread runs
CLAUDE.md CHANGED
@@ -29,16 +29,31 @@ Unit tests: `cargo test`. These are standard Rust tests covering parsers and typ
29
 
30
  WA.exe can deterministically replay recorded games (`.WAgame` files). Each replay test runs the game with the injected DLL and checks that the output matches a baseline log (`*_expected.log`) captured from unmodified WA.exe.
31
 
32
- **Headless** (`.\run-tests.ps1`): Pure CPU simulation, no rendering. Fast, runs in parallel (default 4 concurrent). Validates game logic — a log mismatch means the Rust code caused a desync. Spurious flakes from race conditions can occur; retry with `-j 1` to confirm.
33
 
34
- **Headful** (`/replay-test` skill): Runs with graphics and sound. Requires user interaction. Needed to validate visual/audio hooks — code can pass headless but crash headful (or vice versa) since rendering and sound paths are only exercised headfully.
35
 
36
- **Use replay testing to validate assumptions and test theories.** Implement a hypothesis, run tests, iterate. You can add temporary log statements and see their results by running replay tests.
37
 
38
- **IMPORTANT:** The `*_expected.log` baselines are ground truth from unmodified WA.exe. They must NEVER be deleted or regenerated. If a test fails, the Rust code is wrong.
 
 
39
 
40
  See `crates/openwa-test-runner/CLAUDE.md` for test isolation, crash detection, adding new tests, and env vars. Use `/desync-debug` skill after a test failure to diagnose.
41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
  ## Debug CLI
43
 
44
  Use the `/debug-cli` skill for live memory inspection, struct inspection, frame-level control, and pointer chain traversal.
 
29
 
30
  WA.exe can deterministically replay recorded games (`.WAgame` files). Each replay test runs the game with the injected DLL and checks that the output matches a baseline log (`*_expected.log`) captured from unmodified WA.exe.
31
 
32
+ Two ways to run replay tests:
33
 
34
+ ### Headless (`.\run-tests.ps1`)
35
 
36
+ Pure CPU simulation, no rendering. Fast, runs in parallel (default 4 concurrent). Validates game logic — a log mismatch means the Rust code caused a desync. Spurious flakes from race conditions can occur; retry with `-j 1` to confirm.
37
 
38
+ ### Headful (`.\replay-test.ps1` or `openwa-test headful`)
39
+
40
+ Runs with graphics and sound. The game window must be focused once to start. Needed to validate visual/audio hooks — code can pass headless but crash headful (or vice versa) since rendering and sound paths are only exercised headfully. Checks for crashes, panics, and `[GAMEPLAY PASS/FAIL]` markers. Timeout configurable with `--timeout SECS` (default 150s).
41
 
42
  See `crates/openwa-test-runner/CLAUDE.md` for test isolation, crash detection, adding new tests, and env vars. Use `/desync-debug` skill after a test failure to diagnose.
43
 
44
+ ### Use replay testing to validate assumptions and test theories
45
+
46
+ Implement a hypothesis, run tests, iterate. You can add temporary log statements and see their results by running replay tests.
47
+
48
+ ### **IMPORTANT**: Replay tests can only test two things:
49
+
50
+ - The game simulation matches the original WA.exe, to the extent covered by the replay log and the replay's built-in checksums
51
+ - The game doesn't crash
52
+
53
+ You, as an agent, CANNOT see or hear the game, and therefore you can't verify that anything related to graphics or audio is correct. A passing headful replay test is NOT a guarantee that the rendering is correct. You need to involve the user to verify all graphics and audio related changes before committing or merging.
54
+
55
+ ### **IMPORTANT:** The `*_expected.log` baselines are ground truth from unmodified WA.exe. They must NEVER be deleted or regenerated, unless explicitly requested by the user. If a test fails, the Rust code is wrong.
56
+
57
  ## Debug CLI
58
 
59
  Use the `/debug-cli` skill for live memory inspection, struct inspection, frame-level control, and pointer chain traversal.
crates/openwa-test-runner/CLAUDE.md CHANGED
@@ -1,6 +1,12 @@
1
  # openwa-test-runner
2
 
3
- Headless replay test runner (`openwa-test` binary). Discovers replay tests in `testdata/replays/`, runs them concurrently via WA.exe's `/getlog` mode, and compares output logs byte-for-byte against expected baselines.
 
 
 
 
 
 
4
 
5
  ## Test Isolation
6
 
@@ -18,6 +24,10 @@ All isolation mechanisms are active in headless mode only:
18
 
19
  Tests that crash show `CRASH` (not `FAIL`) with NTSTATUS name and ERRORLOG.TXT content. ERRORLOG.TXT is redirected to the per-test run directory via `OPENWA_ERRORLOG_PATH`.
20
 
 
 
 
 
21
  ## Adding New Replay Tests
22
 
23
  1. Record a game in WA.exe (the replay `.WAgame` file is saved automatically)
@@ -28,11 +38,13 @@ Tests that crash show `CRASH` (not `FAIL`) with NTSTATUS name and ERRORLOG.TXT c
28
  ## Environment Variables
29
 
30
  - `OPENWA_HEADLESS=1` — Headless mode: hooks MessageBoxA to auto-dismiss, launcher uses SW_HIDE, file isolation hook active, semaphore renamed per-PID
 
31
  - `OPENWA_LOG_PATH=<path>` — Override OpenWA.log location (used for per-instance isolation)
32
  - `OPENWA_ERRORLOG_PATH=<path>` — Redirect ERRORLOG.TXT to specified path (crash capture)
33
 
34
  ## Key Paths
35
 
36
  - Replay files + expected logs: `testdata/replays/*.WAgame` + `*_expected.log`
37
- - Per-run output: `testdata/runs/<timestamp>/` (gitignored)
38
- - Convenience script: `run-tests.ps1`
 
 
1
  # openwa-test-runner
2
 
3
+ Replay test runner (`openwa-test` binary). Supports headless (concurrent, log-diff) and headful (sequential, gameplay checks) modes.
4
+
5
+ ## Subcommands
6
+
7
+ - `openwa-test [filter] [-j N] [--no-build] [--wa-path PATH]` — Headless: discovers `testdata/replays/*.WAgame` with `*_expected.log`, runs concurrently via `/getlog`, compares output byte-for-byte.
8
+ - `openwa-test headful [filter|replay.WAgame] [--no-build] [--wa-path PATH] [--timeout SECS]` — Headful: runs with graphics+sound, checks for crashes/panics/gameplay markers. Default timeout 150s.
9
+ - `openwa-test trace-desync <replay.WAgame> [--no-build] [--wa-path PATH]` — Compares per-frame checksums between baseline and hooked runs.
10
 
11
  ## Test Isolation
12
 
 
24
 
25
  Tests that crash show `CRASH` (not `FAIL`) with NTSTATUS name and ERRORLOG.TXT content. ERRORLOG.TXT is redirected to the per-test run directory via `OPENWA_ERRORLOG_PATH`.
26
 
27
+ ## Headful Pass Criteria
28
+
29
+ A headful test passes when: no crash, no `[PANIC]` in OpenWA.log, no `[GAMEPLAY FAIL]`, and at least one `[GAMEPLAY PASS]` marker present.
30
+
31
  ## Adding New Replay Tests
32
 
33
  1. Record a game in WA.exe (the replay `.WAgame` file is saved automatically)
 
38
  ## Environment Variables
39
 
40
  - `OPENWA_HEADLESS=1` — Headless mode: hooks MessageBoxA to auto-dismiss, launcher uses SW_HIDE, file isolation hook active, semaphore renamed per-PID
41
+ - `OPENWA_REPLAY_TEST=1` — Headful mode: enables fast-forward replay (50x speed)
42
  - `OPENWA_LOG_PATH=<path>` — Override OpenWA.log location (used for per-instance isolation)
43
  - `OPENWA_ERRORLOG_PATH=<path>` — Redirect ERRORLOG.TXT to specified path (crash capture)
44
 
45
  ## Key Paths
46
 
47
  - Replay files + expected logs: `testdata/replays/*.WAgame` + `*_expected.log`
48
+ - Per-run output: `testdata/runs/<timestamp>/` (headless), `testdata/runs/headful-<timestamp>/` (headful)
49
+ - Headless script: `run-tests.ps1`
50
+ - Headful script: `replay-test.ps1`
crates/openwa-test-runner/src/main.rs CHANGED
@@ -667,6 +667,334 @@ fn parse_errorlog_exception(content: Option<&str>) -> &'static str {
667
  "crash (SEH handled)"
668
  }
669
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
670
  // ─── Trace-desync subcommand ─────────────────────────────────────────────
671
 
672
  struct TraceDesyncArgs {
@@ -995,10 +1323,18 @@ fn report_startup_check_failures(results: &[TestResult], run_dir: &Path) {
995
  fn main() {
996
  // Check for subcommands before normal arg parsing
997
  let argv: Vec<String> = env::args().skip(1).collect();
998
- if argv.first().map(|s| s.as_str()) == Some("trace-desync") {
999
- let sub_args = parse_trace_desync_args(&argv[1..]);
1000
- run_trace_desync(sub_args);
1001
- return;
 
 
 
 
 
 
 
 
1002
  }
1003
 
1004
  let args = parse_args();
 
667
  "crash (SEH handled)"
668
  }
669
 
670
+ // ─── Headful subcommand ────────────────────────────────────────────────────
671
+
672
+ const DEFAULT_HEADFUL_TIMEOUT_SECS: u64 = 150;
673
+
674
+ struct HeadfulArgs {
675
+ filter_or_replay: Option<String>,
676
+ no_build: bool,
677
+ wa_path: Option<PathBuf>,
678
+ timeout_secs: u64,
679
+ }
680
+
681
+ fn parse_headful_args(argv: &[String]) -> HeadfulArgs {
682
+ let mut args = HeadfulArgs {
683
+ filter_or_replay: None,
684
+ no_build: false,
685
+ wa_path: None,
686
+ timeout_secs: DEFAULT_HEADFUL_TIMEOUT_SECS,
687
+ };
688
+ let mut i = 0;
689
+ while i < argv.len() {
690
+ match argv[i].as_str() {
691
+ "--no-build" => args.no_build = true,
692
+ "--wa-path" => {
693
+ i += 1;
694
+ if i < argv.len() {
695
+ args.wa_path = Some(PathBuf::from(&argv[i]));
696
+ }
697
+ }
698
+ "--timeout" | "-t" => {
699
+ i += 1;
700
+ if i < argv.len() {
701
+ args.timeout_secs = argv[i]
702
+ .parse()
703
+ .unwrap_or(DEFAULT_HEADFUL_TIMEOUT_SECS);
704
+ }
705
+ }
706
+ s if !s.starts_with('-') && args.filter_or_replay.is_none() => {
707
+ args.filter_or_replay = Some(s.to_string());
708
+ }
709
+ other => {
710
+ eprintln!("Unknown argument: {other}");
711
+ eprintln!(
712
+ "Usage: openwa-test headful [filter|replay.WAgame] [--no-build] [--wa-path PATH] [--timeout SECS]"
713
+ );
714
+ std::process::exit(1);
715
+ }
716
+ }
717
+ i += 1;
718
+ }
719
+ args
720
+ }
721
+
722
+ /// Discover replays for headful testing. Unlike headless, doesn't require _expected.log.
723
+ /// If `filter_or_replay` ends with .WAgame and exists, treat it as a direct path.
724
+ fn discover_headful_tests(filter_or_replay: Option<&str>) -> Vec<TestCase> {
725
+ // Direct .WAgame path?
726
+ if let Some(arg) = filter_or_replay {
727
+ let path = Path::new(arg);
728
+ if arg.ends_with(".WAgame") && path.exists() {
729
+ let abs = strip_unc(fs::canonicalize(path).unwrap_or(path.to_path_buf()));
730
+ let stem = abs
731
+ .file_stem()
732
+ .and_then(|s| s.to_str())
733
+ .unwrap_or("unknown")
734
+ .to_string();
735
+ return vec![TestCase {
736
+ name: stem,
737
+ replay_path: abs.clone(),
738
+ expected_log: abs.with_extension("expected.log"),
739
+ output_log: abs.with_extension("log"),
740
+ }];
741
+ }
742
+ }
743
+
744
+ // Scan replays dir with optional filter
745
+ let replays_dir = Path::new(REPLAYS_DIR);
746
+ let entries = match fs::read_dir(replays_dir) {
747
+ Ok(e) => e,
748
+ Err(e) => {
749
+ eprintln!("Cannot read {REPLAYS_DIR}: {e}");
750
+ return Vec::new();
751
+ }
752
+ };
753
+
754
+ let mut tests = Vec::new();
755
+ for entry in entries.flatten() {
756
+ let path = entry.path();
757
+ if path.extension().and_then(|e| e.to_str()) != Some("WAgame") {
758
+ continue;
759
+ }
760
+ let stem = match path.file_stem().and_then(|s| s.to_str()) {
761
+ Some(s) => s.to_string(),
762
+ None => continue,
763
+ };
764
+ if let Some(filter) = filter_or_replay {
765
+ if !stem.contains(filter) {
766
+ continue;
767
+ }
768
+ }
769
+ // No _expected.log requirement for headful
770
+ let abs = strip_unc(fs::canonicalize(&path).unwrap_or(path.clone()));
771
+ tests.push(TestCase {
772
+ name: stem,
773
+ replay_path: abs.clone(),
774
+ expected_log: abs.with_extension("expected.log"),
775
+ output_log: abs.with_extension("log"),
776
+ });
777
+ }
778
+ tests.sort_by(|a, b| a.name.cmp(&b.name));
779
+ tests
780
+ }
781
+
782
+ /// Analyze an OpenWA.log for panics and gameplay check markers.
783
+ struct HeadfulLogAnalysis {
784
+ panics: Vec<String>,
785
+ gameplay_passes: Vec<String>,
786
+ gameplay_fails: Vec<String>,
787
+ }
788
+
789
+ fn analyze_headful_log(log_path: &Path) -> HeadfulLogAnalysis {
790
+ let content = fs::read_to_string(log_path).unwrap_or_default();
791
+ let mut analysis = HeadfulLogAnalysis {
792
+ panics: Vec::new(),
793
+ gameplay_passes: Vec::new(),
794
+ gameplay_fails: Vec::new(),
795
+ };
796
+ for line in content.lines() {
797
+ if line.contains("[PANIC]") {
798
+ analysis.panics.push(line.to_string());
799
+ } else if line.contains("[GAMEPLAY PASS]") {
800
+ analysis.gameplay_passes.push(line.to_string());
801
+ } else if line.contains("[GAMEPLAY FAIL]") {
802
+ analysis.gameplay_fails.push(line.to_string());
803
+ }
804
+ }
805
+ analysis
806
+ }
807
+
808
+ /// Run a single headful replay test.
809
+ fn run_headful_test(
810
+ test: &TestCase,
811
+ launcher: &Path,
812
+ wa_exe: &Path,
813
+ run_dir: &Path,
814
+ timeout_secs: u64,
815
+ ) -> TestResult {
816
+ let start = Instant::now();
817
+
818
+ let openwa_log = run_dir.join(format!("{}.openwa.log", test.name));
819
+ let errorlog_path = run_dir.join(format!("{}.errorlog.txt", test.name));
820
+
821
+ let result = Command::new(launcher)
822
+ .arg("--minimized")
823
+ .arg(wa_exe)
824
+ .arg(&test.replay_path)
825
+ .env("OPENWA_REPLAY_TEST", "1")
826
+ .env("OPENWA_LOG_PATH", &openwa_log)
827
+ .env("OPENWA_ERRORLOG_PATH", &errorlog_path)
828
+ .stdout(std::process::Stdio::null())
829
+ .stderr(std::process::Stdio::null())
830
+ .status();
831
+
832
+ let duration = start.elapsed();
833
+
834
+ match result {
835
+ Err(e) => TestResult {
836
+ name: test.name.clone(),
837
+ passed: false,
838
+ duration,
839
+ diff_lines: Vec::new(),
840
+ error: Some(format!("Failed to launch: {e}")),
841
+ crashed: None,
842
+ },
843
+ Ok(status) => {
844
+ let exit_code = status.code().unwrap_or(0);
845
+
846
+ if duration >= Duration::from_secs(timeout_secs) {
847
+ return TestResult {
848
+ name: test.name.clone(),
849
+ passed: false,
850
+ duration,
851
+ diff_lines: Vec::new(),
852
+ error: Some("Timeout".to_string()),
853
+ crashed: None,
854
+ };
855
+ }
856
+
857
+ // Crash detection (same as headless)
858
+ let errorlog_content = fs::read(&errorlog_path)
859
+ .ok()
860
+ .filter(|b| !b.is_empty())
861
+ .map(|b| String::from_utf8_lossy(&b).into_owned());
862
+
863
+ if is_crash_exit_code(exit_code) || errorlog_content.is_some() {
864
+ let unsigned = exit_code as u32;
865
+ let name = if is_crash_exit_code(exit_code) {
866
+ ntstatus_name(unsigned)
867
+ } else {
868
+ parse_errorlog_exception(errorlog_content.as_deref())
869
+ };
870
+ return TestResult {
871
+ name: test.name.clone(),
872
+ passed: false,
873
+ duration,
874
+ diff_lines: Vec::new(),
875
+ error: None,
876
+ crashed: Some(CrashInfo {
877
+ exit_code: unsigned,
878
+ name,
879
+ errorlog_content,
880
+ }),
881
+ };
882
+ }
883
+
884
+ // Analyze OpenWA.log for panics and gameplay markers
885
+ let analysis = analyze_headful_log(&openwa_log);
886
+
887
+ let mut detail_lines = Vec::new();
888
+ for line in &analysis.panics {
889
+ detail_lines.push(format!("PANIC: {line}"));
890
+ }
891
+ for line in &analysis.gameplay_fails {
892
+ detail_lines.push(format!("FAIL: {line}"));
893
+ }
894
+ for line in &analysis.gameplay_passes {
895
+ detail_lines.push(line.clone());
896
+ }
897
+
898
+ let passed = analysis.panics.is_empty()
899
+ && analysis.gameplay_fails.is_empty()
900
+ && !analysis.gameplay_passes.is_empty();
901
+
902
+ let error = if analysis.gameplay_passes.is_empty()
903
+ && analysis.panics.is_empty()
904
+ && analysis.gameplay_fails.is_empty()
905
+ {
906
+ Some(format!(
907
+ "No gameplay markers found in log (exit code: {exit_code})"
908
+ ))
909
+ } else {
910
+ None
911
+ };
912
+
913
+ TestResult {
914
+ name: test.name.clone(),
915
+ passed,
916
+ duration,
917
+ diff_lines: detail_lines,
918
+ error,
919
+ crashed: None,
920
+ }
921
+ }
922
+ }
923
+ }
924
+
925
+ fn run_headful(args: HeadfulArgs) {
926
+ let tests = discover_headful_tests(args.filter_or_replay.as_deref());
927
+ if tests.is_empty() {
928
+ eprintln!("No replay tests found in {REPLAYS_DIR}/");
929
+ if let Some(f) = &args.filter_or_replay {
930
+ eprintln!(" (filter/path: \"{f}\")");
931
+ }
932
+ std::process::exit(1);
933
+ }
934
+
935
+ // Build
936
+ if !args.no_build {
937
+ eprint!("Building... ");
938
+ match build() {
939
+ Ok(d) => eprintln!("done ({:.1}s)", d.as_secs_f64()),
940
+ Err(e) => {
941
+ eprintln!("FAILED: {e}");
942
+ std::process::exit(1);
943
+ }
944
+ }
945
+ }
946
+
947
+ let wa_exe = find_wa_exe(args.wa_path.as_deref()).unwrap_or_else(|| {
948
+ eprintln!("Cannot find WA.exe. Use --wa-path or set OPENWA_WA_PATH.");
949
+ std::process::exit(1);
950
+ });
951
+ let launcher = find_launcher().unwrap_or_else(|| {
952
+ eprintln!("Cannot find openwa-launcher.exe. Run cargo build first.");
953
+ std::process::exit(1);
954
+ });
955
+
956
+ // Clean stale files from game dir
957
+ if let Some(game_dir) = wa_exe.parent() {
958
+ let _ = fs::remove_file(game_dir.join("ERRORLOG.TXT"));
959
+ let _ = fs::remove_file(game_dir.join("OpenWA.log"));
960
+ }
961
+
962
+ let timestamp = timestamp();
963
+ let run_dir = PathBuf::from(RUNS_DIR).join(format!("headful-{timestamp}"));
964
+ let _ = fs::create_dir_all(&run_dir);
965
+ let run_dir = strip_unc(fs::canonicalize(&run_dir).unwrap_or(run_dir));
966
+
967
+ println!(
968
+ "Running {} headful test{}...\n",
969
+ tests.len(),
970
+ if tests.len() == 1 { "" } else { "s" }
971
+ );
972
+
973
+ let wall_start = Instant::now();
974
+ let mut results = Vec::new();
975
+
976
+ for test in &tests {
977
+ let result = run_headful_test(test, &launcher, &wa_exe, &run_dir, args.timeout_secs);
978
+ print_result(&result);
979
+ results.push(result);
980
+ }
981
+
982
+ let wall_time = wall_start.elapsed();
983
+
984
+ print_summary(&results, wall_time);
985
+ write_summary(&results, wall_time, &run_dir.join("summary.txt"));
986
+
987
+ // Report startup check failures
988
+ report_startup_check_failures(&results, &run_dir);
989
+
990
+ cleanup_temp_files(&wa_exe);
991
+
992
+ let failed = results.iter().any(|r| !r.passed);
993
+ if failed {
994
+ std::process::exit(1);
995
+ }
996
+ }
997
+
998
  // ─── Trace-desync subcommand ─────────────────────────────────────────────
999
 
1000
  struct TraceDesyncArgs {
 
1323
  fn main() {
1324
  // Check for subcommands before normal arg parsing
1325
  let argv: Vec<String> = env::args().skip(1).collect();
1326
+ match argv.first().map(|s| s.as_str()) {
1327
+ Some("headful") => {
1328
+ let sub_args = parse_headful_args(&argv[1..]);
1329
+ run_headful(sub_args);
1330
+ return;
1331
+ }
1332
+ Some("trace-desync") => {
1333
+ let sub_args = parse_trace_desync_args(&argv[1..]);
1334
+ run_trace_desync(sub_args);
1335
+ return;
1336
+ }
1337
+ _ => {}
1338
  }
1339
 
1340
  let args = parse_args();
replay-test.ps1 CHANGED
@@ -1,217 +1,6 @@
1
- # replay-test.ps1 - Automated replay-based testing
2
- # Usage: powershell -File replay-test.ps1 [-Headless] [replay-file]
3
- #
4
- # Modes:
5
- # Default (headful): Fast-forward replay with validation, graphics & sound active.
6
- # -Headless: Uses WA's /getlog mode -- no window, pure simulation.
7
- # Compares output log to expected log for determinism checking.
8
 
9
- param(
10
- [switch]$Headless,
11
- [string]$ReplayFile = "testdata\replays\bots.WAgame"
12
- )
13
-
14
- $ErrorActionPreference = "Stop"
15
-
16
- # Resolve WA installation directory using the same priority order as openwa-config:
17
- # 1. OPENWA_WA_PATH env var (may be WA.exe path or directory)
18
- # 2. HKCU\SOFTWARE\Team17SoftwareLTD\WormsArmageddon, value PATH
19
- function Get-WaDir {
20
- if ($env:OPENWA_WA_PATH) {
21
- $p = $env:OPENWA_WA_PATH
22
- if ($p -match '\.exe$') { return Split-Path $p }
23
- return $p
24
- }
25
- try {
26
- $reg = Get-ItemPropertyValue -Path 'HKCU:\SOFTWARE\Team17SoftwareLTD\WormsArmageddon' -Name 'PATH' -ErrorAction Stop
27
- if ($reg) { return $reg }
28
- } catch {}
29
- return $null
30
- }
31
-
32
- $gameDir = Get-WaDir
33
- if (-not $gameDir) {
34
- Write-Error "Cannot find WA installation. Set OPENWA_WA_PATH or ensure the registry key HKCU\SOFTWARE\Team17SoftwareLTD\WormsArmageddon\PATH is set."
35
- exit 1
36
- }
37
- $waExe = Join-Path $gameDir "WA.exe"
38
- $src = "target\i686-pc-windows-msvc\release"
39
- $logDir = "testdata\logs"
40
-
41
- # Resolve replay file to absolute path
42
- if (-not [System.IO.Path]::IsPathRooted($ReplayFile)) {
43
- $ReplayFile = Join-Path (Get-Location) $ReplayFile
44
- }
45
-
46
- if (-not (Test-Path $ReplayFile)) {
47
- Write-Error "Replay file not found: $ReplayFile"
48
- exit 1
49
- }
50
-
51
- # 1. Build launcher + DLL
52
- Write-Host "Building..." -ForegroundColor Cyan
53
- cargo build --release -p openwa-launcher -p openwa-wormkit
54
- if ($LASTEXITCODE -ne 0) { exit 1 }
55
-
56
- $launcher = "$src\openwa-launcher.exe"
57
-
58
- # 2. Clear old logs
59
- Remove-Item "$gameDir\OpenWA.log" -ErrorAction SilentlyContinue
60
- Remove-Item "$gameDir\OpenWA_validation.log" -ErrorAction SilentlyContinue
61
- Remove-Item "$gameDir\ERRORLOG.TXT" -ErrorAction SilentlyContinue
62
-
63
- # Remove old WormKit DLL if present from a previous deployment
64
- if (Test-Path "$gameDir\wkOpenWA.dll") {
65
- Write-Host "Note: removing legacy $gameDir\wkOpenWA.dll" -ForegroundColor Yellow
66
- Remove-Item "$gameDir\wkOpenWA.dll"
67
- }
68
- if (Test-Path "$gameDir\wkOpenWAValidator.dll") {
69
- Remove-Item "$gameDir\wkOpenWAValidator.dll"
70
- }
71
-
72
- # 3. Launch via launcher with replay, env vars set
73
- if ($Headless) {
74
- Write-Host "Launching WA.exe in HEADLESS mode: $ReplayFile" -ForegroundColor Cyan
75
- Write-Host " (No window -- pure simulation via /getlog)" -ForegroundColor Yellow
76
-
77
- # Remove any previous output log next to the replay file
78
- $replayBase = [System.IO.Path]::ChangeExtension($ReplayFile, ".log")
79
- Remove-Item $replayBase -ErrorAction SilentlyContinue
80
-
81
- $env:OPENWA_HEADLESS = "1"
82
- $proc = Start-Process -FilePath $launcher -ArgumentList "`"$waExe`" /getlog `"$ReplayFile`"" -PassThru
83
- $timeout = 120
84
- } else {
85
- Write-Host "Launching WA.exe with replay: $ReplayFile" -ForegroundColor Cyan
86
- Write-Host " (Fast-forward mode: replay will be auto-advanced, game exits when done)" -ForegroundColor Yellow
87
-
88
- $env:OPENWA_REPLAY_TEST = "1"
89
- $proc = Start-Process -FilePath $launcher -ArgumentList "--minimized `"$waExe`" `"$ReplayFile`"" -PassThru
90
- $timeout = 150
91
- }
92
-
93
- if (-not $proc.WaitForExit($timeout * 1000)) {
94
- Write-Host "WARNING: launcher did not exit within ${timeout}s, killing..." -ForegroundColor Red
95
- $proc.Kill()
96
- $proc.WaitForExit(5000)
97
- }
98
- $env:OPENWA_REPLAY_TEST = $null
99
- $env:OPENWA_HEADLESS = $null
100
- $env:OPENWA_WA_PATH = $null
101
-
102
- Write-Host "Launcher exited with code $($proc.ExitCode)" -ForegroundColor Cyan
103
-
104
- # 4. Copy logs back
105
- if (-not (Test-Path $logDir)) {
106
- New-Item -ItemType Directory -Path $logDir | Out-Null
107
- }
108
-
109
- $timestamp = Get-Date -Format "yyyy-MM-dd_HH-mm-ss"
110
-
111
- if (Test-Path "$gameDir\OpenWA_validation.log") {
112
- Copy-Item "$gameDir\OpenWA_validation.log" "$logDir\validation_$timestamp.log"
113
- Copy-Item "$gameDir\OpenWA_validation.log" "$logDir\validation_latest.log"
114
- Write-Host "Validation log copied." -ForegroundColor Green
115
- } else {
116
- Write-Host "WARNING: No validation log found!" -ForegroundColor Red
117
- }
118
-
119
- if (Test-Path "$gameDir\OpenWA.log") {
120
- Copy-Item "$gameDir\OpenWA.log" "$logDir\openwa_$timestamp.log"
121
- Copy-Item "$gameDir\OpenWA.log" "$logDir\openwa_latest.log"
122
- Write-Host "OpenWA log copied." -ForegroundColor Green
123
- }
124
-
125
- $crashed = $false
126
- if (Test-Path "$gameDir\ERRORLOG.TXT") {
127
- Copy-Item "$gameDir\ERRORLOG.TXT" "$logDir\errorlog_$timestamp.txt"
128
- Copy-Item "$gameDir\ERRORLOG.TXT" "$logDir\errorlog_latest.txt"
129
- Write-Host "ERRORLOG.TXT copied (crash detected)." -ForegroundColor Red
130
- $crashed = $true
131
- } else {
132
- Remove-Item "$logDir\errorlog_latest.txt" -ErrorAction SilentlyContinue
133
- }
134
-
135
- # 5. Crash check (before summary, so it always runs)
136
- if ($crashed) {
137
- Write-Host ""
138
- Write-Host "=== CRASH DETECTED ===" -ForegroundColor Red
139
- Get-Content "$logDir\errorlog_latest.txt" -TotalCount 5 | ForEach-Object {
140
- Write-Host " $_" -ForegroundColor Red
141
- }
142
- Write-Host " ..." -ForegroundColor Red
143
- Write-Host " Full log: $logDir\errorlog_latest.txt" -ForegroundColor Gray
144
- exit 1
145
- }
146
-
147
- # 6. Print summary
148
- Write-Host ""
149
- if ($Headless) {
150
- Write-Host "=== Headless Log Comparison ===" -ForegroundColor Cyan
151
- $replayBase = [System.IO.Path]::ChangeExtension($ReplayFile, ".log")
152
- $expectedLog = [System.IO.Path]::Combine(
153
- [System.IO.Path]::GetDirectoryName($ReplayFile),
154
- [System.IO.Path]::GetFileNameWithoutExtension($ReplayFile) + "_expected.log"
155
- )
156
-
157
- if (-not (Test-Path $replayBase)) {
158
- Write-Host " FAIL: No output log generated at $replayBase" -ForegroundColor Red
159
- exit 1
160
- }
161
- if (-not (Test-Path $expectedLog)) {
162
- Write-Host " No expected log found -- saving output as expected: $expectedLog" -ForegroundColor Yellow
163
- Copy-Item $replayBase $expectedLog
164
- Write-Host " PASS (first run -- expected log created)" -ForegroundColor Green
165
- } else {
166
- $diff = Compare-Object (Get-Content $expectedLog) (Get-Content $replayBase)
167
- if ($diff) {
168
- Write-Host " FAIL: Output differs from expected log" -ForegroundColor Red
169
- Write-Host ""
170
- $diff | ForEach-Object {
171
- $indicator = if ($_.SideIndicator -eq "<=") { "expected" } else { "actual " }
172
- Write-Host " $indicator | $($_.InputObject)" -ForegroundColor $(if ($_.SideIndicator -eq "<=") { "Red" } else { "Yellow" })
173
- }
174
- exit 1
175
- } else {
176
- Write-Host " PASS: Output matches expected log" -ForegroundColor Green
177
- }
178
- }
179
- # Clean up the generated log
180
- Remove-Item $replayBase -ErrorAction SilentlyContinue
181
- } else {
182
- Write-Host "=== Validation Summary ===" -ForegroundColor Cyan
183
- if (Test-Path "$logDir\validation_latest.log") {
184
- $content = Get-Content "$logDir\validation_latest.log" -Raw
185
-
186
- # Static checks (vtables, struct offsets, prologues)
187
- $staticPasses = ([regex]::Matches($content, "\[STATIC PASS\]")).Count
188
- $staticFails = ([regex]::Matches($content, "\[STATIC FAIL\]")).Count
189
- $staticColor = if ($staticFails -gt 0) { "Red" } else { "Green" }
190
- Write-Host " Static checks: $staticPasses pass, $staticFails fail (vtables, struct offsets, prologues)" -ForegroundColor $staticColor
191
-
192
- # Gameplay checks (milestones from frame hook)
193
- $gameplayPasses = ([regex]::Matches($content, "\[GAMEPLAY PASS\]")).Count
194
- $gameplayFails = ([regex]::Matches($content, "\[GAMEPLAY FAIL\]")).Count
195
- if ($gameplayPasses + $gameplayFails -gt 0) {
196
- $gameplayColor = if ($gameplayFails -gt 0) { "Red" } else { "Green" }
197
- Write-Host " Gameplay checks: $gameplayPasses pass, $gameplayFails fail (init, match start, match end)" -ForegroundColor $gameplayColor
198
- } else {
199
- Write-Host " Gameplay checks: not found (DLL may not have detached cleanly)" -ForegroundColor Yellow
200
- }
201
-
202
- if ($content -match "Safety timeout reached") {
203
- Write-Host " Replay: TIMEOUT (safety timeout triggered)" -ForegroundColor Red
204
- }
205
- }
206
- # Check for Rust panics in OpenWA log
207
- if (Test-Path "$logDir\openwa_latest.log") {
208
- $panics = Select-String -Path "$logDir\openwa_latest.log" -Pattern "\[PANIC\]"
209
- if ($panics) {
210
- Write-Host ""
211
- Write-Host "=== RUST PANIC DETECTED ===" -ForegroundColor Red
212
- $panics | ForEach-Object { Write-Host " $($_.Line)" -ForegroundColor Red }
213
- }
214
- }
215
-
216
- Write-Host " Logs: $logDir\" -ForegroundColor Gray
217
- }
 
1
+ # Run a headful replay test via the Rust test runner.
2
+ # Usage: .\replay-test.ps1 [filter|replay.WAgame] [--no-build] [--wa-path PATH]
3
+ cargo build -p openwa-test-runner --release
4
+ if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
 
 
 
5
 
6
+ & "$PSScriptRoot\target\i686-pc-windows-msvc\release\openwa-test.exe" headful @args