File size: 5,664 Bytes
48d8e79 1ab750f 48d8e79 1ab750f 48d8e79 1ab750f 48d8e79 1ab750f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 | ---
title: "ExperimentForge | A/B Testing Intelligence Lab"
emoji: "🧪"
colorFrom: green
colorTo: blue
sdk: static
app_file: index.html
pinned: false
license: mit
short_description: "Offline A/B testing and experiment diagnostics"
tags:
- statistics
- ab-testing
- experimentation
- analytics
- decision-support
- javascript
---
# ExperimentForge | Experiment Intelligence Studio
**A browser-only, reproducible, three-arm A/B/C experiment analysis laboratory.**
ExperimentForge supports teams and researchers exploring binary outcomes such as checkout conversion, course onboarding completion, and customer activation. It is a working statistical app, not a slide mock-up or a connection to a production analytics service.
## Features
| Module | What it does |
| --- | --- |
| Overview | Conversion rates, differences vs control, uncertainty intervals, Holm-adjusted significance and SRM status |
| Variant benchmark | Participant counts, Wilson confidence intervals and treatment comparisons |
| Segment explorer | Exploratory variation across segments, with clear multiple-testing caveats |
| Quality & SRM | Three-arm sample ratio mismatch using the chi-squared test with 2 degrees of freedom |
| Power planner | Approximate sample sizes for a target relative minimum detectable effect and power |
| Data & reports | Local CSV import; download audit JSON, study CSV, comparison CSV, segment CSV, Markdown memo and CSV template |
## How to use
1. Select one of three **synthetic** example datasets or import your own aggregated CSV.
2. Select a population slice and a planned randomization allocation.
3. Inspect the treatment differences and multiplicity-adjusted hypothesis test results.
4. Check the sample ratio mismatch warning. The allocation must be specified from the experimental design.
5. Use the power planner for a *future* trial, not for post-hoc claims of adequate power.
6. Export a reproducible audit before closing the page.
### CSV format
The CSV requires exactly one record for each `segment` / `variant` combination. Every segment must include variants `A`, `B`, and `C`. `A` is the control. Save as UTF-8.
```csv
segment,variant,visitors,conversions
North,A,1000,115
North,B,500,72
North,C,500,61
South,A,1000,108
South,B,500,67
South,C,500,56
```
`visitors` must be a positive integer. `conversions` must be an integer between zero and `visitors`. CSV files are limited to 1 MB and 200 data rows. Invalid uploads do not replace the active data.
## Statistical methods
- **Binary conversion rate:** `conversions / visitors` for each variant.
- **Individual proportion intervals:** Wilson score method.
- **Difference in proportions:** Newcombe-Wilson hybrid interval (independent groups, no continuity correction).
- **Two-sided hypothesis test:** pooled two-proportions normal z-test, with the usual large-sample approximation.
- **Multiple comparisons:** Holm step-down p-value adjustment for the two comparisons **B vs A** and **C vs A** within the currently selected sample.
- **Sample ratio mismatch (SRM):** chi-squared goodness-of-fit test with two degrees of freedom, compared against user-specified expected allocation; a conservative `p < 0.001` alert threshold.
- **Power approximation:** normal-theory equal-size two-group calculation for a specified relative detectable effect, with Bonferroni allocation of familywise alpha across two comparisons. It is an approximate design aid, not an exact trial planner.
## Important limitations
- This application does **not** randomize subjects, collect live events, monitor production agents or prove causality. A causal reading requires a properly designed and executed randomized experiment.
- Fixed-horizon tests are **not** sequentially valid. Repeatedly checking the p-values without an appropriate sequential design increases false-positive risk.
- Holm adjustment applies only to the two predefined comparisons in the currently selected sample. Segment explorations involve more tests and are **exploratory**; their local Holm adjustments do not control the familywise error across all segments and views.
- Displayed confidence intervals are **not** adjusted for multiplicity, even where p-values are Holm-adjusted.
- The application assumes independent binary observations. It is not appropriate for clusters, repeated outcomes, interference, time dependence or observational comparisons without different methods.
- Small or extreme samples may require exact or alternative inference; this demo uses approximate tests.
- The bundled demonstration datasets are fully **synthetic** and are not measurements of commercial products, students or customers.
## Privacy and deployment
The entire app is bundled in one `index.html` and runs in your browser. There are no external JavaScript packages, API keys, tracking scripts, CDN resources or backend. CSV import and analysis remain local to the browser. Downloads are created client-side. Nothing is persisted to a server by ExperimentForge.
Publish to Hugging Face Spaces with `SDK: Static`, template `Blank` and visibility `Public`. Upload the root-level `README.md`, `index.html` and `LICENSE` files.
## Reproducibility and tests
The full code and independent verification scripts are distributed separately from the upload package. To run them locally:
```bash
node tests/core.test.js
python tests/scipy_crosscheck.py
python tests/browser_test.py
```
Browser tests require `playwright` and a Chromium executable at `/usr/bin/chromium`. Numerical reference tests require `scipy` and `statsmodels`. The **app itself** has none of these dependencies.
## License
MIT License. See `LICENSE`.
|