Paper-reported results
Native PyTorch six-round model; native BF16 scores are distinguished from MLX FP16.
Source: S²-MAS: A Society-Scale Multi-Agent Simulation Framework with 500 Million Household Persona Agents. S²-MAS manuscript.
Training and prediction
The grouped continuation training partition contains 2,925,263 examples per language (Methods M6). English is the default evaluation language. Accuracy pools all records; question-macro F1 first averages candidate-class F1 within a question version, then weights versions equally.
| Model and runtime | Precision | Test records | Question versions | Accuracy (%) | Question-macro F1 (%) | Paper source |
|---|---|---|---|---|---|---|
| Fin-Laya 6-round, PyTorch | BF16 | 240,458 | 119 | 74.5586 | 38.3986 | Methods M4 |
| Fin-Laya 6-round, MLX CUDA | FP16 | 240,458 | 119 | 74.5544 | 38.3746 | Methods M5 |
| Fin-Laya 6-round, PyTorch, expanded test | BF16 | 418,597 | — | 67.2728 | 27.8355 | Methods M6 |
| Fin-Laya 6-round, MLX CUDA, expanded test | FP16 | 418,597 | — | 67.2726 | 27.8955 | Methods M6 |
| Jev 1.13 API | Provider-managed | 240,458 | 119 | 65.2584 | 37.4793 | Methods M5 |
The paper's shared-test accuracy difference between Fin-Laya 6-round MLX and Jev is 9.2960 percentage points.
Separate bilingual deployment benchmark
Methods M9 reports the bilingual Fin-Laya checkpoint on three fixed Chinese inputs, with 60 timed calls per arm, batch size and concurrency one. Local FP16 inference uses one A100 PCIe 40GB. The measured interval includes input preparation, tokenization, synchronized prediction and response formatting; Jev includes its API round trip. Loading is excluded. Identical serialized inputs use common mmBERT token counts.
| Measured model and runtime | Mean latency (ms) | Choices/s | Shared input tokens/s |
|---|---|---|---|
| Bilingual Fin-Laya, PyTorch A100 | 22.06 | 45.33 | 15,247.34 |
| Bilingual Fin-Laya, MLX CUDA A100 | 9.10 | 109.88 | 36,955.47 |
| Jev 1.13 API | 206.77 | 4.84 | 1,626.63 |
The bilingual MLX arm's shared-input-token throughput is 22.72× Jev API's and 2.42× the same checkpoint's PyTorch throughput. Prediction accuracy above uses the six-round evaluation; this timing table uses the paper's separate bilingual checkpoint.
The accompanying CSV table contains these paper-reported aggregates. Model-use terms remain in the repository LICENSE.