ya9919 commited on
Commit
17ea3c7
·
verified ·
1 Parent(s): 9d4d590

Update project background and model performance

Browse files
Files changed (1) hide show
  1. README.md +28 -12
README.md CHANGED
@@ -25,17 +25,22 @@ tags:
25
  - lora
26
 
27
  ---
 
28
  # qwen3.5-4b-base-finance-lora-head-200k
29
 
30
- Paper: [S²-MAS](https://github.com/KataDavidXD/s2-mas/blob/main/paper/S2_MAS_Paper.pdf).
31
 
32
- One of ten development-selected financial decision-model release versions in S²-MAS.
33
 
34
- LoRA adapter plus a shared scalar decision head; obtain the official backbone separately.
35
 
36
- The 4B initialization is the official pre-trained-only Base release.
37
 
38
- ## Training and selected checkpoint
 
 
 
 
39
 
40
  - Method: `lora`.
41
 
@@ -49,17 +54,26 @@ The 4B initialization is the official pre-trained-only Base release.
49
 
50
  - A fixed 20,000-example development set selects this checkpoint. The complete 412,873-example development partition compares the two 200k Qwen4B adaptation methods.
51
 
52
- ## Held-out financial QA
 
 
53
 
54
- Evaluation uses the fixed **240,458-record / 119-question-version common test**, with **8,275 respondent/household groups**. Both metrics retain the complete test denominator.
55
 
56
- | Accuracy (%) | Question-macro F1 (%) | Invalid predictions |
57
- |---:|---:|---:|
58
- | 74.1435 | 36.7343 | 0 |
 
59
 
60
- Accuracy pools correct choices across records. Question-macro F1 averages candidate-class F1 within each question version and weights versions equally; every defined answer class is included. Invalid decisions count as incorrect.
61
 
62
- ## Files and precision
 
 
 
 
 
 
63
 
64
  - Model files: `head.pt, adapter/adapter_config.json, adapter/adapter_model.safetensors`.
65
 
@@ -69,6 +83,8 @@ Accuracy pools correct choices across records. Question-macro F1 averages candid
69
 
70
  - Inference scores the full state, question and complete candidate menu.
71
 
 
 
72
  ## License and access
73
 
74
  The project model contribution is offered under the [S²-MAS Controlled Model Research License v1.2](LICENSE): noncommercial scientific use, no redistribution, and no unauthorized reverse engineering, extraction or distillation. Training and evaluation records are not distributed. Source-data access requires the provider's authorization. Permission requests: yli9919@hku.hk.
 
25
  - lora
26
 
27
  ---
28
+
29
  # qwen3.5-4b-base-finance-lora-head-200k
30
 
31
+ Paper: [S²-MAS](https://github.com/KataDavidXD/s2-mas/blob/main/paper/S2_MAS_Paper.pdf). The fixed-budget adaptation results and matched downstream methods define the comparison reported here.
32
 
33
+ ## Project background
34
 
35
+ S²-MAS is a society-scale multi-agent simulation framework for studying how household decisions affect consumption, cash reserves and payments. The paper connects a constructed resource of **500 million household-persona records**, learned decision policies, reusable financial scenarios and a state-transition engine. Its purchase-funding application evaluates **12 million distinct synthetic household agents** under paired conditions. Full-data Fin-Laya supplies the policy for that application; this repository supplies a fixed-budget adaptation checkpoint for the common decision-model comparison. Decision learning uses **real-survey data**. The GENESE/MAS engine implementation remains private.
36
 
37
+ The benchmark groups twelve systems into six decision-model routes: encoder option scoring; language models with learned decision heads; next-token decision SFT; dynamic-schema classification; multimodal encoder decisions; and provider decision interfaces. All receive text inputs in the common benchmark. The **17 model-repository entries** comprise **10 fixed-budget trained variants**—five native Laya and five Qwen variants—plus **six checkpoint/runtime entries** and **one compatibility alias**. The checkpoint/runtime entries include documentation-only records.
38
 
39
+ ## This model's role
40
+
41
+ This checkpoint follows the **language-model plus learned decision-head route**. LoRA adapters and a shared scalar decision head are trained together on Qwen3.5-4B-Base. The 200k head-only counterpart holds the backbone fixed, allowing the comparison to assess representation adaptation. The complete development partition selects this LoRA method before the separate 9B run.
42
+
43
+ ## Training design and checkpoint
44
 
45
  - Method: `lora`.
46
 
 
54
 
55
  - A fixed 20,000-example development set selects this checkpoint. The complete 412,873-example development partition compares the two 200k Qwen4B adaptation methods.
56
 
57
+ ## Performance and matched comparisons
58
+
59
+ **Language:** English. **Inference precision:** BF16 Qwen backbone with FP32 candidate scoring.
60
 
61
+ The common held-out test contains **240,458 QA records**, **119 question versions** and **8,275 respondent/household groups**. The checkpoint is selected using development data before this test.
62
 
63
+ | Checkpoint / reference | Accuracy (%) | Question-macro F1 (%) | Invalid decisions |
64
+ |---|---:|---:|---:|
65
+ | **This checkpoint** | **74.1435** | **36.7343** | **0** |
66
+ | Qwen4B head only, same 200k unique QA | 67.2841 | 30.8995 | 0 |
67
 
68
+ Accuracy weights every test record equally; invalid decisions count as incorrect. Question-macro F1 averages over every defined candidate class within each question version and then weights the 119 versions equally.
69
 
70
+ The head-only reference uses the same pretrained backbone, unique examples and common test, with its backbone fixed. This checkpoint also adapts the backbone representation through LoRA.
71
+
72
+ ## Package and inference
73
+
74
+ LoRA adapter plus a shared scalar decision head; obtain the official backbone separately.
75
+
76
+ The 4B initialization is the official pre-trained-only Base release.
77
 
78
  - Model files: `head.pt, adapter/adapter_config.json, adapter/adapter_model.safetensors`.
79
 
 
83
 
84
  - Inference scores the full state, question and complete candidate menu.
85
 
86
+ The supplied `qwen_decision.py` exposes the candidate-scoring inference interface. Supply the local package and the official backbone and tokenizer named in `decision_config.json`.
87
+
88
  ## License and access
89
 
90
  The project model contribution is offered under the [S²-MAS Controlled Model Research License v1.2](LICENSE): noncommercial scientific use, no redistribution, and no unauthorized reverse engineering, extraction or distillation. Training and evaluation records are not distributed. Source-data access requires the provider's authorization. Permission requests: yli9919@hku.hk.