voraseth commited on
Commit
35929a4
·
verified ·
1 Parent(s): 325f47e

Pronunciation cache limited to the vocabulary words; README wording

Browse files
Files changed (3) hide show
  1. README.md +8 -6
  2. SHA256SUMS +1 -1
  3. g2p_raw_cache.json +0 -0
README.md CHANGED
@@ -31,7 +31,7 @@ writes no คำสร้อย. One sample is drawn per topic at temperature 0.
31
  | `model/config.json`, `TRAINED.json`, `train.log` | model configuration, training record and log |
32
  | `lexicon.json` | the vocabulary: 62,856 words with their pronounced syllables |
33
  | `allow.json` | allowlist of the decoder's stutter ban |
34
- | `g2p_raw_cache.json` | pronunciation cache (word -> phonemes) |
35
  | `config.json`, `configuration_klong4.py`, `modeling_klong4.py` | the `transformers` wrapper |
36
  | `src/klong4/` | the project's code, used by the wrapper |
37
  | `SHA256SUMS` | checksums of the weight and data files above |
@@ -74,21 +74,23 @@ Notes:
74
 
75
  ## Evaluation (from the paper)
76
 
77
- On 75 held-out topics, one sample per topic, judged by three independent runs of an LLM judge (Claude Opus 5.5) in the
78
- same pools as the human stanzas of those topics:
79
 
80
  | | Model | Human verse |
81
  |---|---|---|
82
- | Passes form and all three rhymes | 100% | 100% (selected) |
83
  | Lexically clean | 66.7% | 92.0% |
84
  | On topic (alignment >= 4 of 5) | 72.0% | 88.0% |
85
  | Clean and on topic | 50.7% | 81.3% |
86
 
87
- The difference in the last row is -30.7 points (95% CI [-44.0, -17.3]). No generated line is copied from the corpora.
 
88
 
89
  ## Limitations
90
 
91
- - Every stanza satisfies the form, but a third contain a line the judge considers ill-formed, and the vocabulary is
 
92
  narrower and more repetitive than human verse (6.8% rare words against 13.9%).
93
  - The training set is small (2,001 stanzas).
94
  - The numbers are judgments of one LLM judge family, not of expert readers.
 
31
  | `model/config.json`, `TRAINED.json`, `train.log` | model configuration, training record and log |
32
  | `lexicon.json` | the vocabulary: 62,856 words with their pronounced syllables |
33
  | `allow.json` | allowlist of the decoder's stutter ban |
34
+ | `g2p_raw_cache.json` | pronunciation cache for the 62,856 vocabulary words (word -> phonemes) |
35
  | `config.json`, `configuration_klong4.py`, `modeling_klong4.py` | the `transformers` wrapper |
36
  | `src/klong4/` | the project's code, used by the wrapper |
37
  | `SHA256SUMS` | checksums of the weight and data files above |
 
74
 
75
  ## Evaluation (from the paper)
76
 
77
+ On 75 held-out stanzas and their topics (63 distinct), one sample per item, judged by three independent runs of an LLM
78
+ judge (Claude Opus 5.5) in the same pools as the human stanzas:
79
 
80
  | | Model | Human verse |
81
  |---|---|---|
82
+ | Passes the form checker, three rhymes included | 100% | 100% (selected) |
83
  | Lexically clean | 66.7% | 92.0% |
84
  | On topic (alignment >= 4 of 5) | 72.0% | 88.0% |
85
  | Clean and on topic | 50.7% | 81.3% |
86
 
87
+ The difference in the last row is -30.7 points (95% CI [-44.0, -17.3]). No generated line equals a line of the corpora searched, and
88
+ none shares more than five consecutive syllables with them.
89
 
90
  ## Limitations
91
 
92
+ - Every stanza passes the automatic form checker, which counts syllables as the pronunciation converter reads them
93
+ (it misreads some words), but a third contain a line the judge considers ill-formed, and the vocabulary is
94
  narrower and more repetitive than human verse (6.8% rare words against 13.9%).
95
  - The training set is small (2,001 stanzas).
96
  - The numbers are judgments of one LLM judge family, not of expert readers.
SHA256SUMS CHANGED
@@ -1,4 +1,4 @@
1
  f053475b9be734873050775abc9fcd51cc01f271ce80d22352126ed9e97e8916 model/model.pt
2
  5e47c92e00fd240544a4acf8fe8f95fd30a1b79a7899d08023039ea810e3c228 lexicon.json
3
  fdcb862c4dcab8f370dc0bac46d9066ddee92fd3728a4ffdf7d12b99d9e0eaf5 allow.json
4
- 7177cfba9e087512f0c7e805440ad64533b69f34ad7c8a9803d97d84c8be92b3 g2p_raw_cache.json
 
1
  f053475b9be734873050775abc9fcd51cc01f271ce80d22352126ed9e97e8916 model/model.pt
2
  5e47c92e00fd240544a4acf8fe8f95fd30a1b79a7899d08023039ea810e3c228 lexicon.json
3
  fdcb862c4dcab8f370dc0bac46d9066ddee92fd3728a4ffdf7d12b99d9e0eaf5 allow.json
4
+ 8d4b4ef7c9860fd4181c1eb9cd38856a900b7b5899698f7ff95c01c35c774466 g2p_raw_cache.json
g2p_raw_cache.json CHANGED
The diff for this file is too large to render. See raw diff