Hugging Face's logo

fffoivos
/
apertus-tokenizer-extension

Safetensors
Greek
tokenizers
tokenizer
greek
apertus
continued-pretraining
Model card Files
xet
 Community
apertus-tokenizer-extension / supporting-material /source-code /README.md
fffoivos's picture
fffoivos
Sync release docs referenced by checksums
a4826df verified
preview code
|
Raw
History Blame Contribute Delete
1.47 kB

Source Code

The runnable implementation lives in GitHub:

https://github.com/fffoivos/glossapi-tokenizer-extension/tree/main/subprojects/03_apertus_extension_and_embedding_adaptation

This Hugging Face release keeps artifact payloads, manifests, compact evidence, and hydration instructions. It does not mirror the full scripts tree.

Where to start reading

Canonical entry point on the GitHub side:

subprojects/03_apertus_extension_and_embedding_adaptation/CPT_MASTER_20260526.md

That doc is the master synthesis: mission, plans (v0.12 + v0.7), Apertus fidelity, bakeoff results, discrepancy log, operational reference, and what's needed for production. A copy is also mirrored to this release at ../provenance/decisions/CPT_MASTER_20260526.md.

Important policies

Loss measurement:

../provenance/evals/LOSS_MEASUREMENT_POLICY.md

Raw Megatron lm loss is health-only across different tokenizers. Cross-arm loss conclusions use heldout BPB plus downstream evals. Older files may call BPB BPC; that is a legacy bits-per-byte label.

Greek aggregate rule: explicit MT diagnostics (arc_challenge_mt_el, global_piqa_completions_ell_grek) are excluded from the Greek aggregate per v0.12 §10 Q6. Per-task scores remain visible.

Manifests

Top-level: ../../manifest.json. Per-checkpoint: each experiment-checkpoints/<Arm>-<scale>/manifest.json carries source Clariden paths and human-readable metadata.