Haas-ja-v1.0-mlx-8bit

English version below

้žๅ…ฌๅผใฎๅค‰ๆ›็‰ฉใงใ™ใ€‚ third-intelligence/Haas-ja-v1.0ใฎ็ทšๅฝขๅฑคใ‚’8bit้‡ๅญๅŒ–ใ—ใŸMLX็‰ˆใงใ™ใ€‚Third Intelligence, Inc.ใจใฏๆๆบใƒปๆ‰ฟ่ชใƒปๅพŒๆดใฎ้–ขไฟ‚ใซใ‚ใ‚Šใพใ›ใ‚“ใ€‚

้‡ใฟใฏ็ด„0.73GBใงใ€bf16็‰ˆใฎ็ด„0.59ๅ€ใ€‚ใƒ”ใƒผใ‚ฏใƒกใƒขใƒชใฏๆธ›ใ‚Šใพใ™ใŒใ€ๆคœ่จผๆฉŸใงใฏRTFxไธญๅคฎๅ€คใงbf16ใ‚ˆใ‚Š็ด„14%ไฝŽใ„๏ผˆ้€Ÿใใฏใชใ‚‰ใชใ„๏ผ‰็ตๆžœใงใ—ใŸใ€‚

ๆฆ‚่ฆใƒป็’ฐๅขƒ

  • ๅ…จ็ทšๅฝขๅฑคใ‚’mlx.nn.quantizeใง8bitใ‚ขใƒ•ใ‚ฃใƒณ้‡ๅญๅŒ–๏ผˆใ‚ฐใƒซใƒผใƒ—ใ‚ตใ‚คใ‚บ64๏ผ‰ใ€‚็•ณใฟ่พผใฟใƒปๆญฃ่ฆๅŒ–ๅฑคใƒปๅŸ‹ใ‚่พผใฟใฏbf16ใ‚’็ถญๆŒใ€‚่ชญใฟ่พผใฟๆ™‚ใฎscalesใจbiasesใฏใ‚จใƒณใ‚ณใƒผใƒ€ใƒผใงbf16ใ€ไบˆๆธฌใƒป็ตๅˆใƒใƒƒใƒˆใƒฏใƒผใ‚ฏใงfloat32ใ€‚
  • Apple SiliconใฎMacๅฐ‚็”จใ€‚NeMoใƒปCUDAใƒปTritonใƒปtorchไธ่ฆใ€‚Python 3.12ใจhaas-ja-localใ‚’ไฝฟ็”จใ€‚
  • ๆคœ่จผ็’ฐๅขƒใฏbf16็‰ˆใจๅŒใ˜Apple M5 Max๏ผˆ128GB๏ผ‰ใ€macOS 27.2ใ€mlx 0.32.4ใ€‚ไป–ๆฉŸ็จฎใฏๆœชๆธฌๅฎšใ€‚

ไฝฟใ„ๆ–น

bf16็‰ˆใจๅŒใ˜ใ‚ณใƒผใƒ‰ใงๅ‹•ใใพใ™ใ€‚hfใŒใชใ‘ใ‚Œใฐpip install huggingface_hubใงๅฐŽๅ…ฅใ—ใฆใใ ใ•ใ„ใ€‚

pip install "haas-ja-local[mlx] @ git+https://github.com/basyauma-tech/haas-ja-local"
hf download basyauma-tech/Haas-ja-v1.0-mlx-8bit --local-dir Haas-ja-v1.0-mlx-8bit
from haas_ja_local import load_audio, load_model, transcribe

backend, tokenizer = load_model("Haas-ja-v1.0-mlx-8bit")
result = transcribe(backend, tokenizer, load_audio("audio.wav"))
print(result.text)
haas-ja-local transcribe Haas-ja-v1.0-mlx-8bit audio.wav
haas-ja-local transcribe Haas-ja-v1.0-mlx-8bit audio.wav --json --timestamps

็ฒพๅบฆ

asr-ja-bench v1.1ใ‚’bf16็‰ˆใจๅŒใ˜ๆ‰‹้ †ใง่ฉ•ไพกใ€‚CER strictใฎๅ˜ไฝใฏ%ใ€ๅทฎใฏใƒใ‚คใƒณใƒˆใงใ™ใ€‚bf16ใจใฎๅทฎใฏ่กจใฎไธธใ‚ๅ€คใ‹ใ‚‰็ฎ—ๅ‡บใ—ใฆใ„ใพใ™ใ€‚

ใ‚ตใƒ–ใ‚ปใƒƒใƒˆ ็™บ่ฉฑๆ•ฐ bf16 8bit bf16ใจใฎๅทฎ ๅ…ฌ่กจๅ€ค ๅ…ฌ่กจๅ€คใจใฎๅทฎ
FLEURS 650 3.65 3.68 +0.03 3.65ใ€œ3.67 +0.01
JVNV 1,375 4.51 4.49 -0.02 4.50ใ€œ4.53 -0.01

ๅ…ฌ่กจๅ€คใจใฎๅทฎใฏ็ฏ„ๅ›ฒใฎ่ฟ‘ใ„็ซฏใ‹ใ‚‰่จˆ็ฎ—ใ—ใ€็ฏ„ๅ›ฒๅ†…ใชใ‚‰0ใ€‚8bitใฏๅˆๆ ผๅŸบๆบ–ใฎๅทฎ0.3ใƒใ‚คใƒณใƒˆไปฅๅ†…ใงใ™ใ€‚็ฒพๅบฆ่ชฟๆ•ดใฏใ—ใฆใ„ใพใ›ใ‚“ใ€‚

4bit็‰ˆใฏ็ฒพๅบฆๅŸบๆบ–ใ‚’ๅค–ใ‚ŒใŸใŸใ‚ๅ…ฌ้–‹ใ—ใฆใ„ใพใ›ใ‚“ใ€‚ CER strictใฏFLEURS 3.70%๏ผˆๅ…ฌ่กจๅ€คใจใฎๅทฎ+0.03ใƒใ‚คใƒณใƒˆ๏ผ‰ใ€JVNV 5.00%๏ผˆ+0.47ใƒใ‚คใƒณใƒˆ๏ผ‰ใงใ—ใŸใ€‚

่ฉณ็ดฐใƒปๅ†็พๆ‰‹้ †: ้‡ๅญๅŒ–็‰ˆใฎๆคœ่จผ่จ˜้Œฒใ€‚

ๅฎŸ่ฃ…ใฎ็ขบ่ช

ๅฐใ•ใชไนฑๆ•ฐใƒขใƒ‡ใƒซใงๅค‰ๆ›ใƒป่ชญใฟ่พผใฟใฎๅพ€ๅพฉใ€ใ‚ญใƒผใจๅฝข็Šถใ€้‡ๅญๅŒ–ใ—ใชใ„ๅ€คใฎใƒ“ใƒƒใƒˆไธ€่‡ดใ‚’ๆคœ่จผๆธˆใฟใ€‚8bit็‰ˆใฎๆœฌๆ–‡ใฏbf16็‰ˆใจFLEURS 88.8%ใƒปJVNV 84.9%ใฎ็™บ่ฉฑใงไธ€่‡ดใ—ใพใ—ใŸใ€‚้€†้‡ๅญๅŒ–ใ—ใŸ้‡ใฟใ‚’float32ใงๆŒใคใƒขใƒ‡ใƒซใจใฎ็›ธๅฏพ่ชคๅทฎใฏ1e-5ๆœชๆบ€ใงใ—ใŸใ€‚ๅฎŸใƒขใƒ‡ใƒซใจPyTorch๏ผˆfloat32๏ผ‰ใฎใƒˆใƒผใ‚ฏใƒณๅˆ—ไธ€่‡ดใฏๆœชๆคœ่จผใงใ™ใ€‚

้€Ÿๅบฆใจใƒกใƒขใƒช

ไธŠ่จ˜M5 Maxใ€ใƒใƒƒใƒ1ใ€้€ๆฌก่ช่ญ˜ใ€‚FLEURSใฎๅ…ˆ้ ญ3็™บ่ฉฑใ‚’ใ‚ฆใ‚ฉใƒผใƒ ใ‚ขใƒƒใƒ—ใจใ—ใฆ้™คใ„ใŸ647็™บ่ฉฑใง้›†่จˆใ€‚ใ‚ญใƒฃใƒƒใ‚ทใƒฅไธŠ้™ใฏๆ—ขๅฎšใฎ1GiBใ€‚bf16ใฏๅŒใ˜ๆ™‚้–“ๅธฏใซๅ†ๆธฌๅฎšใ—ใŸๅ€คใงใ™ใ€‚GB๏ผ10^9ใƒใ‚คใƒˆใ€‚

้ …็›ฎ bf16 8bit
RTFx๏ผˆ1็™บ่ฉฑใ”ใจใฎไธญๅคฎๅ€ค๏ผ‰ 260.1 223.3
RTFx๏ผˆๅˆ่จˆ้Ÿณๅฃฐรทๅˆ่จˆๅ‡ฆ็†ๆ™‚้–“๏ผ‰ 263.4 226.6
1็™บ่ฉฑใฎๅ‡ฆ็†ๆ™‚้–“๏ผˆไธญๅคฎๅ€ค๏ผ‰ 0.048็ง’ 0.057็ง’
ใƒ”ใƒผใ‚ฏRSS 1.57GB 1.09GB
mlxใฎใƒ”ใƒผใ‚ฏๅ‰ฒใ‚Šๅฝ“ใฆ 1.79GB 1.23GB
macOSใฎใƒ”ใƒผใ‚ฏใƒกใƒขใƒชใƒ•ใƒƒใƒˆใƒ—ใƒชใƒณใƒˆ 3.20GB 2.63GB

ใƒ•ใ‚กใ‚คใƒซไธ€่ฆง

ใ‚ตใ‚คใ‚บใฏใƒใ‚คใƒˆใ€‚

ใƒ•ใ‚กใ‚คใƒซ ใ‚ตใ‚คใ‚บ SHA-256
model.safetensors 727,881,324 17ec1d12a67b35303c1bbc4defa3077b03f14665550e45d2749ada12ef73368e
config.json 726 0b7ad196af39a5120570d31bdcb0c9194e89d216892c9ea8e4d552170e8df6b3
tokenizer.model 279,711 9bd7bcdff62e43200001cd4c258d22e475b7b3b1e0f64f8be8e6e046f402f38c
LICENSE 11,358 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30
NOTICE 741 cb6145131e3cb48e0bb432aa5b949ae72330ef0dddaf1621de9b0cc65ab8b51b

ๅˆถ้™ไบ‹้ …

bf16็‰ˆใจๅŒใ˜ๅˆถ้™ไบ‹้ …ใŒ้ฉ็”จใ•ใ‚Œใพใ™ใ€‚้‡ๅญๅŒ–็‰ˆใงใฏ้•ทๆ™‚้–“้Ÿณๅฃฐใฎๅˆ†ๅ‰ฒ่ช่ญ˜ใ€็„ก้Ÿณใƒป้›‘้Ÿณๅ…ฅๅŠ›ใ€JSUTใƒปTEDxJP-10KใƒปSpeechBSDใ€ใ‚ฐใƒซใƒผใƒ—ใ‚ตใ‚คใ‚บ64ไปฅๅค–ใ€ใƒกใƒขใƒชใฎๅฐ‘ใชใ„ๆฉŸ็จฎใ€ใƒใƒƒใƒ2ไปฅไธŠใฏๆœชๆธฌๅฎšใงใ™ใ€‚

ใƒฉใ‚คใ‚ปใƒณใ‚นใจๅธฐๅฑž

  • bf16็‰ˆใจๅŒใ˜Apache-2.0๏ผˆ้‡ใฟใƒปๆŽจ่ซ–ใ‚ณใƒผใƒ‰๏ผ‰ใ€‚ๅ…ƒ้‡ใฟใฏthird-intelligence/Haas-ja-v1.0๏ผˆCopyright Third Intelligence, Inc.๏ผ‰ใ€‚ๅธฐๅฑžใฏNOTICEใ€‚ใ€ŒHaasใ€ใฏๅค‰ๆ›ๅ…ƒใ‚’็คบใ™ๅ็งฐใ€‚
  • ๅ‚่€ƒๅฎŸ่ฃ…: NVIDIA NeMoใ€third-intelligence/Haas-ASR๏ผˆๅ…ฌๅผๆŽจ่ซ–ใƒ‘ใƒƒใ‚ฑใƒผใ‚ธ๏ผ‰ใ€matsuurajun/haas-mlxใ€Sanami/haas-ja-onnxใฏApache-2.0ใ€‚flash-linear-attentionใฏMITใ€‚
  • Sanamiใฎfloat32 ONNXใฏๆ•ฐๅ€คๆฏ”่ผƒใฎใฟใงใ€้‡ใฟใฏๅ†้…ๅธƒใ—ใพใ›ใ‚“ใ€‚
  • ่ฉ•ไพกใƒ‡ใƒผใ‚ฟ: FLEURSใฏCC BY 4.0ใ€JVNVใฏCC BY-SA 4.0ใ€‚ๅŒๆขฑใƒปๅ†้…ๅธƒใ—ใพใ›ใ‚“ใ€‚

English

This is an unofficial conversion of third-intelligence/Haas-ja-v1.0 to MLX with 8-bit linear layers. It is not affiliated with, endorsed by, or sponsored by Third Intelligence, Inc.

Weights occupy about 0.73 GB, approximately 0.59 times the bf16 version. Peak memory decreases, but median RTFx is about 14% lower than bf16; quantization does not make inference faster on the tested machine.

Overview and requirements

  • All linear layers use mlx.nn.quantize 8-bit affine quantization with group size 64; convolutions, normalization layers and embeddings stay bf16. At load time, scales and biases are bf16 in the encoder and float32 in the prediction and joint networks.
  • Apple Silicon Macs only; no NeMo, CUDA, Triton or torch. Requires Python 3.12 and haas-ja-local.
  • Same test environment as bf16: Apple M5 Max (128 GB), macOS 27.2, mlx 0.32.4. Other machines are unmeasured.

Usage

Uses the same code as bf16. Install hf with pip install huggingface_hub if needed.

pip install "haas-ja-local[mlx] @ git+https://github.com/basyauma-tech/haas-ja-local"
hf download basyauma-tech/Haas-ja-v1.0-mlx-8bit --local-dir Haas-ja-v1.0-mlx-8bit
from haas_ja_local import load_audio, load_model, transcribe

backend, tokenizer = load_model("Haas-ja-v1.0-mlx-8bit")
result = transcribe(backend, tokenizer, load_audio("audio.wav"))
print(result.text)
haas-ja-local transcribe Haas-ja-v1.0-mlx-8bit audio.wav
haas-ja-local transcribe Haas-ja-v1.0-mlx-8bit audio.wav --json --timestamps

Accuracy

asr-ja-bench v1.1, evaluated with the same procedure as bf16. CER strict values are percentages; differences are percentage points. Differences from bf16 use the rounded values shown.

Subset Utterances bf16 8-bit Difference from bf16 Published Difference from published
FLEURS 650 3.65 3.68 +0.03 3.65โ€“3.67 +0.01
JVNV 1,375 4.51 4.49 -0.02 4.50โ€“4.53 -0.01

Differences from published values use the nearer endpoint, or 0 within the range. The 8-bit version passes the 0.3-point threshold. No accuracy tuning was applied.

The 4-bit version is not published because it failed the accuracy threshold. CER strict was 3.70% on FLEURS (+0.03 points from published) and 5.00% on JVNV (+0.47 points).

Details/reproduction (Japanese): quantization verification.

Implementation verification

Small random-model tests cover conversion/loading round trips, keys, shapes and bit-identical unquantized values. The 8-bit version produced identical text to bf16 for 88.8% of FLEURS utterances and 84.9% of JVNV utterances. Relative error against a model holding dequantized weights in float32 was below 1e-5. Token agreement between the full quantized model and PyTorch (float32) is unverified.

Speed and memory

M5 Max setup above, batch 1, sequential recognition. 647 FLEURS utterances after excluding the first 3 warm-ups; default 1 GiB cache cap. bf16 was remeasured in the same time window. GB = 10^9 bytes.

Item bf16 8-bit
RTFx (median per utterance) 260.1 223.3
RTFx (total audio / total processing time) 263.4 226.6
Median processing time per utterance 0.048 s 0.057 s
Peak RSS 1.57 GB 1.09 GB
mlx peak allocation 1.79 GB 1.23 GB
macOS peak memory footprint 3.20 GB 2.63 GB

Files

Sizes in bytes.

File Size SHA-256
model.safetensors 727,881,324 17ec1d12a67b35303c1bbc4defa3077b03f14665550e45d2749ada12ef73368e
config.json 726 0b7ad196af39a5120570d31bdcb0c9194e89d216892c9ea8e4d552170e8df6b3
tokenizer.model 279,711 9bd7bcdff62e43200001cd4c258d22e475b7b3b1e0f64f8be8e6e046f402f38c
LICENSE 11,358 cfc7749b96f63bd31c3c42b5c471bf756814053e847c10f3eb003417bc523d30
NOTICE 741 cb6145131e3cb48e0bb432aa5b949ae72330ef0dddaf1621de9b0cc65ab8b51b

Limitations

The bf16 limitations also apply. Quantized inference is unmeasured on long audio with splitting, silence/noise, JSUT, TEDxJP-10K, SpeechBSD, group sizes other than 64, low-memory machines and batch sizes โ‰ฅ2.

License and attribution

Downloads last month
107
Safetensors
Model size
0.6B params
Tensor type
BF16
ยท
U32
ยท
MLX
Hardware compatibility
Log In to add your hardware

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for basyauma-tech/Haas-ja-v1.0-mlx-8bit

Quantized
(15)
this model