INTEGRATION.md

Integrating the INT8 Parakeet bundle

Migrating from the FP16 release

Use the root INT8 encoder with root decoder, joint and tokenizer. Their filenames remain stable. Decoder/joint/tokenizer bytes match the previous release. Check release_id and SHA256 in the manifest when invalidating downloaded-file caches. The previous complete release is in legacy/; its documentation describes that historical release, not the current INT8 memory measurements.

The 15-second shape and TDT token/duration contracts are retained. Use the actual tensor names and signature metadata rather than relying on positional ordering across exports. Keep all three interpreters alive and reuse buffers; do not load all precision variants concurrently.

Encoder

Signature: serving_default.

Role	Actual tensor name	Type	Shape
Normalized log-mel input	serving_default_args_0	float32	[1,128,1500]
Actual valid mel length	serving_default_args_1	int32	[1]
Encoded sequence	serving_default_output_0_output	float32	[1,1024,188]
Valid encoder length	serving_default_output_1_output	int32	[1]

Anchor actual audio/features at position zero and zero-pad only at the tail. Pass the true valid length. Use the returned encoded length rather than treating all 188 output positions as valid. Handle empty/no-speech input in the host; this bundle does not include VAD.

Frontend

Match the evaluated NeMo frontend: 16 kHz mono, preemphasis 0.97, FFT 512, nonperiodic Hann window 400 samples, hop 160, 128 Slaney-normalized mel filters, power spectrum, and natural log with additive guard 2**-24. Inference dither is disabled. Normalize each mel channel over valid time frames using the mean and sample standard deviation plus 1e-5; mask invalid frames to zero afterward. Valid mel length is the reference frontend's floor of sample count / 160, not simply every STFT output column.

For the long-recording validation, frontend features were computed independently for each overlapping audio window. Do not normalize across future, unavailable audio for a live implementation. Match centering/padding, mel filter construction and lengths against NeMo before attributing differences to quantization. The legacy README's log(mel + 1e-5) description was not the exact evaluated NeMo log guard; the values above record the actual reference. No frontend graph or new frontend implementation is included here.

Prediction network and joint

Decoder inputs: token int64 [1,1], hidden and cell float32 [2,1,640]. Outputs: prediction float32 [1,1,640], updated hidden/cell of the same state shape.

Joint inputs: encoded frame float32 [1,1024,1], transposed prediction float32 [1,640,1]. Output: float32 [1,1,1,8198]. Split logits into [0:8193] for tokens and [8193:8198] for durations [0,1,2,3,4]. Vocabulary contains 8192 pieces; blank ID is 8192.

Initialize hidden/cell to zero and prime the decoder with blank. On non-blank emission, append the token, update decoder state, and advance by the predicted duration. On blank, advance by at least one frame without updating prediction state. Guard repeated zero-duration emissions, as in the validated host loop. Reset independent-window decoder state; do not pretend this offline encoder has a cache. Do not blindly collapse repeated text or token pieces.

Continuous input

Tested profile: 496 new mel frames per commit, 496 left-context frames, 200 right-context frames. Each mel frame is 10 ms; encoder steps are 80 ms. Re-encode the overlapping window and commit only tokens in its target interval. No left zero-padding at the beginning of the recording. The profile is approximately 5-second updates after approximately 7 seconds of initial buffering, plus compute. Audio/frame timestamps are emission estimates, not verified word alignments. Boundary stitching remains imperfect.

Memory and optional native mapping hint

The default INT8 runtime measured about 1.60 GB process peak on Linux. The optional verified source-page hint measured about 1.46 GB initialization peak and 0.954 GB retained RSS after the complete evaluation. The latter includes all three model objects; it is not a post-teardown measurement. Full English and long-recording token sequences were identical with and without the hint.

mapped_model_file.h is a Linux/Android native example, compiled and tested on Linux only. It owns an immutable file-backed mapping. If the chosen model API explicitly supports an externally owned read-only model buffer:

1. Construct the mapping owner before the model/interpreter and keep it alive until after they are destroyed.
2. Prepare the CPU delegate and run representative encoder warmup.
3. Call DropVerifiedSourcePages() once. It verifies mapped bytes against the file before MADV_DONTNEED.
4. Keep the mapping and file valid. Required pages can fault back in; this does not unmap the model.

This discards redundant resident source pages after weights have been packed. It does not shrink the file, guarantee immediate removal from the global OS file cache, or eliminate startup allocations. Some runtimes may continue reading source weights and refault more pages.

Never apply the hint to an anonymous or writable model ByteBuffer. Do not advise an unrelated second mapping and assume it affects the interpreter. Do not call it while inference is running. A Java file-path-only CompiledModel API may not expose the necessary mapping, so this is not a drop-in Java option. Ordinary INT8 remains usable without this optimization.

On Android, measure total application PSS/RSS, initialization overlap, sustained latency, page faults and thermals. The local ASR figures exclude the UI, microphone frontend, text/audio emotion models and a second ASR model. Keeping Nemotron and Parakeet resident together is not covered by this memory result. GPU/NPU delegates and INT8 coverage must be evaluated separately; retain CPU decoder/joint initially.