arctic-embed-s, NNEF for tract
Snowflake/snowflake-arctic-embed-s (revision e596f507467533e48a2e17c007f0e1dacc837b33), converted to tract's NNEF format with float16 storage. BrowserNote downloads it for search by meaning and related items, and runs it with tract-nnef on the CPU.
This is a conversion. The weights, the training and the license are Snowflake's (Apache 2.0, in LICENSE).
Files
| File | Size | SHA-256 |
|---|---|---|
arctic-s-f16.nnef.tar |
70,451,200 bytes | 45e986857c436a1e0429a1c48b88a313fd938b539366ddd73b20b81fbd36ec49 |
tokenizer.json |
711,649 bytes | 91f1def9b9391fdabe028cd3f3fcc4efd34e5d1f08c3bf2de513ebb5911a1854 |
tokenizer.json is the original model's, unchanged.
The conversion
tract-onnx0.23.8 loadsonnx/model.onnx, types it and declutters it;tract-nnef0.23.8 writes it (deterministic: the same input gives the same bytes).- Decluttering fuses GELU into tract-core's
GeluExact, which tract-nnef 0.23.8 has no NNEF name for. The file names itbrowsernote_gelu_exact, in an extension namedbrowsernote. - Each float32 tensor of 4,096 values or more (62 tensors) is stored as float16, with a
tract_core_castto float32 after it ingraph.nnef. The math stays float32.
Using it with tract
Register the browsernote extension, then replace each float16 constant's cast with a float32 constant before optimizing. Without that step tract converts the weights at every call, which halves the speed on x86.
use tract_nnef::internal::*;
use tract_nnef::tract_core::ops::{cast::Cast, konst::Const, nn::GeluExact};
use tract_transformers::WithTractTransformers;
fn load(path: &str) -> TractResult<std::sync::Arc<TypedRunnableModel>> {
let mut ext = Registry::new("browsernote");
ext.register_unit_element_wise("browsernote_gelu_exact", &GeluExact {});
let mut model = tract_nnef::nnef().with_tract_transformers().with_registry(ext).model_for_path(path)?;
let casts: Vec<(usize, usize)> = model.nodes().iter()
.filter(|n| n.op_as::<Const>().is_some_and(|k| k.val().datum_type() == f16::datum_type()))
.flat_map(|n| n.outputs[0].successors.iter().map(move |s| (n.id, s.node)))
.filter(|(_, s)| model.nodes()[*s].op_as::<Cast>().is_some_and(|c| c.to == f32::datum_type()))
.collect();
for (konst, cast) in casts {
let wide = model.nodes()[konst].op_as::<Const>().unwrap().val().cast_to::<f32>()?.into_owned();
let mut patch = TypedModelPatch::default();
let c = patch.add_const(format!("{}.wide", model.nodes()[cast].name), wide)?;
patch.shunt_outside(&model, cast.into(), c)?;
patch.apply(&mut model)?;
}
model.into_optimized()?.into_runnable()
}
Inputs: input_ids, attention_mask and token_type_ids, each int64 [batch_size, sequence_length], at most 512 tokens. The output is the last hidden state, [batch_size, sequence_length, 384]. The embedding is the first token's state (CLS pooling), normalized to length 1.
For a search query, put Represent this sentence for searching relevant passages: in front of it. Documents need no prefix.
Checked
On 500 chunks of notes and mail (about 300 tokens each), every vector has a cosine of 1.0000 to the original model's in sentence-transformers (float32, CPU).
| Machine | Threads | Chunks a second |
|---|---|---|
| Apple M4 | 4 | 17β22 |
| AMD Ryzen 5 5600X | 4 | 14.4 |
| AMD Ryzen 5 5600X | 2 | 9.5 |
Model tree for n7apollo/arctic-embed-s-nnef
Base model
Snowflake/snowflake-arctic-embed-s