arctic-embed-s, NNEF for tract

Snowflake/snowflake-arctic-embed-s (revision e596f507467533e48a2e17c007f0e1dacc837b33), converted to tract's NNEF format with float16 storage. BrowserNote downloads it for search by meaning and related items, and runs it with tract-nnef on the CPU.

This is a conversion. The weights, the training and the license are Snowflake's (Apache 2.0, in LICENSE).

Files

File Size SHA-256
arctic-s-f16.nnef.tar 70,451,200 bytes 45e986857c436a1e0429a1c48b88a313fd938b539366ddd73b20b81fbd36ec49
tokenizer.json 711,649 bytes 91f1def9b9391fdabe028cd3f3fcc4efd34e5d1f08c3bf2de513ebb5911a1854

tokenizer.json is the original model's, unchanged.

The conversion

  1. tract-onnx 0.23.8 loads onnx/model.onnx, types it and declutters it; tract-nnef 0.23.8 writes it (deterministic: the same input gives the same bytes).
  2. Decluttering fuses GELU into tract-core's GeluExact, which tract-nnef 0.23.8 has no NNEF name for. The file names it browsernote_gelu_exact, in an extension named browsernote.
  3. Each float32 tensor of 4,096 values or more (62 tensors) is stored as float16, with a tract_core_cast to float32 after it in graph.nnef. The math stays float32.

Using it with tract

Register the browsernote extension, then replace each float16 constant's cast with a float32 constant before optimizing. Without that step tract converts the weights at every call, which halves the speed on x86.

use tract_nnef::internal::*;
use tract_nnef::tract_core::ops::{cast::Cast, konst::Const, nn::GeluExact};
use tract_transformers::WithTractTransformers;

fn load(path: &str) -> TractResult<std::sync::Arc<TypedRunnableModel>> {
    let mut ext = Registry::new("browsernote");
    ext.register_unit_element_wise("browsernote_gelu_exact", &GeluExact {});
    let mut model = tract_nnef::nnef().with_tract_transformers().with_registry(ext).model_for_path(path)?;
    let casts: Vec<(usize, usize)> = model.nodes().iter()
        .filter(|n| n.op_as::<Const>().is_some_and(|k| k.val().datum_type() == f16::datum_type()))
        .flat_map(|n| n.outputs[0].successors.iter().map(move |s| (n.id, s.node)))
        .filter(|(_, s)| model.nodes()[*s].op_as::<Cast>().is_some_and(|c| c.to == f32::datum_type()))
        .collect();
    for (konst, cast) in casts {
        let wide = model.nodes()[konst].op_as::<Const>().unwrap().val().cast_to::<f32>()?.into_owned();
        let mut patch = TypedModelPatch::default();
        let c = patch.add_const(format!("{}.wide", model.nodes()[cast].name), wide)?;
        patch.shunt_outside(&model, cast.into(), c)?;
        patch.apply(&mut model)?;
    }
    model.into_optimized()?.into_runnable()
}

Inputs: input_ids, attention_mask and token_type_ids, each int64 [batch_size, sequence_length], at most 512 tokens. The output is the last hidden state, [batch_size, sequence_length, 384]. The embedding is the first token's state (CLS pooling), normalized to length 1.

For a search query, put Represent this sentence for searching relevant passages: in front of it. Documents need no prefix.

Checked

On 500 chunks of notes and mail (about 300 tokens each), every vector has a cosine of 1.0000 to the original model's in sentence-transformers (float32, CPU).

Machine Threads Chunks a second
Apple M4 4 17–22
AMD Ryzen 5 5600X 4 14.4
AMD Ryzen 5 5600X 2 9.5
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for n7apollo/arctic-embed-s-nnef

Finetuned
(12)
this model