Add/update the quantized ONNX model files and README.md for Transformers.js v3

#3
by whitphx HF Staff - opened
Files changed (1) hide show
  1. README.md +18 -2
README.md CHANGED
@@ -8,7 +8,23 @@ license: apache-2.0
8
 
9
  # Grok-1 Tokenizer
10
 
11
- A 🤗-compatible version of the **Grok-1 tokenizer** (adapted from [xai-org/grok-1](https://github.com/xai-org/grok-1)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/xenova/transformers.js).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
  ## Example usage:
14
 
@@ -22,7 +38,7 @@ assert tokenizer.encode('hello world') == [21560, 1135]
22
 
23
  ### Transformers.js
24
  ```js
25
- import { AutoTokenizer } from '@xenova/transformers';
26
 
27
  const tokenizer = await AutoTokenizer.from_pretrained('Xenova/grok-1-tokenizer');
28
  const tokens = tokenizer.encode('hello world'); // [21560, 1135]
 
8
 
9
  # Grok-1 Tokenizer
10
 
11
+ A 🤗-compatible version of the **Grok-1 tokenizer** (adapted from [xai-org/grok-1](https://github.com/xai-org/grok-1)). This means it can be used with Hugging Face libraries including [Transformers](https://github.com/huggingface/transformers), [Tokenizers](https://github.com/huggingface/tokenizers), and [Transformers.js](https://github.com/huggingface/transformers.js).
12
+
13
+ ## Usage (Transformers.js)
14
+
15
+ If you haven't already, you can install the [Transformers.js](https://huggingface.co/docs/transformers.js) JavaScript library from [NPM](https://www.npmjs.com/package/@huggingface/transformers) using:
16
+ ```bash
17
+ npm i @huggingface/transformers
18
+ ```
19
+
20
+ **Example:** Run tokenization.
21
+
22
+ ```js
23
+ import { AutoTokenizer } from '@huggingface/transformers';
24
+
25
+ const tokenizer = await AutoTokenizer.from_pretrained('Xenova/grok-1-tokenizer');
26
+ const tokens = tokenizer.encode('hello world'); // [21560, 1135]
27
+ ```
28
 
29
  ## Example usage:
30
 
 
38
 
39
  ### Transformers.js
40
  ```js
41
+ import { AutoTokenizer } from '@huggingface/transformers';
42
 
43
  const tokenizer = await AutoTokenizer.from_pretrained('Xenova/grok-1-tokenizer');
44
  const tokens = tokenizer.encode('hello world'); // [21560, 1135]