Correct the usage snippet: load_adapter takes a local directory and returns (encoder, metadata); encode_batch for lists
Browse files
README.md
CHANGED
|
@@ -67,26 +67,45 @@ and no difference between them may be read as the cost of anything.
|
|
| 67 |
|
| 68 |
## Using it
|
| 69 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
```python
|
| 71 |
from multilingual_embedding.embedding.neural.adapter import load_adapter
|
| 72 |
|
| 73 |
-
encoder = load_adapter("
|
| 74 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 75 |
```
|
| 76 |
|
| 77 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
|
| 79 |
### Base revision
|
| 80 |
|
| 81 |
-
This adapter's manifest predates our base pinning convention, so
|
| 82 |
-
The base revision it was built against is
|
| 83 |
`614241f622f53c4eeff9890bdc4f31cfecc418b3`, which we establish from the training host's model
|
| 84 |
-
cache rather than from a field the run recorded: that snapshot is the only one present
|
| 85 |
-
predates the run by a month.
|
|
|
|
| 86 |
|
| 87 |
```python
|
| 88 |
-
encoder = load_adapter(
|
| 89 |
-
"
|
| 90 |
revision="614241f622f53c4eeff9890bdc4f31cfecc418b3",
|
| 91 |
)
|
| 92 |
```
|
|
|
|
| 67 |
|
| 68 |
## Using it
|
| 69 |
|
| 70 |
+
Install the package, then download the adapter and load it from the local directory:
|
| 71 |
+
|
| 72 |
+
```bash
|
| 73 |
+
pip install 'quanfire-multilingual-embedding[neural]'
|
| 74 |
+
hf download quanfire-ai/embed-eulaw-multi --revision v1.0.0 --local-dir embed-eulaw-multi
|
| 75 |
+
```
|
| 76 |
+
|
| 77 |
```python
|
| 78 |
from multilingual_embedding.embedding.neural.adapter import load_adapter
|
| 79 |
|
| 80 |
+
encoder, meta = load_adapter("embed-eulaw-multi")
|
| 81 |
+
|
| 82 |
+
texts = [
|
| 83 |
+
"The processing of personal data shall be lawful only if the data subject has given consent.",
|
| 84 |
+
"Die Verarbeitung personenbezogener Daten ist nur rechtmaessig, wenn die betroffene Person ihre Einwilligung erteilt hat.",
|
| 85 |
+
"Member States shall ensure that fishing vessels exceeding 12 metres carry a satellite tracking device.",
|
| 86 |
+
]
|
| 87 |
+
vectors = encoder.encode_batch(texts)
|
| 88 |
```
|
| 89 |
|
| 90 |
+
`load_adapter` returns the encoder together with its metadata, because the metadata carries
|
| 91 |
+
the prefixes needed to use the model correctly. `encode_batch` takes a list of strings;
|
| 92 |
+
`encode` takes a single string.
|
| 93 |
+
|
| 94 |
+
On the three sentences above, the English and German expressions of the same provision sit at
|
| 95 |
+
cosine 0.835, while the unrelated fisheries article sits at 0.379.
|
| 96 |
|
| 97 |
### Base revision
|
| 98 |
|
| 99 |
+
This adapter's manifest predates our base pinning convention, so `meta.checkpoint_revision`
|
| 100 |
+
reads back as `None`. The base revision it was built against is
|
| 101 |
`614241f622f53c4eeff9890bdc4f31cfecc418b3`, which we establish from the training host's model
|
| 102 |
+
cache rather than from a field the run recorded: that snapshot is the only one present there
|
| 103 |
+
and it predates the run by a month. Pass it explicitly if you need the base held still, since
|
| 104 |
+
an upstream repository can change what sits behind a name:
|
| 105 |
|
| 106 |
```python
|
| 107 |
+
encoder, meta = load_adapter(
|
| 108 |
+
"embed-eulaw-multi",
|
| 109 |
revision="614241f622f53c4eeff9890bdc4f31cfecc418b3",
|
| 110 |
)
|
| 111 |
```
|