Spaces:
Build error
Build error
Update README
Browse files
README.md
CHANGED
|
@@ -80,75 +80,67 @@ Given n samples, the value of the Vendi Score ranges between 1 and n, with highe
|
|
| 80 |
|
| 81 |
### Examples
|
| 82 |
|
| 83 |
-
```
|
| 84 |
-
import numpy as np
|
| 85 |
-
vendiscore = evaluate.load("danf0/vendiscore")
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
vendiscore.compute(samples, k)
|
| 91 |
-
|
| 92 |
-
# 2.9999
|
| 93 |
```
|
| 94 |
|
| 95 |
If you already have precomputed a similarity matrix:
|
| 96 |
-
```
|
| 97 |
-
K = np.array([[1.0, 0.9, 0.0],
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
vendiscore.compute(K, score_K=True)
|
|
|
|
|
|
|
| 101 |
|
| 102 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
```
|
| 104 |
|
| 105 |
-
|
| 106 |
-
embeddings
|
| 107 |
-
to compute the Vendi Score using the covariance matrix,
|
| 108 |
-
$\frac{1}{n} \sum_i x_i x_i^{\top}$:
|
| 109 |
-
```python
|
| 110 |
-
vendiscore.compute(X, score_dual=True)
|
| 111 |
```
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
# 1 5.31 3.50
|
| 129 |
-
# 2 12.18 3.62
|
| 130 |
-
# 3 9.97 2.97
|
| 131 |
-
# 4 11.10 3.75
|
| 132 |
-
# 5 13.51 3.16
|
| 133 |
-
# 6 9.06 3.63
|
| 134 |
-
# 7 9.58 4.07
|
| 135 |
-
# 8 9.69 3.74
|
| 136 |
-
# 9 8.56 3.43
|
| 137 |
```
|
| 138 |
|
| 139 |
-
Text
|
| 140 |
-
```
|
| 141 |
-
sents = ["Look, Jane.",
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
ngram_vs = vendiscore.compute(sents, k="ngram_overlap", ns=[1, 2])
|
| 147 |
-
bert_vs = vendiscore.compute(sents, k="text_embeddings", model_path="bert-base-uncased")
|
| 148 |
-
simcse_vs = vendiscore.compute(sents, k="text_embeddings", model_path="princeton-nlp/unsup-simcse-bert-base-uncased")
|
| 149 |
-
print(f"N-grams: {ngram_vs:.02f}, BERT: {bert_vs:.02f}, SimCSE: {simcse_vs:.02f})
|
| 150 |
-
|
| 151 |
-
# N-grams: 3.91, BERT: 1.21, SimCSE: 2.81
|
| 152 |
```
|
| 153 |
|
| 154 |
## Limitations and Bias
|
|
|
|
| 80 |
|
| 81 |
### Examples
|
| 82 |
|
| 83 |
+
```
|
| 84 |
+
>>> import numpy as np
|
| 85 |
+
>>> vendiscore = evaluate.load("danf0/vendiscore")
|
| 86 |
+
>>> samples = [0, 0, 10, 10, 20, 20]
|
| 87 |
+
>>> k = lambda a, b: np.exp(-np.abs(a - b))
|
| 88 |
+
>>> vendiscore.compute(samples, k)
|
| 89 |
+
2.9999
|
|
|
|
|
|
|
|
|
|
| 90 |
```
|
| 91 |
|
| 92 |
If you already have precomputed a similarity matrix:
|
| 93 |
+
```
|
| 94 |
+
>>> K = np.array([[1.0, 0.9, 0.0],
|
| 95 |
+
[0.9, 1.0, 0.0],
|
| 96 |
+
[0.0, 0.0, 1.0]])
|
| 97 |
+
>>> vendiscore.compute(K, score_K=True)
|
| 98 |
+
2.1573
|
| 99 |
+
```
|
| 100 |
|
| 101 |
+
If your similarity function is a dot product between `n` normalized
|
| 102 |
+
`d`-dimensional embeddings `X`, and `d` < `n`, it is faster
|
| 103 |
+
to compute the Vendi Score using the covariance matrix, `X @ X.T`.
|
| 104 |
+
(If the rows of `X` are not normalized, set `normalize = True`.)
|
| 105 |
+
```
|
| 106 |
+
>>> X = np.array([[100, 0], [99, 1], [1, 99], [0, 100])
|
| 107 |
+
>>> vendiscore.compute(X, score_dual=True, normalize=True)
|
| 108 |
+
1.9989...
|
| 109 |
```
|
| 110 |
|
| 111 |
+
Image similarity can be calculated using inner products between pixel vectors or between embeddings from a neural network.
|
| 112 |
+
The default embeddings are from the pool-2048 layer of the torchvision version of the Inception v3 model; other embedding functions can be passed to the `model` argument.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
```
|
| 114 |
+
>>> from torchvision import datasets
|
| 115 |
+
>>> mnist = datasets.MNIST("data/mnist", train=False, download=True)
|
| 116 |
+
>>> digits = [[x for x, y in mnist if y == c] for c in range(10)]
|
| 117 |
+
>>> pixel_vs = [vendiscore.compute(imgs, k="pixels") for imgs in digits]
|
| 118 |
+
>>> inception_vs = [vendiscore.compute(imgs, k="image_embeddings", batch_size=64, device="cuda") for imgs in digits]
|
| 119 |
+
>>> for y, (pvs, ivs) in enumerate(zip(pixel_vs, inception_vs)): print(f"{y}\t{pvs:.02f}\t{ivs:02f}")
|
| 120 |
+
0 7.68 3.45
|
| 121 |
+
1 5.31 3.50
|
| 122 |
+
2 12.18 3.62
|
| 123 |
+
3 9.97 2.97
|
| 124 |
+
4 11.10 3.75
|
| 125 |
+
5 13.51 3.16
|
| 126 |
+
6 9.06 3.63
|
| 127 |
+
7 9.58 4.07
|
| 128 |
+
8 9.69 3.74
|
| 129 |
+
9 8.56 3.43
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
```
|
| 131 |
|
| 132 |
+
Text similarity can be calculated using n-gram overlap or using inner products between embeddings from a neural network.
|
| 133 |
+
```
|
| 134 |
+
>>> sents = ["Look, Jane.",
|
| 135 |
+
"See Spot.",
|
| 136 |
+
"See Spot run.",
|
| 137 |
+
"Run, Spot, run.",
|
| 138 |
+
"Jane sees Spot run."]
|
| 139 |
+
>>> ngram_vs = vendiscore.compute(sents, k="ngram_overlap", ns=[1, 2])
|
| 140 |
+
>>> bert_vs = vendiscore.compute(sents, k="text_embeddings", model_path="bert-base-uncased")
|
| 141 |
+
>>> simcse_vs = vendiscore.compute(sents, k="text_embeddings", model_path="princeton-nlp/unsup-simcse-bert-base-uncased")
|
| 142 |
+
>>> print(f"N-grams: {ngram_vs:.02f}, BERT: {bert_vs:.02f}, SimCSE: {simcse_vs:.02f})
|
| 143 |
+
N-grams: 3.91, BERT: 1.21, SimCSE: 2.81
|
|
|
|
| 144 |
```
|
| 145 |
|
| 146 |
## Limitations and Bias
|