Spaces:
Running
Running
File size: 2,602 Bytes
23680f2 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
# Datasets
## Creating a Dataset
```python
import hyperview as hv
# Persistent dataset (default) - survives restarts
dataset = hv.Dataset("my_dataset")
# In-memory dataset - lost when process exits
dataset = hv.Dataset("my_dataset", persist=False)
```
**Storage location:** `~/.hyperview/datasets/` (configurable via `HYPERVIEW_DATABASE_DIR`)
Internally, each dataset is stored as two Lance tables (directories) inside that folder:
- `hyperview_{dataset_name}.lance/` (samples)
- `hyperview_{dataset_name}_meta.lance/` (metadata like label colors)
## Adding Samples
### From HuggingFace
```python
dataset.add_from_huggingface(
"uoft-cs/cifar100",
split="train",
image_key="img",
label_key="fine_label",
max_samples=1000,
)
```
### From Directory
```python
dataset.add_images_dir("/path/to/images", label_from_folder=True)
```
## Persistence Model: Additive
HyperView uses an **additive** persistence model:
| Action | Behavior |
|--------|----------|
| Add samples | New samples inserted, existing skipped by ID |
| Request fewer than exist | Existing samples preserved (no deletion) |
| Request more than exist | Only new samples added |
| Embeddings | Cached per-sample, reused across sessions |
| Projections | Recomputed when new samples added (UMAP requires refit) |
**Example:**
```python
dataset = hv.Dataset("my_dataset")
dataset.add_from_huggingface(..., max_samples=200) # 200 samples
dataset.add_from_huggingface(..., max_samples=400) # +200 new → 400 total
dataset.add_from_huggingface(..., max_samples=300) # no change → 400 total
dataset.add_from_huggingface(..., max_samples=500) # +100 new → 500 total
```
Samples are **never implicitly deleted**. Use `hv.Dataset.delete("name")` for explicit removal.
## Computing Embeddings
```python
# High-dimensional embeddings (CLIP/ResNet)
dataset.compute_embeddings(model="clip", show_progress=True)
# 2D projections for visualization
dataset.compute_visualization() # UMAP to Euclidean + Hyperbolic
```
Embeddings are stored per-sample. If a sample already has embeddings, it's skipped.
## Listing & Deleting Datasets
```python
# List all persistent datasets
hv.Dataset.list_datasets() # ['cifar100_demo', 'my_dataset', ...]
# Delete a dataset
hv.Dataset.delete("my_dataset")
# Check existence
hv.Dataset.exists("my_dataset") # True/False
```
## Dataset Info
```python
len(dataset) # Number of samples
dataset.name # Dataset name
dataset.labels # Unique labels
dataset.samples # Iterator over all samples
dataset[sample_id] # Get sample by ID
```
|