Update README.md
Browse files
README.md
CHANGED
|
@@ -4,13 +4,13 @@ tags:
|
|
| 4 |
- multimodal
|
| 5 |
- classification
|
| 6 |
datasets:
|
| 7 |
-
-
|
| 8 |
-
-
|
| 9 |
---
|
| 10 |
|
| 11 |
AstroM³ is a self-supervised multimodal model for astronomy that integrates time-series photometry, spectra, and metadata into a unified embedding space
|
| 12 |
-
for classification and other downstream tasks. AstroM³ is trained on [AstroM3Processed](https://huggingface.co/datasets/
|
| 13 |
-
which is the pre-processed version of [AstroM3Dataset](https://huggingface.co/datasets/
|
| 14 |
For more details on the AstroM³ architecture, training, and results, please refer to the [paper](https://arxiv.org/abs/2411.08842).
|
| 15 |
|
| 16 |
<p align="center">
|
|
@@ -43,7 +43,7 @@ from datasets import load_dataset
|
|
| 43 |
from src.data import process_photometry
|
| 44 |
|
| 45 |
# Load the test dataset
|
| 46 |
-
test_dataset = load_dataset('
|
| 47 |
|
| 48 |
# Process photometry to have a fixed sequence length of 200 (center-cropped)
|
| 49 |
test_dataset = test_dataset.map(process_photometry, batched=True, fn_kwargs={'seq_len': 200, 'how': 'center'})
|
|
@@ -55,7 +55,7 @@ import torch
|
|
| 55 |
from src.model import AstroM3
|
| 56 |
|
| 57 |
# Load the base AstroM3-CLIP model
|
| 58 |
-
model = AstroM3.from_pretrained('
|
| 59 |
|
| 60 |
# Retrieve the first sample (batch size = 1)
|
| 61 |
sample = test_dataset[0:1]
|
|
@@ -80,22 +80,22 @@ print('Multimodal Embedding (Spectra Missing):', multimodal_emb_missing)
|
|
| 80 |
from src.model import AstroM3, Informer, GalSpecNet, MetaModel
|
| 81 |
|
| 82 |
# Photometry classification
|
| 83 |
-
photo_model = Informer.from_pretrained('
|
| 84 |
prediction = photo_model(photometry, photometry_mask).argmax(dim=1).item()
|
| 85 |
print('Photometry Classification:', test_dataset.features['label'].int2str(prediction))
|
| 86 |
|
| 87 |
# Spectra classification
|
| 88 |
-
spectra_model = GalSpecNet.from_pretrained('
|
| 89 |
prediction = spectra_model(spectra).argmax(dim=1).item()
|
| 90 |
print('Spectra Classification:', test_dataset.features['label'].int2str(prediction))
|
| 91 |
|
| 92 |
# Metadata classification
|
| 93 |
-
meta_model = MetaModel.from_pretrained('
|
| 94 |
prediction = meta_model(metadata).argmax(dim=1).item()
|
| 95 |
print('Metadata Classification:', test_dataset.features['label'].int2str(prediction))
|
| 96 |
|
| 97 |
# Multimodal classification
|
| 98 |
-
all_model = AstroM3.from_pretrained('
|
| 99 |
prediction = all_model(photometry, photometry_mask, spectra, metadata).argmax(dim=1).item()
|
| 100 |
print('Multimodal Classification:', test_dataset.features['label'].int2str(prediction))
|
| 101 |
```
|
|
@@ -104,11 +104,11 @@ print('Multimodal Classification:', test_dataset.features['label'].int2str(predi
|
|
| 104 |
|
| 105 |
| # Model | # Description |
|
| 106 |
| :--- | :--- |
|
| 107 |
-
| [AstroM3-CLIP](https://huggingface.co/
|
| 108 |
-
| [AstroM3-CLIP-meta](https://huggingface.co/
|
| 109 |
-
| [AstroM3-CLIP-spectra](https://huggingface.co/
|
| 110 |
-
| [AstroM3-CLIP-photo](https://huggingface.co/
|
| 111 |
-
| [AstroM3-CLIP-all](https://huggingface.co/
|
| 112 |
|
| 113 |
## AstroM3-CLIP Variants
|
| 114 |
These variants of the base AstroM3-CLIP model are trained using different random seeds (42, 0, 66, 12, 123);
|
|
@@ -116,11 +116,11 @@ ensure that the dataset is loaded with the corresponding seed for consistency.
|
|
| 116 |
|
| 117 |
| # Model | # Description |
|
| 118 |
| :--- | :--- |
|
| 119 |
-
| [AstroM3-CLIP-42](https://huggingface.co/
|
| 120 |
-
| [AstroM3-CLIP-0](https://huggingface.co/
|
| 121 |
-
| [AstroM3-CLIP-66](https://huggingface.co/
|
| 122 |
-
| [AstroM3-CLIP-12](https://huggingface.co/
|
| 123 |
-
| [AstroM3-CLIP-123](https://huggingface.co/
|
| 124 |
|
| 125 |
## Using your own data
|
| 126 |
|
|
@@ -132,5 +132,5 @@ If you want to use the model with your own data, you must pre-process it in the
|
|
| 132 |
3. **Metadata**: Scalar metadata is transformed via domain-specific functions (e.g., absolute magnitude, log, sin/cos), then normalized using dataset-level statistics.
|
| 133 |
|
| 134 |
For a detailed description, read the [paper](https://arxiv.org/abs/2411.08842).
|
| 135 |
-
To see exactly how we performed this preprocessing, refer to [`preprocess.py`](https://huggingface.co/datasets/
|
| 136 |
|
|
|
|
| 4 |
- multimodal
|
| 5 |
- classification
|
| 6 |
datasets:
|
| 7 |
+
- AstroMLCore/AstroM3Processed
|
| 8 |
+
- AstroMLCore/AstroM3Dataset
|
| 9 |
---
|
| 10 |
|
| 11 |
AstroM³ is a self-supervised multimodal model for astronomy that integrates time-series photometry, spectra, and metadata into a unified embedding space
|
| 12 |
+
for classification and other downstream tasks. AstroM³ is trained on [AstroM3Processed](https://huggingface.co/datasets/AstroMLCore/AstroM3Processed),
|
| 13 |
+
which is the pre-processed version of [AstroM3Dataset](https://huggingface.co/datasets/AstroMLCore/AstroM3Dataset).
|
| 14 |
For more details on the AstroM³ architecture, training, and results, please refer to the [paper](https://arxiv.org/abs/2411.08842).
|
| 15 |
|
| 16 |
<p align="center">
|
|
|
|
| 43 |
from src.data import process_photometry
|
| 44 |
|
| 45 |
# Load the test dataset
|
| 46 |
+
test_dataset = load_dataset('AstroMLCore/AstroM3Processed', name='full_42', split='test')
|
| 47 |
|
| 48 |
# Process photometry to have a fixed sequence length of 200 (center-cropped)
|
| 49 |
test_dataset = test_dataset.map(process_photometry, batched=True, fn_kwargs={'seq_len': 200, 'how': 'center'})
|
|
|
|
| 55 |
from src.model import AstroM3
|
| 56 |
|
| 57 |
# Load the base AstroM3-CLIP model
|
| 58 |
+
model = AstroM3.from_pretrained('AstroMLCore/AstroM3-CLIP')
|
| 59 |
|
| 60 |
# Retrieve the first sample (batch size = 1)
|
| 61 |
sample = test_dataset[0:1]
|
|
|
|
| 80 |
from src.model import AstroM3, Informer, GalSpecNet, MetaModel
|
| 81 |
|
| 82 |
# Photometry classification
|
| 83 |
+
photo_model = Informer.from_pretrained('AstroMLCore/AstroM3-CLIP-photo')
|
| 84 |
prediction = photo_model(photometry, photometry_mask).argmax(dim=1).item()
|
| 85 |
print('Photometry Classification:', test_dataset.features['label'].int2str(prediction))
|
| 86 |
|
| 87 |
# Spectra classification
|
| 88 |
+
spectra_model = GalSpecNet.from_pretrained('AstroMLCore/AstroM3-CLIP-spectra')
|
| 89 |
prediction = spectra_model(spectra).argmax(dim=1).item()
|
| 90 |
print('Spectra Classification:', test_dataset.features['label'].int2str(prediction))
|
| 91 |
|
| 92 |
# Metadata classification
|
| 93 |
+
meta_model = MetaModel.from_pretrained('AstroMLCore/AstroM3-CLIP-meta')
|
| 94 |
prediction = meta_model(metadata).argmax(dim=1).item()
|
| 95 |
print('Metadata Classification:', test_dataset.features['label'].int2str(prediction))
|
| 96 |
|
| 97 |
# Multimodal classification
|
| 98 |
+
all_model = AstroM3.from_pretrained('AstroMLCore/AstroM3-CLIP-all')
|
| 99 |
prediction = all_model(photometry, photometry_mask, spectra, metadata).argmax(dim=1).item()
|
| 100 |
print('Multimodal Classification:', test_dataset.features['label'].int2str(prediction))
|
| 101 |
```
|
|
|
|
| 104 |
|
| 105 |
| # Model | # Description |
|
| 106 |
| :--- | :--- |
|
| 107 |
+
| [AstroM3-CLIP](https://huggingface.co/AstroMLCore/AstroM3-CLIP) | The base model pre-trained using the trimodal CLIP approach. |
|
| 108 |
+
| [AstroM3-CLIP-meta](https://huggingface.co/AstroMLCore/AstroM3-CLIP-meta) | Fine-tuned for metadata-only classification. |
|
| 109 |
+
| [AstroM3-CLIP-spectra](https://huggingface.co/AstroMLCore/AstroM3-CLIP-spectra) | Fine-tuned for spectra-only classification. |
|
| 110 |
+
| [AstroM3-CLIP-photo](https://huggingface.co/AstroMLCore/AstroM3-CLIP-photo) | Fine-tuned for photometry-only classification. |
|
| 111 |
+
| [AstroM3-CLIP-all](https://huggingface.co/AstroMLCore/AstroM3-CLIP-all) | Fine-tuned for multimodal classification. |
|
| 112 |
|
| 113 |
## AstroM3-CLIP Variants
|
| 114 |
These variants of the base AstroM3-CLIP model are trained using different random seeds (42, 0, 66, 12, 123);
|
|
|
|
| 116 |
|
| 117 |
| # Model | # Description |
|
| 118 |
| :--- | :--- |
|
| 119 |
+
| [AstroM3-CLIP-42](https://huggingface.co/AstroMLCore/AstroM3-CLIP-42) | The base model pre-trained with random seed 42 (identical to AstroM3-CLIP). |
|
| 120 |
+
| [AstroM3-CLIP-0](https://huggingface.co/AstroMLCore/AstroM3-CLIP-0) | AstroM3-CLIP pre-trained with random seed 0 (use dataset with seed 0). |
|
| 121 |
+
| [AstroM3-CLIP-66](https://huggingface.co/AstroMLCore/AstroM3-CLIP-66) | AstroM3-CLIP pre-trained with random seed 66 (use dataset with seed 66). |
|
| 122 |
+
| [AstroM3-CLIP-12](https://huggingface.co/AstroMLCore/AstroM3-CLIP-12) | AstroM3-CLIP pre-trained with random seed 12 (use dataset with seed 12). |
|
| 123 |
+
| [AstroM3-CLIP-123](https://huggingface.co/AstroMLCore/AstroM3-CLIP-123) | AstroM3-CLIP pre-trained with random seed 123 (use dataset with seed 123). |
|
| 124 |
|
| 125 |
## Using your own data
|
| 126 |
|
|
|
|
| 132 |
3. **Metadata**: Scalar metadata is transformed via domain-specific functions (e.g., absolute magnitude, log, sin/cos), then normalized using dataset-level statistics.
|
| 133 |
|
| 134 |
For a detailed description, read the [paper](https://arxiv.org/abs/2411.08842).
|
| 135 |
+
To see exactly how we performed this preprocessing, refer to [`preprocess.py`](https://huggingface.co/datasets/AstroMLCore/AstroM3Dataset/blob/main/preprocess.py) in the AstroM3Dataset repo.
|
| 136 |
|