AlekseyCalvin/LyricalEmbeddingGemma_v1

SentenceTransformer based on google/embeddinggemma-300m

This is a sentence-transformers model finetuned from google/embeddinggemma-300m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
queries = [
    "(\u041d\u0430\u0437\u0432\u0430\u043d\u0438\u0435:) \u0420\u0415\u0419\u0413\u0410\u041d-\u041f\u0420\u041e\u0412\u041e\u041a\u0410\u0422\u041e\u0420 \n(\u041f\u043e\u044d\u0442:) \u0410\u043d\u0434\u0440\u0435\u0439 \u201c\u0421\u0432\u0438\u043d\u201d \u041f\u0430\u043d\u043e\u0432 \n\n\u041e\u043f\u044f\u0442\u044c \u043d\u0435\u043b\u0451\u0442\u043d\u0430\u044f \u043f\u043e\u0433\u043e\u0434\u0430, \n\u0418 \u043e\u0442 \u0441\u0443\u0434\u044c\u0431\u044b \u043d\u0435 \u0443\u0431\u0435\u0436\u0430\u0442\u044c. \n\u041c\u044b \u043e\u0431\u044a\u044f\u0432\u043b\u044f\u0435\u043c \u043c\u043e\u0440\u0430\u0442\u043e\u0440\u0438\u0439 \u2014 \n\u0410\u043c\u0435\u0440\u0438\u043a\u0430\u043d\u0446\u0430\u043c \u043d\u0430\u043f\u043b\u0435\u0432\u0430\u0442\u044c\u2026 \n\u0412 \u0416\u0435\u043d\u0435\u0432\u0435 \u0432\u0441\u0435 \u043f\u0435\u0440\u0435\u0433\u043e\u0432\u043e\u0440\u044b \n\u0414\u0430\u0432\u043d\u043e \u0443\u0436\u0435 \u0437\u0430\u0448\u043b\u0438 \u0432 \u0442\u0443\u043f\u0438\u043a. \n\u0420\u0435\u0439\u043a\u044c\u044f\u0432\u0438\u043a \u0434\u0430\u043b \u043f\u043e\u043d\u044f\u0442\u044c \u043d\u0430\u0440\u043e\u0434\u0443 \u2014 \n\u041a\u0430\u043a \u0413\u043e\u0440\u0431\u0430\u0447\u0451\u0432 \u0443 \u043d\u0430\u0441 \u0432\u0435\u043b\u0438\u043a. \n\n\u0410\u2026 \n\u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \n\u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \n\n\u0412 \u041d\u0435\u0432\u0430\u0434\u0435 \u0442\u0430\u043a \u0436\u0435, \u043a\u0430\u043a \u0438 \u0440\u0430\u043d\u044c\u0448\u0435, \n\u0412\u0437\u0440\u044b\u0432\u0430\u044e\u0442 \u043c\u043d\u043e\u0433\u043e \u043a\u0438\u043b\u043e\u0442\u043e\u043d\u043d. \n\u0420\u044d\u0439-\u0433\u0430\u043d \u0443\u0441\u0442\u0440\u043e\u0438\u043b \u0438\u0437 \u043f\u043b\u0430\u043d\u0435\u0442\u044b \n\u041e\u0433\u0440\u043e\u043c\u043d\u044b\u0439 \u043c\u043e\u0449\u043d\u044b\u0439 \u043f\u043e\u043b\u0438\u0433\u043e\u043d. \n\u041d\u0430\u0440\u043e\u0434\u044b \u0432 \u0433\u043e\u043b\u043e\u0434\u0435 \u0438 \u0432 \u0432\u043e\u0439\u043d\u0430\u0445 \n\u0422\u0435\u0440\u044f\u044e\u0442 \u0442\u044b\u0441\u044f\u0447\u0438 \u0436\u0438\u0437\u043d\u0435\u0301\u0439; \n\u0410 \u0432 \u0412\u0430\u0448\u0438\u043d\u0433\u0442\u043e\u043d\u0435, \u0411\u0435\u043b\u043e\u043c \u0434\u043e\u043c\u0435 \n\u0420\u0435\u0448\u0430\u044e\u0442 \u043a\u0430\u043a \u0443\u0431\u0438\u0442\u044c \u0434\u0435\u0442\u0435\u0439. \n\n\u0422\u043e\u0442\u2026 \n\u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \n\u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440! \u0420\u0435\u0439\u0433\u0430\u043d-\u043f\u0440\u043e\u0432\u043e\u043a\u0430\u0442\u043e\u0440!",
]
documents = [
    "(Title:) REAGAN PROVOCATEUR \n(Poet:) Andrey “Swine” Panov \n\nAgain a weather barring travel, \nBut fate not easily escaped, \nOur moratorium is blaring — \nBut USA don't give a shit… \nNegotiations in Geneva, \nAchieve an impasse again, \nReykjavik really showed the people — \nOur Gorbachev the greater man. \n\nAnd… \nReagan — provocateur! Reagan — provocateur! \nReagan — provocateur! Reagan — provocateur! \n\nWhile in Nevada, same as ever, \nThey blow up many kilotons, \nRay-gun is splitting up our planet \nIn mighty weapons testing zones. \nAmid the war and hunger, peoples \nAre losing millions of lives; \nWhile back in Washington, the White House\nTries to ensure no child survives. \n\nExcept… \nReagan — provocateur! Reagan — provocateur! \nReagan — provocateur! Reagan — provocateur!",
    "THAT PROVOCATIVE REAGAN \n(Poet:) Andrew “The Pig” Panoff \n\nOnce again, the weather is unfavorable for flying, \nAnd there is no escaping fate. \nWe declare a moratorium— \nThe Americans don't care… \nIn Geneva, all negotiations \nHave long since reached an impasse. \nReykjavik made it clear to the people — \nHow great Gorbachev is to us. \n\nAh... \nReagan the provocateur! Provocateur Reagan! \nReagan the provocateur! Provocateur Reagan! \n\nIn Nevada, just like before, \nThey detonate thousands of kilotons. \nReagan has turned the planet \nInto a huge, powerful testing ground. \nPeople are starving and fighting wars, \nLosing thousands of lives; \nAnd in Washington, in the White House, \nThey decide how to kill children. \n\nThat... \nProvocateur Reagan! Reagan the provocateur! \nHe... Provocateur Reagan! Reagan the provocateur!",
    '(Title:) I GROW BITCHY \n(Songwriter: Yanka Dyagileva) \n\nI irrevocably grow bitchy \nEvery night, with every chuckle, \nEvery emptied out glass cup \nI go on boarding up the doors \nAnd letting mean and hungry dogs \nFrom all the chains run freely \nWhat else could we do – \nWe who inherit only kneecaps blistered over \nI irrevocably grow bitchy every time I \n\nI’m educated \nTo be iron barrel’s latched continuation \nOf a rifle the arm shaft \nSit if you wanna \nHave a smoke beside me on a little bench – into the ground staring \n\nWhere else could we go – we who inherit only dirtiest of pathways \nI irrevocably grow bitchy by the hour \n\nI’m irrevocably made bitchy \nEvery sighting of a cop hat, or a fancy mink fur hat \nOut where the wartime never ends, \nWhere springtime never really sets, where childhood never continues, \nWhere else could we turn – we who are left with only dreams and conversations, \nI irrevocably grow bitchy by the hour \nI irrevocably grow bitchy every step I \nI irrevocably grow bitchy every time I',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.1512, -0.5952,  0.8725]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 43 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 43 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 8 tokens mean: 235.65 tokens max: 448 tokens	min: 8 tokens mean: 219.33 tokens max: 442 tokens	min: 8 tokens mean: 193.65 tokens max: 361 tokens

• Samples:

  anchor	positive	negative
  (Название:) ЗВЕЗДА ПО ИМЕНИ СОЛНЦЕ (Поэт:) Виктор Цой Белый снег, серый лёд На растрескавшейся земле Одеялом лоскутным на ней Город в дорожной петле А над городом – плывут облака Закрывая небесный свет А над городом жёлтый дым Городу две тысячи лет Прожитых под светом звезды по имени Солнце И две тысячи лет война Война без особых причин Война дело молодых Лекарство против морщин Красная-красная кровь Через час уже просто земля Через два — на ней цветы и трава Через три — она снова жива И согрета лучами звезды по имени Солнце И мы знаем, что так было всегда Что судьбою больше любим Кто живёт по законам другим И кому умирать молодым Он не помнит слово «да» и слово «нет» Он не помнит ни чинов, ни имён И способен дотянуться до звёзд Не считая, что это сон И упасть опалённым звездой по имени Солнце	(Title:) STAR KNOWN AS THE SUN (Poet:) Viktor Tsoy White of snow, grey of ice, Over earth splintered dry with cracks As a blanket unrolling on there — Noosed in roads, a metropolis. While above it – rolling clouds’ slow tide All the heavenly light conceals. On the city fall yellow fumes – It had stood for two thousand years. Lived in the light of that star known as the sun there. And for two thousand years, there’s war – And this war has no special cause. To war – is the work for the young, Common treatment against wrinkles. Blood runs crimson and blood runs red – In an hour, as it were, mere earth, After two – grass, flowers on it, And in three it’s, once more alive, And is warmed by the light of that star known as the sun still. And we know it had always been thus, That the fates tend to love and approve Those who live by differing laws, And reach death while still with youth. Knowing not of words like “yes" or “no", Who forget all the ranks and roles A...	(Title:) THAT SUN NAMED STAR (Poet:) Viktor Tsoi Snow is white, while ice is gray When it’s on cracking ground A big blanket covering it A city in a traffic loop And clouds swim above the city Blocking tout he light of the sky And the yellow smoke above the city This city has already turned two thousand years old Lived under the light of a star called the Sun And thousand years long also is a war War without any particular reason War is an enterprise perfect for the young A cure for wrinkles Red, red seems the blood An hour later, just simply soil Two hours later, flowers and grass on it Three hours later, it is alive again And warmed by the rays of a star called Sun And we know that it has always been something like this That by fate is categorically preferred That fellow who lives by other sorts of laws And who is supposed to die young He does not recall what is that word “yes” or that word “no” He does not remember hierarchical formal ranks or names A...

  | (Название:) “Вечером желтым как зрелый колос…“
(Поэт:) Константин Вагинов

Вечером желтым как зрелый колос,
Средь случайных дорожных берез,
Цыганенок плакал голый,
Вспоминал он имя свое;

Но не мог никак он вспомнить –
Кто, откуда, зачем он здесь;
Слышал матери шепот любовный;
Но не видел ее нигде.

На дороге воробьи чирикают –
Чирик, чирик и по дороге скок;
И девушки уносят землянику;
Но завтра солнце озарит восток. | (Title:) “Come one evening, as yellow as ripe grain…“ 
 (Poet:) Konstantin Vaginov

Come one evening, as yellow as ripe grain,
Between birches there strewn by the road,
Wept a small gypsy boy, bare-naked,
For his name he no longer recalled;

How he tried to, but couldn’t remember –
Wherefrom, or wherefore, he’d fared;
Heard his mother whispering tender;
He would look, but she wasn’t there.

On the road, baby sparrows chirp past him –
Chirpy-chirpy, up the road they would frisk;
Girls are passing with strawberry baskets;
But tomorrow, the sun would flood the east. | At Night…
(Poetry:) Vaginoff

At night, as yellowy as ripened ears of corn,
Amidst random roadside birches,
A naked gypsy-wanderer child cried.
He remembered his name,

But he couldn't remember
Who he was, where he came from, why he was here.
He heard his mommy’s loving speaking under her breath,
But he couldn't see her around him.

Sparrows squeak on the highway
Squeaking, squeaking, and bounce along the highway –
And women carry away strawberries,
But the very next day the sun will shine in the eastern direction. | | (Название:) НА ЧËРНЫЙ ДЕНЬ
(Поэт:) Янка Дягилева

На чёрный день усталый танец пьяных глаз, дырявых рук
Второй упал, четвёртый сел, восьмого вывели на круг
На провода из-под колёс да на три буквы из-под асфальта
В тихий омут буйной головой
В холодный пот — расходятся круги

Железный конь, защитный цвет, резные гусеницы в ряд
Аттракцион для новичков — по кругу лошади летят
А заводной калейдоскоп гремит кривыми зеркалами
Колесо вращается быстрей
Под звуки марша головы долой

Поела моль цветную шаль, на картах тройка и семёрка
Бык хвостом сгоняя мух с тяжёлым сердцем лезет в горку
Лбов бильярдные шары от столкновения раскатились
Пополам по обе стороны
Да по углам просторов и широт

А за осколками витрин обрывки праздничных нарядов
Под полозьями саней живая плоть чужих раскладов
За прилавком попугай из шапки достаёт билеты на трамвай
До ближнего моста
На вертолёт без окон и дверей —
В тихий омут буйной головой —
Колесо вращается быстрей... | (Title:) COME RAINY DAY
(Poet:) Yanka Dyaghileva

Come rainy day a weary dance of drunken eyes, of clumsy hands
Odd person fell, each fourth was penned, each eighth was rounded for a spin
Across the wiring under wheels and triple letters under asphalt
Quiet washers whirlpool rowdy heads
Unto cold sweat — their spinners gather steam

The iron horse, its armored shade, etched rows of caterpillar tracks
A ride for novices designed — such horses flying endless rounds
While the kaleidoscope on cogs flips grating crooked fun house mirrors
And the wheel is spinning faster yet
Off with their heads, off to the marching band

A moth consumed the rainbow shawl, a three then seven in the cards
While swatting flies under its tail a bull grave hearted climbs a mount
Of forehead lobes, like billiard balls from a collision scattered 
 Over either side in equal portioned parts
And into nooks of latitudes and views

While over shattered store displays hang scraps of celebration wear
And... | (Title:) ON A DARK DAY
(Poet:) Yanka Dyagileva

On a dark day, the exhausted jig of intoxicated looks, hole-covered hands
The second collapsed, the fourth sat down, every eighth was led into the circle
On wires from under the tracks and on three letters from under the concrete
Into a quiet pool with a wild head
In chilled sweat frothing — circles spread

An metallic horsey, protective color, carved caterpillars in a row
An attraction for beginners — horses fly in a circle
And a wind-up kaleidoscope rattles with crooked mirrors
The wheel spins faster
To the sounds of the march, heads down

The moth ate the colorful fabric, on the cards appear a three and then a seven
The bull brushes flies with its tail and climbs the hill with a heavy spirit
Pool balls roll away from the bounce
In half on both sides
And in the corners of spaces and meridians

And behind the shards of shop windows, scraps of festive costumes
Under the runners of the sleigh, the living flesh of stranger... |
• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1
• learning_rate: 2e-05
• num_train_epochs: 4
• warmup_ratio: 0.1
• prompts: task: classification | query:

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 4
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: task: classification | query:
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
1.0	43	0.231
2.0	86	0.0008
3.0	129	0.0002
4.0	172	0.0

Framework Versions

• Python: 3.13.12
• Sentence Transformers: 5.2.3
• Transformers: 4.57.0.dev0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}