| ## Descripción | |
| La evaluación de un modelo del lenguaje es una tarea de investigación porque las soluciones propuestas no son perfectas: | |
| - **Las métricas convencionales** (BLEU,BERTScore ...) son muy rígidas en cuanto a la representación de la palabra y fallan en alinearse con las preferencias humanas. | |
| - **LLM-as-judge** (RAGAS,DeepEval...) las soluciones que delegan la evaluación en un LLM tienen el problema de que dos ejecuciones consecutivas no dan el mismo valor en sus métricas. | |
| La presente propuesta pasa por quedarnos con lo mejor de los dos mundos: preditibilidad y flexibilidad en la elección de palabras. Para ello hemos propuesto evaluar la distancia coseno con un modelo de embeddings especializado. | |
| Nos hemos concentrado en detectar si un modelo del lenguaje tiene dominio sobre los refranes de habla hispana. Por ello presentamos un modelo de embeddings para detectar la similitud entre dos frases, especializado en refranes de habla hispana. | |
| Entrenado con refranes españoles y su significado. | |
| Empleando el dataset [es-paremias-variantes-antonimos](https://huggingface.co/datasets/somosnlp-hackathon-2025/es-paremias-variantes-antonimos) se calcula el coeficiente de Pearson para medir el alineamiento con las preferencias humanas, en este caso, tener conocimiento de refranes de habla hispana. | |
| Los coeficientes de los modelos estudiados para esta tarea son: | |
| <table> | |
| <tr> | |
| <th>Modelo</th> | |
| <th>Coeficiente de Pearson</th> | |
| </tr> | |
| <tr> | |
| <td>GTE Qwen 1.5B </td> | |
| <td>-0.31</td> | |
| </tr> | |
| <tr> | |
| <td>Multilingual E5 - instruct </td> | |
| <td>0.17</td> | |
| </tr> | |
| <tr> | |
| <td>Multilingual E5 </td> | |
| <td>0.21</td> | |
| </tr> | |
| <tr> | |
| <td>Cohere embed -v4.0 dim 256 </td> | |
| <td>0.67</td> | |
| </tr> | |
| <tr> | |
| <td>Cohere embed -v4.0 dim 512 </td> | |
| <td>0.68</td> | |
| </tr> | |
| <tr> | |
| <td>Cohere embed -v4.0 dim 1024 </td> | |
| <td>0.68</td> | |
| </tr> | |
| <tr> | |
| <td>Cohere embedd multilingual v3.0-light dim 384 </td> | |
| <td><b>0.72*</b></td> | |
| </tr> | |
| <tr> | |
| <td>Cohere embedd multilingual v3.0-light dim 1024 </td> | |
| <td><b>0.70</b></td> | |
| </tr> | |
| <tr> | |
| <td>Multilingual E5 refranes (actual)</td> | |
| <td><b>0.40</b></td> | |
| </tr> | |
| </table> | |
| --- | |
| tags: | |
| - sentence-transformers | |
| - sentence-similarity | |
| - feature-extraction | |
| - generated_from_trainer | |
| - dataset_size:290 | |
| - loss:OnlineContrastiveLoss | |
| base_model: intfloat/multilingual-e5-large | |
| widget: | |
| - source_sentence: Antes se coge al mentiroso que al cojo | |
| sentences: | |
| - A escudero pobre, taza de plata y cántaro de cobre | |
| - En río revuelto, pesca abundante | |
| - Se ayuda primero al necesitado que al engañador. | |
| - source_sentence: Asno de muchos, lobos lo comen | |
| sentences: | |
| - Sabio entre sabios, amigos lo respetan. | |
| - El que mucho madruga más hace que el que Dios ayuda. | |
| - Se pilla antes a un mentiroso que a un cojo | |
| - source_sentence: Al buey por el asta, y al hombre por la palabra | |
| sentences: | |
| - Si no quieres arroz con leche, toma tres tazas | |
| - Al hombre por la palabra, y al buey por el cuerno ata | |
| - Ese no es tu amigo, sino alguien que siempre busca estar rodeado de bullicio y | |
| actividad. | |
| - source_sentence: Al médico, confesor y letrado, hablarles claro | |
| sentences: | |
| - Al médico, confesor y letrado, no le hayas engañado | |
| - Más vale a quien Dios ayuda que quien mucho madruga | |
| - Al que anda entre la miel, algo se le pega | |
| - source_sentence: A muertos y a idos, no hay amigos | |
| sentences: | |
| - Al buen callar llaman santo | |
| - A los vivos y presentes, siempre hay amigos. | |
| - Al que de prestado se viste, en la calle lo desnudan | |
| pipeline_tag: sentence-similarity | |
| library_name: sentence-transformers | |
| metrics: | |
| - pearson_cosine | |
| - spearman_cosine | |
| model-index: | |
| - name: SentenceTransformer based on intfloat/multilingual-e5-large | |
| results: | |
| - task: | |
| type: semantic-similarity | |
| name: Semantic Similarity | |
| dataset: | |
| name: Unknown | |
| type: unknown | |
| metrics: | |
| - type: pearson_cosine | |
| value: 0.8323883964862309 | |
| name: Pearson Cosine | |
| - type: spearman_cosine | |
| value: 0.8261627064456549 | |
| name: Spearman Cosine | |
| --- | |
| # SentenceTransformer based on intfloat/multilingual-e5-large | |
| This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) on the csv dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. | |
| ## Model Details | |
| ### Model Description | |
| - **Model Type:** Sentence Transformer | |
| - **Base model:** [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) <!-- at revision 0dc5580a448e4284468b8909bae50fa925907bc5 --> | |
| - **Maximum Sequence Length:** 512 tokens | |
| - **Output Dimensionality:** 1024 dimensions | |
| - **Similarity Function:** Cosine Similarity | |
| - **Training Dataset:** | |
| - csv | |
| <!-- - **Language:** Unknown --> | |
| <!-- - **License:** Unknown --> | |
| ### Model Sources | |
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) | |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) | |
| - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) | |
| ### Full Model Architecture | |
| ``` | |
| SentenceTransformer( | |
| (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel | |
| (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) | |
| (2): Normalize() | |
| ) | |
| ``` | |
| ## Usage | |
| ### Direct Usage (Sentence Transformers) | |
| First install the Sentence Transformers library: | |
| ```bash | |
| pip install -U sentence-transformers | |
| ``` | |
| Then you can load this model and run inference. | |
| ```python | |
| from sentence_transformers import SentenceTransformer | |
| # Download from the 🤗 Hub | |
| model = SentenceTransformer("sentence_transformers_model_id") | |
| # Run inference | |
| sentences = [ | |
| 'A muertos y a idos, no hay amigos', | |
| 'A los vivos y presentes, siempre hay amigos.', | |
| 'Al buen callar llaman santo', | |
| ] | |
| embeddings = model.encode(sentences) | |
| print(embeddings.shape) | |
| # [3, 1024] | |
| # Get the similarity scores for the embeddings | |
| similarities = model.similarity(embeddings, embeddings) | |
| print(similarities.shape) | |
| # [3, 3] | |
| ``` | |
| <!-- | |
| ### Direct Usage (Transformers) | |
| <details><summary>Click to see the direct usage in Transformers</summary> | |
| </details> | |
| --> | |
| <!-- | |
| ### Downstream Usage (Sentence Transformers) | |
| You can finetune this model on your own dataset. | |
| <details><summary>Click to expand</summary> | |
| </details> | |
| --> | |
| <!-- | |
| ### Out-of-Scope Use | |
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* | |
| --> | |
| ## Evaluation | |
| ### Metrics | |
| #### Semantic Similarity | |
| * Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator) | |
| | Metric | Value | | |
| |:--------------------|:-----------| | |
| | pearson_cosine | 0.8324 | | |
| | **spearman_cosine** | **0.8262** | | |
| <!-- | |
| ## Bias, Risks and Limitations | |
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* | |
| --> | |
| <!-- | |
| ### Recommendations | |
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* | |
| --> | |
| ## Training Details | |
| ### Training Dataset | |
| #### csv | |
| * Dataset: csv | |
| * Size: 290 training samples | |
| * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code> | |
| * Approximate statistics based on the first 290 samples: | |
| | | sentence1 | sentence2 | label | | |
| |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------| | |
| | type | string | string | int | | |
| | details | <ul><li>min: 7 tokens</li><li>mean: 11.68 tokens</li><li>max: 22 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 17.01 tokens</li><li>max: 44 tokens</li></ul> | <ul><li>0: ~50.00%</li><li>1: ~50.00%</li></ul> | | |
| * Samples: | |
| | sentence1 | sentence2 | label | | |
| |:------------------------------------------------------------|:--------------------------------------------------------------------------------------------------|:---------------| | |
| | <code>Gota a gota, la mar se agota.</code> | <code>Con el pasar del tiempo se llega a alcanzar cualquier meta.</code> | <code>1</code> | | |
| | <code>Dime de qué presumes y te diré de qué careces.</code> | <code>Dime de qué careces y te diré de qué dispones.</code> | <code>0</code> | | |
| | <code>Cómo se vive, se muere.</code> | <code>De aquella forma que hemos vivido nuestra vida será de la forma en la que moriremos.</code> | <code>1</code> | | |
| * Loss: [<code>OnlineContrastiveLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#onlinecontrastiveloss) | |
| ### Evaluation Dataset | |
| #### Unnamed Dataset | |
| * Size: 1,006 evaluation samples | |
| * Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code> | |
| * Approximate statistics based on the first 1000 samples: | |
| | | sentence1 | sentence2 | label | | |
| |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------| | |
| | type | string | string | int | | |
| | details | <ul><li>min: 7 tokens</li><li>mean: 12.51 tokens</li><li>max: 25 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 14.82 tokens</li><li>max: 38 tokens</li></ul> | <ul><li>0: ~49.70%</li><li>1: ~50.30%</li></ul> | | |
| * Samples: | |
| | sentence1 | sentence2 | label | | |
| |:---------------------------------------------|:-----------------------------------------------------------------------|:---------------| | |
| | <code>¿Adónde irá el buey que no are?</code> | <code>¿A dó irá el buey que no are?</code> | <code>1</code> | | |
| | <code>¿Adónde irá el buey que no are?</code> | <code>¿Adónde irá el buey que no are ni la mula que no cargue?</code> | <code>1</code> | | |
| | <code>¿Adónde irá el buey que no are?</code> | <code>¿Adónde irá el buey que no are, sino al matadero?</code> | <code>1</code> | | |
| * Loss: [<code>OnlineContrastiveLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#onlinecontrastiveloss) | |
| ### Training Hyperparameters | |
| #### Non-Default Hyperparameters | |
| - `eval_strategy`: steps | |
| - `learning_rate`: 1e-05 | |
| - `num_train_epochs`: 1 | |
| - `lr_scheduler_type`: constant | |
| - `load_best_model_at_end`: True | |
| - `eval_on_start`: True | |
| - `batch_sampler`: no_duplicates | |
| #### All Hyperparameters | |
| <details><summary>Click to expand</summary> | |
| - `overwrite_output_dir`: False | |
| - `do_predict`: False | |
| - `eval_strategy`: steps | |
| - `prediction_loss_only`: True | |
| - `per_device_train_batch_size`: 8 | |
| - `per_device_eval_batch_size`: 8 | |
| - `per_gpu_train_batch_size`: None | |
| - `per_gpu_eval_batch_size`: None | |
| - `gradient_accumulation_steps`: 1 | |
| - `eval_accumulation_steps`: None | |
| - `torch_empty_cache_steps`: None | |
| - `learning_rate`: 1e-05 | |
| - `weight_decay`: 0.0 | |
| - `adam_beta1`: 0.9 | |
| - `adam_beta2`: 0.999 | |
| - `adam_epsilon`: 1e-08 | |
| - `max_grad_norm`: 1.0 | |
| - `num_train_epochs`: 1 | |
| - `max_steps`: -1 | |
| - `lr_scheduler_type`: constant | |
| - `lr_scheduler_kwargs`: {} | |
| - `warmup_ratio`: 0.0 | |
| - `warmup_steps`: 0 | |
| - `log_level`: passive | |
| - `log_level_replica`: warning | |
| - `log_on_each_node`: True | |
| - `logging_nan_inf_filter`: True | |
| - `save_safetensors`: True | |
| - `save_on_each_node`: False | |
| - `save_only_model`: False | |
| - `restore_callback_states_from_checkpoint`: False | |
| - `no_cuda`: False | |
| - `use_cpu`: False | |
| - `use_mps_device`: False | |
| - `seed`: 42 | |
| - `data_seed`: None | |
| - `jit_mode_eval`: False | |
| - `use_ipex`: False | |
| - `bf16`: False | |
| - `fp16`: False | |
| - `fp16_opt_level`: O1 | |
| - `half_precision_backend`: auto | |
| - `bf16_full_eval`: False | |
| - `fp16_full_eval`: False | |
| - `tf32`: None | |
| - `local_rank`: 0 | |
| - `ddp_backend`: None | |
| - `tpu_num_cores`: None | |
| - `tpu_metrics_debug`: False | |
| - `debug`: [] | |
| - `dataloader_drop_last`: False | |
| - `dataloader_num_workers`: 0 | |
| - `dataloader_prefetch_factor`: None | |
| - `past_index`: -1 | |
| - `disable_tqdm`: False | |
| - `remove_unused_columns`: True | |
| - `label_names`: None | |
| - `load_best_model_at_end`: True | |
| - `ignore_data_skip`: False | |
| - `fsdp`: [] | |
| - `fsdp_min_num_params`: 0 | |
| - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} | |
| - `tp_size`: 0 | |
| - `fsdp_transformer_layer_cls_to_wrap`: None | |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} | |
| - `deepspeed`: None | |
| - `label_smoothing_factor`: 0.0 | |
| - `optim`: adamw_torch | |
| - `optim_args`: None | |
| - `adafactor`: False | |
| - `group_by_length`: False | |
| - `length_column_name`: length | |
| - `ddp_find_unused_parameters`: None | |
| - `ddp_bucket_cap_mb`: None | |
| - `ddp_broadcast_buffers`: False | |
| - `dataloader_pin_memory`: True | |
| - `dataloader_persistent_workers`: False | |
| - `skip_memory_metrics`: True | |
| - `use_legacy_prediction_loop`: False | |
| - `push_to_hub`: False | |
| - `resume_from_checkpoint`: None | |
| - `hub_model_id`: None | |
| - `hub_strategy`: every_save | |
| - `hub_private_repo`: None | |
| - `hub_always_push`: False | |
| - `gradient_checkpointing`: False | |
| - `gradient_checkpointing_kwargs`: None | |
| - `include_inputs_for_metrics`: False | |
| - `include_for_metrics`: [] | |
| - `eval_do_concat_batches`: True | |
| - `fp16_backend`: auto | |
| - `push_to_hub_model_id`: None | |
| - `push_to_hub_organization`: None | |
| - `mp_parameters`: | |
| - `auto_find_batch_size`: False | |
| - `full_determinism`: False | |
| - `torchdynamo`: None | |
| - `ray_scope`: last | |
| - `ddp_timeout`: 1800 | |
| - `torch_compile`: False | |
| - `torch_compile_backend`: None | |
| - `torch_compile_mode`: None | |
| - `dispatch_batches`: None | |
| - `split_batches`: None | |
| - `include_tokens_per_second`: False | |
| - `include_num_input_tokens_seen`: False | |
| - `neftune_noise_alpha`: None | |
| - `optim_target_modules`: None | |
| - `batch_eval_metrics`: False | |
| - `eval_on_start`: True | |
| - `use_liger_kernel`: False | |
| - `eval_use_gather_object`: False | |
| - `average_tokens_across_devices`: False | |
| - `prompts`: None | |
| - `batch_sampler`: no_duplicates | |
| - `multi_dataset_batch_sampler`: proportional | |
| </details> | |
| ### Training Logs | |
| | Epoch | Step | Training Loss | Validation Loss | spearman_cosine | | |
| |:----------:|:------:|:-------------:|:---------------:|:---------------:| | |
| | 0 | 0 | - | 0.1095 | 0.7843 | | |
| | 0.1351 | 5 | 0.6784 | 0.0765 | 0.8123 | | |
| | 0.2703 | 10 | 0.5088 | 0.0533 | 0.8303 | | |
| | 0.4054 | 15 | 0.4364 | 0.0475 | 0.8339 | | |
| | **0.5405** | **20** | **0.3456** | **0.0435** | **0.8345** | | |
| | 0.6757 | 25 | 0.1423 | 0.0424 | 0.8324 | | |
| | 0.8108 | 30 | 0.2852 | 0.0443 | 0.8271 | | |
| | 0.9459 | 35 | 0.2616 | 0.0514 | 0.8262 | | |
| * The bold row denotes the saved checkpoint. | |
| ### Framework Versions | |
| - Python: 3.12.9 | |
| - Sentence Transformers: 3.4.1 | |
| - Transformers: 4.50.0 | |
| - PyTorch: 2.6.0+cpu | |
| - Accelerate: 1.6.0 | |
| - Datasets: 3.5.0 | |
| - Tokenizers: 0.21.1 | |
| ## Citation | |
| ### BibTeX | |
| #### Sentence Transformers | |
| ```bibtex | |
| @inproceedings{reimers-2019-sentence-bert, | |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", | |
| author = "Reimers, Nils and Gurevych, Iryna", | |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", | |
| month = "11", | |
| year = "2019", | |
| publisher = "Association for Computational Linguistics", | |
| url = "https://arxiv.org/abs/1908.10084", | |
| } | |
| ``` | |
| <!-- | |
| ## Glossary | |
| *Clearly define terms in order to be accessible across audiences.* | |
| --> | |
| <!-- | |
| ## Model Card Authors | |
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* | |
| --> | |
| <!-- | |
| ## Model Card Contact | |
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* | |
| --> |