README.md

## Descripción

La evaluación de un modelo del lenguaje es una tarea de investigación porque las soluciones propuestas no son perfectas:

- **Las métricas convencionales** (BLEU,BERTScore ...) son muy rígidas en cuanto a la representación de la palabra y fallan en alinearse con las preferencias humanas.
- **LLM-as-judge** (RAGAS,DeepEval...) las soluciones que delegan la evaluación en un LLM tienen el problema de que dos ejecuciones consecutivas no dan el mismo valor en sus métricas.

La presente propuesta pasa por quedarnos con lo mejor de los dos mundos: preditibilidad y flexibilidad en la elección de palabras. Para ello hemos propuesto evaluar la distancia coseno con un modelo de embeddings especializado.

Nos hemos concentrado en detectar si un modelo del lenguaje tiene dominio sobre los refranes de habla hispana. Por ello presentamos un modelo de embeddings para detectar la similitud entre dos frases, especializado en refranes de habla hispana. 

Entrenado con refranes españoles y su significado. 
Empleando el dataset [es-paremias-variantes-antonimos](https://huggingface.co/datasets/somosnlp-hackathon-2025/es-paremias-variantes-antonimos) se calcula el coeficiente de Pearson para medir el alineamiento con las preferencias humanas, en este caso, tener conocimiento de refranes de habla hispana.

Los coeficientes de los modelos estudiados para esta tarea son:
<table>
  <tr>
    <th>Modelo</th>
    <th>Coeficiente de Pearson</th>
  </tr>
  <tr>
    <td>GTE Qwen 1.5B </td>
    <td>-0.31</td>
  </tr>
  <tr>
    <td>Multilingual E5 - instruct </td>
    <td>0.17</td>
  </tr>
  <tr>
    <td>Multilingual E5 </td>
    <td>0.21</td>
  </tr>
  <tr>
    <td>Cohere embed -v4.0 dim 256 </td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>Cohere embed -v4.0 dim 512 </td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Cohere embed -v4.0 dim 1024 </td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>Cohere embedd multilingual v3.0-light dim 384 </td>
    <td><b>0.72*</b></td>
  </tr>
  <tr>
    <td>Cohere embedd multilingual v3.0-light dim 1024 </td>
    <td><b>0.70</b></td>
  </tr>
  <tr>
    <td>Multilingual E5 refranes (actual)</td>
    <td><b>0.40</b></td>
  </tr>
</table>


---
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:290
- loss:OnlineContrastiveLoss
base_model: intfloat/multilingual-e5-large
widget:
- source_sentence: Antes se coge al mentiroso que al cojo
  sentences:
  - A escudero pobre, taza de plata y cántaro de cobre
  - En río revuelto, pesca abundante
  - Se ayuda primero al necesitado que al engañador.
- source_sentence: Asno de muchos, lobos lo comen
  sentences:
  - Sabio entre sabios, amigos lo respetan.
  - El que mucho madruga más hace que el que Dios ayuda.
  - Se pilla antes a un mentiroso que a un cojo
- source_sentence: Al buey por el asta, y al hombre por la palabra
  sentences:
  - Si no quieres arroz con leche, toma tres tazas
  - Al hombre por la palabra, y al buey por el cuerno ata
  - Ese no es tu amigo, sino alguien que siempre busca estar rodeado de bullicio y
    actividad.
- source_sentence: Al médico, confesor y letrado, hablarles claro
  sentences:
  - Al médico, confesor y letrado, no le hayas engañado
  - Más vale a quien Dios ayuda que quien mucho madruga
  - Al que anda entre la miel, algo se le pega
- source_sentence: A muertos y a idos, no hay amigos
  sentences:
  - Al buen callar llaman santo
  - A los vivos y presentes, siempre hay amigos.
  - Al que de prestado se viste, en la calle lo desnudan
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
- pearson_cosine
- spearman_cosine
model-index:
- name: SentenceTransformer based on intfloat/multilingual-e5-large
  results:
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: Unknown
      type: unknown
    metrics:
    - type: pearson_cosine
      value: 0.8323883964862309
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.8261627064456549
      name: Spearman Cosine
---

# SentenceTransformer based on intfloat/multilingual-e5-large

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) on the csv dataset. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [intfloat/multilingual-e5-large](https://huggingface.co/intfloat/multilingual-e5-large) <!-- at revision 0dc5580a448e4284468b8909bae50fa925907bc5 -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 1024 dimensions
- **Similarity Function:** Cosine Similarity
- **Training Dataset:**
    - csv
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'A muertos y a idos, no hay amigos',
    'A los vivos y presentes, siempre hay amigos.',
    'Al buen callar llaman santo',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Semantic Similarity

* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)

| Metric              | Value      |
|:--------------------|:-----------|
| pearson_cosine      | 0.8324     |
| **spearman_cosine** | **0.8262** |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Dataset

#### csv

* Dataset: csv
* Size: 290 training samples
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
* Approximate statistics based on the first 290 samples:
  |         | sentence1                                                                         | sentence2                                                                         | label                                           |
  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|
  | type    | string                                                                            | string                                                                            | int                                             |
  | details | <ul><li>min: 7 tokens</li><li>mean: 11.68 tokens</li><li>max: 22 tokens</li></ul> | <ul><li>min: 7 tokens</li><li>mean: 17.01 tokens</li><li>max: 44 tokens</li></ul> | <ul><li>0: ~50.00%</li><li>1: ~50.00%</li></ul> |
* Samples:
  | sentence1                                                   | sentence2                                                                                         | label          |
  |:------------------------------------------------------------|:--------------------------------------------------------------------------------------------------|:---------------|
  | <code>Gota a gota, la mar se agota.</code>                  | <code>Con el pasar del tiempo se llega a alcanzar cualquier meta.</code>                          | <code>1</code> |
  | <code>Dime de qué presumes y te diré de qué careces.</code> | <code>Dime de qué careces y te diré de qué dispones.</code>                                       | <code>0</code> |
  | <code>Cómo se vive, se muere.</code>                        | <code>De aquella forma que hemos vivido nuestra vida será de la forma en la que moriremos.</code> | <code>1</code> |
* Loss: [<code>OnlineContrastiveLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#onlinecontrastiveloss)

### Evaluation Dataset

#### Unnamed Dataset

* Size: 1,006 evaluation samples
* Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>label</code>
* Approximate statistics based on the first 1000 samples:
  |         | sentence1                                                                         | sentence2                                                                         | label                                           |
  |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------|
  | type    | string                                                                            | string                                                                            | int                                             |
  | details | <ul><li>min: 7 tokens</li><li>mean: 12.51 tokens</li><li>max: 25 tokens</li></ul> | <ul><li>min: 6 tokens</li><li>mean: 14.82 tokens</li><li>max: 38 tokens</li></ul> | <ul><li>0: ~49.70%</li><li>1: ~50.30%</li></ul> |
* Samples:
  | sentence1                                    | sentence2                                                              | label          |
  |:---------------------------------------------|:-----------------------------------------------------------------------|:---------------|
  | <code>¿Adónde irá el buey que no are?</code> | <code>¿A dó irá el buey que no are?</code>                             | <code>1</code> |
  | <code>¿Adónde irá el buey que no are?</code> | <code>¿Adónde irá el buey que no are ni la  mula que no cargue?</code> | <code>1</code> |
  | <code>¿Adónde irá el buey que no are?</code> | <code>¿Adónde irá el buey que no are, sino al matadero?</code>         | <code>1</code> |
* Loss: [<code>OnlineContrastiveLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#onlinecontrastiveloss)

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: steps
- `learning_rate`: 1e-05
- `num_train_epochs`: 1
- `lr_scheduler_type`: constant
- `load_best_model_at_end`: True
- `eval_on_start`: True
- `batch_sampler`: no_duplicates

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 8
- `per_device_eval_batch_size`: 8
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 1e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 1
- `max_steps`: -1
- `lr_scheduler_type`: constant
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.0
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: True
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `tp_size`: 0
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: None
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: True
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch      | Step   | Training Loss | Validation Loss | spearman_cosine |
|:----------:|:------:|:-------------:|:---------------:|:---------------:|
| 0          | 0      | -             | 0.1095          | 0.7843          |
| 0.1351     | 5      | 0.6784        | 0.0765          | 0.8123          |
| 0.2703     | 10     | 0.5088        | 0.0533          | 0.8303          |
| 0.4054     | 15     | 0.4364        | 0.0475          | 0.8339          |
| **0.5405** | **20** | **0.3456**    | **0.0435**      | **0.8345**      |
| 0.6757     | 25     | 0.1423        | 0.0424          | 0.8324          |
| 0.8108     | 30     | 0.2852        | 0.0443          | 0.8271          |
| 0.9459     | 35     | 0.2616        | 0.0514          | 0.8262          |

* The bold row denotes the saved checkpoint.

### Framework Versions
- Python: 3.12.9
- Sentence Transformers: 3.4.1
- Transformers: 4.50.0
- PyTorch: 2.6.0+cpu
- Accelerate: 1.6.0
- Datasets: 3.5.0
- Tokenizers: 0.21.1

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->