Miya67/aiq-scoring-e5-small-wiki-absolute

CrossEncoder based on hotchpotch/japanese-reranker-cross-encoder-small-v1

This is a Cross Encoder model finetuned from hotchpotch/japanese-reranker-cross-encoder-small-v1 using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Base model: hotchpotch/japanese-reranker-cross-encoder-small-v1
• Maximum Sequence Length: 514 tokens
• Number of Output Labels: 1 label

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("Miya67/aiq-scoring-e5-small-wiki-absolute")
# Get scores for pairs of texts
pairs = [
    ['問題: 俳優の哀川翔、女優の小西真奈美、歌手の長渕剛の出身都道府県はどこでしょう? / 想定解: 鹿児島県', '熊毛地域'],
    ['問題: 和名を「トウショウブ」や「オランダショウブ」という、剣のように尖った葉が特徴的なアヤメ科の植物は何でしょう? / 想定解: グラジオラス', 'グラジオラス属'],
    ['問題: 月見そばで、卵の黄身が表しているものは月ですが、白身が表しているものは何でしょう? / 想定解: 雲', '分子雲'],
    ['問題: 1950年に第1回日本シリーズの第1戦が行われた、現在はヤクルトスワローズが本拠地とする野球場はどこでしょう? / 想定解: 明治神宮野球場', '明治神宮野球大会'],
    ['問題: オーストラリアの6つの州の中で、最も面積が大きいのは西オーストラリア州ですが、最も面積が小さいのは何州でしょう? / 想定解: タスマニア州', 'たすまにあしゅうそうとく'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    '問題: 俳優の哀川翔、女優の小西真奈美、歌手の長渕剛の出身都道府県はどこでしょう? / 想定解: 鹿児島県',
    [
        '熊毛地域',
        'グラジオラス属',
        '分子雲',
        '明治神宮野球大会',
        'たすまにあしゅうそうとく',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

Unnamed Dataset

• Size: 104,687 training samples
• Columns: sentence_0, sentence_1, and label
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	label
  type	string	string	float
  details	min: 29 characters mean: 64.2 characters max: 108 characters	min: 1 characters mean: 7.33 characters max: 30 characters	min: 0.0 mean: 0.59 max: 1.0

• Samples:

  sentence_0	sentence_1	label
  問題: 俳優の哀川翔、女優の小西真奈美、歌手の長渕剛の出身都道府県はどこでしょう? / 想定解: 鹿児島県	熊毛地域	1.0
  問題: 和名を「トウショウブ」や「オランダショウブ」という、剣のように尖った葉が特徴的なアヤメ科の植物は何でしょう? / 想定解: グラジオラス	グラジオラス属	1.0
  問題: 月見そばで、卵の黄身が表しているものは月ですが、白身が表しているものは何でしょう? / 想定解: 雲	分子雲	0.0

• Loss: BinaryCrossEntropyLoss with these parameters:

  {
      "activation_fn": "torch.nn.modules.linear.Identity",
      "pos_weight": null
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• num_train_epochs: 4
• per_device_eval_batch_size: 32

All Hyperparameters

Click to expand

• per_device_train_batch_size: 32
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: no
• per_device_eval_batch_size: 32
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.1528	500	0.6662
0.3056	1000	0.6092
0.4584	1500	0.5728
0.6112	2000	0.5385
0.7641	2500	0.5165
0.9169	3000	0.4984
1.0697	3500	0.4844
1.2225	4000	0.4681
1.3753	4500	0.4557
1.5281	5000	0.4539
1.6809	5500	0.4460
1.8337	6000	0.4402
1.9866	6500	0.4317
2.1394	7000	0.4071
2.2922	7500	0.3955
2.4450	8000	0.3957
2.5978	8500	0.3874
2.7506	9000	0.3996
2.9034	9500	0.3933
3.0562	10000	0.3788
3.2090	10500	0.3635
3.3619	11000	0.3613
3.5147	11500	0.3633
3.6675	12000	0.3584
3.8203	12500	0.3592
3.9731	13000	0.3555

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.3
• Transformers: 5.3.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}