crossencoder-airline-refine

• This model is trained on open source airline related dataset.
• The base model is "cross-encoder/stsb-roberta-large"

Model description

• Cross encoder is useful when we want to calculate the similarity between search query and data items.
• If a Cross-Encoder model is trained on a representative training set, it achieves higher accuracy than Bi-Encoders.
• A Cross-Encoder does not produce a sentence embedding. Also, we are not able to pass individual sentences to a Cross-Encoder.

Intended uses & limitations

• The model is finetuned on limited data.
• It might not produce right result in airline related text.
• Model will be finetuned increamentally based on the availablity of the data.

Training and evaluation data

• Below is the example of training data format for cross encoder.
• Training data has sentence1, sentence2 and the similarity score between the two sentence.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	2	2.0393
No log	2.0	4	1.3405
No log	3.0	6	0.9373

Framework versions

• Transformers 4.35.0
• Pytorch 2.1.0
• Datasets 2.14.6
• Tokenizers 0.14.1