pj-mathematician/JobSkillGTE-7b-lora

Job-Skill matching fintuned Alibaba-NLP/gte-Qwen2-7B-instruct lora

Top performing model on TalentCLEF 2025 Task B. Use it for job title <-> skill set matching

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: Alibaba-NLP/gte-Qwen2-7B-instruct
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 3584 dimensions
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 3584, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pj-mathematician/JobSkillGTE-7b-lora")
# Run inference
sentences = [
    "An insulation supervisor, regardless of the specific type of insulation material or installation area, requires strong project management skills, knowledge of building codes and safety regulations, and expertise in insulation techniques to oversee the installation process effectively and ensure quality standards are met.\n['insulation supervisor', 'supervisor of installation of insulating materials', 'supervisor of insulation materials installation', 'supervisor of installation of insulation', 'solid wall insulation installation supervisor', 'insulation installers supervisor', 'cavity wall insulation installation supervisor', 'loft insulation installation supervisor']",
    "The skill of installing insulation material is primarily required by job roles such as insulation workers, HVAC technicians, and construction specialists, who are responsible for improving energy efficiency and thermal comfort in buildings by correctly fitting and fixing insulation materials in various structures.\n['install insulation material', 'insulate structure', 'fix insulation', 'insulation material installation', 'installation of insulation material', 'fitting insulation', 'insulating structure', 'installing insulation material', 'fixing insulation', 'fit insulation']",
    "Job roles such as Food Safety Inspector, Public Health Officer, and Environmental Health Specialist require the skill of taking action on food safety violations to ensure compliance with health regulations and maintain public safety standards.\n['take action on food safety violations', 'invoke action on food safety violations', 'agree action on food safety violations', 'pursue action on food safety violations', 'determine action on food safety violations']",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 3584]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 114,699 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 73 tokens mean: 133.53 tokens max: 333 tokens	min: 44 tokens mean: 104.56 tokens max: 236 tokens

• Samples:

  anchor	positive
  A technical director or any of its synonyms requires a strong blend of technical expertise and leadership skills, including the ability to oversee technical operations, manage teams, and ensure the successful execution of technical projects while maintaining operational efficiency and innovation. ['technical director', 'technical and operations director', 'head of technical', 'director of technical arts', 'head of technical department', 'technical supervisor', 'technical manager']	Job roles that require promoting health and safety include occupational health and safety specialists, safety managers, and public health educators, all of whom work to ensure safe and healthy environments in workplaces and communities. ['promote health and safety', 'promote importance of health and safety', 'promoting health and safety', 'advertise health and safety']
  A technical director or any of its synonyms requires a strong blend of technical expertise and leadership skills, including the ability to oversee technical operations, manage teams, and ensure the successful execution of technical projects while maintaining operational efficiency and innovation. ['technical director', 'technical and operations director', 'head of technical', 'director of technical arts', 'head of technical department', 'technical supervisor', 'technical manager']	Job roles that require organizing rehearsals include directors, choreographers, and conductors in theater, dance, and music ensembles, who must efficiently plan and schedule practice sessions to prepare performers for a successful final performance. ['organise rehearsals', 'organise rehearsal', 'organize rehearsals', 'plan rehearsals', 'arrange rehearsals', 'organising rehearsals', 'schedule rehearsals']
  A technical director or any of its synonyms requires a strong blend of technical expertise and leadership skills, including the ability to oversee technical operations, manage teams, and ensure the successful execution of technical projects while maintaining operational efficiency and innovation. ['technical director', 'technical and operations director', 'head of technical', 'director of technical arts', 'head of technical department', 'technical supervisor', 'technical manager']	Job roles such as Health and Safety Managers, Environmental Health Officers, and Risk Management Specialists often require the skill of negotiating health and safety issues with third parties to ensure compliance and protection standards are met across different organizations and sites. ['negotiate health and safety issues with third parties', 'agree with third parties on health and safety', 'negotiate issues on health and safety with third parties', 'negotiate with third parties on health and safety issues', 'negotiate health and safety matters with third parties']

• Loss: CachedGISTEmbedLoss with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01, 'mini_batch_size': 48, 'margin_strategy': 'absolute', 'margin': 0.0}
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• gradient_accumulation_steps: 2
• num_train_epochs: 2
• warmup_ratio: 0.05
• log_on_each_node: False
• fp16: True
• dataloader_num_workers: 4
• fsdp: ['full_shard', 'auto_wrap']
• fsdp_config: {'transformer_layer_cls_to_wrap': ['Qwen2DecoderLayer'], 'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• ddp_find_unused_parameters: True
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 2
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: False
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 4
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: ['full_shard', 'auto_wrap']
• fsdp_min_num_params: 0
• fsdp_config: {'transformer_layer_cls_to_wrap': ['Qwen2DecoderLayer'], 'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: True
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss
0.0156	1	21.5186
0.0312	2	21.4075
0.0469	3	21.0309
0.0625	4	20.7294
0.0781	5	20.9851
0.0938	6	21.3215
0.1094	7	19.8458
0.125	8	18.52
0.1406	9	17.622
0.1562	10	17.5794
0.1719	11	15.8784
0.1875	12	14.5842
0.2031	13	13.3324
0.2188	14	12.3194
0.2344	15	11.2523
0.25	16	10.7172
0.2656	17	10.0063
0.2812	18	9.5643
0.2969	19	9.2463
0.3125	20	8.6533
0.3281	21	8.0588
0.3438	22	8.1866
0.3594	23	7.6767
0.375	24	6.9832
0.3906	25	6.7932
0.4062	26	6.292
0.4219	27	6.1263
0.4375	28	5.8976
0.4531	29	5.7214
0.4688	30	5.6451
0.4844	31	5.6232
0.5	32	5.2984
0.5156	33	5.0322
0.5312	34	4.9435
0.5469	35	4.737
0.5625	36	4.4266
0.5781	37	4.5082
0.5938	38	4.315
0.6094	39	4.269
0.625	40	4.2473
0.6406	41	4.2054
0.6562	42	4.2172
0.6719	43	3.8311
0.6875	44	4.0803
0.7031	45	4.2809
0.7188	46	4.1843
0.7344	47	3.9913
0.75	48	3.9465
0.7656	49	4.0828
0.7812	50	4.0018
0.7969	51	3.8023
0.8125	52	3.897
0.8281	53	3.8941
0.8438	54	3.7708
0.8594	55	3.8051
0.875	56	3.7117
0.8906	57	3.8584
0.9062	58	3.6421
0.9219	59	3.7097
0.9375	60	3.6906
0.9531	61	3.7011
0.9688	62	3.744
0.9844	63	3.6493
1.0	64	3.5659

Framework Versions

• Python: 3.11.11
• Sentence Transformers: 4.1.0
• Transformers: 4.51.2
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 3.5.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}