Instructions to use Geofront/BGE-Reranker-v2-M3-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Geofront/BGE-Reranker-v2-M3-GGUF with sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Geofront/BGE-Reranker-v2-M3-GGUF")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium."
]
embeddings = model.encode(sentences)

similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Transformers

How to use Geofront/BGE-Reranker-v2-M3-GGUF with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Geofront/BGE-Reranker-v2-M3-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Geofront/BGE-Reranker-v2-M3-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use Geofront/BGE-Reranker-v2-M3-GGUF with Ollama:
```
ollama run hf.co/Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Geofront/BGE-Reranker-v2-M3-GGUF with Docker Model Runner:
```
docker model run hf.co/Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M
```

Lemonade

How to use Geofront/BGE-Reranker-v2-M3-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Geofront/BGE-Reranker-v2-M3-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.BGE-Reranker-v2-M3-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

BGE-Reranker-v2-M3-GGUF / README.md

lucasbortolozzi

Update README.md

f427681 verified 5 months ago

preview code

Raw

History Blame Contribute Delete

17.2 kB

	---
	license: apache-2.0
	base_model:
	- BAAI/bge-reranker-v2-m3
	tags:
	- transformers
	- sentence-transformers
	- text-embeddings-inference
	language:
	- multilingual
	---
	# Reranker

	More details please refer to our Github: [FlagEmbedding](https://github.com/FlagOpen/FlagEmbedding/tree/master).

	- [Model List](#model-list)
	- [Usage](#usage)
	- [Fine-tuning](#fine-tune)
	- [Evaluation](#evaluation)
	- [Citation](#citation)

	Different from embedding model, reranker uses question and document as input and directly output similarity instead of embedding.
	You can get a relevance score by inputting query and passage to the reranker.
	And the score can be mapped to a float value in [0,1] by sigmoid function.


	## Model List

	\| Model \| Base model \| Language \| layerwise \| feature \|
	\|:--------------------------------------------------------------------------\|:--------:\|:-----------------------------------------------------------------------------------------------------------------------------------:\|:----------------------------------------------------------------------------------------------:\|:----------------------------------------------------------------------------------------------:\|
	\| [BAAI/bge-reranker-base](https://huggingface.co/BAAI/bge-reranker-base) \| [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) \| Chinese and English \| - \| Lightweight reranker model, easy to deploy, with fast inference. \|
	\| [BAAI/bge-reranker-large](https://huggingface.co/BAAI/bge-reranker-large) \| [xlm-roberta-large](https://huggingface.co/FacebookAI/xlm-roberta-large) \| Chinese and English \| - \| Lightweight reranker model, easy to deploy, with fast inference. \|
	\| [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) \| [bge-m3](https://huggingface.co/BAAI/bge-m3) \| Multilingual \| - \| Lightweight reranker model, possesses strong multilingual capabilities, easy to deploy, with fast inference. \|
	\| [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma) \| [gemma-2b](https://huggingface.co/google/gemma-2b) \| Multilingual \| - \| Suitable for multilingual contexts, performs well in both English proficiency and multilingual capabilities. \|
	\| [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) \| [MiniCPM-2B-dpo-bf16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-bf16) \| Multilingual \| 8-40 \| Suitable for multilingual contexts, performs well in both English and Chinese proficiency, allows freedom to select layers for output, facilitating accelerated inference. \|


	You can select the model according your senario and resource.
	- For multilingual, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)

	- For Chinese or English, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).

	- For efficiency, utilize [BAAI/bge-reranker-v2-m3](https://huggingface.co/BAAI/bge-reranker-v2-m3) and the low layer of [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise).

	- For better performance, recommand [BAAI/bge-reranker-v2-minicpm-layerwise](https://huggingface.co/BAAI/bge-reranker-v2-minicpm-layerwise) and [BAAI/bge-reranker-v2-gemma](https://huggingface.co/BAAI/bge-reranker-v2-gemma)

	## Usage
	### Using FlagEmbedding

	```
	pip install -U FlagEmbedding
	```

	#### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )

	Get relevance scores (higher scores indicate more relevance):

	```python
	from FlagEmbedding import FlagReranker
	reranker = FlagReranker('BAAI/bge-reranker-v2-m3', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
	score = reranker.compute_score(['query', 'passage'])
	print(score) # -5.65234375
	# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
	score = reranker.compute_score(['query', 'passage'], normalize=True)
	print(score) # 0.003497010252573502
	scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
	print(scores) # [-8.1875, 5.26171875]
	# You can map the scores into 0-1 by set "normalize=True", which will apply sigmoid function to the score
	scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], normalize=True)
	print(scores) # [0.00027803096387751553, 0.9948403768236574]
	```

	#### For LLM-based reranker

	```python
	from FlagEmbedding import FlagLLMReranker
	reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
	# reranker = FlagLLMReranker('BAAI/bge-reranker-v2-gemma', use_bf16=True) # You can also set use_bf16=True to speed up computation with a slight performance degradation
	score = reranker.compute_score(['query', 'passage'])
	print(score)
	scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']])
	print(scores)
	```

	#### For LLM-based layerwise reranker

	```python
	from FlagEmbedding import LayerWiseFlagLLMReranker
	reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_fp16=True) # Setting use_fp16 to True speeds up computation with a slight performance degradation
	# reranker = LayerWiseFlagLLMReranker('BAAI/bge-reranker-v2-minicpm-layerwise', use_bf16=True) # You can also set use_bf16=True to speed up computation with a slight performance degradation
	score = reranker.compute_score(['query', 'passage'], cutoff_layers=[28]) # Adjusting 'cutoff_layers' to pick which layers are used for computing the score.
	print(score)
	scores = reranker.compute_score([['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']], cutoff_layers=[28])
	print(scores)
	```

	### Using Huggingface transformers

	#### For normal reranker (bge-reranker-base / bge-reranker-large / bge-reranker-v2-m3 )

	Get relevance scores (higher scores indicate more relevance):

	```python
	import torch
	from transformers import AutoModelForSequenceClassification, AutoTokenizer
	tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-m3')
	model = AutoModelForSequenceClassification.from_pretrained('BAAI/bge-reranker-v2-m3')
	model.eval()
	pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
	with torch.no_grad():
	inputs = tokenizer(pairs, padding=True, truncation=True, return_tensors='pt', max_length=512)
	scores = model(**inputs, return_dict=True).logits.view(-1, ).float()
	print(scores)
	```

	#### For LLM-based reranker

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
	if prompt is None:
	prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
	sep = "\n"
	prompt_inputs = tokenizer(prompt,
	return_tensors=None,
	add_special_tokens=False)['input_ids']
	sep_inputs = tokenizer(sep,
	return_tensors=None,
	add_special_tokens=False)['input_ids']
	inputs = []
	for query, passage in pairs:
	query_inputs = tokenizer(f'A: {query}',
	return_tensors=None,
	add_special_tokens=False,
	max_length=max_length * 3 // 4,
	truncation=True)
	passage_inputs = tokenizer(f'B: {passage}',
	return_tensors=None,
	add_special_tokens=False,
	max_length=max_length,
	truncation=True)
	item = tokenizer.prepare_for_model(
	[tokenizer.bos_token_id] + query_inputs['input_ids'],
	sep_inputs + passage_inputs['input_ids'],
	truncation='only_second',
	max_length=max_length,
	padding=False,
	return_attention_mask=False,
	return_token_type_ids=False,
	add_special_tokens=False
	)
	item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
	item['attention_mask'] = [1] * len(item['input_ids'])
	inputs.append(item)
	return tokenizer.pad(
	inputs,
	padding=True,
	max_length=max_length + len(sep_inputs) + len(prompt_inputs),
	pad_to_multiple_of=8,
	return_tensors='pt',
	)
	tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-gemma')
	model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-gemma')
	yes_loc = tokenizer('Yes', add_special_tokens=False)['input_ids'][0]
	model.eval()
	pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
	with torch.no_grad():
	inputs = get_inputs(pairs, tokenizer)
	scores = model(**inputs, return_dict=True).logits[:, -1, yes_loc].view(-1, ).float()
	print(scores)
	```

	#### For LLM-based layerwise reranker

	```python
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer
	def get_inputs(pairs, tokenizer, prompt=None, max_length=1024):
	if prompt is None:
	prompt = "Given a query A and a passage B, determine whether the passage contains an answer to the query by providing a prediction of either 'Yes' or 'No'."
	sep = "\n"
	prompt_inputs = tokenizer(prompt,
	return_tensors=None,
	add_special_tokens=False)['input_ids']
	sep_inputs = tokenizer(sep,
	return_tensors=None,
	add_special_tokens=False)['input_ids']
	inputs = []
	for query, passage in pairs:
	query_inputs = tokenizer(f'A: {query}',
	return_tensors=None,
	add_special_tokens=False,
	max_length=max_length * 3 // 4,
	truncation=True)
	passage_inputs = tokenizer(f'B: {passage}',
	return_tensors=None,
	add_special_tokens=False,
	max_length=max_length,
	truncation=True)
	item = tokenizer.prepare_for_model(
	[tokenizer.bos_token_id] + query_inputs['input_ids'],
	sep_inputs + passage_inputs['input_ids'],
	truncation='only_second',
	max_length=max_length,
	padding=False,
	return_attention_mask=False,
	return_token_type_ids=False,
	add_special_tokens=False
	)
	item['input_ids'] = item['input_ids'] + sep_inputs + prompt_inputs
	item['attention_mask'] = [1] * len(item['input_ids'])
	inputs.append(item)
	return tokenizer.pad(
	inputs,
	padding=True,
	max_length=max_length + len(sep_inputs) + len(prompt_inputs),
	pad_to_multiple_of=8,
	return_tensors='pt',
	)
	tokenizer = AutoTokenizer.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True)
	model = AutoModelForCausalLM.from_pretrained('BAAI/bge-reranker-v2-minicpm-layerwise', trust_remote_code=True, torch_dtype=torch.bfloat16)
	model = model.to('cuda')
	model.eval()
	pairs = [['what is panda?', 'hi'], ['what is panda?', 'The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China.']]
	with torch.no_grad():
	inputs = get_inputs(pairs, tokenizer).to(model.device)
	all_scores = model(**inputs, return_dict=True, cutoff_layers=[28])
	all_scores = [scores[:, -1].view(-1, ).float() for scores in all_scores[0]]
	print(all_scores)
	```

	## Fine-tune

	### Data Format

	Train data should be a json file, where each line is a dict like this:

	```
	{"query": str, "pos": List[str], "neg":List[str], "prompt": str}
	```

	`query` is the query, and `pos` is a list of positive texts, `neg` is a list of negative texts, `prompt` indicates the relationship between query and texts. If you have no negative texts for a query, you can random sample some from the entire corpus as the negatives.

	See [toy_finetune_data.jsonl](https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_reranker/toy_finetune_data.jsonl) for a toy data file.

	### Train

	You can fine-tune the reranker with the following code:

	For llm-based reranker

	```shell
	torchrun --nproc_per_node {number of gpus} \
	-m FlagEmbedding.llm_reranker.finetune_for_instruction.run \
	--output_dir {path to save model} \
	--model_name_or_path google/gemma-2b \
	--train_data ./toy_finetune_data.jsonl \
	--learning_rate 2e-4 \
	--num_train_epochs 1 \
	--per_device_train_batch_size 1 \
	--gradient_accumulation_steps 16 \
	--dataloader_drop_last True \
	--query_max_len 512 \
	--passage_max_len 512 \
	--train_group_size 16 \
	--logging_steps 1 \
	--save_steps 2000 \
	--save_total_limit 50 \
	--ddp_find_unused_parameters False \
	--gradient_checkpointing \
	--deepspeed stage1.json \
	--warmup_ratio 0.1 \
	--bf16 \
	--use_lora True \
	--lora_rank 32 \
	--lora_alpha 64 \
	--use_flash_attn True \
	--target_modules q_proj k_proj v_proj o_proj
	```

	For llm-based layerwise reranker

	```shell
	torchrun --nproc_per_node {number of gpus} \
	-m FlagEmbedding.llm_reranker.finetune_for_layerwise.run \
	--output_dir {path to save model} \
	--model_name_or_path openbmb/MiniCPM-2B-dpo-bf16 \
	--train_data ./toy_finetune_data.jsonl \
	--learning_rate 2e-4 \
	--num_train_epochs 1 \
	--per_device_train_batch_size 1 \
	--gradient_accumulation_steps 16 \
	--dataloader_drop_last True \
	--query_max_len 512 \
	--passage_max_len 512 \
	--train_group_size 16 \
	--logging_steps 1 \
	--save_steps 2000 \
	--save_total_limit 50 \
	--ddp_find_unused_parameters False \
	--gradient_checkpointing \
	--deepspeed stage1.json \
	--warmup_ratio 0.1 \
	--bf16 \
	--use_lora True \
	--lora_rank 32 \
	--lora_alpha 64 \
	--use_flash_attn True \
	--target_modules q_proj k_proj v_proj o_proj \
	--start_layer 8 \
	--head_multi True \
	--head_type simple \
	--lora_extra_parameters linear_head
	```

	Our rerankers are initialized from [google/gemma-2b](https://huggingface.co/google/gemma-2b) (for llm-based reranker) and [openbmb/MiniCPM-2B-dpo-bf16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-bf16) (for llm-based layerwise reranker), and we train it on a mixture of multilingual datasets:

	- [bge-m3-data](https://huggingface.co/datasets/Shitao/bge-m3-data)
	- [quora train data](https://huggingface.co/datasets/quora)
	- [fever train data](https://fever.ai/dataset/fever.html)

	## Evaluation

	- llama-index.

	![image-20240317193909373](./assets/llama-index.png)


	- BEIR.

	rereank the top 100 results from bge-en-v1.5 large.

	![image-20240317174633333](./assets/BEIR-bge-en-v1.5.png)

	rereank the top 100 results from e5 mistral 7b instruct.

	![image-20240317172949713](./assets/BEIR-e5-mistral.png)

	- CMTEB-retrieval.
	It rereank the top 100 results from bge-zh-v1.5 large.

	![image-20240317173026235](./assets/CMTEB-retrieval-bge-zh-v1.5.png)

	- miracl (multi-language).
	It rereank the top 100 results from bge-m3.

	![image-20240317173117639](./assets/miracl-bge-m3.png)



	## Citation

	If you find this repository useful, please consider giving a star and citation

	```bibtex
	@misc{li2023making,
	title={Making Large Language Models A Better Foundation For Dense Retrieval},
	author={Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
	year={2023},
	eprint={2312.15503},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
	}
	@misc{chen2024bge,
	title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
	author={Jianlv Chen and Shitao Xiao and Peitian Zhang and Kun Luo and Defu Lian and Zheng Liu},
	year={2024},
	eprint={2402.03216},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
	}
	```