How to use nisavid/MemReranker-4B-OptiQ-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("nisavid/MemReranker-4B-OptiQ-4bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True)
How to use nisavid/MemReranker-4B-OptiQ-4bit with sentence-transformers:
from sentence_transformers import CrossEncoder model = CrossEncoder("nisavid/MemReranker-4B-OptiQ-4bit") query = "Which planet is known as the Red Planet?" passages = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", "Saturn, famous for its rings, is sometimes mistaken for the Red Planet." ] scores = model.predict([(query, passage) for passage in passages]) print(scores)
How to use nisavid/MemReranker-4B-OptiQ-4bit with MLX LM:
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "nisavid/MemReranker-4B-OptiQ-4bit"
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "nisavid/MemReranker-4B-OptiQ-4bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nisavid/MemReranker-4B-OptiQ-4bit", "messages": [ {"role": "user", "content": "Hello"} ] }'
