Instructions to use mlx-community/Qwen2-Math-72B-Instruct-bf16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mlx-community/Qwen2-Math-72B-Instruct-bf16 with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Qwen2-Math-72B-Instruct-bf16")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

MLX LM

How to use mlx-community/Qwen2-Math-72B-Instruct-bf16 with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "mlx-community/Qwen2-Math-72B-Instruct-bf16"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "mlx-community/Qwen2-Math-72B-Instruct-bf16"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "mlx-community/Qwen2-Math-72B-Instruct-bf16",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

[WIP] Upload folder using huggingface_hub (multi-commit 03bf74113129436c620132f64db3f63fa0e5351377d4af064482c4b6479d1f01)

by prince-canuma - opened Aug 9, 2024

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+81

-0

This PR is in draft mode

prince-canuma

MLX Community org Aug 9, 2024

•

edited Aug 9, 2024

Upload folder using huggingface_hub

Multi commit ID: 03bf74113129436c620132f64db3f63fa0e5351377d4af064482c4b6479d1f01

Scheduled commits:

Upload 1 file(s) totalling 5.0G (f9b9cf17687e4b0215a5f3406280deb961c64a05586ba057925b0bf5e12e12d1)
Upload 1 file(s) totalling 5.3G (12e092bd8f49fc38e8d189dd8d21bd9eca2ca74d52fc0bfe11adfa5abf171b5b)
Upload 1 file(s) totalling 5.3G (74de202e71f71406af7907d780687b1825b67d448dc18b50f8bb4a628358b0dc)
Upload 1 file(s) totalling 5.3G (4e888c52b978d4fd488c5f07afb90a4c1e99b35cbd80aabecc93557fe402edd2)
Upload 1 file(s) totalling 5.3G (84ab543f4845fdaa7b82382711760062dfcab5b0ff6dddd63e5191c586bbc24d)
Upload 1 file(s) totalling 5.3G (99127673e188e50e7be083c557e558deef21927f3cb71631b4f2de499441f76f)
Upload 1 file(s) totalling 5.3G (9e0e54e49a7e7006eeeb65120cfcf5d0e3c6a713ac2ee2b260fc24e154f277cf)
Upload 1 file(s) totalling 5.3G (c2fbe2a3471944d1dfdaf5454c6534a9fab452a5fe86685ed5f39bb870d1d7f4)
Upload 1 file(s) totalling 5.3G (4f11af2a0cc7a1ca06a0661080ff95f7248423ea0acde91742546e7afa99159a)
Upload 1 file(s) totalling 5.3G (af3353111859fdfb776f1c4d57f3a500deed5dcf9af81d862a2f627205a81a9b)
Upload 1 file(s) totalling 5.3G (1c1cec3b0ba5f4d15410d023b59aef19ec24c727708b2da3ea9e87f57ef1c943)
Upload 1 file(s) totalling 5.3G (752b712266d9080cb7010045e5397627671e40a7c6fff79f412dc2a901be25d0)
Upload 1 file(s) totalling 5.3G (44ba9e2eb8b6981690079c59d52856de2d98d2715955b3c1d2aaf71ba7dc669e)
Upload 1 file(s) totalling 5.3G (bcfd39dc6019cc42180bd76555ebbfdd7327bb54de2d92885622941b0a7412c1)
Upload 1 file(s) totalling 5.3G (6f331ed04f5bf8858f16e316e50091486d0746a80a06bb55d181e302f0b7b69b)
Upload 1 file(s) totalling 5.3G (7ee1923fef35c0d9cb8637b624eafdcc28959f930c79894b1c431c5d99692727)
Upload 1 file(s) totalling 5.3G (480c3fbef7c3f29e3a55a07ce981b3fda3db5598ba2c47983333abdbed5b3a06)
Upload 1 file(s) totalling 5.3G (65736469d42f47a751d380a91d93b8d5c9789353d1e62097e2fae7839ca33903)
Upload 1 file(s) totalling 5.3G (de6372e0a09fd46bdaedaa2e4cedefe46a05bcb9e2d68c3b36ae6f97a91eb638)
Upload 1 file(s) totalling 5.3G (fbdc3d8b2e5c3f79b708b8c64ea1a226bacabbd0dd36b1e27bae2fdcffea4dfb)
Upload 1 file(s) totalling 5.3G (212df1b9714e97f3a0014fc7354d3d831a2f94e74b8ba787269e3aa095dad956)
Upload 1 file(s) totalling 5.3G (d4996b5e8bc381b22e6428d898a7a613eddc0fd079895e10a19fe3b6b92d0481)
Upload 1 file(s) totalling 5.3G (6268904c8f87a32bd820bd5aa8fd98adea1baf27b0f5e3f7d8fac7cc520b5288)
Upload 1 file(s) totalling 5.3G (04b629088b4a4888b822735aa4f606fa5dff67a28741efada2c2e7713e3b5338)
Upload 1 file(s) totalling 5.3G (dfd7ce808f492e7f9d8c75c64a392ccac72cd0fab7c1ae3da5ac0e28fb5ce03f)
Upload 1 file(s) totalling 5.3G (45a5326a90c03e5d9586e6e1748c42ac0eb29a486a92ededd2cdc690ef1ede6b)
Upload 1 file(s) totalling 3.5G (3fc3d11d529d327f86902c4beae92510081f71247bcfcb14ccd49fcf9faa36a5)
Upload 10 file(s) totalling 11.6M (4bbfbd6f157c0a526333498157103b5d3d810ea0f654431c8a066f13f71ee04f)

This is a PR opened using the huggingface_hub library in the context of a multi-commit. PR can be commented as a usual PR. However, please be aware that manually updating the PR description, changing the PR status, or pushing new commits, is not recommended as it might corrupt the commit process. Learn more about multi-commits in this guide.